why a better performance (silhouette score) in python 2.7 than 3.6?
I read a lot on SoF about the difference in speed between Python 2.7 and 3.6. but my question is more about performance between the two versions.
I used for document clustering: TF-IDF + KMeans and score silhouette to evaluate the homogeneity of my clusters.
By switching from Python 3.6 to Python 2.7, my silhouette score has increased by +0.20!
**Would someone have an explanation? ** Thanks!
code :
tfidf = TfidfVectorizer(
stop_words=my_stopwords_str,
max_df=0.95,
min_df=5,
token_pattern=r'w3,',
max_features=20)
tfidf.fit(data_final.all_text)
data_vect = tfidf.transform(data_final.all_text)
num_clusters = 15
kmeans = KMeans(n_clusters=num_clusters, init='k-means++',
max_iter=300).fit(data_vect_lsa)
kmeans_predict = KMeans(n_clusters=num_clusters, init='k-means++', max_iter=300).fit_predict(data_vect_lsa)
silhouette_score(data_vect, labels = kmeans_predict, metric='euclidean')
The output for Python 2.7 is :
0.58234789374593758
The output for Python 3.6 is :
0.37524101598378656
python python-3.x python-2.7 cluster-analysis
asked Nov 12 at 14:16
Themis
62
add a comment |
I read a lot on SoF about the difference in speed between Python 2.7 and 3.6. but my question is more about performance between the two versions.
I used for document clustering: TF-IDF + KMeans and score silhouette to evaluate the homogeneity of my clusters.
By switching from Python 3.6 to Python 2.7, my silhouette score has increased by +0.20!
**Would someone have an explanation? ** Thanks!
code :
tfidf = TfidfVectorizer(
stop_words=my_stopwords_str,
max_df=0.95,
min_df=5,
token_pattern=r'w3,',
max_features=20)
tfidf.fit(data_final.all_text)
data_vect = tfidf.transform(data_final.all_text)
num_clusters = 15
kmeans = KMeans(n_clusters=num_clusters, init='k-means++',
max_iter=300).fit(data_vect_lsa)
kmeans_predict = KMeans(n_clusters=num_clusters, init='k-means++', max_iter=300).fit_predict(data_vect_lsa)
silhouette_score(data_vect, labels = kmeans_predict, metric='euclidean')
The output for Python 2.7 is :
0.58234789374593758
The output for Python 3.6 is :
0.37524101598378656
python python-3.x python-2.7 cluster-analysis
asked Nov 12 at 14:16
Themis
62
1
It's very hard to answer this without more details (preferably including code).
– Mark Dickinson
Nov 12 at 14:34
thanks for your advice, I edit my post !
– Themis
Nov 12 at 16:34
What library doesTfidfVectorizerandKMeanscome from? In general, something to look for might be divisions -- the behavior of the division operator/with integers changed from floor to true division in Python3, and if there's a hidden division with two integers somewhere in your code, that might explain the numerical discrepancy
– Linuxios
Nov 12 at 16:48
Likely, it is an issue that depends on the libraries you are using. What versions? Another thing, try to se the random-seed explicitly.
– juanpa.arrivillaga
Nov 12 at 18:32
add a comment |
I read a lot on SoF about the difference in speed between Python 2.7 and 3.6. but my question is more about performance between the two versions.
I used for document clustering: TF-IDF + KMeans and score silhouette to evaluate the homogeneity of my clusters.
By switching from Python 3.6 to Python 2.7, my silhouette score has increased by +0.20!
**Would someone have an explanation? ** Thanks!
code :
tfidf = TfidfVectorizer(
stop_words=my_stopwords_str,
max_df=0.95,
min_df=5,
token_pattern=r'w3,',
max_features=20)
tfidf.fit(data_final.all_text)
data_vect = tfidf.transform(data_final.all_text)
num_clusters = 15
kmeans = KMeans(n_clusters=num_clusters, init='k-means++',
max_iter=300).fit(data_vect_lsa)
kmeans_predict = KMeans(n_clusters=num_clusters, init='k-means++', max_iter=300).fit_predict(data_vect_lsa)
silhouette_score(data_vect, labels = kmeans_predict, metric='euclidean')
The output for Python 2.7 is :
0.58234789374593758
The output for Python 3.6 is :
0.37524101598378656
python python-3.x python-2.7 cluster-analysis
asked Nov 12 at 14:16
Themis
62
I read a lot on SoF about the difference in speed between Python 2.7 and 3.6. but my question is more about performance between the two versions.
I used for document clustering: TF-IDF + KMeans and score silhouette to evaluate the homogeneity of my clusters.
By switching from Python 3.6 to Python 2.7, my silhouette score has increased by +0.20!
**Would someone have an explanation? ** Thanks!
code :
tfidf = TfidfVectorizer(
stop_words=my_stopwords_str,
max_df=0.95,
min_df=5,
token_pattern=r'w3,',
max_features=20)
tfidf.fit(data_final.all_text)
data_vect = tfidf.transform(data_final.all_text)
num_clusters = 15
kmeans = KMeans(n_clusters=num_clusters, init='k-means++',
max_iter=300).fit(data_vect_lsa)
kmeans_predict = KMeans(n_clusters=num_clusters, init='k-means++', max_iter=300).fit_predict(data_vect_lsa)
silhouette_score(data_vect, labels = kmeans_predict, metric='euclidean')
The output for Python 2.7 is :
0.58234789374593758
The output for Python 3.6 is :
0.37524101598378656
python python-3.x python-2.7 cluster-analysis
python python-3.x python-2.7 cluster-analysis
asked Nov 12 at 14:16
Themis
62
asked Nov 12 at 14:16
Themis
62
edited Nov 12 at 16:33
asked Nov 12 at 14:16
Themis
62
asked Nov 12 at 14:16
Themis
62
asked Nov 12 at 14:16
Themis
62
62
1
It's very hard to answer this without more details (preferably including code).
– Mark Dickinson
Nov 12 at 14:34
thanks for your advice, I edit my post !
– Themis
Nov 12 at 16:34
What library doesTfidfVectorizerandKMeanscome from? In general, something to look for might be divisions -- the behavior of the division operator/with integers changed from floor to true division in Python3, and if there's a hidden division with two integers somewhere in your code, that might explain the numerical discrepancy
– Linuxios
Nov 12 at 16:48
Likely, it is an issue that depends on the libraries you are using. What versions? Another thing, try to se the random-seed explicitly.
– juanpa.arrivillaga
Nov 12 at 18:32
add a comment |
1
It's very hard to answer this without more details (preferably including code).
– Mark Dickinson
Nov 12 at 14:34
thanks for your advice, I edit my post !
– Themis
Nov 12 at 16:34
What library doesTfidfVectorizerandKMeanscome from? In general, something to look for might be divisions -- the behavior of the division operator/with integers changed from floor to true division in Python3, and if there's a hidden division with two integers somewhere in your code, that might explain the numerical discrepancy
– Linuxios
Nov 12 at 16:48
Likely, it is an issue that depends on the libraries you are using. What versions? Another thing, try to se the random-seed explicitly.
– juanpa.arrivillaga
Nov 12 at 18:32
1
1
It's very hard to answer this without more details (preferably including code).
– Mark Dickinson
Nov 12 at 14:34
It's very hard to answer this without more details (preferably including code).
– Mark Dickinson
Nov 12 at 14:34
thanks for your advice, I edit my post !
– Themis
Nov 12 at 16:34
thanks for your advice, I edit my post !
– Themis
Nov 12 at 16:34
What library does
TfidfVectorizer and KMeans come from? In general, something to look for might be divisions -- the behavior of the division operator / with integers changed from floor to true division in Python3, and if there's a hidden division with two integers somewhere in your code, that might explain the numerical discrepancy– Linuxios
Nov 12 at 16:48
What library does
TfidfVectorizer and KMeans come from? In general, something to look for might be divisions -- the behavior of the division operator / with integers changed from floor to true division in Python3, and if there's a hidden division with two integers somewhere in your code, that might explain the numerical discrepancy– Linuxios
Nov 12 at 16:48
Likely, it is an issue that depends on the libraries you are using. What versions? Another thing, try to se the random-seed explicitly.
– juanpa.arrivillaga
Nov 12 at 18:32
Likely, it is an issue that depends on the libraries you are using. What versions? Another thing, try to se the random-seed explicitly.
– juanpa.arrivillaga
Nov 12 at 18:32
add a comment |
1 Answer
1
active
oldest
votes
Try again. A single sample is not enough.
K-means begins with a random setting, and may find a local optimum only.
It's fairly common to see different results when running it multiple times.
answered Nov 12 at 18:24
Anony-Mousse
57.2k796159
in particular, random seeds might be different across different versions.
– juanpa.arrivillaga
Nov 12 at 18:32
1
Well, right now the seed is not fixed at all, so every run could be different. But of course version differences can also change random values generated.
– Anony-Mousse
Nov 12 at 18:33
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53264034%2fwhy-a-better-performance-silhouette-score-in-python-2-7-than-3-6%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Try again. A single sample is not enough.
K-means begins with a random setting, and may find a local optimum only.
It's fairly common to see different results when running it multiple times.
answered Nov 12 at 18:24
Anony-Mousse
57.2k796159
in particular, random seeds might be different across different versions.
– juanpa.arrivillaga
Nov 12 at 18:32
1
Well, right now the seed is not fixed at all, so every run could be different. But of course version differences can also change random values generated.
– Anony-Mousse
Nov 12 at 18:33
add a comment |
Try again. A single sample is not enough.
K-means begins with a random setting, and may find a local optimum only.
It's fairly common to see different results when running it multiple times.
answered Nov 12 at 18:24
Anony-Mousse
57.2k796159
in particular, random seeds might be different across different versions.
– juanpa.arrivillaga
Nov 12 at 18:32
1
Well, right now the seed is not fixed at all, so every run could be different. But of course version differences can also change random values generated.
– Anony-Mousse
Nov 12 at 18:33
add a comment |
Try again. A single sample is not enough.
K-means begins with a random setting, and may find a local optimum only.
It's fairly common to see different results when running it multiple times.
answered Nov 12 at 18:24
Anony-Mousse
57.2k796159
Try again. A single sample is not enough.
K-means begins with a random setting, and may find a local optimum only.
It's fairly common to see different results when running it multiple times.
answered Nov 12 at 18:24
Anony-Mousse
57.2k796159
answered Nov 12 at 18:24
Anony-Mousse
57.2k796159
answered Nov 12 at 18:24
Anony-Mousse
57.2k796159
answered Nov 12 at 18:24
Anony-Mousse
57.2k796159
57.2k796159
in particular, random seeds might be different across different versions.
– juanpa.arrivillaga
Nov 12 at 18:32
1
Well, right now the seed is not fixed at all, so every run could be different. But of course version differences can also change random values generated.
– Anony-Mousse
Nov 12 at 18:33
add a comment |
in particular, random seeds might be different across different versions.
– juanpa.arrivillaga
Nov 12 at 18:32
1
Well, right now the seed is not fixed at all, so every run could be different. But of course version differences can also change random values generated.
– Anony-Mousse
Nov 12 at 18:33
in particular, random seeds might be different across different versions.
– juanpa.arrivillaga
Nov 12 at 18:32
in particular, random seeds might be different across different versions.
– juanpa.arrivillaga
Nov 12 at 18:32
1
1
Well, right now the seed is not fixed at all, so every run could be different. But of course version differences can also change random values generated.
– Anony-Mousse
Nov 12 at 18:33
Well, right now the seed is not fixed at all, so every run could be different. But of course version differences can also change random values generated.
– Anony-Mousse
Nov 12 at 18:33
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53264034%2fwhy-a-better-performance-silhouette-score-in-python-2-7-than-3-6%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
It's very hard to answer this without more details (preferably including code).
– Mark Dickinson
Nov 12 at 14:34
thanks for your advice, I edit my post !
– Themis
Nov 12 at 16:34
What library does
TfidfVectorizerandKMeanscome from? In general, something to look for might be divisions -- the behavior of the division operator/with integers changed from floor to true division in Python3, and if there's a hidden division with two integers somewhere in your code, that might explain the numerical discrepancy– Linuxios
Nov 12 at 16:48
Likely, it is an issue that depends on the libraries you are using. What versions? Another thing, try to se the random-seed explicitly.
– juanpa.arrivillaga
Nov 12 at 18:32