Python/Gensim - What is the meaning of syn0 and syn0norm?
I know that in gensims KeyedVectors-model, one can access the embedding matrix by the attribute model.syn0. There is also a syn0norm, which doesn't seem to work for the glove model I recently loaded. I think I also have seen syn1 somewhere previously.
I haven't found a doc-string for this and I'm just wondering what's the logic behind this?
So if syn0 is the embedding matrix, what is syn0norm? What would then syn1 be and generally, what does syn stand for?
python deep-learning nlp gensim word-embedding
asked Nov 14 '18 at 13:56
blue-phoenoxblue-phoenox
4,216101743
add a comment |
I know that in gensims KeyedVectors-model, one can access the embedding matrix by the attribute model.syn0. There is also a syn0norm, which doesn't seem to work for the glove model I recently loaded. I think I also have seen syn1 somewhere previously.
I haven't found a doc-string for this and I'm just wondering what's the logic behind this?
So if syn0 is the embedding matrix, what is syn0norm? What would then syn1 be and generally, what does syn stand for?
python deep-learning nlp gensim word-embedding
asked Nov 14 '18 at 13:56
blue-phoenoxblue-phoenox
4,216101743
add a comment |
I know that in gensims KeyedVectors-model, one can access the embedding matrix by the attribute model.syn0. There is also a syn0norm, which doesn't seem to work for the glove model I recently loaded. I think I also have seen syn1 somewhere previously.
I haven't found a doc-string for this and I'm just wondering what's the logic behind this?
So if syn0 is the embedding matrix, what is syn0norm? What would then syn1 be and generally, what does syn stand for?
python deep-learning nlp gensim word-embedding
asked Nov 14 '18 at 13:56
blue-phoenoxblue-phoenox
4,216101743
I know that in gensims KeyedVectors-model, one can access the embedding matrix by the attribute model.syn0. There is also a syn0norm, which doesn't seem to work for the glove model I recently loaded. I think I also have seen syn1 somewhere previously.
I haven't found a doc-string for this and I'm just wondering what's the logic behind this?
So if syn0 is the embedding matrix, what is syn0norm? What would then syn1 be and generally, what does syn stand for?
python deep-learning nlp gensim word-embedding
python deep-learning nlp gensim word-embedding
asked Nov 14 '18 at 13:56
blue-phoenoxblue-phoenox
4,216101743
asked Nov 14 '18 at 13:56
blue-phoenoxblue-phoenox
4,216101743
edited Nov 14 '18 at 19:21
blue-phoenox
asked Nov 14 '18 at 13:56
blue-phoenoxblue-phoenox
4,216101743
asked Nov 14 '18 at 13:56
blue-phoenoxblue-phoenox
4,216101743
asked Nov 14 '18 at 13:56
blue-phoenoxblue-phoenox
4,216101743
4,216101743
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
These names were inherited from the original Google word2vec.c implementation, upon which the gensim Word2Vec class was based. (I believe syn0 only exists in recent versions for backward-compatbility.)
The syn0 array essentially holds raw word-vectors. From the perspective of the neural-network used to train word-vectors, these vectors are a 'projection layer' that can convert a one-hot encoding of a word into a dense embedding-vector of the right dimensionality.
Similarity operations tend to be done on the unit-normalized versions of the word-vectors. That is, vectors that have all been scaled to have a magnitude of 1.0. (This makes the cosine-similarity calculation easier.) The syn0norm array is filled with these unit-normalized vectors, the first time they're needed.
This syn0norm will be empty until either you do an operation (like most_similar()) that requires it, or you explicitly do an init_sims() call. If you explicitly do an init_sims(replace=True) call, you'll actually clobber the raw vectors, in-place, with the unit-normed vectors. This saves the memory that storing both vectors for every word would otherwise require. (However, some word-vector uses may still be interested in the original raw vectors of varying magnitudes, so only do this when you're sure most_similar() cosine-similarity operations are all you'll need.)
The syn1 (or syn1neg in the more common case of negative-sampling training) properties, when they exist on a full model (and not for a plain KeyedVectors object of only word-vectors), are the model neural network's internal 'hidden' weights leading to the output nodes. They're needed during model training, but not a part of the typical word-vectors collected after training.
I believe the syn prefix is just a convention from neural-network variable-naming, likely derived from 'synapse'.
answered Nov 16 '18 at 7:12
gojomogojomo
19.8k64467
Great explanation, thank you!
– blue-phoenox
Nov 16 '18 at 7:21
@gojomo When flattening my word2vec model, I was using vecotrs = model.wv.syn0 and I am getting a deprecation warning. Is there a new way to extract the vector matrix?
– Tyler Russell
Feb 6 at 16:00
@TylerRussell What was the text of the deprecation warning?
– gojomo
Feb 6 at 17:46
DeprecationWarning: Call to deprecatedsyn0(Attribute will be removed in 4.0.0, use self.vectors instead). I played around with my IDE and found model.wv.vectors and have been assuming that's what the message wanted me to use. Is that wrong?
– Tyler Russell
Feb 6 at 21:02
Yes, that message means you should use.vectorsinstead of.syn0.
– gojomo
Feb 6 at 23:05
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53301916%2fpython-gensim-what-is-the-meaning-of-syn0-and-syn0norm%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
These names were inherited from the original Google word2vec.c implementation, upon which the gensim Word2Vec class was based. (I believe syn0 only exists in recent versions for backward-compatbility.)
The syn0 array essentially holds raw word-vectors. From the perspective of the neural-network used to train word-vectors, these vectors are a 'projection layer' that can convert a one-hot encoding of a word into a dense embedding-vector of the right dimensionality.
Similarity operations tend to be done on the unit-normalized versions of the word-vectors. That is, vectors that have all been scaled to have a magnitude of 1.0. (This makes the cosine-similarity calculation easier.) The syn0norm array is filled with these unit-normalized vectors, the first time they're needed.
This syn0norm will be empty until either you do an operation (like most_similar()) that requires it, or you explicitly do an init_sims() call. If you explicitly do an init_sims(replace=True) call, you'll actually clobber the raw vectors, in-place, with the unit-normed vectors. This saves the memory that storing both vectors for every word would otherwise require. (However, some word-vector uses may still be interested in the original raw vectors of varying magnitudes, so only do this when you're sure most_similar() cosine-similarity operations are all you'll need.)
The syn1 (or syn1neg in the more common case of negative-sampling training) properties, when they exist on a full model (and not for a plain KeyedVectors object of only word-vectors), are the model neural network's internal 'hidden' weights leading to the output nodes. They're needed during model training, but not a part of the typical word-vectors collected after training.
I believe the syn prefix is just a convention from neural-network variable-naming, likely derived from 'synapse'.
answered Nov 16 '18 at 7:12
gojomogojomo
19.8k64467
Great explanation, thank you!
– blue-phoenox
Nov 16 '18 at 7:21
@gojomo When flattening my word2vec model, I was using vecotrs = model.wv.syn0 and I am getting a deprecation warning. Is there a new way to extract the vector matrix?
– Tyler Russell
Feb 6 at 16:00
@TylerRussell What was the text of the deprecation warning?
– gojomo
Feb 6 at 17:46
DeprecationWarning: Call to deprecatedsyn0(Attribute will be removed in 4.0.0, use self.vectors instead). I played around with my IDE and found model.wv.vectors and have been assuming that's what the message wanted me to use. Is that wrong?
– Tyler Russell
Feb 6 at 21:02
Yes, that message means you should use.vectorsinstead of.syn0.
– gojomo
Feb 6 at 23:05
add a comment |
These names were inherited from the original Google word2vec.c implementation, upon which the gensim Word2Vec class was based. (I believe syn0 only exists in recent versions for backward-compatbility.)
The syn0 array essentially holds raw word-vectors. From the perspective of the neural-network used to train word-vectors, these vectors are a 'projection layer' that can convert a one-hot encoding of a word into a dense embedding-vector of the right dimensionality.
Similarity operations tend to be done on the unit-normalized versions of the word-vectors. That is, vectors that have all been scaled to have a magnitude of 1.0. (This makes the cosine-similarity calculation easier.) The syn0norm array is filled with these unit-normalized vectors, the first time they're needed.
This syn0norm will be empty until either you do an operation (like most_similar()) that requires it, or you explicitly do an init_sims() call. If you explicitly do an init_sims(replace=True) call, you'll actually clobber the raw vectors, in-place, with the unit-normed vectors. This saves the memory that storing both vectors for every word would otherwise require. (However, some word-vector uses may still be interested in the original raw vectors of varying magnitudes, so only do this when you're sure most_similar() cosine-similarity operations are all you'll need.)
The syn1 (or syn1neg in the more common case of negative-sampling training) properties, when they exist on a full model (and not for a plain KeyedVectors object of only word-vectors), are the model neural network's internal 'hidden' weights leading to the output nodes. They're needed during model training, but not a part of the typical word-vectors collected after training.
I believe the syn prefix is just a convention from neural-network variable-naming, likely derived from 'synapse'.
answered Nov 16 '18 at 7:12
gojomogojomo
19.8k64467
Great explanation, thank you!
– blue-phoenox
Nov 16 '18 at 7:21
@gojomo When flattening my word2vec model, I was using vecotrs = model.wv.syn0 and I am getting a deprecation warning. Is there a new way to extract the vector matrix?
– Tyler Russell
Feb 6 at 16:00
@TylerRussell What was the text of the deprecation warning?
– gojomo
Feb 6 at 17:46
DeprecationWarning: Call to deprecatedsyn0(Attribute will be removed in 4.0.0, use self.vectors instead). I played around with my IDE and found model.wv.vectors and have been assuming that's what the message wanted me to use. Is that wrong?
– Tyler Russell
Feb 6 at 21:02
Yes, that message means you should use.vectorsinstead of.syn0.
– gojomo
Feb 6 at 23:05
add a comment |
These names were inherited from the original Google word2vec.c implementation, upon which the gensim Word2Vec class was based. (I believe syn0 only exists in recent versions for backward-compatbility.)
The syn0 array essentially holds raw word-vectors. From the perspective of the neural-network used to train word-vectors, these vectors are a 'projection layer' that can convert a one-hot encoding of a word into a dense embedding-vector of the right dimensionality.
Similarity operations tend to be done on the unit-normalized versions of the word-vectors. That is, vectors that have all been scaled to have a magnitude of 1.0. (This makes the cosine-similarity calculation easier.) The syn0norm array is filled with these unit-normalized vectors, the first time they're needed.
This syn0norm will be empty until either you do an operation (like most_similar()) that requires it, or you explicitly do an init_sims() call. If you explicitly do an init_sims(replace=True) call, you'll actually clobber the raw vectors, in-place, with the unit-normed vectors. This saves the memory that storing both vectors for every word would otherwise require. (However, some word-vector uses may still be interested in the original raw vectors of varying magnitudes, so only do this when you're sure most_similar() cosine-similarity operations are all you'll need.)
The syn1 (or syn1neg in the more common case of negative-sampling training) properties, when they exist on a full model (and not for a plain KeyedVectors object of only word-vectors), are the model neural network's internal 'hidden' weights leading to the output nodes. They're needed during model training, but not a part of the typical word-vectors collected after training.
I believe the syn prefix is just a convention from neural-network variable-naming, likely derived from 'synapse'.
answered Nov 16 '18 at 7:12
gojomogojomo
19.8k64467
These names were inherited from the original Google word2vec.c implementation, upon which the gensim Word2Vec class was based. (I believe syn0 only exists in recent versions for backward-compatbility.)
The syn0 array essentially holds raw word-vectors. From the perspective of the neural-network used to train word-vectors, these vectors are a 'projection layer' that can convert a one-hot encoding of a word into a dense embedding-vector of the right dimensionality.
Similarity operations tend to be done on the unit-normalized versions of the word-vectors. That is, vectors that have all been scaled to have a magnitude of 1.0. (This makes the cosine-similarity calculation easier.) The syn0norm array is filled with these unit-normalized vectors, the first time they're needed.
This syn0norm will be empty until either you do an operation (like most_similar()) that requires it, or you explicitly do an init_sims() call. If you explicitly do an init_sims(replace=True) call, you'll actually clobber the raw vectors, in-place, with the unit-normed vectors. This saves the memory that storing both vectors for every word would otherwise require. (However, some word-vector uses may still be interested in the original raw vectors of varying magnitudes, so only do this when you're sure most_similar() cosine-similarity operations are all you'll need.)
The syn1 (or syn1neg in the more common case of negative-sampling training) properties, when they exist on a full model (and not for a plain KeyedVectors object of only word-vectors), are the model neural network's internal 'hidden' weights leading to the output nodes. They're needed during model training, but not a part of the typical word-vectors collected after training.
I believe the syn prefix is just a convention from neural-network variable-naming, likely derived from 'synapse'.
answered Nov 16 '18 at 7:12
gojomogojomo
19.8k64467
answered Nov 16 '18 at 7:12
gojomogojomo
19.8k64467
answered Nov 16 '18 at 7:12
gojomogojomo
19.8k64467
answered Nov 16 '18 at 7:12
gojomogojomo
19.8k64467
19.8k64467
Great explanation, thank you!
– blue-phoenox
Nov 16 '18 at 7:21
@gojomo When flattening my word2vec model, I was using vecotrs = model.wv.syn0 and I am getting a deprecation warning. Is there a new way to extract the vector matrix?
– Tyler Russell
Feb 6 at 16:00
@TylerRussell What was the text of the deprecation warning?
– gojomo
Feb 6 at 17:46
DeprecationWarning: Call to deprecatedsyn0(Attribute will be removed in 4.0.0, use self.vectors instead). I played around with my IDE and found model.wv.vectors and have been assuming that's what the message wanted me to use. Is that wrong?
– Tyler Russell
Feb 6 at 21:02
Yes, that message means you should use.vectorsinstead of.syn0.
– gojomo
Feb 6 at 23:05
add a comment |
Great explanation, thank you!
– blue-phoenox
Nov 16 '18 at 7:21
@gojomo When flattening my word2vec model, I was using vecotrs = model.wv.syn0 and I am getting a deprecation warning. Is there a new way to extract the vector matrix?
– Tyler Russell
Feb 6 at 16:00
@TylerRussell What was the text of the deprecation warning?
– gojomo
Feb 6 at 17:46
DeprecationWarning: Call to deprecatedsyn0(Attribute will be removed in 4.0.0, use self.vectors instead). I played around with my IDE and found model.wv.vectors and have been assuming that's what the message wanted me to use. Is that wrong?
– Tyler Russell
Feb 6 at 21:02
Yes, that message means you should use.vectorsinstead of.syn0.
– gojomo
Feb 6 at 23:05
Great explanation, thank you!
– blue-phoenox
Nov 16 '18 at 7:21
Great explanation, thank you!
– blue-phoenox
Nov 16 '18 at 7:21
@gojomo When flattening my word2vec model, I was using vecotrs = model.wv.syn0 and I am getting a deprecation warning. Is there a new way to extract the vector matrix?
– Tyler Russell
Feb 6 at 16:00
@gojomo When flattening my word2vec model, I was using vecotrs = model.wv.syn0 and I am getting a deprecation warning. Is there a new way to extract the vector matrix?
– Tyler Russell
Feb 6 at 16:00
@TylerRussell What was the text of the deprecation warning?
– gojomo
Feb 6 at 17:46
@TylerRussell What was the text of the deprecation warning?
– gojomo
Feb 6 at 17:46
DeprecationWarning: Call to deprecated
syn0 (Attribute will be removed in 4.0.0, use self.vectors instead). I played around with my IDE and found model.wv.vectors and have been assuming that's what the message wanted me to use. Is that wrong?– Tyler Russell
Feb 6 at 21:02
DeprecationWarning: Call to deprecated
syn0 (Attribute will be removed in 4.0.0, use self.vectors instead). I played around with my IDE and found model.wv.vectors and have been assuming that's what the message wanted me to use. Is that wrong?– Tyler Russell
Feb 6 at 21:02
Yes, that message means you should use
.vectors instead of .syn0.– gojomo
Feb 6 at 23:05
Yes, that message means you should use
.vectors instead of .syn0.– gojomo
Feb 6 at 23:05
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53301916%2fpython-gensim-what-is-the-meaning-of-syn0-and-syn0norm%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
