Keras: How does embedding layers in Keras work?

[Yingyingzhang15]

Hi, I'm new to keras, and I feel quite confused when learning about the embedding layers. Can it be used to prepare data for LSTM? Or it can only be used in the NLP?
Waiting for your answer!Thank you!

[ymcui]

The embedding layer is a way to convert word indices into dense representations.
for e.g.
your input X should like [2,4,3]
and after embedding layer(say dim=3), the output should like [[0.12,0.3.-0.2],[0.4,-0.22,0.01],[0.12,0.32,0.5]]

[EderSantana]

just like @ymcui said it convert integers to vectors. It is not limited to NLP, but it is limited to integer sequences with finite range.

[Yingyingzhang15]

Thank you all for your replies!

[wangliangguo]

@ymcui So the value of the dense representations is random choose? How can i load the pretrained Embedding such as goole-word-vector?

[ymcui]

@chenwangliangguo

So the value of the dense representations is random choose?

yes, randomly initialized, and trained during BP.

How can i load the pretrained Embedding such as goole-word-vector?

using weights argument in Embedding, see http://keras.io/layers/embeddings/

[Sandy4321]

Thanks
On Mar 25, 2016 00:23, "Yiming Cui" [email protected] wrote:

So the value of the dense representations is random choose?
yes, randomly initialized, and trained during BP.
How can i load the pretrained Embedding such as goole-word-vector?
using weights argument in Embedding, see
http://keras.io/layers/embeddings/

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
https://github.com/fchollet/keras/issues/2034#issuecomment-201130792

[PiranjaF]

Are there any papers on how this works? I know quite a bit about word2vec, but I haven't seen any papers describing embedding layers as an integrated part of the network. Particularly, it could be interesting if there were papers on using this kind of layer for other tasks than NLP.

[braingineer]

Anytime you have a number of discrete things, you can use embedding layers. NLP uses it the most (and extensively.. a recent paper called it a 'veritable cottage industry') because we inherently are dealing with discrete data. Most other domains start from the continuous data and try to abstract to the discrete.

I've used embeddings, though, for state features in an RL environment, for example. For each of the RL state variables, I have a multi-hot vector which then indexes an embedding matrix (though, because it's a multihot rather than one-hot, it has to multiply the matrix). I then concatenated these embedded representations to get a state feature vector that can be used with policy/value gradient methods.

[snowxiaoru]

I using wordvec as embedding ，the input is [2,4,3]
and the result is [[0.12,0.3.-0.2],[0.4,-0.22,0.01],[0.12,0.32,0.5]]，and next I want to use dense layer as the next layer ,but the input is not the only one wordvec([0.12,0.3.-0.2]) ,but the all ,how can I build the model ?

[450586509]

the next layer's input is [0.12,0.3.-0.2,0.4,-0.22,0.01,0.12,0.32,0.5] or the mean of the three vector?

[graydan]

Do our inputs need to fill the range of 0 to input_dim?

I'm getting weird behavior and I think that's why:
My possible inputs to the embedding layer are: 0, 45, 48 to 57, 97 to 122
Thats ascii for numbers, lowercase letters, and then just 0 for padding. Also 45 for '-' character.

Training goes smoothly if I set input_dim around 128 or above. There are only 38 features, but setting input_dim to this goes horribly wrong

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs, but feel free to re-open a closed issue if needed.

[asturkmani]

@braingineer could you share your code for multi-hot indexing? I would like to create a layer that does something similar (matrix multiplies the input vector by the embedding matrix to create a sum of embedding rows)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs, but feel free to re-open a closed issue if needed.

[calcqu]

question: when embedding layer changes integers to vectors for input, how do i correspondingly change the target integers? I have made an attempt, but found the embedding layer created a 3D vector, while my target is 2D...should I just recast the target? Have not found any examples that show fitting the target (train_ydata), just train_xdata. @calcqu on twitter [email protected] - anxiously awaiting replies - many thanks!