About Attention for Memory Addressig

[simaiden]

In the paper at section 3.3.2, in eq 4 shows that in MemAE each weight is computed using softmax operation and cosine similarity, but I can't find this in the code, so where this operation actually is used?

Thanks

[Wolfybox]

Cosine Similarity is not implemented in the code. Instead, the weight computing process was replaced as applying Softmax on the inner product of z and m. I tried modified it to the way as written in the paper, however I found using cosine similarity only got me to an attention weight with all elements down to zero. So I guess the cosine distance measure was not applicable as it was said in the paper, which explained why they replaced it with inner product similarity in the actual implementation.