-
|
Hi! Please correct me if I am wrong, but I saw here that |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
|
When you look at the definition of lm_head you'll see the forward pass is dense -> nonlinearities -> output layer. So the 480,480 is the first of a 2layer MLP. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you! |
Beta Was this translation helpful? Give feedback.
When you look at the definition of lm_head you'll see the forward pass is dense -> nonlinearities -> output layer. So the 480,480 is the first of a 2layer MLP.
And the output layer is shared with the transformer input token embedding.