論文出處：Attention Is All You Need

Model Architecture

Scaled Dot-Product Attention

$$\mathrm {Attention}(Q, K, V) = \mathrm {softmax}(\frac {QK^T}{\sqrt {d_k}})V$$

參考連結

Multi-Head Attention

$$\mathrm {MultiHead}(Q, K, V) = \mathrm {Concate(head_1, ...., head_h)}W^O$$

where $\mathrm {head_i}$ = $\mathrm {Attention}(QW_i^Q, KW_i^K, VW_i^V)$