Effective Approaches to Attention-based Neural Machine Translation

Hidden states from neurons going in to the encoder are also provided to the encoder. However there is a layer which assigns a weight to each hidden context and those hidden contexts are all merged proportional to this weight before being fed in to the encoder-decoder layer.

Conjecture: they are finally starting to approximate the dendrite connections in brains. Clusters of cortical neurons whos purpose is to contextually inhibit firing of their columns.

Search help: "neural transformers" paper.