Engrams: the unit of memory

The unit of a single thought form (the engram) is, basically, a large pool of toggles where only some small number of them are active at a given time.

• --

Google's [new generalist agent](https://storage.googleapis.com/deepmind-media/A%20Generalist%20Agent/Generalist%20Agent.pdf) is basically a study in using large linear maps to translate a given input (posed as one of these engrams, synthesized) to an output (another engram, synthesized.) There is nothing new about "autoencoders" but now one has been used to deliberately "solve" a number of distinct problems using just one bigger codebook.

[Doc2Vec](https://arxiv.org/pdf/1405.4053v2.pdf) have looked in to how engrams encoding the meaning of words can be determined by trying to force one to predict output words and documents. They then just take the middle layer and use it for other shenanigans like seeing how similar two documents are.

[Butterfly Transforms](https://dawn.cs.stanford.edu/2019/06/13/butterfly/) have examined how linear mapping not only solves a multitude of real world complex math problems but how they can be rearranged in a particular way to use the same model with different parameters to efficiently do it.

[Numenta's cortical "spatial pooling"](https://numenta.com/resources/biological-and-machine-intelligence/spatial-pooling-algorithm/) and sparse distributed representations attempt to model memory from studying actual brains of living systems. The cortical machinery has a very complex repeating circuit but most of that circuitry seems to exist to conditionally fire or inhibit outputs which, when viewed from far away, create a kind of huge bitmap of which parts of the machinery are active given some input.

[Cortical's "Semantic Folding Theory"](https://iceworks.cc/z/ckjxo2z5r00001dis1dsfftl2.html) tries to immitate what Word2Vec and Doc models do but for Numenta's cortical format. While Google models just sort of accidentally do it while trying to learn to immitate outputs from inputs the Cortical one uses some deliberate steps to create the bitmap from a system of input words. I've privately replicated at least the creation of the initial maps but haven't done much more with these yet.

It's interesting that the hip neural network school and the more classical AI approaches are converging on the same ideas of how individual thought forms are held. It seems that conjecture was right in that thoughts are stored in a large format where they are defined solely in relation to one another. There are also other inevitable and sometimes hilarious problems like the taste of an apple being similar to a color. You wouldn't think it makes any sense but some number of ons and offs have to go somewhere and the distance between those patterns has to exist because of it.

Personally I'm looking in to Numenta and classical Hidden Markov models still. Google has impressive results but they are still reliant on piling an unreasonable amount of compute behind everything. Numenta and Hidden Markovs *are* capable of real-time learning.