AGI Hardware is Here: Synapse, Neuron, Memristor, and DRAM (3/X)

The four basic circuit components are the capacitor, memristor, resistor, and inductor. For folks who haven’t followed more theoretical electrical engineering, memristor is like a resistor whose resistance can be increased by applying a current or a EM field in one direction or reversed in another direction.

DRAM stores a single bit of information with 1 transistor and 1 capacitor. These by convention store only binary values, but likely in theoretical aspects, we could store more information, as from what I recall, the bit value of 0 is probably only 20%-30% of capacitor capacity, whereas bit value of 1 is 70-80% of capacitor capacity.

Biological synapses are essentially a bundle of memristors that link two neurons together. From a ANN-perspective, these memristors store information as resistances are floating point values rather than decimal. If you’ve dabbled in software engineering, to store more precision, you’ll need more memory.

The “emergent” behavior behind bigger models seems to be holding true, albeit architectures have changed over the years.

For example, seemingly massive amount of information can be “compressed” into a neural network, e.g. 0.040B parameters for AlphaGo Zero, though it assembles a Monte Carlo Tree Search during run time.

Fast-forward to 2022 and later, ChatGPT 3 had around 175B parameters, ChatGPT 3.5 has 20B, though this seems to be distilled. GPT-4 had ~1750B parameters using a mixture of experts model.

Just this week, California now allows Waymo to be used throughout the Bay Area, and my AI-assisted best guess is FSD 14 is around 5-10B parameters.

Compare that with the neocortex, which has 160T!