Why Brain-like Computers Are Hard

🌈 Abstract

The article discusses the challenges in building brain-like or neuromorphic computers, which aim to replicate the capabilities of the human brain. It compares the key differences between the brain and traditional computer architectures, such as the brain's low power consumption, parallelism, and ability to operate amidst chaos. The article then explores various efforts to create "silicon neurons" and memristor-based neuromorphic hardware, as well as the challenges faced in these approaches. Finally, it suggests the potential for hybrid systems that combine neuromorphic and traditional computing architectures.

🙋 Q&A

[01] Why Brain-like Computers Are Hard

1. What are the key differences between the brain and traditional computer architectures?

• The brain operates at low power (12-20 watts) and low clock frequency (around 10 Hz), yet is highly capable, while traditional computers like desktop PCs and AI accelerators use much more power (175-700 watts).
• The brain uses massive parallelism, with 86 billion neurons firing their signals independently without a central clock, in contrast to the synchronized, centralized operation of digital circuits like CPUs.
• The brain embraces "chaos" and noise, with many neuron spikes going nowhere, while traditional computers work hard to eliminate any noise or errors.

2. What are the key challenges in replicating the brain's capabilities in hardware?

• Implementing the tight integration of memory, computation, and communication in the brain, rather than the separate memory and CPU components in traditional Von Neumann architecture.
• Capturing the diversity of neuron properties, such as differences in signal propagation speeds and preferred stimuli.
• Replicating the brain's analog, asynchronous, and noisy behavior in digital hardware.

[02] Going Neuromorphic

1. How do "silicon neuron" chips like IBM's TrueNorth attempt to mimic the brain's architecture?

• TrueNorth has 4,096 "neurosynaptic cores", each with 256 neurons and 256 axons fully connected in a crossbar structure, similar to the brain's neurons and synapses.
• The cores operate asynchronously, with neurons updating their "membrane potentials" and firing spikes based on the incoming spikes, without a global clock.
• To introduce randomness and noise, each core has a random number generator that can affect spike transmission.

2. What are the limitations of the CMOS-based "silicon neuron" approach?

• There are tradeoffs between energy efficiency and accuracy, with more power needed to achieve competitive accuracy.
• The circuits are relatively large, as each core needs its own memory, unlike the dense central memory in Von Neumann architectures.
• The digital nature of CMOS is not well-suited to replicating the brain's analog behavior.

3. How do memristor-based approaches aim to address the limitations of CMOS-based neuromorphic hardware?

• Memristors can act as non-volatile, analog memory elements that can mimic the behavior of biological synapses.
• The resistance of memristors can be programmed and changed based on the history of voltage applied, allowing them to store and process information in a more brain-like manner.
• However, memristor-based approaches face challenges in manufacturability, uniformity, and programming the non-linear resistance changes.

[03] Conclusion

1. What is the key challenge that neuromorphic computing faces in competing with traditional AI hardware?

• The rapid progress in traditional AI hardware, such as GPUs, following "Huang's Law" where AI inference performance doubles every two years, making it difficult for neuromorphic approaches to catch up.

2. What is the suggested approach to address this challenge?

• Developing hybrid systems that combine neuromorphic and traditional computing architectures, leveraging the strengths of both approaches.