Construct a Synthetic Neural Network

In the beginning of this year (2015) I finished building and testing a single asynchronous digital neuron. I've now finished building a network of 36 of them. Most of this work has been in building the support hardware for the synthetic neural network. Training the network is going to be another several months of work, of course, and the subject of future writing.

The build cost was low: c < $150 for me (I used scavenged parts for the case and the fancy giant switch)

$200 or less estimated cost for someone who already owns a soldering iron and wire.

The device is an asynchronous neural network in a tiny synthetic universe. It also contains a connectome control module with which to design learning algorithms.

Build and debugging time for me to reach this point with this specific device -- starting from nothing -- was about 200 hours according to my time log.

Core system components:

FPGA

Chaos clock

Connectome control unit

Universe control unit

Peripheral support system components:

FT232H USB-to-serial bridge

Voltage Regulator

AVR dragon (AVR ISP)

USB blaster (FPGA JTAG)

USB hub

This entire mess began because I was thinking about how free will in humans might be argued for.* The argument ended up involving continuity of time.

Whether or not that argument holds water (it doesn't : no arguments do : only empirical test data passes muster for me), it led me to wonder whether an artificial neural network could be made to be continuous in time. Most neural networks are implemented in software in discretized-time systems (though memristors, PAAs, and their ilk allow for some neuromorphic architecture in continuous time) and I am curious if any interesting chaotic effects arise when the neural network is allowed to be asynchronous.

The individual asynchronous digital neuron was developed in an Altera EP4CE6E22C8N. After adding the required multiplexing logic, the network of 36 neurons occupies roughly 4000 logic elements, 65% of the chip. But that's just for the 36 neurons and associated synapse multiplexers, i.e., a blank network that can be rewired however based on external commands, with I/O lines that map directly to synapse input lines (e.g. nerve_splice_into_layer_0_axon_in_0) and to axon output lines (e.g. nerve_tap_from_layer_5_axon_out_3). There are 6 output lines, 1 DTACK (data transfer acknowledge) line, and 18 input lines. "Sounds like neat hardware" you say. Indeed it is, imaginary reader! However, to actually USE it I need to have some way to interface with the thing (I named it Xenolux!), and so our saga begins...

What's necessary to use the synthetic neural network? For starters, a Connectome Control Unit. There must be an external processor which tells the network how to wire. Every neuron has 6 synapses; there is an excitatory and an inhibitory synapse of each of the following weights: 0.17, 0.5, and 1.0. Each synapse has a tiny bit (4 bits, so a nibble, actually) of RAM. The bitcode in the synapse's RAM controls the synapse's multiplexer, which decides where it is connected. Each synapse may be connected to nothing (null, ground, no signal), to any of the 6 axons coming in from the previous layer of synthetic neurons, or to a "stim_line", which supplies a constant stimulating signal (kind of the opposite of a null connection).

Synapses are accessed individually by setting 9 address bits which are formatted as follows: 3 bits to select which LAYER of 6 neurons, 3 bits to select which NEURON out of the 6 within the selected layer, and 3 bits to select which SYNAPSE of the 6 within the neuron. The "mux_write_enable" data line (which does exactly what it says on the tin) is asserted a little bit after the synapse's address and the desired synapse bitcode are applied.

*based on nondeterminism caused by the absence of local hidden variables
such that the precise moments in which wavefunctions collapse would be truly random, and that this shift in time would enable modification of network operation in a chaotic nonlinear fashion.

The design was implemented on an Altera EP4CE6E22C8N. A development board with this chip can be found cheaply on ebay.

The FPGA .bdf files are hosted at the github repository.

A shy creature, the Connectome Control Unit tugs on the address lines of xenolux in order to determine which synapse multiplexer should have what bytecode. Later an evolutionary algorithm will be implemented in this processor in order to enable guided learning. Much later, unguided. Currently you can use the splice(); command to connect the network, which is good fun all its own.

The universe control unit can send pulses into the network and receive pulses from the network. Currently[in which current shall hereby be known as 2015-12-01], pulses are sent into nerve_splices by a simple direct OR gate, but later implementations may use a rising edge detector to narrow the input pulses and avoid latching the neurons for too long.

Pulses are read from nerve_taps, which latch SR latches connected to AVRuc_alpha; the individual pulses the network uses are too narrow to reliably read by my equipment or by the AVR, so I have the signals set a bank of SR latches which the AVR periodically polls and resets.

Since the synthetic neural network on the FPGA is asynchronous, it is possible to use a signal source for the FPGA's internal stim_line which bears no resemblance to a traditional clock signal. This "chaos clock" signal is derived from two basic 74XX inverter oscillators and a UJT relaxation oscillator. The outputs from these three normal and sane clock signal sources are then XORed together in XOR gates constructed from NOT, AND, and OR gates (XOR made by implementing the sum-of-products solution to the XOR function's Karnaugh map). Due to delays in the physical wiring, and the fact that three unsynchronized and frequency-drift-prone oscillators are the signal source, the output signal from the chaos clock ends up looking like a mountain range, a nonperiodic mountain range at that. Extraction of randomness from the fabric of reality itself may enable the manifestation of free will in the resulting device.

The system has some support components.

A voltage regulator (runs 2.6V to the AVRs so they can talk to the 2.5V logic of the FPGA and HOPEFULLY the 3.3V logic of the.. FT232H (UM232H-B module) {if not, I've got a bunch more soldering to do | for a level shifter}

The FT232H provides data lines between the computer and the ATMEGA32s (Connectome Control Unit and Universe Control Unit).

A USB Blaster programs the FPGA.

An AVR Dragon programs the AVRs.

A scavenged parallel switch switches which AVR is programmed.

A scavenged ammo can houses the entire device.

This step is boring unless you're building one of these! These are pictures of my design log of every data transmitting connection in the device, and the bytecodes used by the connectome control unit to access individual synapses. In the connectome control unit's software these codes are abstracted away inside the function splice();

Next? After sleep? Software! This system needs algorithms... I will be starting by reading carefully and readapting the methods used within the paper "Evolvable hardware: genetic search in a physical realm".