← Journal
NOTE 14·Methods

Running the full network on our own server

The full cat visual-cortex model, 108,150 neurons and 99.4 million connections, was built and briefly run on our existing four-vCPU, 8 GB research server using swap. A full-length run projects to about 13 hours there, plus steps the test could not time; protocol 007 freezes it, and the run started on 22 October.

The full model needs far more memory than our research server has. Rather than rent a larger machine, we tested whether the existing server could run it anyway. That server has four vCPUs and about 7.8 GiB of memory, against the 16 vCPUs and 128 GiB the frozen plan had assumed. We added a 64 GiB swap file, which extends memory onto disk, and ran three checks. The complete network, 108,150 neurons and 99,352,247 connections, was built in 3.0 hours, and a 0.52-second run of spontaneous activity completed without runaway activity. Memory use peaked at 7.5 GiB plus 24.0 GiB of swap. At the measured simulation speed, the released spontaneous protocol, 40.8 seconds of model time including blank periods, would take about 13 hours on this server, plus some steps the test could not time.

This follows the plan in note 12 and the recording-set correction in note 13. The model is the published LSV1M circuit of cat primary visual cortex, layers 4 and 2/3, not a whole brain. The short run is a feasibility test, not the experiment.

Line chart over 4.1 hours of host measurements. Memory in use jumps to about 4 GiB within the first minute, reaches about 7.4 GiB by 40 minutes and then stays between about 7.0 and 7.5 GiB, just below the dashed line for the 7.8 GiB of physical memory. Swap in use starts growing after about 40 minutes, reaches about 20.6 GiB when the network is built at 3.0 hours, and peaks at 24.0 GiB during the short run, which ends at 4.1 hours. Notes below the chart: built in 3.0 hours; the 0.52 second run took 68 minutes; NEST's loop ran at about 760 seconds per simulated second in the last 150 milliseconds. No neural activity is shown.
Figure 1. Host memory and swap in use, sampled every 15 seconds while the full network was built and briefly run. Plotting code · Samples.

Splitting the work differently

The authors ran the model as 16 processes. NEST, the simulator underneath, divides the work into 16 "virtual processes"; according to its documentation, its random numbers depend on that count, not on how the virtual processes are distributed. Sixteen separate processes would each hold their own copy of the framework's data, so we expected them to need even more memory than one process; we did not try them at full size on this server. One process with 16 threads shares that memory. We first checked on the small upstream test network whether the split changes the results. With the same 16 (or 2) virtual processes split as 16 × 1, 4 × 4 or 1 × 16 processes × threads (or 1 × 2 and 2 × 1), none of the 2,208 recorded spike trains differed. Voltages and conductances agreed to within rounding, apart from the first recorded voltage sample in the segments of the opening blank period. Those differed by up to 13 millivolts per neuron between the 16 × 1 and 1 × 16 splits, for reasons we have not investigated. As a control, changing the number of virtual processes from 2 to 16 changed 2,205 of the 2,208 spike trains. One practical detail mattered a lot: with more threads than cores, waiting threads must sleep rather than spin. With spinning threads, the 4 × 4 split took 1,104 seconds, about 18 times as long as 16 × 1 (62 seconds); with sleeping threads, 1 × 16 took 92 seconds.

This was a small network over 1.3 seconds. Over a long run, rounding differences could eventually shift individual spikes, so the full run's results belong to the 1 × 16 split and are not a byte-for-byte copy of a 16-process run.

Building and running the full network

The test ran the unchanged model with the released spontaneous settings and seeds, shortening only the measured spontaneous segment from 40,320 to 70 milliseconds. Including the workflow's three 150-millisecond blank periods, NEST simulated 520.1 milliseconds of model time.

Construction took 10,767 seconds. Almost all of it went into creating the 16 groups of connections: mostly the framework computing and sampling them, plus about 28 minutes in NEST storing and indexing them. The largest group, 48 million connections within excitatory layer 2/3, took 44 minutes alone. Memory use reached about 7.4 GiB, close to the 7.8 GiB installed, after about 40 minutes, and from then on the kernel moved data between memory and disk: over the whole test, 125 GiB was read back from swap.

The short run then took 68 minutes. Most of that was not NEST's simulation loop. NEST resets its loop timer at every call, so its last reading covers only the final 150 milliseconds: 114 seconds, or about 760 seconds per simulated second. Most of the rest went to Python-side work that slows down badly under swap. That work prepares the input for each segment, sets up the recorders, and retrieves and converts the recordings after each segment.

Activity stayed bounded. Right after start-up, the recorded neurons fired briefly at high rates: layer 2/3 inhibitory cells averaged 40 spikes per second over the first 150 milliseconds. In the later windows, recorded excitatory neurons averaged 1.4–2.4 spikes per second and inhibitory neurons 4.5–9.6. These windows are 70–150 milliseconds long and immediately follow start-up. They cannot be compared with the paper's statistics, which use 40 seconds.

What the test cannot tell us

The 13-hour figure is a linear projection: 3.0 hours of construction, 8.6 hours of NEST's loop at the measured speed, and 1.0 hour of the test's other overhead. It does not include retrieving, converting and saving 40 seconds of recordings, which comprise about 190 million voltage and conductance samples and 31,000 spike trains. Nor does it cover any change in speed as activity settles and the recordings grow in memory. The run could take noticeably longer.

Parts of the analysis were tested at scale. The frozen analysis program ran on the test's recordings, which include every recorded neuron but only 70 milliseconds each, in 7 minutes with a peak of 5.6 GiB. All 36 statistics that are defined for a 70-millisecond window agreed between our independent code and the authors' figure code, to within 10⁻¹⁵. The remaining 12 need at least seven spikes per neuron and are undefined at this length. The correlation step grows with recording length, so we also timed it on synthetic random spike trains of the full size and length, which are not neural data. It took about half a minute and under 4 GiB. A full-length analysis holds all of these at once on 40 seconds of recordings, and its peak memory has not been measured; protocol 007 expects it to exceed the server's physical memory and use swap.

Protocol 007

Protocol 007 freezes the full run on this server as a versioned follow-up. Protocol 006 stays unchanged and unrun. The changes: the released spontaneous recording settings, as explained in note 13 (a correction to that note: the two settings also differ in their internal file references and in the position of one unchanged entry in three files, with no differing values); 16 virtual processes as one process with 16 sleeping-wait threads; host-specific memory and swap gates; a 72-hour time limit instead of 3 hours; and an analysis export that stores traces as numerical arrays rather than text. The model files, seeds, simulation steps, the six measurements, the 48 numerical cross-checks and the descriptive comparison with the paper's printed Figure 4 values are unchanged. A declared secondary analysis will compute the spike-based measures for the neurons that both recording settings record: 836, 152, 856 and 173 of the 838, 152, 862 and 173 that protocol 006 would have spike-recorded. A review before freezing caught a defect that would have stopped the run at its first step: the timing wrapper tried to patch NEST's module, which NEST 3.4 does not allow. The wrapper now sits on the run step of PyNN, the interface between the modelling framework and NEST, and passed a short test on the server.

The full run started on 22 October at 06:12 UTC on this server. Its results will follow in a later note.

Limits

These are single measurements on one server, and swap speed depends on its disk. The speed estimate comes from one 150-millisecond call. The split check covered only the small test network; for the full network we checked only that neuron positions and recorder selections are identical. Swapping changes speed, not arithmetic. No full-model result, biological validation or controller work exists yet.

Evidence and sources