# Felisyn feasibility 010: the full network on the existing server

Recorded 22 October 2026 (Europe/Vienna). The work ran on 22 October 2026 between 00:36 and 05:55 UTC on the project's existing 4-vCPU research server, at no additional cost.

The owner decided not to rent new servers. This package tests whether the full model can run on the server the project already has. That server has 4 vCPUs and about 7.8 GiB of memory; the declared protocol had assumed a 16-vCPU, 128 GiB machine. Four results:

1. **Splitting the 16 virtual processes differently left the tiny model's spike trains unchanged.** On the upstream tiny model, runs with the same number of virtual processes gave identical spike trains, however the processes were split into MPI ranks and threads. Voltage and conductance traces agreed to rounding level, except some first samples, whose cause was not investigated.
2. **The full network fits, with swap.** All 108,150 neurons and 99,352,247 connections were built in 3.0 h on one MPI rank with 16 threads. The run peaked at 7.5 GiB of memory plus 24.0 GiB of swap. A 0.52 s spontaneous run completed in 68 min, with no runaway activity.
3. **A full-length run looks practical.** NEST's simulation loop ran at about 760 s of wall time per simulated second. A linear projection for the released 40.77 s protocol is about 12.6 h, excluding steps the probe could not measure. The analysis program peaked at 5.6 GiB on the probe's 70 ms recordings (full neuron count), and the correlation step at 3.6 GiB at full length on synthetic data; a full-length analysis was not measured, and protocol-007 expects it to use swap.
4. **The full model's neurons are placed identically under the new split.** Built as 1 × 16, the model's neuron positions and recorder selections are identical to the 16-process build of recipe-check-009.

This is a feasibility study. It is not the full-model experiment, a paper reproduction, biological validation or controller evidence. The 0.52 s run is far too short for the Figure 4 statistics.

## Part 1: does splitting the 16 virtual processes differently change results?

NEST's documentation on parallel computing (not retained in this package) states that a simulation with a given number of virtual processes yields the same results however those processes are split into MPI ranks and threads. Mozaik draws one connection seed per target neuron from a random stream shared by all ranks (`mozaik.get_seeds(len(target.pop))`, then the local subset), so its sampled connectivity should not depend on the split either. We tested this on the unchanged upstream tiny LSV1M step-current model, the one used by the small-model regression. It simulates 1,275 ms of model time: 105 ms of spontaneous activity, two 210 ms gratings and five 150 ms blanks.

| Split (ranks × threads) | Virtual processes | Wall time | Exit |
| --- | --- | --- | --- |
| 1 × 2 (upstream reference setting) | 2 | 79 s | 0 |
| 2 × 1 | 2 | 46 s | 0 |
| 16 × 1 (protocol-006 setting) | 16 | 62 s | 0 |
| 4 × 4, OpenMP active waiting | 16 | 1,104 s | 0 |
| 1 × 16, OpenMP passive waiting | 16 | 92 s | 0 |

Results (`receipts/decomp/decomp-analysis.json`, `decomp_analysis.py`):

- **Same number of virtual processes:** 0 of 2,208 spike trains differ (16×1 vs 4×4, 16×1 vs 1×16, 1×2 vs 2×1). Apart from the first sample of each segment, all voltage and conductance traces agree within 2.2 × 10⁻⁷ (16×1 vs 4×4) and 6.5 × 10⁻⁹ (the others), consistent with floating-point summation order.
- **First recorded sample of a segment:** in 136 of 144 signal groups per pair, the first-sample values are the same set of values assigned to different neurons. Of the other 8 groups per pair, 4 differ by at most 8.2 × 10⁻¹⁰. The remaining 4 are membrane-voltage groups in segments 8–11: the cortical segments of the 150 ms blank that opens the run (stored after the first stimulus's own segments) and recorded as 151 samples. Their first-sample values differ by up to 10.0 mV between the sorted value sets, and by up to 16.3 mV per neuron (13.0 mV for 16 × 1 versus 1 × 16). All later samples agree as above. `receipts/decomp/sample0-groups.json` (`sample0_groups.py`) records this per group and supersedes the interactive check summarised in `sample0-note.json`. The cause was not investigated; the label "bookkeeping effect" in the docstring of `decomp_analysis.py`, kept as run, is not established. In the full protocol such effects can touch only the first sample of each segment per neuron.
- **Negative control (1×2 vs 16×1, different virtual-process counts):** 2,205 of 2,208 spike trains differ and traces differ by up to 40 mV. The comparison is therefore sensitive to real differences.
- **Speed:** with 16 threads on four vCPUs, OpenMP's default active waiting made the 4×4 split about 18 times slower than 16×1. Passive waiting (`OMP_WAIT_POLICY=PASSIVE`, which changes scheduling only) brought 1×16 to 92 s. The 1×16 run in the table is that rerun; the earlier active-waiting attempt was stopped (`receipts/decomp/runs-notes.tsv`).

A strict byte-for-byte comparator (`compare_raw_output.py`) reports every pair as not identical. Its receipts are kept for completeness. The differences it detects are the rounding-level and first-sample effects described above.

**Limits.** This is one tiny model over 1.275 s. Over the 40.32 s full protocol, rounding-level differences could eventually change individual spike times in a recurrent network. An alternative split is therefore treated as equivalent only to the extent tested, not as bitwise identical to a 16 × 1 run.

## Part 2: the full network on the existing server

**Setup.** `probe_feasibility.py` runs the unchanged model with the released spontaneous recipe (`param_spont/defaults`), the released seeds and 16 virtual processes as 1 MPI rank × 16 threads. It uses Mozaik's own workflow functions. The only change is the length of the single spontaneous segment: 70 ms instead of 40,320 ms. `run_probe.sh` ran it as a systemd unit with `MemoryHigh=7000M`, passive OpenMP waiting and a temporary 64 GiB swap file. It sampled host memory, swap, load and swap traffic every 15 s (`receipts/full-1x16-70ms/resources.csv`). NEST reported 520.1 ms of model time at the end: a 150 ms blank (plus PyNN's extra 0.1 ms step on the first call), the 70 ms segment, a 150 ms blank and a 150 ms terminal blank.

**Construction.** Mozaik created 7,200 + 7,200 thalamic (LGN) neurons and 37,500 + 9,375 excitatory and inhibitory neurons in each of layers 4 and 2/3: 108,150 neurons. NEST counted 99,352,247 connections in 16 projections; Mozaik's logged counts sum to the same number within their six-digit rounding. Construction took 10,767 s (2 h 59 min). Mozaik's per-projection timers account for 10,697 s of this. The largest projection, layer 2/3 excitatory to excitatory with 48.0 million connections, took 2,622 s. Memory in use reached about 7.4 GiB, just under the 7.8 GiB installed, after about 40 minutes, when swap use began (figure).

**The short run.** `run_experiments` took 4,053 s for 520.1 ms of model time. Most of that is not NEST's simulation loop. NEST resets its `time_simulate` timer at every `Simulate` call (pinned NEST 3.4 source), so the reported 114.0 s covers only the last call, the 150 ms terminal blank. That is about **760 s per simulated second**. A further 245 s went to rebuilding NEST's connection tables after the recorders were attached. The rest went to Mozaik and PyNN work under swap: preparing the visual input for each segment, setting up recorders, and retrieving and converting the recordings after each segment. Mozaik's own "simulation time" (1,344 s and 999 s for the two presentations) includes such work, so it is not a measure of NEST's loop speed.

**Memory.** Host-level peaks: 7.51 GiB of memory in use, 24.0 GiB of swap, 31.4 GiB together. Over the 4.1 h the kernel swapped in 125 GiB and swapped out 133 GiB. GNU `time` reports a peak resident size of 6.9 GiB and 13.2 million major page faults. The probe process forks one worker per thread for each projection (`mozaik/connectors/modular.py`). A per-user resident-memory sum is therefore misleading during construction, because shared pages are counted repeatedly. The analysis uses host `MemAvailable` and `SwapFree` instead.

**Activity.** Mozaik's explosion monitor did not trigger, and all recorded voltage and conductance traces are finite. Mean firing rates of the recorded neurons (`receipts/full-1x16-70ms/activity.json`):

| Segment (model time) | L4 exc | L4 inh | L2/3 exc | L2/3 inh |
| --- | --- | --- | --- | --- |
| Blank after start-up (0–150 ms) | 4.47 Hz | 28.70 Hz | 10.98 Hz | 39.78 Hz |
| Spontaneous segment (150–220 ms) | 1.61 Hz | 9.64 Hz | 2.01 Hz | 5.54 Hz |
| Blank (220–370 ms) | 1.58 Hz | 8.36 Hz | 2.37 Hz | 6.21 Hz |
| Terminal blank (370–520 ms) | 1.38 Hz | 7.46 Hz | 2.04 Hz | 4.49 Hz |

The first window is the network's start-up transient. Later windows are much quieter and stay in a similar range. These rates come from windows of 70–150 ms immediately after start-up. They cannot be compared with the paper's 40 s statistics and are not evidence about them. The rates logged by Mozaik after each presentation match the recomputed values. Two warnings in the log (`overflow encountered in power` in the retina model and `Recorded at least one spike after t_stop` in PyNN) also appear in every tiny-model run.

## Part 3: projection and analysis scale

**Projection for the released protocol** (`probe-analysis.json`): construction 3.0 h, plus NEST's loop for 40,770 ms at the probe rate, 8.6 h. Add the probe's other run overhead, 1.0 h, assumed not to grow with simulated time. Total: **about 12.6 h**. Not measured: whether the loop rate changes later in the run, when activity has settled and the recordings grow in memory, and the time to retrieve, convert and save 40 s of recordings under swap. The recordings comprise about 190 million voltage and conductance samples and roughly 31,000 spike trains. For the analog values of the measured segment alone, NEST holds about 1.55 GB as float64 values.

**Analysis at full size.** The spontaneous recording set has about 15 times as many spike trains as the set protocol-006 had planned. Two checks measured parts of the analysis: the whole program on the probe's 70 ms recordings, and the correlation step at full length on synthetic data. Loading and processing 40 s of recordings was not tested, and the peak of a full-length analysis was not measured; protocol-007 expects it to exceed physical memory and use swap.

- Protocol-006's unchanged analysis program (`validate_datastore.py`) ran on the probe's saved recordings in 7 min 22 s, with a peak resident size of 5.6 GiB. It covered all 31,258 spike trains and 1,597 voltage and conductance neurons (`receipts/analysis-scale/`). With only 70 ms per neuron, 12 of the 48 statistics are undefined: ISI CV and correlation need at least 7 spikes, and only 2 neurons, both layer 4 inhibitory, had that many. All 36 defined statistics agree between the independent helpers and the authors' Figure 4 code, with a largest difference of 8.9 × 10⁻¹⁶. The program therefore exits as failed, as designed for incomplete comparisons. The input files were unchanged.
- The correlation step grows with recording length. `scale_test_metrics.py` ran it on synthetic Poisson spike trains, not neural data, for 12,176 neurons over 40,320 ms (4,032 bins). The independent helper took 33 s and peaked at 3.2 GiB. The numerical core of Mozaik's correlation analysis took 30 s and peaked at 3.6 GiB, holding a 1.1 GiB matrix (`receipts/analysis-scale/synthetic/`).

Protocol-006's program exports every voltage and conductance sample as JSON lists. At 40 s, the roughly 190 million samples would become several GiB of Python objects. Protocol-007 therefore keeps the definitions and comparisons and writes those traces as NumPy arrays instead. Run on the same probe recordings, protocol-007's version produced an identical `analysis-parity.json` apart from its date (7 min 19 s, 5.6 GiB). Its array files hold exactly the same neuron IDs, values and timing as protocol-006's JSON export, and its spike export is identical. `compare_exports.py` checks this and wrote `receipts/analysis-007-test/export-equivalence.json`. Protocol-007's secondary analysis passed its checks on the same data in 16 s: the probe's neuron positions match the frozen hashes, and all 836, 152, 856 and 173 subset neurons are recorded. Its values over a 70 ms window are a software check, not results.

## Part 4: the same neurons and recorders under the 1 × 16 split

`subset_indices.py` repeats recipe-check-009's sheets-only construction under 1 MPI rank × 16 threads. It stops before the first connection and applies both released recorder recipes. The neuron positions of all four cortical sheets have the same SHA-256 hashes as in recipe-check-009's 16 × 1 run. So do the recorder selections of both recipes, and the counts are equal (`receipts/subset-1x16/`). It took 4 min 10 s. The script also records which spike-recorded neurons the two recipes share: 836, 152, 856 and 173 of protocol-006's 838, 152, 862 and 173. Protocol-007 freezes these neurons for a declared secondary analysis. After the output was written, the log shows a segmentation-fault report from another process during shutdown. GNU `time` reports exit status 0 for the Python process. We did not investigate the report.

## Limits

- One probe on one host. Timings are single observations without uncertainty. Swap behaviour depends on the disk and on other load.
- The 0.52 s run includes the start-up transient. The loop rate comes from one 150 ms call and may not hold for 40 s.
- The decomposition check covers the tiny model only.
- The probe used `param_spont`, the recipe identified in recipe-check-009. Protocol-006 declared `param/`. The two released trees differ in self-references, recorder definitions and the position of an unchanged `'native_nest': False` entry in three files (`l4_cortex_inh`, `l23_cortex_exc`, `l23_cortex_inh`); their values are otherwise identical (protocol-007 README).

## Evidence map

- Part 1: `decomp_check.sh`, `decomp_rest.sh`, `compare_decomp.sh`, `compare_raw_output.py`, `diag_compare.py`, `decomp_analysis.py`, `sample0_groups.py`; receipts in `receipts/decomp/` (run logs, `runs.tsv`, comparisons, `decomp-analysis.json`, `sample0-groups.json`, and the earlier `sample0-note.json`).
- Part 2: `probe_feasibility.py`, `run_probe.sh` (the unit command is in `receipts/full-1x16-70ms/unit-command.txt`); receipts in `receipts/full-1x16-70ms/`: `probe.log.gz` (complete log, gzip), `probe.json`, `resources.csv`, `time.txt`, start/end/exit files, `host.json` and `activity.json` (from `probe_activity.py`). `results-sha256.txt` lists the saved recordings (83 MB, 24 segment files), which are retained privately and on the server.
- Part 3: `analyze_probe.py` → `probe-analysis.json`; `scale_test_metrics.py` and the analysis receipts in `receipts/analysis-scale/`; protocol-007's analysis test, `compare_exports.py` and its receipts in `receipts/analysis-007-test/` (the exported arrays stay on the server; their hashes are in `exports-sha256.txt`).
- Part 4: `subset_indices.py` (a variant of recipe-check-009's `count_recorders.py`); receipts in `receipts/subset-1x16/`.
- `draw_probe.py` → `figures/probe-memory.png` and `.svg`, plotted from `resources.csv`.
- `manifest.json`: SHA-256 and size for every file except itself, the zip and the X drafts. `feasibility-evidence.zip` contains the manifested files and the manifest.

## Attribution

Model and S5 Table: Antolík J, Cagnol R, Rózsa T, Monier C, Frégnac Y, Davison AP (2024), *A comprehensive data-driven model of cat primary visual cortex*, [PLOS Computational Biology](https://doi.org/10.1371/journal.pcbi.1012342). The article and supplements are published under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/); `subset_indices.py` and `receipts/subset-1x16/subset-indices.json` contain S5 Table counts transcribed from supplement S5 (see recipe-check-009). LSV1M v1.0 is under its MIT license. Framework: [Mozaik v0.4.0](https://github.com/antolikjan/mozaik/tree/f5c09a0eccc5786daf78881d883bbce586a00e6d) (CeCILL), PyNN and NEST 3.4 as pinned in protocol-006. The figure is a new plot of retained host measurements. It shows no neural activity and reproduces no paper figure. No animal recordings were collected.
