# Felisyn rehearsal 008: the fresh-host recipe in a clean userland

Recorded 22 October 2026 (Europe/Vienna). The rehearsal ran on 21 October 2026 from 21:56:59 to 22:16:54 UTC.

The consolidated execution-003 installation recipe was run unchanged inside a clean Ubuntu 24.04 userland. All **21 steps** ended with their expected outcome. The rebuilt stack reproduced the unchanged upstream small-model regression (**10 of 10** exact-equality comparisons), and it passed the protocol-006 preparation checks: **22/22** unit tests, **78/78** runtime checks including **187** installed source files, and **82/82** parameter schemas. The full-model launcher's hardware preflight refused this small host, as intended. No full-model simulation was run.

This is an installation and software-regression result. It is not a full-model run, a paper reproduction, biological validation or controller evidence.

## Why

A full-model run should not be spent debugging installation. The protocol-006 README noted that the consolidated recipe had not been executed end to end on a second clean machine. This rehearsal provides a clean userland on the same machine, at no additional cost. It does not provide a second machine.

## Method

1. `rehearse.sh` (host side, as root) downloaded the official Ubuntu 24.04 server cloud-image root filesystem, serial `release-20261017`. It verified the signed `SHA256SUMS` with the Ubuntu cloud-image keyring ("Good signature from UEC Image Automatic Signing Key") and checked the SHA-256 of `ubuntu-24.04-server-cloudimg-amd64-root.tar.xz` (`ec6972af…a39f9a03`). Records are in `cloud-image/` and `rehearse.log`.
2. The image was unpacked into a new directory. The published execution-003 and protocol-006 packages were copied in after all 135 + 78 files were checked against their manifests on the server (check output not retained).
3. `in-chroot.sh` ran inside a `chroot` within private mount and PID namespaces (`unshare --mount --pid --fork`). The host's own `/opt/felisyn` runtime was neither used nor changed. The recipe's scripts ran unchanged. Two adaptations were needed for a chroot: the host's upstream DNS resolvers were copied into the image, because a chroot has no `systemd-resolved`, and `/proc`, `/sys` and `/dev` were mounted inside the private namespace. No mounts remained afterwards.
4. Inside, the script followed the execution-003 README's "Fresh-host recipe" as written: `bootstrap-host.sh`, the lock files, `build-python.sh`, `fetch-runtime-sources.sh` and `build-runtime.sh`. It then ran the small-model regression, inspection, plotting and environment capture. Next came the protocol-006 README's preparation commands as the unprivileged `felisyn` user. The launcher was run in preflight mode only.
5. Separately, the script ran the identity checks that `launch.py --execute` performs after its hardware gate (37 packaged model-file hashes, 5 source commits, clean tracked trees). It also ran two Open MPI checks: 16 ranks on this 4-core host, expected to be refused, and a 4-rank `mpi4py` + NEST import smoke test.

`compare_environment.py` compares the retained receipts with the original execution-003 passing build. `draw_rehearsal.py` plots the measured step times and outcomes from the receipts.

## Results

| Step group | Wall-clock time | Outcome |
| --- | --- | --- |
| Install system packages | 72 s | pass |
| Build Python 3.9.20 | 162 s | pass |
| Fetch five pinned sources | 9 s | all commits verified |
| Build NEST 3.4, step-current module and Python packages | 841 s | pass; `pip check` clean |
| Small-model regression, diagnostics, plot, environment | 104 s | 10/10 exact comparisons |
| protocol-006 and launch-path checks | 7 s | as expected (see below) |
| **Total** | **1,195 s** | 21/21 expected outcomes |

About half a minute before these steps went to fetching, verifying and unpacking the image (`rehearse.log`).

- Regression: status `upstream_exact_regression_pass`, 10 observed cases, 94.2 s for the test command. The example figure data and the full diagnostics file are **byte-identical** to the passing run of 7 October (UTC) (`figure-data.json` SHA-256 `e2f68c2a…c0353a`). The 48 saved segment files were decoded in the matching environment and compared with the original raw output. Their content is identical: 2,208 spike trains and 144 analog signals, with the same values, times, units, recorded-neuron identifiers and segment annotations (`raw-output-comparison.json`, `compare_raw_output.py`). All 48 segment files nevertheless differ at the byte level; the cause was not investigated. Three further files differ: `log` and `sim_info.json` contain run dates and times, and `datastore.recordings.pickle` was not decoded. Seven files are byte-identical.
- Python environment: all 66 pinned PyPI package versions are identical to the original build, and all 72 installed package versions match the contract. The five source commits are identical.
- System packages: the key build dependencies have the same versions, including GCC 11.5.0, CMake 3.28.3, Open MPI 4.1.6-7ubuntu2, GSL, Boost, BLAS/LAPACK and glibc. The exception is the kernel headers (`linux-libc-dev` 6.8.0-142 → 6.8.0-147). After bootstrap, the clean userland had 769 packages and the original provider image 821; 50 shared packages differ in version.
- protocol-006: 22/22 unit tests, 78/78 runtime checks with 187 matching installed Python files, and 82/82 parameter schemas. Identity checks: all 37 model files and 5 source commits match. The hardware preflight refused the host (4 CPUs and about 7.8 GiB, below the declared floors), and no simulation started.
- Open MPI: 16 ranks were refused with "not enough slots available" on this host, as expected, while 4 ranks started and imported NEST 3.4.

## Findings for the full-model run

1. **The frozen launch needs 16 cores that the guest can see.** By default (no `--oversubscribe` or `--use-hwthread-cpus`), Ubuntu's Open MPI 4.1.6 allocates one slot per core visible to the virtual machine, not per hardware thread. The frozen launcher runs a plain `mpirun -np 16`. By that documented default, a host that presents 16 vCPUs as 8 cores × 2 threads would refuse the frozen launch before any simulation. This host has one thread per core, so the rehearsal did not test that case. Any host for the frozen launcher must be checked for its presented topology (`lscpu`), not just its vCPU count. The protocol is unchanged; this is a host requirement.
2. **Physical cores may matter for the timeout.** The released model README reports about 1 h 30 min for the spontaneous protocol, run with `param_spont`, on "16 processes on a EPYC 7302 CPU with 128 GB of RAM". The authors' Slurm launcher requests `--hint=nomultithread`, meaning one rank per physical core. On many cloud providers one vCPU is one hardware thread, so a 16-vCPU machine may have only 8 physical cores. Running 16 ranks on 8 cores could approach the frozen 3-hour process timeout. This is an inference, not a measurement.
3. **Documentation correction.** `check_compatibility.py` changes into the model directory before writing its output. The protocol-006 README's relative `--output compatibility-check.json` therefore writes `model/compatibility-check.json`, which is what happened here. The receipt was retrieved from that location afterwards (`receipts/protocol-006/compatibility-check.json`, SHA-256 `92be61ae…f4474e3bbc`). This does not affect the launcher, which passes absolute paths, or the contract hashes, which cover the listed model files only. Use absolute output paths. The published protocol-006 package is not edited.
4. **Setup time.** On this 4-vCPU host, the recipe built in about 18 minutes. The build scripts use four parallel jobs, so a larger host will not necessarily be much faster.

## Limits

- The rehearsal shared the original virtual machine, kernel and CPU model. It is a clean **userland**, not a second clean host, so hardware and kernel differences remain untested.
- Ubuntu repositories are not snapshotted. Installing later will resolve newer system package versions.
- One rehearsal: the timings are single observations without uncertainty.
- The regression is the upstream tiny step-current test. It cannot establish anything about full-model dynamics, memory use or runtime.

## Evidence map

- `rehearse.sh`, `in-chroot.sh`: the exact scripts that ran, byte-identical to the copies on the server.
- `rehearse.log`: host-side log with image serial, signature verification, checksum, start and end.
- `cloud-image/`: `image.json` (serial, file, SHA-256), the signed `SHA256SUMS` and `SHA256SUMS.gpg`.
- `receipts/logs/`: one log per step and `steps.tsv` with UTC start, end, seconds and exit codes.
- `receipts/build-evidence/`: package reports, `requirements-freeze.txt`, `system-packages-initial.tsv`, source commits, environment. The NEST source export tar was omitted; its hash is kept in `nest-source-export.sha256`.
- `receipts/regression-run/`: `run.json`, JUnit XML, pytest log, GNU time resources, diagnostics, figure data, regression figure and the unmodified raw output. Pickle files require the matching environment.
- `receipts/protocol-006/`: runtime and compatibility receipts.
- `receipts-archive.tar.gz` and `.sha256`: the archive as produced inside the rehearsal (SHA-256 `915dae66…2e5850e24`), identical on server and download. The extracted files above come from it; the compatibility receipt was retrieved separately, as described.
- `comparison.json` (environment and derived regression data) and `raw-output-comparison.json` (decoded recordings), with the scripts that produced them.
- `result-summary.json`, `figures/rehearsal-steps.png` and `.svg`.
- `manifest.json`: SHA-256 and size for every file except itself, the zip and the X drafts; `rehearsal-evidence.zip` contains the manifested files and the manifest.

## Reproduce

On an Ubuntu 24.04 x86_64 host with `unshare`, as root, place `rehearse.sh` in `/srv/felisyn-rehearsal-008/` with the execution-003 and protocol-006 packages under `kit/` and `in-chroot.sh` at `kit/in-chroot.sh`. Then run `bash rehearse.sh`. The newest cloud-image serial will be used, and Ubuntu packages will have moved on. To recompute the comparison and figure from this package: `python compare_environment.py --original ../execution-003 --output comparison.json`, then `python draw_rehearsal.py` (matplotlib).

## Attribution

Model: Antolík et al. (2024), *A comprehensive data-driven model of cat primary visual cortex*, [PLOS Computational Biology](https://doi.org/10.1371/journal.pcbi.1012342); LSV1M v1.0 under its MIT license. Framework and test reference data: [Mozaik v0.4.0](https://github.com/antolikjan/mozaik/tree/f5c09a0eccc5786daf78881d883bbce586a00e6d) (CeCILL). The figure is a new plot of retained timing and check receipts. It depicts no neural activity and reproduces no paper figure. No animal recordings were collected. Ubuntu is a trademark of Canonical Ltd.; the image was used unmodified apart from the DNS file and the packages copied into `/tmp`.
