Before the first simulation

This is an experiment plan. No simulation run has started, and no acceptance criterion has been met.

The first question is deliberately modest: can the original candidate implementation reproduce one specified quantitative result under its published conditions?

Preserve the baseline

We will attempt the original implementation before porting, reducing or adapting it. Otherwise, a mismatch could come from our changes as easily as from an incomplete reference setup.

The baseline record needs the exact code version, data files, dependency versions and commands. We will also record the hardware, numerical solver settings, stimulus protocol, initial conditions and random seeds. A runnable environment matters because a description that omits these details can leave several plausible implementations.

If the original environment cannot be made to run, that failure will stay in the record. Any replacement implementation will have its own version and an explicit list of differences.

Choose the comparison in advance

The precise figure or analysis is not selected yet. That decision depends on the package audit. We need to know what reference outputs exist and which parts of the analysis can actually be reproduced.

Possible quantities include firing rates, membrane responses or response distributions. We will choose the measure that matches the selected result, rather than selecting a metric because it makes the implementation look successful.

The acceptance tolerance must also have a reason. Reference variability, numerical precision and stochastic sampling can justify a range of expected differences. A convenient percentage selected after seeing our results cannot.

Keep fitting and validation distinct

When parameters are fitted, the fitting objective and dataset will be recorded. Where the available data permit it, independent recording units or preparations should be reserved for validation.

Matching the same data used to choose the parameters tells us something about the fit. It does not provide an independent test of the model.

What the first result will contain

The first execution report should include:

1. The question, exact reference result and comparison criteria.
2. Source versions, file checksums, environment and commands.
3. Parameters, units, input protocol and numerical settings.
4. Raw outputs and the scripts that generate each plot or summary.
5. Runtime, peak memory, warnings, failures and any exclusions.
6. The measured discrepancy and whether it meets the declared criterion.
7. The claim supported by that comparison and the remaining limitations.

A successful program exit establishes that the program ran. Reproducing a reference result, agreeing with biological measurements and working as a controller each need additional evidence.

Next step: complete the package audit, then commit to a reference result and comparison protocol before changing model parameters.