2026-11-11T09:58:51Z confirm queue start (felisyn-controller-023-confirm, runs folder /felisyn-local/run-023r2/runs); attempts may start for 172800 s 2026-11-11T09:58:51Z guard passed (before s8-a1): no other queue, model run or memcap job is alive; 9001228 kB free on the disk 2026-11-11T09:58:52Z s8-a1 start (pynn_seed 8) 2026-11-11T11:15:56Z s8-a1: the server check passed 2026-11-11T11:16:22Z s8-a1: its Mozaik result folder archived, verified, recorded and removed 2026-11-11T11:16:22Z guard passed (before s8-a2): no other queue, model run or memcap job is alive; 8863068 kB free on the disk 2026-11-11T11:16:23Z s8-a2 start (pynn_seed 8) 2026-11-11T13:40:12Z s8-a2: the server check passed 2026-11-11T13:40:22Z s8-a1-a2: the two runs do not agree 2026-11-11T13:41:30Z s8-a2: its Mozaik result folder archived, verified, recorded and removed 2026-11-11T13:41:30Z guard passed (before s8-a3): no other queue, model run or memcap job is alive; 6074400 kB free on the disk 2026-11-11T13:41:32Z s8-a3 start (pynn_seed 8) 2026-11-11T16:45:05Z s8-a3: the server check passed 2026-11-11T16:45:10Z s8-a1-a3: the two runs agree, and their server checks are the same 2026-11-11T16:45:57Z s8-a3: its Mozaik result folder archived, verified, recorded and removed 2026-11-11T16:45:57Z s8 settled: agreed a1 a3 2026-11-11T16:45:57Z guard passed (before s9-a1): no other queue, model run or memcap job is alive; 6394268 kB free on the disk 2026-11-11T16:46:00Z s9-a1 start (pynn_seed 9) 2026-11-11T18:08:37Z s9-a1 was stopped at the disk floor of 3 GiB: it is not counted, and the queue waits for free disk 2026-11-11T18:08:37Z the archive of s9-a1 was stopped at the disk floor of 3 GiB: the queue waits for free disk and makes it again 2026-11-11T18:08:37Z waiting for free disk (after the archive of s9-a1 stopped at the disk floor): 3129372 kB free; the queue goes on at 4194304 kB, 1 GiB above the floor of 3 GiB, and checks every 5m 2026-11-11T18:13:37Z free disk (after the archive of s9-a1 stopped at the disk floor): 5531224 kB after 300 s; the queue goes on 2026-11-11T18:13:40Z s9-a1: its Mozaik result folder archived, verified, recorded and removed 2026-11-11T18:13:40Z guard passed (before s9-a2): no other queue, model run or memcap job is alive; 5525288 kB free on the disk 2026-11-11T18:13:42Z s9-a2 start (pynn_seed 9) 2026-11-11T19:21:39Z s9-a2 was stopped at the disk floor of 3 GiB: it is not counted, and the queue waits for free disk 2026-11-11T19:21:43Z s9-a2: its Mozaik result folder archived, verified, recorded and removed 2026-11-11T19:21:43Z guard passed (before s9-a3): no other queue, model run or memcap job is alive; 5024180 kB free on the disk 2026-11-11T19:21:51Z s9-a3 start (pynn_seed 9) 2026-11-11T19:38:28Z s9-a3 was stopped at the disk floor of 3 GiB: it is not counted, and the queue waits for free disk 2026-11-11T19:38:28Z the archive of s9-a3 was stopped at the disk floor of 3 GiB: the queue waits for free disk and makes it again 2026-11-11T19:38:28Z waiting for free disk (after the archive of s9-a3 stopped at the disk floor): 2706808 kB free; the queue goes on at 4194304 kB, 1 GiB above the floor of 3 GiB, and checks every 5m 2026-11-11T19:43:29Z free disk (after the archive of s9-a3 stopped at the disk floor): 5624828 kB after 301 s; the queue goes on 2026-11-11T19:43:31Z s9-a3: its Mozaik result folder archived, verified, recorded and removed 2026-11-11T19:43:31Z guard passed (before s9-a4): no other queue, model run or memcap job is alive; 5624460 kB free on the disk 2026-11-11T19:43:33Z s9-a4 start (pynn_seed 9) 2026-11-11T20:41:33Z s9-a4 was stopped at the disk floor of 3 GiB: it is not counted, and the queue waits for free disk 2026-11-11T20:41:34Z the archive of s9-a4 was stopped at the disk floor of 3 GiB: the queue waits for free disk and makes it again 2026-11-11T20:41:34Z waiting for free disk (after the archive of s9-a4 stopped at the disk floor): 2969280 kB free; the queue goes on at 4194304 kB, 1 GiB above the floor of 3 GiB, and checks every 5m 2026-11-11T20:46:34Z free disk (after the archive of s9-a4 stopped at the disk floor): 5522244 kB after 300 s; the queue goes on 2026-11-11T20:46:36Z s9-a4: its Mozaik result folder archived, verified, recorded and removed 2026-11-11T20:46:36Z guard passed (before s9-a5): no other queue, model run or memcap job is alive; 5520112 kB free on the disk 2026-11-11T20:46:39Z s9-a5 start (pynn_seed 9) 2026-11-11T21:06:36Z s9-a5 was stopped at the disk floor of 3 GiB: it is not counted, and the queue waits for free disk 2026-11-11T21:06:36Z the archive of s9-a5 was stopped at the disk floor of 3 GiB: the queue waits for free disk and makes it again 2026-11-11T21:06:36Z waiting for free disk (after the archive of s9-a5 stopped at the disk floor): 3128048 kB free; the queue goes on at 4194304 kB, 1 GiB above the floor of 3 GiB, and checks every 5m 2026-11-11T21:11:38Z free disk (after the archive of s9-a5 stopped at the disk floor): 4672776 kB after 302 s; the queue goes on 2026-11-11T21:11:47Z s9-a5: its Mozaik result folder archived, verified, recorded and removed 2026-11-11T21:11:48Z guard passed (before s9-a6): no other queue, model run or memcap job is alive; 4662164 kB free on the disk 2026-11-11T21:11:56Z s9-a6 start (pynn_seed 9) 2026-11-11T21:19:03Z s9-a6 was stopped at the disk floor of 3 GiB: it is not counted, and the queue waits for free disk 2026-11-11T21:19:04Z the archive of s9-a6 was stopped at the disk floor of 3 GiB: the queue waits for free disk and makes it again 2026-11-11T21:19:04Z waiting for free disk (after the archive of s9-a6 stopped at the disk floor): 3120232 kB free; the queue goes on at 4194304 kB, 1 GiB above the floor of 3 GiB, and checks every 5m 2026-11-11T21:34:04Z free disk (after the archive of s9-a6 stopped at the disk floor): 7740852 kB after 900 s; the queue goes on 2026-11-11T21:34:08Z s9-a6: its Mozaik result folder archived, verified, recorded and removed 2026-11-11T21:34:09Z guard passed (before s9-a7): no other queue, model run or memcap job is alive; 7917312 kB free on the disk 2026-11-11T21:34:25Z s9-a7 start (pynn_seed 9) 2026-11-11T21:59:20Z s9-a7 was stopped at the disk floor of 3 GiB: it is not counted, and the queue waits for free disk 2026-11-11T21:59:23Z s9-a7: its Mozaik result folder archived, verified, recorded and removed 2026-11-11T21:59:23Z guard passed (before s9-a8): no other queue, model run or memcap job is alive; 5273976 kB free on the disk 2026-11-11T21:59:26Z s9-a8 start (pynn_seed 9) 2026-11-11T23:45:38Z s9-a8 was stopped at the disk floor of 3 GiB: it is not counted, and the queue waits for free disk 2026-11-11T23:45:41Z s9-a8: its Mozaik result folder archived, verified, recorded and removed 2026-11-11T23:45:41Z waiting for free disk (after s9-a8 stopped at the disk floor, before s9-a9): 3949288 kB free; the queue goes on at 4194304 kB, 1 GiB above the floor of 3 GiB, and checks every 5m 2026-11-11T23:50:41Z free disk (after s9-a8 stopped at the disk floor, before s9-a9): 6135672 kB after 300 s; the queue goes on 2026-11-11T23:50:41Z guard passed (before s9-a9): no other queue, model run or memcap job is alive; 6135572 kB free on the disk 2026-11-11T23:50:43Z s9-a9 start (pynn_seed 9) 2026-11-12T00:26:48Z s9-a9 was stopped at the disk floor of 3 GiB: it is not counted, and the queue waits for free disk 2026-11-12T00:26:50Z the archive of s9-a9 was stopped at the disk floor of 3 GiB: the queue waits for free disk and makes it again 2026-11-12T00:26:50Z waiting for free disk (after the archive of s9-a9 stopped at the disk floor): 2658896 kB free; the queue goes on at 4194304 kB, 1 GiB above the floor of 3 GiB, and checks every 5m 2026-11-12T00:31:50Z free disk (after the archive of s9-a9 stopped at the disk floor): 5189324 kB after 300 s; the queue goes on 2026-11-12T00:31:52Z s9-a9: its Mozaik result folder archived, verified, recorded and removed 2026-11-12T00:31:52Z s9 settled: attempts used up 2026-11-12T00:31:52Z guard passed (before s10-a1): no other queue, model run or memcap job is alive; 5190980 kB free on the disk 2026-11-12T00:31:54Z s10-a1 start (pynn_seed 10) 2026-11-12T00:42:27Z s10-a1 was stopped at the disk floor of 3 GiB: it is not counted, and the queue waits for free disk 2026-11-12T00:42:27Z the archive of s10-a1 was stopped at the disk floor of 3 GiB: the queue waits for free disk and makes it again 2026-11-12T00:42:27Z waiting for free disk (after the archive of s10-a1 stopped at the disk floor): 2930668 kB free; the queue goes on at 4194304 kB, 1 GiB above the floor of 3 GiB, and checks every 5m 2026-11-12T00:47:27Z free disk (after the archive of s10-a1 stopped at the disk floor): 5047724 kB after 300 s; the queue goes on 2026-11-12T00:47:29Z s10-a1: its Mozaik result folder archived, verified, recorded and removed 2026-11-12T00:47:30Z guard passed (before s10-a2): no other queue, model run or memcap job is alive; 5056940 kB free on the disk 2026-11-12T00:47:32Z s10-a2 start (pynn_seed 10) 2026-11-12T01:13:30Z s10-a2 was stopped at the disk floor of 3 GiB: it is not counted, and the queue waits for free disk 2026-11-12T01:13:31Z the archive of s10-a2 was stopped at the disk floor of 3 GiB: the queue waits for free disk and makes it again 2026-11-12T01:13:31Z waiting for free disk (after the archive of s10-a2 stopped at the disk floor): 3122856 kB free; the queue goes on at 4194304 kB, 1 GiB above the floor of 3 GiB, and checks every 5m 2026-11-12T01:18:31Z free disk (after the archive of s10-a2 stopped at the disk floor): 6187548 kB after 300 s; the queue goes on 2026-11-12T01:18:33Z s10-a2: its Mozaik result folder archived, verified, recorded and removed 2026-11-12T01:18:33Z guard passed (before s10-a3): no other queue, model run or memcap job is alive; 6183272 kB free on the disk 2026-11-12T01:18:35Z s10-a3 start (pynn_seed 10) 2026-11-12T03:13:09Z s10-a3: the server check passed 2026-11-12T03:13:41Z s10-a3: its Mozaik result folder archived, verified, recorded and removed 2026-11-12T03:13:42Z guard passed (before s10-a4): no other queue, model run or memcap job is alive; 9142868 kB free on the disk 2026-11-12T03:13:43Z s10-a4 start (pynn_seed 10) 2026-11-12T04:41:36Z s10-a4: the server check passed 2026-11-12T04:41:38Z s10-a3-a4: the two runs agree, and their server checks are the same 2026-11-12T04:42:07Z s10-a4: its Mozaik result folder archived, verified, recorded and removed 2026-11-12T04:42:07Z s10 settled: agreed a3 a4 2026-11-12T04:42:07Z guard passed (before s11-a1): no other queue, model run or memcap job is alive; 11092868 kB free on the disk 2026-11-12T04:42:08Z s11-a1 start (pynn_seed 11) 2026-11-12T05:56:03Z s11-a1: the server check passed 2026-11-12T05:56:40Z s11-a1: its Mozaik result folder archived, verified, recorded and removed 2026-11-12T05:56:40Z guard passed (before s11-a2): no other queue, model run or memcap job is alive; 11556592 kB free on the disk 2026-11-12T05:56:41Z s11-a2 start (pynn_seed 11) 2026-11-12T07:17:19Z s11-a2: the server check passed 2026-11-12T07:17:21Z s11-a1-a2: the two runs agree, and their server checks are the same 2026-11-12T07:17:51Z s11-a2: its Mozaik result folder archived, verified, recorded and removed 2026-11-12T07:17:51Z s11 settled: agreed a1 a2 2026-11-12T07:17:51Z guard passed (before s12-a1): no other queue, model run or memcap job is alive; 11012108 kB free on the disk 2026-11-12T07:17:53Z s12-a1 start (pynn_seed 12) 2026-11-12T08:55:48Z s12-a1: the server check passed 2026-11-12T08:56:41Z s12-a1: its Mozaik result folder archived, verified, recorded and removed 2026-11-12T08:56:41Z guard passed (before s12-a2): no other queue, model run or memcap job is alive; 7873228 kB free on the disk 2026-11-12T08:56:43Z s12-a2 start (pynn_seed 12) 2026-11-12T10:36:24Z s12-a2: the server check passed 2026-11-12T10:36:26Z s12-a1-a2: the two runs agree, and their server checks are the same 2026-11-12T10:36:57Z s12-a2: its Mozaik result folder archived, verified, recorded and removed 2026-11-12T10:36:57Z s12 settled: agreed a1 a2 2026-11-12T10:36:57Z guard passed (before s13-a1): no other queue, model run or memcap job is alive; 9265768 kB free on the disk 2026-11-12T10:36:58Z s13-a1 start (pynn_seed 13) 2026-11-12T11:49:42Z s13-a1: the server check passed 2026-11-12T11:50:10Z s13-a1: its Mozaik result folder archived, verified, recorded and removed 2026-11-12T11:50:10Z guard passed (before s13-a2): no other queue, model run or memcap job is alive; 10271484 kB free on the disk 2026-11-12T11:50:11Z s13-a2 start (pynn_seed 13) 2026-11-12T13:00:55Z s13-a2: the server check refused the export (a one-off fault unless an agreeing pair repeats it) 2026-11-12T13:00:59Z s13-a1-a2: the two runs do not agree 2026-11-12T13:01:27Z s13-a2: its Mozaik result folder archived, verified, recorded and removed 2026-11-12T13:01:27Z guard passed (before s13-a3): no other queue, model run or memcap job is alive; 10292856 kB free on the disk 2026-11-12T13:01:28Z s13-a3 start (pynn_seed 13) 2026-11-12T14:28:32Z s13-a3: the server check passed 2026-11-12T14:28:34Z s13-a1-a3: the two runs agree, and their server checks are the same 2026-11-12T14:29:11Z s13-a3: its Mozaik result folder archived, verified, recorded and removed 2026-11-12T14:29:11Z s13 settled: agreed a1 a3 2026-11-12T14:29:11Z reserve 14 for slot 9: the attempts used up 2026-11-12T14:29:11Z guard passed (before s14-a1): no other queue, model run or memcap job is alive; 7892736 kB free on the disk 2026-11-12T14:29:13Z s14-a1 start (pynn_seed 14) 2026-11-12T16:02:22Z s14-a1 was stopped at the disk floor of 3 GiB: it is not counted, and the queue waits for free disk 2026-11-12T16:02:23Z the archive of s14-a1 was stopped at the disk floor of 3 GiB: the queue waits for free disk and makes it again 2026-11-12T16:02:23Z waiting for free disk (after the archive of s14-a1 stopped at the disk floor): 2425496 kB free; the queue goes on at 4194304 kB, 1 GiB above the floor of 3 GiB, and checks every 5m 2026-11-12T16:07:23Z free disk (after the archive of s14-a1 stopped at the disk floor): 6383688 kB after 300 s; the queue goes on 2026-11-12T16:07:26Z s14-a1: its Mozaik result folder archived, verified, recorded and removed 2026-11-12T16:07:26Z guard passed (before s14-a2): no other queue, model run or memcap job is alive; 6376460 kB free on the disk 2026-11-12T16:07:27Z s14-a2 start (pynn_seed 14) 2026-11-12T17:15:29Z s14-a2 was stopped at the disk floor of 3 GiB: it is not counted, and the queue waits for free disk 2026-11-12T17:15:30Z the archive of s14-a2 was stopped at the disk floor of 3 GiB: the queue waits for free disk and makes it again 2026-11-12T17:15:30Z waiting for free disk (after the archive of s14-a2 stopped at the disk floor): 3117572 kB free; the queue goes on at 4194304 kB, 1 GiB above the floor of 3 GiB, and checks every 5m 2026-11-12T17:20:30Z free disk (after the archive of s14-a2 stopped at the disk floor): 5450764 kB after 300 s; the queue goes on 2026-11-12T17:20:33Z s14-a2: its Mozaik result folder archived, verified, recorded and removed 2026-11-12T17:20:33Z guard passed (before s14-a3): no other queue, model run or memcap job is alive; 5448308 kB free on the disk 2026-11-12T17:20:34Z s14-a3 start (pynn_seed 14) 2026-11-12T17:38:37Z s14-a3 was stopped at the disk floor of 3 GiB: it is not counted, and the queue waits for free disk 2026-11-12T17:38:37Z the archive of s14-a3 was stopped at the disk floor of 3 GiB: the queue waits for free disk and makes it again 2026-11-12T17:38:37Z waiting for free disk (after the archive of s14-a3 stopped at the disk floor): 3136804 kB free; the queue goes on at 4194304 kB, 1 GiB above the floor of 3 GiB, and checks every 5m 2026-11-12T17:43:37Z free disk (after the archive of s14-a3 stopped at the disk floor): 5452108 kB after 300 s; the queue goes on 2026-11-12T17:43:39Z s14-a3: its Mozaik result folder archived, verified, recorded and removed 2026-11-12T17:43:40Z guard passed (before s14-a4): no other queue, model run or memcap job is alive; 5449676 kB free on the disk 2026-11-12T17:43:41Z s14-a4 start (pynn_seed 14) 2026-11-12T18:08:45Z s14-a4 was stopped at the disk floor of 3 GiB: it is not counted, and the queue waits for free disk 2026-11-12T18:08:45Z the archive of s14-a4 was stopped at the disk floor of 3 GiB: the queue waits for free disk and makes it again 2026-11-12T18:08:45Z waiting for free disk (after the archive of s14-a4 stopped at the disk floor): 3142868 kB free; the queue goes on at 4194304 kB, 1 GiB above the floor of 3 GiB, and checks every 5m 2026-11-12T18:13:46Z free disk (after the archive of s14-a4 stopped at the disk floor): 5242232 kB after 301 s; the queue goes on 2026-11-12T18:13:48Z s14-a4: its Mozaik result folder archived, verified, recorded and removed 2026-11-12T18:13:48Z guard passed (before s14-a5): no other queue, model run or memcap job is alive; 5235820 kB free on the disk 2026-11-12T18:13:49Z s14-a5 start (pynn_seed 14) 2026-11-12T20:52:40Z s14-a5: the server check passed 2026-11-12T20:53:39Z s14-a5: its Mozaik result folder archived, verified, recorded and removed 2026-11-12T20:53:39Z guard passed (before s14-a6): no other queue, model run or memcap job is alive; 8977620 kB free on the disk 2026-11-12T20:53:41Z s14-a6 start (pynn_seed 14) 2026-11-12T23:53:35Z s14-a6: the server check passed 2026-11-12T23:53:39Z s14-a5-a6: the two runs agree, and their server checks are the same 2026-11-12T23:54:28Z s14-a6: its Mozaik result folder archived, verified, recorded and removed 2026-11-12T23:54:28Z s14 settled: agreed a5 a6 2026-11-12T23:54:28Z confirm queue done: s8 agreed a1 a3; s9 attempts used up; s10 agreed a3 a4; s11 agreed a1 a2; s12 agreed a1 a2; s13 agreed a1 a3; s14 agreed a5 a6;