Development for LHC@home

I have many many of the herwig7's and pythia's over 2 days and even over 4 days
My longest herwig7 was Run time 22 days 3 hours

The powheg-box are usually the only ones that average 7 hours

In reply to boboviz's message of 19 Jun 2026:

I have 2 wus running for 3 days.
VERY long runners :-P

Now 5 days and 12hrs
But i'm near to deadline.... :-(

Laufzeit 3 Tage 2 Stunden 57 min. 17 sek.
CPU Zeit 2 Tage 23 Stunden 40 min. 20 sek.
Priority 0
Prüfungsstatus Gültig
Punkte 4,648.39

finished this Morning.

In reply to boboviz's message of 22 Jun 2026:

In reply to boboviz's message of 19 Jun 2026:

I have 2 wus running for 3 days.
VERY long runners :-P

Now 5 days and 12hrs
But i'm near to deadline.... :-(

No need to get nervous.

I have a running task with walltime (so far) >8 days:

===> [runRivet] Mon Jun 15 01:48:38 UTC 2026 [boinc pp z1j 13000 45 - herwig7 7.2.0 nlo-pw-dipole 100000 974]
...
91900 events processed

... and running fine.

Looking at long walltimes only(!) does not indicate a task got stuck.

Last info from the mcplots team:
They do not have a timeout that rejects any result.

BOINC also does not reject tasks that started before the deadline.


Most important:
It is possible to pause/resume Theory docker tasks but each full restart - e.g. due to a reboot - will cause the tasks to start from scratch.

In reply to computezrmle's message of 23 Jun 2026:

Last info from the mcplots team: They do not have a timeout that rejects any result.
BOINC also does not reject tasks that started before the deadline.

So, what is the sense of the deadline?

Most important:
It is possible to pause/resume Theory docker tasks but each full restart - e.g. due to a reboot - will cause the tasks to start from scratch.

:-(
I turn off my pcs often and it's not a realy chechpoint...

In reply to boboviz's message of 23 Jun 2026:

In reply to computezrmle's message of 23 Jun 2026:

Last info from the mcplots team: They do not have a timeout that rejects any result.
BOINC also does not reject tasks that started before the deadline.

So, what is the sense of the deadline?

Most important:
It is possible to pause/resume Theory docker tasks but each full restart - e.g. due to a reboot - will cause the tasks to start from scratch.

:-(
I turn off my pcs often and it's not a realy chechpoint...

One reason (for BOINC) is to avoid huge local work buffers that can't be processed before the deadline.
So, after the deadline BOINC returns unstarted tasks as errors.

The vbox app uses snapshots, this doesn't work for the docker app.
See:
https://github.com/BOINC/boinc/blob/master/samples/docker_wrapper/checkpoint_notes

In reply to computezrmle's message of 23 Jun 2026:

One reason (for BOINC) is to avoid huge local work buffers that can't be processed before the deadline.
So, after the deadline BOINC returns unstarted tasks as errors.

I'm not speaking about not-started-wus. I'm speaking about wus that arrive at deadline without completion.
What's the sense of checkpoint (for reboot, turn off of the pc, etc) if it restarted calculation from scratch?

The vbox app uses snapshots, this doesn't work for the docker app.
See:
https://github.com/BOINC/boinc/blob/master/samples/docker_wrapper/checkpoint_notes

Uh, i didn't see that point, thank you!!
I hope that, before or later, docker/podman will be more friendly about checkpoint/restart

Computer ID 4639
Laufzeit 6 Tage 5 Stunden 2 min. 52 sek.
CPU Zeit 5 Tage 22 Stunden 16 min. 55 sek.
Endstatus 0 (0x00000000)

3790564 5495 20 Jun 2026, 14:16:24 UTC 21 Jun 2026, 20:35:01 UTC Abgebrochen 103,871.66 101,751.50 --- Theory Simulation v7.68 (docker)
windows_x86_64
3791330 5406 21 Jun 2026, 20:38:06 UTC 24 Jun 2026, 12:08:36 UTC Abgebrochen 180,786.57 180,107.30 --- Theory Simulation v7.68 (docker)
x86_64-pc-linux-gnu
3792513 4639 24 Jun 2026, 14:49:01 UTC 30 Jun 2026, 19:52:29 UTC Fertig und Bestätigt 536,572.88 512,215.00 9,243.35 Theory Simulation v7.68 (docker)
windows_x86_64

Nice 6 day Valid Axel
Run time 6 days 5 hours 2 min 52 sec
CPU time 5 days 22 hours 16 min 55 sec
Priority 0
Validate state Valid - herwig7
Credit 9,243.35

It‘s your 1.001th Message in -dev. Thank you to watch it.

https://lhcathomedev.cern.ch/lhcathome-dev/result.php?resultid=3810113
This is my first long running WU
Run time 2 days 4 hours 55 min 41 sec
CPU time 2 days 1 hours 31 min 34 sec
Priority 0
Validate state Valid
Credit 4,199.21

Anwendung Theory Simulation 7.69 (docker)
Name Theory_2922-4777184-1262_1787080866
Status Aktiv
erhalten 18.08.2026 21:37:33
Ablaufdatum 28.08.2026 21:37:30
Geschätzter Berechnungsaufwand 3.600 GFLOPs
Prozessorzeit 6d 08:15:50
Prozessor-Zeit seit dem letzten Checkpoint ---
bisherige Laufzeit 6d 16:49:47
Geschätzte verbleibende Zeit ---
Fortschritt 100,000%
benötigter Arbeitsspeicher 2,16 MB
Größe des Arbeitspakets 95,37 MB
Verzeichnis slots/4
Prozess-Nr. 2012
Fortschrittsrate 0,720% pro Stunde
Ausführbare Datei docker_wrapper_22b_windows_x86_64.exe
Application Name Theory
Plan Class docker

You can see this application has been running for over 6 days now. According to the website (https://lhcathomedev.cern.ch/lhcathome-dev/results.php?userid=264&offset=0&show_names=0&state=6&appid=), the task has been aborted. What happens to this task after more than 500,000 seconds of processing time? Will it still be counted or not? I'll let it run to see when it finishes. Isn't it important to the operator to receive completely finished tasks? Why is the time limit for completing tasks so short when they know there are tasks that take considerably longer?

Anwendung Theory Simulation 7.69 (docker)
Name Theory_2922-4777184-1262_1787080866
Status Aktiv
erhalten 18.08.2026 21:37:33
Ablaufdatum 28.08.2026 21:37:30
Geschätzter Berechnungsaufwand 3.600 GFLOPs
Prozessorzeit 10d 21:05:40
Prozessor-Zeit seit dem letzten Checkpoint ---
bisherige Laufzeit 11d 10:58:22
Geschätzte verbleibende Zeit ---
Fortschritt 100,000%
benötigter Arbeitsspeicher 2,23 MB
Größe des Arbeitspakets 95,37 MB
Verzeichnis slots/4
Prozess-Nr. 29012
Fortschrittsrate 0,360% pro Stunde
Ausführbare Datei docker_wrapper_22b_windows_x86_64.exe
Application Name Theory
Plan Class docker

Still aktive since 11,5 Days

This bad boy has been running for almost 7 days and is at 3 200 / 10 000 events generated. As you can see, it has already failed once after about 9 days and a half. I checked here and "pp mb-inelastic 7000 - - pythia8 8.301 vincia-default" has a track record of 0 successes, 3 failures and 10 unknown.

According to htop, all 7 Theory processes (including this one) are using the exact same amount of resources so I assume it shouldn't be a zombie.

Is there any hope?

---

vincia is one of those once in a while event generators over the years.

Are you checking the slot runRivet for that one to make sure it is running?

In reply to Magic Quantum Mechanic's message of 3 Sep 2026:

vincia is one of those once in a while event generators over the years.

Are you checking the slot runRivet for that one to make sure it is running?

Uh, if you mean runRivet.log in within the task's slot, then it shows the same content, i.e. 3200 events. Guess that means the task is dead? No change for well over a day at least.

---

The way to always make sure they are running is to check the runRivet for the slot of the questionable task and highlight the last lines at the bottom of that page and X out and then go back and check it an hour later and see if it is still the same lines that you highlighted on the previous check at the bottom of that runRivet page.

Just checking it once and figuring it is running or just running several days and aborting because you *think* it isn't running is a waste of time.

I always make sure they are running and save all of the evidence which is why I have many long Dockers that are Valid and don't just waste time letting them run without checking to see if it is running.

Like this one today is over 100 hours and Valid https://lhcathomedev.cern.ch/lhcathome-dev/result.php?resultid=3822230

Received 20 Sep 2026, 6:35:40 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 5326
Run time 4 days 15 hours 56 min 11 sec
CPU time 4 days 10 hours 40 min 24 sec
Priority 0
Validate state Valid
Credit 4,564.05

For theory tasks that run longer than 1 day, I check the `runRivet.log` file to see if any more events are being processed:

cat /var/lib/boinc/slots/6/shared/runRivet.log

At the end of the log file, you’ll see something like this:

Event 70000 (40m 50s elapsed / 17m 30s left) -> ETA: Sun Sep 20 11:24
XS = 5.02052e+07 pb ± (189827 pb = 0.37%)
70,000 events processed
70,100 events processed
70,200 events processed
70,300 events processed
70,400 events processed
70,500 events processed
70,600 events processed
70,700 events processed
70,800 events processed
70,900 events processed
71,000 events processed
71,100 events processed
71,200 events processed
71,300 events processed
71,400 events processed

If the number increases after some time (1 hour or so), the task is still running.

Yesterday, a 5-day job ended successfully:

https://lhcathomedev.cern.ch/lhcathome-dev/result.php?resultid=3821877

https://lhcathomedev.cern.ch/lhcathome-dev/workunit.php?wuid=2841183
This Task running for 12 days and would be finished today,
but no Task in -dev defined. 10 day limit have deleted the Task entry!
Is it possible to reactivate this task in the Database?

1,110,473.23 1,056,024.00 15,976.55
Thank you for this result after 12 days of running :-))