Development for LHC@home

I am getting these errors:

12/05/16 16:12:01 (pid:4505) ReliSock::put_file_with_permissions(): Failed to stat file '/var/lib/condor/execute/dir_4505/pilot.out': No such file or directory (errno: 2, si_error: 1)
12/05/16 16:12:02 (pid:4505) DoUpload: (Condor error code 13, subcode 2) STARTER at 10.0.2.15 failed to send file(s) to <188.184.94.254:9618>: error reading from /var/lib/condor/execute/dir_4505/pilot.out: (errno 2) No such file or directory; SHADOW failed to receive file(s) from <95.222.151.3:52834>
12/05/16 16:12:02 (pid:4505) JICShadow::notifyJobTermination(): Sending mock terminate event.
12/05/16 16:12:02 (pid:4505) JIC::transferOutput() failed, waiting for job lease to expire or for a reconnect attempt
12/05/16 16:12:02 (pid:4505) Returning from CStarter::JobReaper()
12/05/16 16:12:02 (pid:4505) Got SIGQUIT. Performing fast shutdown.
12/05/16 16:12:02 (pid:4505) ShutdownFast all jobs.
12/05/16 16:12:02 (pid:4505) Lost connection to shadow, waiting 86300 secs for reconnect
12/05/16 16:12:02 (pid:4505) Failed to send job exit status to shadow
12/05/16 16:12:02 (pid:4505) **** condor_starter (condor_STARTER) pid 4505 EXITING WITH STATUS 0

Thanks for reporting this. We have noticed them too. Am investigating ...

Appears to be working again, as of about 1h ago.

Yes, we have been having some trouble with the LHCb submission recently. On top of that it looks like an LHCb software version was bumped and the application was removed from CVMFS. The error you saw was due to the pilots failing and so the output not being available.

Tasks failing after about 3.5h-- Heartbeat missing.

http://lhcathomedev.cern.ch/vLHCathome-dev/result.php?resultid=291697

There is this interesting post on the topic. Once the consolidation is stable, we can focus on things like this.

How can it be, that after 13h it decides, that the heartbeat is missing???

The only way to get a valid result is to shut a task down manually, before it reaches the end.


http://lhcathomedev.cern.ch/vLHCathome-dev/result.php?resultid=291794

The VM freezes. This could be related to the post. If the system is loaded in such a way that the internal clock slows down, this could result in the heartbeat slowing down to a point where the external monitor thinks it is dead.

Would it not be better to check the heartbeat file for ANY change?
Even if it is not up to date, but if it has changed since the last time it was checked, that should be sufficient to declare the VM is still alive.

Disregard this post.

The "LHCb jobs" graphs are not working.