Development for LHC@home

No-one's commented yet, but some of you may have noticed that this batch of jobs are longer than the last. That's because I want to explore if we do have problems with longer jobs or larger result files. There has been some suggestion in the past that bigger files may lead to transfer time-outs. So, I'm increasing the number of events in each job to try to provoke failures, to get some idea of where our limits are. I'll probably push on until we get 5-10% failures (i.e. approaching Atlas's failure rate), but at present this batch is looking hopeful.

---

Thanks for the warning. Could this explain the unexplained "validate errors" that
are causing some disquiet over there, one wonders. Atlas themselves don't say
much.

Thanks for the warning. Could this explain the unexplained "validate errors" that
are causing some disquiet over there, one wonders. Atlas themselves don't say
much.

No, I can't comment. I know next-to-nothing about the Atlas project.

---

No-one's commented yet, but some of you may have noticed that this batch of jobs are longer than the last. That's because I want to explore if we do have problems with longer jobs or larger result files. There has been some suggestion in the past that bigger files may lead to transfer time-outs. So, I'm increasing the number of events in each job to try to provoke failures, to get some idea of where our limits are. I'll probably push on until we get 5-10% failures (i.e. approaching Atlas's failure rate), but at present this batch is looking hopeful.

Already noticed the increase from 25 to 50 events, but on average an event is a bit faster processed in this batch, so the total runtime per job is about 160% of a job from the previous batch.

Is the result file about 66MB and transferred to you without compression?
No problem here - was transferred within ~32 seconds.

if necessary, you can make it selectable for the user if he wants to process only small or even larger Jobs

Already noticed the increase from 25 to 50 events, but on average an event is a bit faster processed in this batch, so the total runtime per job is about 160% of a job from the previous batch.

Is the result file about 66MB and transferred to you without compression?
No problem here - was transferred within ~32 seconds.

Yes to the first; not sure about the second, I've not seen anything explicit but it may be a default (not much point in compressing though, the ROOT files are already compressed).

---

Is there a chance to make it possible to use 2 or more cores?
I have tried trough the app_info.xml, but it really only uses one, even though the virtual box has two assigned.

I had a very long delay between jobs.
One hour between the end of event 50 and event 1 of the next job.

Normal is about 20 minutes.

Which log might contain any info on that? All console windows did not show anything unusual.

I had a very long delay between jobs.
One hour between the end of event 50 and event 1 of the next job.

Normal is about 20 minutes.

Which log might contain any info on that? All console windows did not show anything unusual.

There's a glide-in log, isn't there? I can't run jobs at home, due to my poor ADSL, so I have to rely on memory. Check to see if it was within one glide-in instance (which calls several successive jobs) or between glide-ins. The logs I get back are some combination of Condor and application logs, on a per-job basis so I don't think they have much inter-job information. We seem to be running smoothly at the moment, so it might have been a network blockage somewhere.

---

glidein download started 6 min before the 1 event of the following run.
This would explain 6 of the extra 40 min it took to start the next job.

If there was a network outage of some kind, would that not be logging the retries?

If there was a network outage of some kind, would that not be logging the retries?

No idea, really, I'm not a network expert. ...but I have seen connections hang for some time and come back up when the physical link is restored. As I recall, Linux is(was) far more resilient than Windows in this.

---

The delay happened before 17.12h .
If it helps, look at the bold section(or show it to someone).
I will keep an eye on it to see, if this happens again.
It is not a serious issue, as it keeps running, but if you( or someone)has the time...

Run-1 master.log

10/27/15 11:10:05 (pid:7632) ******************************************************
10/27/15 11:10:05 (pid:7632) ** condor_master (CONDOR_MASTER) STARTING UP
10/27/15 11:10:05 (pid:7632) ** /home/boinc/CMSRun/glide_6VklMu/main/condor/sbin/condor_master
10/27/15 11:10:05 (pid:7632) ** SubsystemInfo: name=MASTER type=MASTER(2) class=DAEMON(1)
10/27/15 11:10:05 (pid:7632) ** Configuration: subsystem:MASTER local:<NONE> class:DAEMON
10/27/15 11:10:05 (pid:7632) ** $CondorVersion: 8.2.3 Sep 30 2014 BuildID: 274619 $
10/27/15 11:10:05 (pid:7632) ** $CondorPlatform: x86_64_RedHat5 $
10/27/15 11:10:05 (pid:7632) ** PID = 7632
10/27/15 11:10:05 (pid:7632) ** Log last touched time unavailable (No such file or directory)
10/27/15 11:10:05 (pid:7632) ******************************************************
10/27/15 11:10:05 (pid:7632) Using config source: /home/boinc/CMSRun/glide_6VklMu/condor_config
10/27/15 11:10:05 (pid:7632) config Macros = 213, Sorted = 213, StringBytes = 10574, TablesBytes = 7708
10/27/15 11:10:05 (pid:7632) CLASSAD_CACHING is OFF
10/27/15 11:10:05 (pid:7632) Daemon Log is logging: D_ALWAYS D_ERROR
10/27/15 11:10:05 (pid:7632) DaemonCore: command socket at <10.0.2.15:56373?noUDP>
10/27/15 11:10:05 (pid:7632) DaemonCore: private command socket at <10.0.2.15:56373>
10/27/15 11:10:05 (pid:7632) CCBListener: registered with CCB server lcggwms02.gridpp.rl.ac.uk:9622 as ccbid 130.246.180.120:9622#108616
10/27/15 11:10:05 (pid:7632) Master restart (GRACEFUL) is watching /home/boinc/CMSRun/glide_6VklMu/main/condor/sbin/condor_master (mtime:1445940567)
10/27/15 11:10:05 (pid:7632) Started DaemonCore process "/home/boinc/CMSRun/glide_6VklMu/main/condor/sbin/condor_startd", pid and pgroup = 7635
10/27/15 17:12:47 (pid:7632) Got SIGTERM. Performing graceful shutdown.
10/27/15 17:12:47 (pid:7632) Sent SIGTERM to STARTD (pid 7635)
10/27/15 17:12:49 (pid:7632) AllReaper unexpectedly called on pid 7635, status 25344.
10/27/15 17:12:49 (pid:7632) The STARTD (pid 7635) exited with status 99 (daemon will not restart automatically)
10/27/15 17:12:49 (pid:7632) All daemons are gone. Exiting.
10/27/15 17:12:49 (pid:7632) **** condor_master (condor_MASTER) pid 7632 EXITING WITH STATUS 0

Run-2 master log

10/27/15 17:15:13 (pid:27729) ******************************************************
10/27/15 17:15:13 (pid:27729) ** condor_master (CONDOR_MASTER) STARTING UP
10/27/15 17:15:13 (pid:27729) ** /home/boinc/CMSRun/glide_I5t966/main/condor/sbin/condor_master
10/27/15 17:15:13 (pid:27729) ** SubsystemInfo: name=MASTER type=MASTER(2) class=DAEMON(1)
10/27/15 17:15:13 (pid:27729) ** Configuration: subsystem:MASTER local:<NONE> class:DAEMON
10/27/15 17:15:13 (pid:27729) ** $CondorVersion: 8.2.3 Sep 30 2014 BuildID: 274619 $
10/27/15 17:15:13 (pid:27729) ** $CondorPlatform: x86_64_RedHat5 $
10/27/15 17:15:13 (pid:27729) ** PID = 27729
10/27/15 17:15:13 (pid:27729) ** Log last touched time unavailable (No such file or directory)
10/27/15 17:15:13 (pid:27729) ******************************************************
10/27/15 17:15:13 (pid:27729) Using config source: /home/boinc/CMSRun/glide_I5t966/condor_config
10/27/15 17:15:13 (pid:27729) config Macros = 213, Sorted = 213, StringBytes = 10579, TablesBytes = 7708
10/27/15 17:15:13 (pid:27729) CLASSAD_CACHING is OFF
10/27/15 17:15:13 (pid:27729) Daemon Log is logging: D_ALWAYS D_ERROR
10/27/15 17:15:13 (pid:27729) DaemonCore: command socket at <10.0.2.15:35944?noUDP>
10/27/15 17:15:13 (pid:27729) DaemonCore: private command socket at <10.0.2.15:35944>
10/27/15 17:15:14 (pid:27729) CCBListener: registered with CCB server lcggwms02.gridpp.rl.ac.uk:9620 as ccbid 130.246.180.120:9620#108541
10/27/15 17:15:14 (pid:27729) Master restart (GRACEFUL) is watching /home/boinc/CMSRun/glide_I5t966/main/condor/sbin/condor_master (mtime:1445962467)
10/27/15 17:15:14 (pid:27729) Started DaemonCore process "/home/boinc/CMSRun/glide_I5t966/main/condor/sbin/condor_startd", pid and pgroup = 27732

OK, I've submitted another 1,000 jobs, twice as large again (i.e. 100 events per job).

---

OK, I've submitted another 1,000 jobs, twice as large again (i.e. 100 events per job).

Looks like ~128 MB per job in <= 2 hours for most machines. Does that sound right?

---

OK, I've submitted another 1,000 jobs, twice as large again (i.e. 100 events per job).

Looks like ~128 MB per job in <= 2 hours for most machines. Does that sound right?

Only done one so far (74). I think 136MB and 2h 50mins wall time from Dashboard but
it took nearly an hour to get the job. Don't know how to find out how long
upload actually took but speed normally ca 900k so say 2.5 mins.

Edit. The following job (20) has already failed "stageout 60311" twice so it will
be interesting to see if it works here.... but I'll be asleep by then.

OK, I've submitted another 1,000 jobs, twice as large again (i.e. 100 events per job).

Looks like ~128 MB per job in <= 2 hours for most machines. Does that sound right?

Only done one so far (74). I think 136MB and 2h 50mins wall time from Dashboard but
it took nearly an hour to get the job. Don't know how to find out how long
upload actually took but speed normally ca 900k so say 2.5 mins.

Edit. The following job (20) has already failed "stageout 60311" twice so it will
be interesting to see if it works here.... but I'll be asleep by then.

Job 20 finished here OK. Each try is shown on Dashboard with a different IP,
presumably the public IP of the machine on which it ran.

OK, I've submitted another 1,000 jobs, twice as large again (i.e. 100 events per job).

Looks like ~128 MB per job in <= 2 hours for most machines. Does that sound right?

The 1st job in a new created VM takes always a bit longer.
On my not the fastest i7 3 hours and 17 minutes.
At the completion of the job 129.92 MB was uploaded within 75 seconds.

We are getting a lot of stage-out errors. The precise reason isn't evident in the logs to my not-very-practised eye. There were some network problems at CERN last night but it seems the higher error rate is continuing.

---

We are getting a lot of stage-out errors. The precise reason isn't evident in the logs to my not-very-practised eye. There were some network problems at CERN last night but it seems the higher error rate is continuing.

I think it's obvious we've hit a limit at this point, even before any careful analysis (Laurence has a new student to look at things like this, perhaps he can do the actual culling of statistics, they are all there in Dashboard). Next batch I'll step back to 75 ev/job to fill in an intermediate point -- the apparent step-up from 50 to 100 was a bit unexpected.

---

My task 68516, work unit 63658, took 3 days to complete - spent most of it's time in "Waiting to run" state in spite of CMS-Dev being my highest BOINC priority project. Looking at the log, it is mostly

2015-10-27 21:46:47 (608): Status Report: virtualbox.exe/vboxheadless.exe is no longer running.

Looks like when this happens the task attempts a retry MANY hours after the failure; don't understand VBox at all (can't get it to run on my Mac, where I do know what I'm doing, unlike Windows...).

Would be nice if either was notified something was wrong so it could be fixed on my end, or if task would just abort with an error, rather than keeping that CPU from running CMS indefinitely. (Until reboot?) That machine (Phoenix) was overheating with a GPU problem, fixed yesterday, or it might still be cycling the same task...