Development for LHC@home

The CMS jobs graph shows rising number of crab jobs, but the errors are staying the same(or even falling).
That is good news.

Has anything changed?

It's just recovery from when we lost Condor on Tuesday night, I think. The CRAB jobs are going just to LHC@Home now (-dev and Laurence's cluster are munching on the WMAgent jobs, which show up as "unknown" for some unknown reason). Many hosts would have quota-ed out due to continued failure, so they will be gradually getting their quota back, while those that fell back to Test4Theory wouldn't return to us for the 18-20 hours their tasks run. We're only now back up to the 750-800 running CRAB jobs that we had before the disturbance.

---

Tasks have stopped (no cpu activity) and the following in the stderr.log:
CMS jobs graphs not accessible,also.

INFO:root:Beginning report processing for step logArch1
ERROR:root:Cannot find report for step logArch1 in space /var/lib/condor/execute/dir_10487/job/WMTaskSpace/logArch1
INFO:root:Beginning report processing for step cmsRun1
/var/lib/condor/execute/dir_10487/startup_environment.sh: line 2: BASHOPTS: readonly variable
/var/lib/condor/execute/dir_10487/startup_environment.sh: line 9: BASH_VERSINFO: readonly variable
/var/lib/condor/execute/dir_10487/startup_environment.sh: line 13: EUID: readonly variable
/var/lib/condor/execute/dir_10487/startup_environment.sh: line 25: PPID: readonly variable
/var/lib/condor/execute/dir_10487/startup_environment.sh: line 29: SHELLOPTS: readonly variable
/var/lib/condor/execute/dir_10487/startup_environment.sh: line 36: UID: readonly variable
***ERROR*** in propagateError, *newErr is not NULL impossible to overwrite ... old error wasHTTP 404 : File not found
***ERROR*** in propagateError, *newErr is not NULL impossible to overwrite ... old error wasHTTP 404 : File not found
***ERROR*** in propagateError, *newErr is not NULL impossible to overwrite ... old error wasHTTP 404 : File not found
***ERROR*** in propagateError, *newErr is not NULL impossible to overwrite ... old error wasHTTP 404 : File not found
gfal-copy error: 13 (Permission denied) - TRANSFER Authentication error, reached maximum number of attempts
/var/lib/condor/execute/dir_9939/startup_environment.sh: line 2: BASHOPTS: readonly variable
/var/lib/condor/execute/dir_9939/startup_environment.sh: line 9: BASH_VERSINFO: readonly variable
/var/lib/condor/execute/dir_9939/startup_environment.sh: line 13: EUID: readonly variable
/var/lib/condor/execute/dir_9939/startup_environment.sh: line 25: PPID: readonly variable
/var/lib/condor/execute/dir_9939/startup_environment.sh: line 29: SHELLOPTS: readonly variable
/var/lib/condor/execute/dir_9939/startup_environment.sh: line 36: UID: readonly variable
***ERROR*** in propagateError, *newErr is not NULL impossible to overwrite ... old error wasHTTP 404 : File not found

Tasks have stopped (no cpu activity) and the following in the stderr.log:
CMS jobs graphs not accessible,also.

INFO:root:Beginning report processing for step logArch1
ERROR:root:Cannot find report for step logArch1 in space /var/lib/condor/execute/dir_10487/job/WMTaskSpace/logArch1
INFO:root:Beginning report processing for step cmsRun1
/var/lib/condor/execute/dir_10487/startup_environment.sh: line 2: BASHOPTS: readonly variable
/var/lib/condor/execute/dir_10487/startup_environment.sh: line 9: BASH_VERSINFO: readonly variable
/var/lib/condor/execute/dir_10487/startup_environment.sh: line 13: EUID: readonly variable
/var/lib/condor/execute/dir_10487/startup_environment.sh: line 25: PPID: readonly variable
/var/lib/condor/execute/dir_10487/startup_environment.sh: line 29: SHELLOPTS: readonly variable
/var/lib/condor/execute/dir_10487/startup_environment.sh: line 36: UID: readonly variable
***ERROR*** in propagateError, *newErr is not NULL impossible to overwrite ... old error wasHTTP 404 : File not found
***ERROR*** in propagateError, *newErr is not NULL impossible to overwrite ... old error wasHTTP 404 : File not found
***ERROR*** in propagateError, *newErr is not NULL impossible to overwrite ... old error wasHTTP 404 : File not found
***ERROR*** in propagateError, *newErr is not NULL impossible to overwrite ... old error wasHTTP 404 : File not found
gfal-copy error: 13 (Permission denied) - TRANSFER Authentication error, reached maximum number of attempts
/var/lib/condor/execute/dir_9939/startup_environment.sh: line 2: BASHOPTS: readonly variable
/var/lib/condor/execute/dir_9939/startup_environment.sh: line 9: BASH_VERSINFO: readonly variable
/var/lib/condor/execute/dir_9939/startup_environment.sh: line 13: EUID: readonly variable
/var/lib/condor/execute/dir_9939/startup_environment.sh: line 25: PPID: readonly variable
/var/lib/condor/execute/dir_9939/startup_environment.sh: line 29: SHELLOPTS: readonly variable
/var/lib/condor/execute/dir_9939/startup_environment.sh: line 36: UID: readonly variable
***ERROR*** in propagateError, *newErr is not NULL impossible to overwrite ... old error wasHTTP 404 : File not found

Things look OK here at the moment, tho' there seems to have been a glitch at LHC@Home overnight, which appears to have led to a spike of "echo" failures once the problem cleared. I'll keep checking for a while, but I discovered when I came in this morning that the University is actually closed today, so I'll probably declare it POET'S Day[1] soon.

[1] Traditional Aussie approach to any Friday, not just the one before Christmas -- P*ss Off Early, Tomorrow's Saturday.

---