Development for LHC@home

CMS running jobs are falling. I haven't seen anything yet. Anyone? A link to a failed WU?
It's way past my bed-time. See you in 6+ hours and hope it's a false alarm...

---

Laurence send yesterday a message about a dev Server upgrade today.

Laurence send yesterday a message about a dev Server upgrade today.

That shouldn't have started to affect us last night,
In any event, the trend didn't continue. We've fewer jobs running now, but still more than this time last week.

---

I guess, the high numbers last week were down to the fact, that Theory was down.

I guess, the high numbers last week were down to the fact, that Theory was down.

Exactly. Laurence, Ben, et al. opened CMS@Home up to "production" status at vLHC@Home so users there started gobbling down jobs. For a while we nearly tripled our previous job rate, we're now settling down at about twice.

BTW, the new workflow we've been running the last month or so is now approaching 10 billion events processed. I've heard back from the people who requested the jobs that the results look good, but no reports of analysis data yet.

---

I've heard back from the people who requested the jobs that the results look good,

It is nice, that it is actually being used. Unlike Seti, where it is very doubtful, that there will ever be a "result" of any kind.

In CMS Jobs-dashboard is since midnight more red than green shown. Is this ok?

In CMS Jobs-dashboard is since midnight more red than green shown. Is this ok?

We're not sure where that is coming from. Other monitoring doesn't show a problem. It went away after a while, but may be coming back.

---

Has anyone spotted where the sudden increase of running CMS jobs today has come from? I checked earlier and there wasn't a spike in new sign-ups in -dev or production projects.

---

There was an issue with the Theory jobs so CMS has stolen some extra resources :)

EDIT: Shouldn't this be in an anti-dip thread?

There was an issue with the Theory jobs so CMS has stolen some extra resources :)

Ah, OK.

EDIT: Shouldn't this be in an anti-dip thread?

Yes, well, ...but...

---

I've heard back from the people who requested the jobs that the results look good,

It is nice, that it is actually being used. Unlike Seti, where it is very doubtful, that there will ever be a "result" of any kind.

You may have noticed a reference to my trip to Ambleside last week for a collaboration meeting for GRIDPP, the organisation of Universities and Laboratories which runs the UK Grid computing network for the LHC experiments (plus some extra for other science projects).
I gave a talk on CMS@Home, including a partial analysis of the data we have been generating. The talk is publicly available, though obviously without my running commentary on the slides. You should be able to view it as PDF or PowerPoint(TM).

---

No jobs?

I am trying a new OS.

Is it me, or is there a problem?

No jobs?

I am trying a new OS.

Is it me, or is there a problem?

There are jobs. I'm puzzled by the dip, I'll take a closer look.

---

Puzzled by this. Anyone seen it before in StartLog?
09/07/16 21:22:19 PERMISSION DENIED to submit-side@matchsession from host 130.246.180.120 for command 442 (REQUEST_CLAIM), access level DAEMON: reason: DAEMON authorization policy contains no matching ALLOW entry for this request; identifiers used for this host: 130.246.180.120,lcggwms02.gridpp.rl.ac.uk, hostname size = 1, original ip address = 130.246.180.120
09/07/16 21:22:19 Request accepted.
09/07/16 21:22:19 Remote owner is cms005@lcggwms02.gridpp.rl.ac.uk
09/07/16 21:22:19 State change: claiming protocol successful
09/07/16 21:22:19 Changing state: Unclaimed -> Claimed
09/07/16 21:22:20 PERMISSION DENIED to submit-side@matchsession from host 130.246.180.120 for command 501 (DELEGATE_GSI_CRED_STARTD), access level DAEMON: reason: cached result for DAEMON; see first case for the full reason
09/07/16 21:22:20 GLEXEC_STARTER is false, cancelling delegation
09/07/16 21:22:20 PERMISSION DENIED to submit-side@matchsession from host 130.246.180.120 for command 444 (ACTIVATE_CLAIM), access level DAEMON: reason: cached result for DAEMON; see first case for the full reason
09/07/16 21:22:20 Got activate_claim request from shadow (130.246.180.120)
09/07/16 21:22:20 Remote job ID is 1375089.0
09/07/16 21:22:20 Got universe "VANILLA" (5) from request classad
09/07/16 21:22:20 State change: claim-activation protocol successful
09/07/16 21:22:20 Changing activity: Idle -> Busy

---

No, that must be a red herring. I've got a CMS job now and it's running -- admittedly on the production project, not here on

-dev

. So, jobs are available.

---

It might be a reporting problem again. The graphs we use for CMS Jobs show job completion down to ~40/hour but I'm seeing twice that on the condor server, though it was down a bit earlier on, but still above 60/hour. For the last three full hours:
[cms005@lcggwms02:~] > ls -l 160903_112049:ireid_crab_BPH-RunIISummer15GS-00046_T/job*.txt.gz|grep '7 20'|wc
85 765 10285
[cms005@lcggwms02:~] > ls -l 160903_112049:ireid_crab_BPH-RunIISummer15GS-00046_T/job*.txt.gz|grep '7 19'|wc
65 585 7865
[cms005@lcggwms02:~] > ls -l 160903_112049:ireid_crab_BPH-RunIISummer15GS-00046_T/job*.txt.gz|grep '7 18'|wc
68 612 8228

---

Did you find out, what caused the last dip a few days ago?

Did you find out, what caused the last dip a few days ago?

Yeah, that was my mistake. Somehow I misread the queue status last Friday night, after a rather stressful day, and went to bed without submitting a new batch when I should have... Woke up to find it empty, so a quick CRAB3 submission of another 10,000 jobs quickly ensued. That reminds me, I'd better submit a new batch tonight, too!

---

There is something definitely wrong with the Dashboard reporting. The headline numbers on the "CMS Jobs" page come (in)directly from the condor server and correspond with what I see when I log into the machine. The Dashboard plots are not consistent with those data. I see the number of jobs in WNPostProc reported by Dashboard are much higher than usual, but I'm not sure if that's enough to account for the discrepancies.
I still think it's a Dashboard communications problem.

---