Development for LHC@home

[cms005@lcggwms02:~] > cat stats.sh
#!/bin/bash
grep 'NodeStatus ' $1/node_state.txt|sort|uniq -c
Mon May 23 12:30:46
[cms005@lcggwms02:~] > ./stats.sh 160518_203523:ireid_crab_CMS_at_Home_TTbar_50ev_prodB
4173 NodeStatus = 1; /* "STATUS_READY" */
1075 NodeStatus = 3; /* "STATUS_SUBMITTED" */
4745 NodeStatus = 5; /* "STATUS_DONE" */
7 NodeStatus = 6; /* "STATUS_ERROR" */

---

Thanks for the info.
How can there be fewer "submitted" than "Ready" or "Done"?

Thanks for the info.
How can there be fewer "submitted" than "Ready" or "Done"?

When I send a batch - 10,000 in this case - all the jobs are "ready". Then, up to ~1,000 are moved into the queue and become "submitted". As jobs are taken up by processes, jobs move into the queue to replace them and thus go out of "ready" into "submitted"; I think that jobs which are re-queued for retry also are "submitted". I'm not sure what state running jobs are in, probably also submitted since the sum of "idle" and "running" is about the number submitted. And of course jobs which are successful get to "done", those which fail three tries (or other errors) go into "error". Note that all four categories add up to the total number in the batch.

---

Thanks for the explanation Ivan.
I thought, submitted means the same as in dashboard.

No, that's from Condor itself. Confusing terminology, I must admit...

---

How is the proxy lease?
Does it not need a refresh. soon?

How is the proxy lease?
Does it not need a refresh. soon?

160518_203523, so due on the 25th. Current stats:
3776 NodeStatus = 1; /* "STATUS_READY" */
1084 NodeStatus = 3; /* "STATUS_SUBMITTED" */
1 NodeStatus = 4; /* "STATUS_POSTRUN" */
5132 NodeStatus = 5; /* "STATUS_DONE" */
7 NodeStatus = 6; /* "STATUS_ERROR" */

---

Previous large batch:
[cms005@lcggwms02:~] > ./stats.sh 160509_200134:ireid_crab_CMS_at_Home_TTbar_50ev_prodA
9881 NodeStatus = 5; /* "STATUS_DONE" */
119 NodeStatus = 6; /* "STATUS_ERROR" */

---

Currrent batch:
[cms005@lcggwms02:~] > ./stats.sh 160518_203523:ireid_crab_CMS_at_Home_TTbar_50ev_prodB
1771 NodeStatus = 1; /* "STATUS_READY" */
1095 NodeStatus = 3; /* "STATUS_SUBMITTED" */
7126 NodeStatus = 5; /* "STATUS_DONE" */
8 NodeStatus = 6; /* "STATUS_ERROR" */

Proxy renewed today. :-)

---

Thanks, Ivan!

[cms005@lcggwms02:~] > ./stats.sh 160518_203523:ireid_crab_CMS_at_Home_TTbar_50ev_prodB
617 NodeStatus = 3; /* "STATUS_SUBMITTED" */
9356 NodeStatus = 5; /* "STATUS_DONE" */
27 NodeStatus = 6; /* "STATUS_ERROR" */

I'll have to submit a new batch sometime over the (long) weekend.

---

New batch (5,000 x 50 TTbar events) submitted. Let's see if Dashboard is fully operational again...

---

Final status:
[cms005@lcggwms02:~] > ./stats.sh 160518_203523:ireid_crab_CMS_at_Home_TTbar_50ev_prodB
9968 NodeStatus = 5; /* "STATUS_DONE" */
32 NodeStatus = 6; /* "STATUS_ERROR" */

---

Congrats!
Best one, yet.

Only 0.32% ERROR.

Congrats!
Best one, yet.

Only 0.32% ERROR.

Looks like we lost a few in the system, though -- I only count 9,816 result files on the data-bridge. Some may yet turn up, but it's doubtful.

---

Looks like we lost a few in the system, though -- I only count 9,816 result files on the data-bridge. Some may yet turn up, but it's doubtful.

Does that include the error jobs?

I prefer 99.68% SUCCESS. We should try to understand what happened to the 32 jobs that failed.

If you can give me some examples of missing files I can investigate.

Looks like we lost a few in the system, though -- I only count 9,816 result files on the data-bridge. Some may yet turn up, but it's doubtful.

Does that include the error jobs?

Yes, I presume they didn't make it all the way through for some reason or another.

---

If you can give me some examples of missing files I can investigate.

I'll see what I can dig out, Laurence. It may be a bit messy, but that's what bash, awk and python are for...

---