Message boards : CMS Application : Busy for a bit...
Message board moderation

To post messages, you must log in.

AuthorMessage
Profile ivan
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Avatar

Send message
Joined: 20 Jan 15
Posts: 1129
Credit: 7,870,419
RAC: 595
Message 3024 - Posted: 25 Apr 2016, 20:23:36 UTC

Just to say that I'll be preoccupied with other CMS matters for the rest of the week. I have to fly into CERN on Friday (at great taxpayer expense) to give two presentations, so I'll be busy the next three days generating data and sweating over PowerPoint. I'll keep an eye out for problems, but I may be slow to respond. Priorities, and all that... I must remember to renew the proxy for batch 160419_185901:ireid_crab_CMS_at_Home_MinBias_250ev10Ke tomorrow though, to avoid sudden failures after 1959 local.
ID: 3024 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Rasputin42
Volunteer tester

Send message
Joined: 16 Aug 15
Posts: 966
Credit: 1,211,816
RAC: 0
Message 3025 - Posted: 25 Apr 2016, 20:28:03 UTC

Good Luck,Ivan.

Show them, where to go--.
ID: 3025 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile ivan
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Avatar

Send message
Joined: 20 Jan 15
Posts: 1129
Credit: 7,870,419
RAC: 595
Message 3039 - Posted: 26 Apr 2016, 15:34:32 UTC - in response to Message 3024.  

I must remember to renew the proxy for batch 160419_185901:ireid_crab_CMS_at_Home_MinBias_250ev10Ke tomorrow though, to avoid sudden failures after 1959 local.

Proxy updated. :-)
ID: 3039 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile ivan
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Avatar

Send message
Joined: 20 Jan 15
Posts: 1129
Credit: 7,870,419
RAC: 595
Message 3109 - Posted: 30 Apr 2016, 0:21:37 UTC - in response to Message 3039.  

Update: just got back from CERN -- at 1 AM local! (Lots of woes on the way back: plane 90 mins late; then another 10 minutes or so on tarmac at LHR waiting for ground power to be connected 'cos the onboard generator was borked so the engines had to be kept running for electrickery; then 30 minutes to slowly snake through the non-EU Immigration queue. The 2nd-last 350 bus of the night was about to leave when I got to the stand at 0011; then a 14 minute wait for the last U5 bus at West Drayton -- the last U3 was right behind it, both get me to my nearest stop.) A long day, after getting up at 0430 to get to the airport in good time in case security was a nightmare.
Had lunch and later beers with Laurence today, sandwiching my CMS Uprade meeting -- we thought things were running well, until I looked at these boards just now...
I'll look at issues raised when I surface later today, must watch the BBC news on catch-up now to see what they say about "The Weasel that Killed the LHC"!
ID: 3109 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile ivan
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Avatar

Send message
Joined: 20 Jan 15
Posts: 1129
Credit: 7,870,419
RAC: 595
Message 3110 - Posted: 30 Apr 2016, 3:48:09 UTC - in response to Message 3109.  
Last modified: 30 Apr 2016, 3:48:26 UTC

must watch the BBC news on catch-up now to see what they say about "The Weasel that Killed the LHC"!

Ah, it wasn't on the 6-O'clock news, but it was on their news website, as well as the Telegraph and The Register.
ID: 3110 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Rasputin42
Volunteer tester

Send message
Joined: 16 Aug 15
Posts: 966
Credit: 1,211,816
RAC: 0
Message 3122 - Posted: 30 Apr 2016, 12:35:35 UTC

No or very little upload after job finishes.
It used to upload several 10th of MB. Now nearly nothing.
Is that normal?
Jobs are showing as "finished" on dashboard.Are they?
ID: 3122 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Phil

Send message
Joined: 9 Apr 15
Posts: 57
Credit: 230,221
RAC: 0
Message 3197 - Posted: 3 May 2016, 16:18:51 UTC - in response to Message 3109.  

Update: just got back from CERN -- at 1 AM local! (Lots of woes on the way back: plane 90 mins late; then another 10 minutes or so on tarmac at LHR waiting for ground power to be connected 'cos the onboard generator was borked so the engines had to be kept running for electrickery; then 30 minutes to slowly snake through the non-EU Immigration queue.

Usually its the opposite - wait an hour in departure lounge because they cant start the plane (sorry, the AUX generator wont go, we are waiting for a mobile supply) of course they have a dozen mobile generators but they're all for Airbus and this is the only Boeing flight the airline has, and the only generator with a Boeing plug on it is at another terminal 4 miles away.

The 2nd-last 350 bus of the night was about to leave when I got to the stand at 0011; then a 14 minute wait for the last U5 bus at West Drayton -- the last U3 was right behind it, both get me to my nearest stop.)

I usually arrive West Drayton late on a train, and stand around wondering if all the Last Busses have gone.
In the lonely darkness can be seen an advert for a taxi company, and usually while contemplating dialing, The Last Bus arrives.
Recently though, it has been supplemented with a big LED Display that flashes CALL NOW FOR TAXI followed by a 12-digit number thats visible for about 250ms and impossible to remember - weirdly its totally different from the phone number on the painted sign. While wondering of this is an update, the sign then flashes MERRY XMAS so I'm not sure if this new sign is 4 months out of date or 8 months into the future.
Sometimes I have the opposite problem - arrive at Uxbridge and look for a bus toward West Drayton. There will be 20-30 buses stood around at Uxbridge, all with hopeful-looking numbers on the front. But of course, they are all just queued to be put into the garage for the night. Theres usually around a dozen Drunks&Wierdos who keep clambering aboard each one in turn, shouting, vomiting and worse. Aah, well.

I'll look at issues raised when I surface later today, must watch the BBC news on catch-up now to see what they say about "The Weasel that Killed the LHC"!

Seems The Weasel Surge has arrived here, I've just discovered a RAM board is faulty, so thats off for Lifetime Warranty replacement (Sorry but this has No Life Left, so it isnt under warranty}.
ID: 3197 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile ivan
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Avatar

Send message
Joined: 20 Jan 15
Posts: 1129
Credit: 7,870,419
RAC: 595
Message 3203 - Posted: 3 May 2016, 19:49:47 UTC - in response to Message 3122.  

No or very little upload after job finishes.
It used to upload several 10th of MB. Now nearly nothing.
Is that normal?
Jobs are showing as "finished" on dashboard.Are they?

Sorry, missed this for some reason. Hassen has been running WMAgent-sumbmission jobs lately, and I believe they have small uploads. Although, there is something strange lately that I can't pin down yet. Dashboard thinks I'm having very few failures. but the graph under "CMS Jobs" (for all T3_CH_Volunteer jobs) thinks otherwise.
ID: 3203 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Rasputin42
Volunteer tester

Send message
Joined: 16 Aug 15
Posts: 966
Credit: 1,211,816
RAC: 0
Message 3209 - Posted: 3 May 2016, 20:38:21 UTC
Last modified: 3 May 2016, 20:40:35 UTC

Hassen has been running WMAgent-sumbmission jobs lately,


There have not been an WMAgent jobs. These where CMS jobs.

Only one in about 10 jobs had the correct upload size.

I suspect, despite the job declared a "success" on dashboard, the results where not(completly) uploaded.

Jobs are, for example, 8146, 8329, 8477 of the previous batch.
Only one (8448) had the proper upload size.

EDIT batch....250ev10Kk
ID: 3209 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Rasputin42
Volunteer tester

Send message
Joined: 16 Aug 15
Posts: 966
Credit: 1,211,816
RAC: 0
Message 3210 - Posted: 3 May 2016, 20:47:20 UTC

I just started new cms-tasks a couple of hours ago, and the first uploads seemed to be the corrrect size.
ID: 3210 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
m
Volunteer tester

Send message
Joined: 20 Mar 15
Posts: 243
Credit: 875,205
RAC: 450
Message 3224 - Posted: 3 May 2016, 23:47:05 UTC - in response to Message 3203.  

No or very little upload after job finishes.
It used to upload several 10th of MB. Now nearly nothing.
Is that normal?
Jobs are showing as "finished" on dashboard.Are they?

Sorry, missed this for some reason. Hassen has been running WMAgent-sumbmission jobs lately, and I believe they have small uploads. Although, there is something strange lately that I can't pin down yet. Dashboard thinks I'm having very few failures. but the graph under "CMS Jobs" (for all T3_CH_Volunteer jobs) thinks otherwise.


You can get results for crab and wmagent jobs separately from Dashboard.

To save you a lot of clicking:-
The last 24hrs and the last week for the crab3 jobs, whilst there are wmagent results for the last 24hrs and for the last week.
ID: 3224 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile ivan
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Avatar

Send message
Joined: 20 Jan 15
Posts: 1129
Credit: 7,870,419
RAC: 595
Message 3238 - Posted: 4 May 2016, 10:03:17 UTC - in response to Message 3209.  

Hassen has been running WMAgent-sumbmission jobs lately,


There have not been an WMAgent jobs. These where CMS jobs.

Only one in about 10 jobs had the correct upload size.

I suspect, despite the job declared a "success" on dashboard, the results where not(completly) uploaded.

Jobs are, for example, 8146, 8329, 8477 of the previous batch.
Only one (8448) had the proper upload size.

EDIT batch....250ev10Kk

Hmm, OK, I might have given up on that batch too quickly after my proxy error and starting up a new batch. I stopped the batch and archived the log to save disk space; looks like a few jobs were still running.
ID: 3238 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Rasputin42
Volunteer tester

Send message
Joined: 16 Aug 15
Posts: 966
Credit: 1,211,816
RAC: 0
Message 3239 - Posted: 4 May 2016, 10:38:13 UTC

Hmm, OK, I might have given up on that batch too quickly after my proxy error and starting up a new batch. I stopped the batch and archived the log to save disk space; looks like a few jobs were still running.


The jobs, i mentioned were finished before the proxy error and i was just wondering, how many jobs had incomplete uploads and therefore useless.
(A job, without result file is useless, isn't it?)

And if they did not have a result file uploaded the dashboard declartation as a success is wrong.
ID: 3239 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile ivan
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Avatar

Send message
Joined: 20 Jan 15
Posts: 1129
Credit: 7,870,419
RAC: 595
Message 3240 - Posted: 4 May 2016, 10:53:27 UTC - in response to Message 3239.  

Hmm, OK, I might have given up on that batch too quickly after my proxy error and starting up a new batch. I stopped the batch and archived the log to save disk space; looks like a few jobs were still running.


The jobs, i mentioned were finished before the proxy error and i was just wondering, how many jobs had incomplete uploads and therefore useless.
(A job, without result file is useless, isn't it?)

Yes.

And if they did not have a result file uploaded the dashboard declartation as a success is wrong.

Yes, but as we have seen over time, Dashboard sometimes takes a while to catch up, and sometimes is just wrong.
ID: 3240 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote

Message boards : CMS Application : Busy for a bit...


©2024 CERN