Message boards : News : CMS job queue to drain this weekend (21/08/2021)
Message board moderation

To post messages, you must log in.

AuthorMessage
Profile ivan
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Avatar

Send message
Joined: 20 Jan 15
Posts: 1101
Credit: 7,021,236
RAC: 60
Message 7241 - Posted: 20 Aug 2021, 15:32:25 UTC
Last modified: 20 Aug 2021, 15:32:59 UTC

CMS is about to release a new version of WMAgent based entirely on python 3. They have asked that they be able to update our agent by Monday evening (23/08), so I will not inject any new workflows before the upgrade. I expect the job queue to drain by late on Sunday.
Please set your CMS application to no new tasks by then.
ID: 7241 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Magic Quantum Mechanic
Avatar

Send message
Joined: 8 Apr 15
Posts: 596
Credit: 8,503,937
RAC: 1,901
Message 7242 - Posted: 20 Aug 2021, 22:17:23 UTC - in response to Message 7241.  

When will they move multi-core CMS over to the public?
The current version worked fine with 1,2,3,4,and 8 cores when I ran hundreds of them (running 8's and 4'rs right now)
ID: 7242 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile ivan
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Avatar

Send message
Joined: 20 Jan 15
Posts: 1101
Credit: 7,021,236
RAC: 60
Message 7243 - Posted: 21 Aug 2021, 0:04:19 UTC - in response to Message 7242.  

When will they move multi-core CMS over to the public?
The current version worked fine with 1,2,3,4,and 8 cores when I ran hundreds of them (running 8's and 4'rs right now)

Are you sure it works now? We have a new VM on -dev that uses a different glide-in mechanism, to prepare for when we move to CMS production (like fusion energy, that's always just "this" far away...). Last I checked, you could specify multi-core still, but the task only ran one job. Unless Laurence tweaked it without telling me.
We hope to move the new VM to the public app when both Laurence and Federica are back from summer holidays -- I think that's next week.
ID: 7243 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile ivan
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Avatar

Send message
Joined: 20 Jan 15
Posts: 1101
Credit: 7,021,236
RAC: 60
Message 7244 - Posted: 21 Aug 2021, 0:06:49 UTC

Oh, b****r, I posted that twice here and not to the main app as intended. Sorry...
ID: 7244 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Magic Quantum Mechanic
Avatar

Send message
Joined: 8 Apr 15
Posts: 596
Credit: 8,503,937
RAC: 1,901
Message 7245 - Posted: 21 Aug 2021, 0:58:31 UTC

No I'm not sure since I haven't been watching them run like I used to with the single cores with this same version ( because the previous versions couldn't be trusted)
But next batch tonight I will look at that ( since I do d/l the Extension Pack and can watch all of that)

Before they just crashed and became errors but these finish Valid and many run longer CPU time than Run time when I look at ones by other members who do not hide their pc's

But just looking stderr's here don't show the separate cores and jobs running as I remember over the years.
ID: 7245 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
computezrmle
Avatar

Send message
Joined: 28 Jul 16
Posts: 287
Credit: 353,387
RAC: 0
Message 7246 - Posted: 21 Aug 2021, 7:30:21 UTC - in response to Message 7243.  

None of the multicore VMs here ever ran 2 or more CMS subtasks concurrently during the past weeks.
This is simply not enabled in the job templates.
The fact that some computers show CPU times higher than runtimes is caused by some auxiliary processes running beside the scientific app, e.g. the CVMFS client.

Hence, configuring a VM with more than 2 cores currently makes no sense since it would just allocate additional cores and RAM.
The cores would be marked as used in BOINC but in reality they would run idle, the RAM would be allocated by the VM but never be returned for other usage until the VM completely shuts down.

The VM image file can be used by a singlecore VM and also by a multicore VM.
It's like dismounting a real disk from one computer case and mounting it into another case.
It also runs fine in the prod environment (strictly as singlecore).
This is due to the fact that most of the job settings are downloaded via CVMFS instead of being hardwired in the vdi file.
ID: 7246 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Magic Quantum Mechanic
Avatar

Send message
Joined: 8 Apr 15
Posts: 596
Credit: 8,503,937
RAC: 1,901
Message 7247 - Posted: 21 Aug 2021, 14:25:22 UTC - in response to Message 7246.  
Last modified: 21 Aug 2021, 14:27:54 UTC

Yes I know that Stephan, you don't need to post it since I am usually the only one reading this.
And I also run the prod version singles at the same time.

I am just talking about how these aren't crashing like they did with the versions Version 60.30- 60.20 - 60.10 - 50.00
And I got up early today to check the files here to see what is going on with this current version.

It isn't up to any of us to make these project multi-cores to run several jobs per cpu core.
We run what we get just like it has been for over 10 yeas now and I have been doing these every day since the beginning in 2011
ID: 7247 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile ivan
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Avatar

Send message
Joined: 20 Jan 15
Posts: 1101
Credit: 7,021,236
RAC: 60
Message 7250 - Posted: 23 Aug 2021, 15:52:12 UTC

The update is done and jobs are available again. Feel free to continue.
ID: 7250 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote

Message boards : News : CMS job queue to drain this weekend (21/08/2021)


©2021 CERN