Message boards : News : Graceful Shutdown Now Implemented
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3

AuthorMessage
Crystal Pellet
Volunteer tester

Send message
Joined: 13 Feb 15
Posts: 1188
Credit: 862,257
RAC: 50
Message 1846 - Posted: 4 Feb 2016, 10:54:08 UTC - in response to Message 1758.  

The graceful shutdown of VMs has now been implemented. When the VM is older than 24 hours,
after the current run has finished the VM will shut itself down using the completion_trigger_file method.

It's not working as expected.

Run 4 ended after 27 hours and 50 minutes elapsed time and 25 hours and 12 minutes cpu time.

Me waiting for the shutdown, however:

Run 5 started and after 2 minutes
Run 6 started and after 2 minutes
Run 7 started and after 2 minutes
Run 8 started.

Enough waiting, so I shutdown the VM myself graceful with a trigger file.
ID: 1846 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile PDW

Send message
Joined: 20 May 15
Posts: 217
Credit: 6,193,119
RAC: 3,208
Message 1875 - Posted: 5 Feb 2016, 10:16:09 UTC - in response to Message 1846.  

The forced shutdown at 36 hours works.

Shame it had got as far as the 223rd record on the last job but hopefully it shouldn't usually run for that long.
ID: 1875 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Crystal Pellet
Volunteer tester

Send message
Joined: 13 Feb 15
Posts: 1188
Credit: 862,257
RAC: 50
Message 1897 - Posted: 6 Feb 2016, 12:22:13 UTC - in response to Message 1875.  

The forced shutdown at 36 hours works.

Shame it had got as far as the 223rd record on the last job but hopefully it shouldn't usually run for that long.

The kill after 36 hours wallclock is just for when the graceful method did not work.
So far I did not notice the graceful shutdown myself, but appearently it's working for others:

2016-02-04 22:13:04 (69032): Status Report: Elapsed Time: '96000.000000'
2016-02-04 22:13:04 (69032): Status Report: CPU Time: '70718.410000'
2016-02-04 22:56:28 (69032): Guest Log: [INFO] CMS glidein Run 5 ended
2016-02-04 22:57:28 (69032): Guest Log: [INFO] Time exceeded. Shutting down!
2016-02-04 22:57:28 (69032): VM Completion File Detected.
2016-02-04 22:57:28 (69032): Powering off VM.
ID: 1897 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Crystal Pellet
Volunteer tester

Send message
Joined: 13 Feb 15
Posts: 1188
Credit: 862,257
RAC: 50
Message 1899 - Posted: 6 Feb 2016, 16:32:19 UTC - in response to Message 1897.  

So far I did not notice the graceful shutdown myself, but appearently it's working for others

Now I had 2 tasks ending gracefully myself:

2016-02-06 15:20:13 (10296): Status Report: Elapsed Time: '85725.281227'
2016-02-06 15:20:13 (10296): Status Report: CPU Time: '82063.249643'
2016-02-06 16:55:12 (10296): Guest Log: [INFO] CMS glidein Run 10 ended
2016-02-06 16:56:12 (10296): Guest Log: [INFO] Time exceeded. Shutting down!
2016-02-06 16:56:12 (10296): VM Completion File Detected.
2016-02-06 16:56:12 (10296): Powering off VM.

2016-02-06 15:20:38 (14984): Status Report: Elapsed Time: '85152.827036'
2016-02-06 15:20:38 (14984): Status Report: CPU Time: '81513.798520'
2016-02-06 16:11:40 (14984): Guest Log: [INFO] CMS glidein Run 13 ended
2016-02-06 16:12:40 (14984): Guest Log: [INFO] Time exceeded. Shutting down!
2016-02-06 16:12:40 (14984): VM Completion File Detected.
2016-02-06 16:12:40 (14984): Powering off VM.
ID: 1899 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
rbpeake

Send message
Joined: 15 Apr 15
Posts: 38
Credit: 227,251
RAC: 0
Message 1900 - Posted: 6 Feb 2016, 16:52:01 UTC - in response to Message 1899.  

How are you managing to get new jobs? I recently get the message that I need a huge amount of memory.
ID: 1900 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Crystal Pellet
Volunteer tester

Send message
Joined: 13 Feb 15
Posts: 1188
Credit: 862,257
RAC: 50
Message 1901 - Posted: 6 Feb 2016, 17:52:19 UTC - in response to Message 1900.  

How are you managing to get new jobs? I recently get the message that I need a huge amount of memory.

That were CMS-tasks from the VirtualLHC@home project using the same VM and software.
A few hundreds of CMS-tasks were submitted over there yesterday and quickly gone.
ID: 1901 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Rasputin42
Volunteer tester

Send message
Joined: 16 Aug 15
Posts: 966
Credit: 1,211,816
RAC: 0
Message 1902 - Posted: 6 Feb 2016, 17:59:57 UTC

How are you managing to get new jobs? I recently get the message that I need a huge amount of memory.


It is actually disk space.

Server is messed up.
ID: 1902 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Previous · 1 · 2 · 3

Message boards : News : Graceful Shutdown Now Implemented


©2024 CERN