Message boards : Theory Application : x509 proxy error
Message board moderation

To post messages, you must log in.

AuthorMessage
Profile PDW

Send message
Joined: 20 May 15
Posts: 217
Credit: 5,193,337
RAC: 8,222
Message 3297 - Posted: 10 May 2016, 7:16:11 UTC

After trying 6 times (per project site) to request a credential tasks are failing...

2016-05-10 07:18:48 (12888): Guest Log: [ERROR] Could not get an x509 credential
2016-05-10 07:18:48 (12888): Guest Log: [ERROR] The x509 proxy creation failed.
2016-05-10 07:18:48 (12888): Guest Log: [INFO] Shutting Down.
2016-05-10 07:18:48 (12888): VM Completion File Detected.
2016-05-10 07:18:48 (12888): VM Completion Message: The x509 proxy creation failed.
ID: 3297 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Rasputin42
Volunteer tester

Send message
Joined: 16 Aug 15
Posts: 966
Credit: 1,211,816
RAC: 0
Message 3298 - Posted: 10 May 2016, 7:54:14 UTC

Same for CMS-tasks.
ID: 3298 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile PDW

Send message
Joined: 20 May 15
Posts: 217
Credit: 5,193,337
RAC: 8,222
Message 3300 - Posted: 10 May 2016, 8:46:00 UTC

The host is now told that it has completed its quota for the day...

10/05/2016 09:36:49 | vLHCathome-dev | Sending scheduler request: Requested by user.
10/05/2016 09:36:49 | vLHCathome-dev | Requesting new tasks for CPU
10/05/2016 09:36:50 | vLHCathome-dev | Scheduler request completed: got 0 new tasks
10/05/2016 09:36:50 | vLHCathome-dev | No tasks sent
10/05/2016 09:36:50 | vLHCathome-dev | No tasks are available for Theory Simulation
10/05/2016 09:36:50 | vLHCathome-dev | This computer has finished a daily quota of 1 tasks

I assume this is a result of the invalid tasks being reported due to the system failure to get x509 credential.
Is this what you expect to happen regarding host backoff ?

Does this not mean if every user gets this system error all machines will be backed off for a day ?
ID: 3300 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile PDW

Send message
Joined: 20 May 15
Posts: 217
Credit: 5,193,337
RAC: 8,222
Message 3301 - Posted: 10 May 2016, 8:59:28 UTC - in response to Message 3300.  

Gave LHCb a go and that fails also (as I expected) but did see this in the output after the 'Could not get an X509 credential' message...

/usr/sbin/boinc-shutdown: line 31: [: too many arguments

Perhaps some sort of anger management is required ?
ID: 3301 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile PDW

Send message
Joined: 20 May 15
Posts: 217
Credit: 5,193,337
RAC: 8,222
Message 3306 - Posted: 10 May 2016, 21:44:41 UTC - in response to Message 3301.  

LHCb task running okay now, will be tomorrow before I can run a Theory task !
ID: 3306 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Crystal Pellet
Volunteer tester

Send message
Joined: 13 Feb 15
Posts: 1178
Credit: 810,581
RAC: 1,976
Message 3730 - Posted: 18 Jul 2016, 7:10:43 UTC
Last modified: 18 Jul 2016, 7:11:20 UTC

Started a new task and got no credentials for running a Theory mt-mcore VM - 206 (0xce) EXIT_INIT_FAILURE

2016-07-18 07:37:48 (5740): Guest Log: [INFO] Reading volunteer information
2016-07-18 07:37:48 (5740): Guest Log: [INFO] Volunteer: Crystal Pellet (38) Host: 37
2016-07-18 07:37:48 (5740): Guest Log: [INFO] VMID: f1b43d2f-577d-4de1-aea7-eac1787823b8
2016-07-18 07:37:48 (5740): Guest Log: [INFO] Requesting an X509 credential from vLHC@home
2016-07-18 07:37:48 (5740): Guest Log: [INFO] Requesting an X509 credential from vLHC@home-dev
2016-07-18 07:38:19 (5740): Guest Log: [INFO] Requesting an X509 credential from vLHC@home
2016-07-18 07:38:19 (5740): Guest Log: [INFO] Requesting an X509 credential from vLHC@home-dev
2016-07-18 07:38:49 (5740): Guest Log: [INFO] Requesting an X509 credential from vLHC@home
2016-07-18 07:38:49 (5740): Guest Log: [INFO] Requesting an X509 credential from vLHC@home-dev
2016-07-18 07:39:19 (5740): Guest Log: [INFO] Requesting an X509 credential from vLHC@home
2016-07-18 07:39:19 (5740): Guest Log: [INFO] Requesting an X509 credential from vLHC@home-dev
2016-07-18 07:39:49 (5740): Guest Log: [INFO] Requesting an X509 credential from vLHC@home
2016-07-18 07:39:49 (5740): Guest Log: [INFO] Requesting an X509 credential from vLHC@home-dev
2016-07-18 07:40:29 (5740): Guest Log: [INFO] Requesting an X509 credential from vLHC@home
2016-07-18 07:40:29 (5740): Guest Log: [INFO] Requesting an X509 credential from vLHC@home-dev
2016-07-18 07:40:59 (5740): Guest Log: [ERROR] Could not get an x509 credential
2016-07-18 07:40:59 (5740): Guest Log: [ERROR] The x509 proxy creation failed.
2016-07-18 07:40:59 (5740): Guest Log: [INFO] Shutting Down.
ID: 3730 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Laurence
Project administrator
Project developer
Project tester
Avatar

Send message
Joined: 12 Sep 14
Posts: 1064
Credit: 325,950
RAC: 278
Message 3731 - Posted: 18 Jul 2016, 8:09:16 UTC - in response to Message 3730.  

I confirm there is a problem with the authentication server and am working to fix it.
ID: 3731 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Laurence
Project administrator
Project developer
Project tester
Avatar

Send message
Joined: 12 Sep 14
Posts: 1064
Credit: 325,950
RAC: 278
Message 3732 - Posted: 18 Jul 2016, 8:56:35 UTC - in response to Message 3731.  
Last modified: 18 Jul 2016, 9:59:28 UTC

The online CA that provides the user certificates to the proxy server is down. We are working to bring it back up ASAP.
ID: 3732 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile ivan
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Avatar

Send message
Joined: 20 Jan 15
Posts: 1128
Credit: 7,870,419
RAC: 595
Message 3733 - Posted: 18 Jul 2016, 9:41:10 UTC - in response to Message 3300.  

The host is now told that it has completed its quota for the day...

10/05/2016 09:36:49 | vLHCathome-dev | Sending scheduler request: Requested by user.
10/05/2016 09:36:49 | vLHCathome-dev | Requesting new tasks for CPU
10/05/2016 09:36:50 | vLHCathome-dev | Scheduler request completed: got 0 new tasks
10/05/2016 09:36:50 | vLHCathome-dev | No tasks sent
10/05/2016 09:36:50 | vLHCathome-dev | No tasks are available for Theory Simulation
10/05/2016 09:36:50 | vLHCathome-dev | This computer has finished a daily quota of 1 tasks

I assume this is a result of the invalid tasks being reported due to the system failure to get x509 credential.
Is this what you expect to happen regarding host backoff ?

Does this not mean if every user gets this system error all machines will be backed off for a day ?

Sorry for not spotting that sooner, but luckily Laurence was ahead of me...

Yes, that is what BOINC does when it spots errored-out tasks. Unfortunately, since we have a small limit anyhow the quota bombs out quickly. And, yes, a lot of hosts will be without tasks for up to a day if they quotaed-out. I've got one host that's dry and another running one task where it normally has two (one -dev, one production).
ID: 3733 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Rasputin42
Volunteer tester

Send message
Joined: 16 Aug 15
Posts: 966
Credit: 1,211,816
RAC: 0
Message 3734 - Posted: 18 Jul 2016, 9:49:23 UTC

anyhow the quota bombs out quickly


I thought, if you produce a number of valid tasks, this builds up to counter the quota.Apparently, that does notwork or i am wrong.
ID: 3734 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Laurence
Project administrator
Project developer
Project tester
Avatar

Send message
Joined: 12 Sep 14
Posts: 1064
Credit: 325,950
RAC: 278
Message 3735 - Posted: 18 Jul 2016, 10:03:19 UTC - in response to Message 3734.  

I have just increased the limit. Will decrease it again tomorrow.
ID: 3735 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile ivan
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Avatar

Send message
Joined: 20 Jan 15
Posts: 1128
Credit: 7,870,419
RAC: 595
Message 3736 - Posted: 18 Jul 2016, 10:34:41 UTC - in response to Message 3734.  
Last modified: 18 Jul 2016, 10:36:21 UTC

anyhow the quota bombs out quickly


I thought, if you produce a number of valid tasks, this builds up to counter the quota.Apparently, that does notwork or i am wrong.

Yes, but if you are at the point that your quota has run out, you need to wait up to one day before you can get another task, to start building up your quota again. Anyway, the graph of running jobs is starting to rise again
ID: 3736 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Rasputin42
Volunteer tester

Send message
Joined: 16 Aug 15
Posts: 966
Credit: 1,211,816
RAC: 0
Message 3737 - Posted: 18 Jul 2016, 10:44:45 UTC - in response to Message 3736.  

Well, if you have done 20 consecutive valid task before, you should be able to produce 20 failed tasks+ quota, before you are not allowed any more tasks.

Credentials are working, but i cannot get any job (CMS task).
ID: 3737 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Laurence
Project administrator
Project developer
Project tester
Avatar

Send message
Joined: 12 Sep 14
Posts: 1064
Credit: 325,950
RAC: 278
Message 3738 - Posted: 18 Jul 2016, 11:01:03 UTC - in response to Message 3737.  

Yes, but if 20 tasks have failed, then the no more tasks will be given. I haven't found the magic flag in the DB to reset this.
ID: 3738 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote

Message boards : Theory Application : x509 proxy error


©2024 CERN