Development for LHC@home

Today the Native Theory application has been added to the production project.

I would like to take this opportunity to thank everyone who has been involved in the testing. Without your efforts it would be difficult to catch all the issues that occur in the different environments. It means that the production application is much more robust that it would be otherwise.

The application is currently in beta (test) until we can confirm that it is working for everyone, so please could you shift your focus to the production project.

Thanks again for all your effort and feedback!

Once again I would like to say a big thank you to all those who have been involved in testing this application. From my perspective the release to production went very well. So far it has not been necessary to update the version that we originally deployed which means that we managed to catch the issues during testing. Also I would like to thank those who have used their experience from the dev project to help volunteers in the production project with their problems and questions. Before we move the application out of beta, I would like to address some server-side scheduling issues to restrict the tasks given out to hosts that fail tasks due to CVMFS not being available. In the meantime, please let me know if anything needs to be improved in the sticky or CVMFS documentation. Also, if there is any other feedback, please let me know, don't be shy.

Hi Laurence,

You may have noticed that, since the introduction of the native Theory at the production site, a problem is (re)introduced with requesting tasks with the max # of jobs and max # of CPUs. I just tested it again and as an example

Max # of jobs - no limit
Max # of CPUs - 1

I don't get new tasks when I have 2 tasks running and an idle core.

There after I tried

Max # of jobs - no limit
Max # of CPUs - 2

and I got 2 tasks, but not more, where I expect at least 2 x 3 (available cores) is 6.

So somehow the Max # of CPUs is influencing the number of jobs, where it should not, cause in principle we run the native Theory as single cores.
With higher # of CPU's BOINC reserves cores for nothing. The user can create a work around with app_config.xml, but app_config.xml should not be used to work around bugs/failures.

Some comments

As CMS is also back at the production server I tried to get both (CMS and Theory native) on the same client.
Unfortunately this client only gets Theory native tasks from the server.
Hence it needs manual intervention to get CMS tasks.


The credits the server rewards for Theory native is rather poor.
Seems only 10% of normal credit and it looks like it doesn't normalize over the time.
It should be clarified why.

The native theory app seems to be running quite nicely in production. Is there anything that needs to be addressed before we take it out of beta?

... Is there anything that needs to be addressed before we take it out of beta?

Yes.

1. Task deadlines should be extended to give longrunners a better chance to finish.

2. A method to write the job parameters to stderr.txt early should be included.
A possible solution can be found here:
https://lhcathomedev.cern.ch/lhcathome-dev/forum_thread.php?id=467

3. Credits given for Theory native are still very poor, especially compared to Theory vbox.
This may result in a bad acceptance.

Some comments

As CMS is also back at the production server I tried to get both (CMS and Theory native) on the same client.
Unfortunately this client only gets Theory native tasks from the server.
Hence it needs manual intervention to get CMS tasks.


The credits the server rewards for Theory native is rather poor.
Seems only 10% of normal credit and it looks like it doesn't normalize over the time.
It should be clarified why.

The same applies to Theory native and Atlas (this is from the production server). The server will only send Atlas if it can't send Theory, either because of preference settings or because there are no Theory tasks available (as happened a few weeks ago)
This is a " VBox free" host where there have been >50 Theory tasks in a row without the server sending an Atlas task. I thought that the server may be trying to equalise the credit between the sub-projects but this is greater than the difference in credit; unless it's trying to make up the backlog which, for me, will take a long time unless the credit for Theory increases considerably.

Also,
The need to run hosts continuously is a problem for me and, I expect, for many others as well, especially if the project wants to widen it's potential pool of volunteers. I know that with a bit of babysitting and some clever control scripts this can be overcome to some extent, but we're getting a long way from the original ideas of BOINC.

... Is there anything that needs to be addressed before we take it out of beta?

Yes.

1. Task deadlines should be extended to give longrunners a better chance to finish.

Do no know what parameter needs to be set? Is it rsc_fpops_est?

2. A method to write the job parameters to stderr.txt early should be included.
A possible solution can be found here:
https://lhcathomedev.cern.ch/lhcathome-dev/forum_thread.php?id=467

I am looking at this and will respond in that thread.

3. Credits given for Theory native are still very poor, especially compared to Theory vbox.
This may result in a bad acceptance.

It think this changes with rsc_fpops_est.

The same applies to Theory native and Atlas (this is from the production server). The server will only send Atlas if it can't send Theory, either because of preference settings or because there are no Theory tasks available (as happened a few weeks ago)
This is a " VBox free" host where there have been >50 Theory tasks in a row without the server sending an Atlas task. I thought that the server may be trying to equalise the credit between the sub-projects but this is greater than the difference in credit; unless it's trying to make up the backlog which, for me, will take a long time unless the credit for Theory increases considerably.

The scheduling is a bit of a black box.

Also,
The need to run hosts continuously is a problem for me and, I expect, for many others as well, especially if the project wants to widen it's potential pool of volunteers. I know that with a bit of babysitting and some clever control scripts this can be overcome to some extent, but we're getting a long way from the original ideas of BOINC.

It is possible to suspend the Theory tasks but not the ATLAS tasks.

I know that with a bit of babysitting and some clever control scripts this can be overcome to some extent, but we're getting a long way from the original ideas of BOINC.

Glad to see I'm not alone in that thought.

Here's my comments in another thread: https://lhcathomedev.cern.ch/lhcathome-dev/forum_thread.php?id=465&postid=6324#6324

1. Task deadlines should be extended to give longrunners a better chance to finish.

Do no know what parameter needs to be set? Is it rsc_fpops_est?

The server parameter that sets the client's "report_deadline".

I'm not familiar with the server templates but I doubt it's "rsc_fpops_est".
My guess it could be "delay_bound".

1. Task deadlines should be extended to give longrunners a better chance to finish.

Do no know what parameter needs to be set? Is it rsc_fpops_est?

The server parameter that sets the client's "report_deadline".

I'm not familiar with the server templates but I doubt it's "rsc_fpops_est".
My guess it could be "delay_bound".

delay_bound is currently set to 200000 which is 55.6 hours. What would you suggest setting it to?

delay_bound is currently set to 200000 which is 55.6 hours. What would you suggest setting it to?

The longest running successful job I could find was 9.85 days, but out of 1,584.735 jobs only 3 were running longer than a week.

delay_bound for vbox tasks seems to be 2592000 (30 days).
If nobody disagrees I would suggest to set it to 432000 (5 days) or 600000 (a bit less than a week).
Are there other limits we are not aware of in this thread (see: https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=4028&postid=38669)?

<edit>
May be obsolete due to CP's post.
</edit>

Also,
The need to run hosts continuously is a problem for me and, I expect, for many others as well, especially if the project wants to widen it's potential pool of volunteers. I know that with a bit of babysitting and some clever control scripts this can be overcome to some extent, but we're getting a long way from the original ideas of BOINC.

It is possible to suspend the Theory tasks but not the ATLAS tasks.

Hosts here shut down (aka switch off) during the day.and run overnight (cheaper electricity, unmetered internet and I can use the heat)
There must be many (potential) volunteers who want or need to shut their computers down overnght, over the weekend or whatever without a lot of manual attention..

... Is there anything that needs to be addressed ...

Tried to integrate Theory native into my existing cgroups hierarchy but so far without full success.
Hence I either run it in my own (sub)cgroup to get a balanced scheduling among all projects or I run it in the project's cgroup to get suspend/resume running.

Needs more knowledge/testing.

There must be many (potential) volunteers who want or need to shut their computers down overnght, over the weekend or whatever without a lot of manual attention..

Starting June 1 to September 30, the electric here will have peak hour rates, 9x greater than off-peak from 10am-7pm so all the BOINCMgr will be set to suspend all work during those hours. The electric company is pushing to have those new meters be default installs in all homes.

There must be many (potential) volunteers who want or need to shut their computers down overnght, over the weekend or whatever without a lot of manual attention..

Starting June 1 to September 30, the electric here will have peak hour rates, 9x greater than off-peak from 10am-7pm so all the BOINCMgr will be set to suspend all work during those hours. The electric company is pushing to have those new meters be default installs in all homes.

Where's the problem?
There are already 2 possible solutions:

1. Run a LHC vbox app
2. Set up a self defined VM and inside this VM run a native app.


Both can be paused during the expensive periods by BOINC or a timer of your choice.

Tried to integrate Theory native into my existing cgroups hierarchy but so far without full success.
Hence I either run it in my own (sub)cgroup to get a balanced scheduling among all projects or I run it in the project's cgroup to get suspend/resume running.

Needs more knowledge/testing.

From you or us?

delay_bound is currently set to 200000 which is 55.6 hours. What would you suggest setting it to?

The longest running successful job I could find was 9.85 days, but out of 1,584.735 jobs only 3 were running longer than a week.

That is surprising. We’re those jobs fine otherwise?