Development for LHC@home

Laurence asked this question, Magic:

What are your views on the ATLAS native application and this potential new direction?

...The proposal would be that we focus on providing native applications for Linux built upon containers. The purpose of virtualization would therefore be to provide the Linux environment for running the boinc client, CVMFS and the container. For dedicated resources, a VM could be started using any hypervisor. For shared resources, the boinc client on the host could be used to start and manage the VM but the jobs would be managed native in the guest by another instance of the boinc client.

What are your views on the ATLAS native application and this potential new direction?

Have SL69 in use inside of Virtualbox for Atlas-native. Since 2-3 weeks with openhtc.io.
Since of the beginning Atlas-native some small errors, but with more than would say 95% successful tasks on 4 Computer.
A very good stability. Atlas need a lot of network-traffic for down/upload.

It would be a good idea to transfer this know how to Theory, LHCb or CMS and if possible Alice.
Singularity is needed for other Linux outside of SL69 or CentOS.

It is a great step to find one way for all LHC-projects.

Not 100% clear what you meant. I suppose you do not allow the user to login into your provided VM else...
In my opinion you could provide a VM with the best/lightest Linux OS for CERN purposes with all most needed instances installed like CVMFS, singularity and BOINC without projects connected.
No need for the user without much Linux knowledge to install and config packages. The VM-image can be used for several host OS's and hypervisors.
No need to control this VM by the host BOINC, cause the VM should be managed by the user him/herself who only have to add the LHC project with his username/pw.
He could even add other BOINC-projects only having Linux apps or where the Linux apps outperform the Windows/Mac-apps.
I have a Mint VM and was, although following the instructions, not able to get native ATLAS to run, so gave up.
... you have to run the inside BOINC with the user credentials.

As before any new idea can be tested at the Atlas Alpha site first and if found to work then moved here for more testing by more people and then on to LHC and as before I would be testing the Windows version at the Atlas Alpha site instead of just getting everyone running versions that may or may not even work and as I said.....ending up with fewer cores working for Cern.

It doesn't take much to get members to just go to a project such as Einstein when there is rarely any problem and work is done 24/7 by twice as many people and computers.

The main problem with VB is it can never be trusted to work without problems all the time and as I already said (many times) that is all it takes to see computers and members to go to a project that can be trusted to actually work without being watched all the time.

Right now none of the Cern projects can be trusted to run without problems like we have right now at LHC

I'm not 100% clear on what's proposed either. [thinking it through while I type so might take a couple of edits]
Not tried the Native Alas as I don't run Linux, and anyway my machines really just aren't powerful enough for what Atlas requires.
I did briefly sample Linux to try some earlier apps here (VMs inside a VM) but couldn't get my head around the command line stuff.

Let's see if I've got this right;
Windows hosts (pobably the majority of private users) run Boinc which controls the VM inside of which runs the computation which downloads the relevant science package (Pythia, Sherpa etc.) as required, which all gets destroyed each time Boinc finishes a Task.
Is the new approach for the host to support a Linux guest VM which contains Boinc, which woud download AND STORE the packages and keep them for whenever they are needed next, with Boinc then controlling the Native apps?

Seems there is potential for reduced bandwidth requirement after the initial setup.
Would the user need to set up Boinc within theVM?

How to earn credits if there isn't a requirement for Boinc to end and report Tasks? (Like the the Christmas Challenges of a few years ago where we gathered M^CPlots but no Boincs.) Might be a turnoff for some.

Would there be issues with the user pausing or shutting down the Guest VM?

One of my mahines has a noisy fan so I turn Boinc down to 50% overnight. Could I still do that without getting too Linuxy

- Would you like to change the way how jobs are distributed as well? So switch from the HTCondor approach to a similar one like ATLAS where the boinc server is responsible for distributing the jobs?

One of the issues with the current approach is that as the VMs are rebooted, the local CVMFS cache is destroyed and over time the cache in the image degrades.

What are your views on the ATLAS native application and this potential new direction?

I still have one question:

- Would you like to change the way how jobs are distributed as well? So switch from the HTCondor approach to a similar one like ATLAS where the boinc server is responsible for distributing the jobs?

I just had an idea (don't know if that would work): Why not place the CVMFS cache directory within one of the directories that is shared with the host system and does not get deleted (maybe the scratch directory within the project's folder within the Boinc Data directory) and configure CVMFS to use this directory as cache. When a new VM is started, include this directory and the already existing cache can be used again. This way the cache won't get deleted once the VM has shutdown.

I really like the native ATLAS app. I think the direction you proposed is good since everything that makes it easier for volunteers to set everything up is good (if that is the case). Keeping it as simple as possible for the volunteers is the most important thing!

Have you considered a similar approach as boinc2docker, so something like boinc2singularity? Again, no idea if that approach would work: So providing a very small VM image that runs CVMFS (again place the CVMFS cache directory into a shared directory if thats possible) and is able to execute a singularity image. Then download/execute the prebuild singularity image which then do the actual calculations.

I don't/won't run LINUX so this great idea sounds like a really efficient way to lose volunteers like me. I've already trashed CPDN because it has become an unstable mess and no one over there seems to be able to fix it. Looks to me like LHC is reading their roadmap to oblivion.

Why is it that projects reach a point of reliable stability, then someone gets a wild hare to start dinking around with the architecture?

I am really sorry to read you will start messing with Windows after this LINUX thing is in place.

Something like boinc2singularity is what we are after.

There is still the overhead of the time to boot the VM.

Actually more like boinc2runc