Development for LHC@home

One of the main headaches for many people seems to be related to VirtualBox. First of all there is the overhead of actually installing it and then there are the errors is produces. For Linux we could avoid Virtulization by using docker and for Windows we could try Hyper-V. However, these approaches many just swap one set of issues for another. Comments?

Cafe-talk, I see.

Counting with my thumbs, I guess there are between 800 and 1000 crunchers using VirtualBox for CERN's BOINC projects and the Cosmology project.

Maybe only 1-2% seem to have troubles with the setup.

I think you know my opinion by now.

I think you know my opinion by now.

No, what is it?

Looking at Theory during the past 24 hours, on vLHC 460 hosts are fine and 168 are producing errors. For LHC it is 402 and 286 respectability. Many of those errors are due to Virtual Box. This does not provide any insight into those who were either put of by installing VritualBox or didn't manage to even get a task.

So what can we do to improve the situation? Where should we direct our effort? Three reasonably smart people have tried to get Virtual Box working but in my view we haven't yet reach a level of quality where this suitable for the non-technical volunteer. My suggestion to try Hyper-V is that is bundled with the OS so should workout-of-the-box. Just read this and now think Hyper-V is out of question.

The Hyper-V role cannot be installed on Windows 10 Home.

AFAIK Hyper-V is a Microsoft product and will not run on Linux?
And you noted it yourself: not available on Home-versions.
You can imagine the volunteers don't want to pay for a needed upgrade.

BOINC programmers have developed vboxwrapper round VirtualBox and is supporting most 32- and 64-bit OS's/platforms.

AFAIK Hyper-V is a Microsoft product and will not run on Linux?
And you noted it yourself: not available on Home-versions.
You can imagine the volunteers don't want to pay for a needed upgrade.

BOINC programmers have developed vboxwrapper round VirtualBox and is supporting most 32- and 64-bit OS's/platforms.

I don't see a FREE upgrade to Win 10 Pro anywhere.

Four of mine are Win10 home version and the other 3 are Win7 and the one XP Pro

And as you can tell here and vLHC and LHC and Atlas (and Atlas-test) I have done thousands of these THeory and CMS tasks over the last 6 years.

My only problem has always been my DSL not letting me d/l all 7 of these with new tasks and add those vdi updates it is over the limit for 99% of DSL's in this country.

So just starting my tasks have to be done at a certain time when my DSL is running at a good enough speed (not throttled down)

I would not pay to update to Win 10 Pro (I have spent thousands already just doing this)

---

Mad Scientist For Life

AFAIK Hyper-V is a Microsoft product and will not run on Linux?

That is correct. We would end up with different wrappers for each OS. More effort on our side but worth it if the user experience improves.

BOINC programmers have developed vboxwrapper round VirtualBox and is supporting most 32- and 64-bit OS's/platforms.

It came from here before the BOINC developers took over. Now that the NSF funding for BOINC stopped, it is an Open Source project so we are all BOINC developers :)

What's about the power of GPU's?
There was a idea for Atlas to work with GPU's, too complicate.
http://atlasathome.cern.ch/forum_thread.php?id=463&postid=3846#3846
http://atlasathome.cern.ch/forum_thread.php?id=303&postid=2572#2572

It came from here before the BOINC developers took over. Now that the NSF funding for BOINC stopped, it is an Open Source project so we are all BOINC developers :)

I was here/there too in 2011 and the cernvmwrapper was rather buggy. I must say, I was very surprised how many bugs were not found during the Alpha-testing.
The main advantage for developers and science was the introduction of running the application in a Virtual Machine by the volunteers.

Maybe the choice for BOINC to control the VM was/is not the best and surely not the only way.

Why not provide the Science VM's directly to the volunteers and let them choose their preferred Virtual Machine Manager to run the VM('s).
During CERN's summer challenge I had several standalone VM's running outside of BOINC or web-interface.

With MC-Plots you already have a system for appreciation for the Theory jobs.
For ATLAS, CMS etc. something like that could be developed.

Just a bit of day-dreaming and maybe this post is gone when I wake-up.

With MC-Plots you already have a system for appreciation for the Theory jobs.
For ATLAS, CMS etc. something like that could be developed.

Ivan told about using MCPlot for CMS

https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=4110&postid=28698#28698

Just a bit of day-dreaming and maybe this post is gone when I wake-up.

Dreaming is the most important part. We have to imagine something before it can be realized.

Why not provide the Science VM's directly to the volunteers and let them choose their preferred Virtual Machine Manager to run the VM('s).

We are pretty close to being able to do this. The manual setup was discussed in this thread. The VMs already exist and can be used with other hypervisors. Just this week I was experimenting with KVM on Linux. We need to inject a userid into the VM. This can either be done with a shared directory or a CDROM image. Accounting for HTCondor already exists for our internal batch system and we already have plans to use it for these resources too. So with very little effort we could provide something that can be used to run a VM locally.

The two things missing are something to automagically setup the VM, i.e. give it the correct amount of memory, control it and a portal to tie it all together. This is was the Challenge Web portal did and maybe the time is now right to resurrect it. The BOINC User ID is something that we would need to replicate. We have the ability to both create light-weight accounts or to use Google/Facebook etc. via our [url=
https://home.web.cern.ch/Shibboleth.sso/?target=https%3A%2F%2Fhome.web.cern.ch%2Fshib_login%2Fnode%2F186]SSO login[/url].

I am interested to know peoples visions, dreams and wishes. We can turn Science Fiction into Science Fact. If there is interest, we could arrange a conference call with something like Google Hangouts at some point in the near future.

What's about the power of GPU's?
There was a idea for Atlas to work with GPU's, too complicate.
http://atlasathome.cern.ch/forum_thread.php?id=463&postid=3846#3846
http://atlasathome.cern.ch/forum_thread.php?id=303&postid=2572#2572

There has been some work done on porting CMSSW to GPUs, but nothing that's been released for production work. In general, only parts of it are parallelisable, and Amdahl's Law will get you most of the time. The code is very heavily object-oriented and lots of small objects are created and destroyed in short order, meaning lots of alloc/dealloc. Moving these into and out of (current-generation) GPUs would also be a troublesome overhead.

---

A lot of I/O-traffic is a great argument against GPU.

OpenCL is now a standard for using GPU's.
Have a Radeon R7 combined with R7 200.

They do Integer-Work for Collatz (4 EXA-Flops per day).

Is CernVM File System a alternate for VM's?

Sorry, but have no background therefore.

Dreaming is the most important part. We have to imagine something before it can be realized.

The large majority out there a windows machines.

The best would be to have stand alone apps with no network traffic apart from up/download.(no vbox)

Requires writing the apps, but is the most reliable.

Are there any statistics for the docker method?
How reliable is that?

Is CernVM File System a alternate for VM's?

No, it's "a scalable, reliable and low-maintenance software distribution service". It communicates with software repositories using HTTP (often through caching proxies) and maintains a local cache of files. As far as user-space is concerned it's read-only.

---

Got yesterday a update for GPU's and saw this information for GPU-programming:

https://www.khronos.org/opencl/
https://github.com/KhronosGroup/OpenCL-Headers