Development for LHC@home

Try this command for testing:

unshare -U /bin/bash

The output is

unshare: unshare failed: Operation not permitted

running it with sudo gives a new line in the terminal with

nobody@debian:

with "nobody" not being my normal user

This is related to rootless containers. It is a newish feature so may not work with older kernel versions. Let me know if you figure it out otherwise I will investigate after looking into Opensuse.

I guess we are running real scientific simulations, e.g. Herwig++, rather than dummy subtasks, right?
Hence the different runtimes of the tasks.

Yes. I am copying jobs from the production Theory queue. Will look at sending the results back to MCPlots later this week.

m wrote:

.... trying to get an app_config to get this host to only download one task at a time, but I can't even get that right now..... says "missing start tag"

An app_config will not control the number of tasks that get downloaded, it controls the number of tasks that run concurrently. If you want to limit the number of tasks that get downloaded, use the "Max # Jobs" setting in your project preferences.

If you still want to use an app_config for another purpose, post it here and maybe someone else here can help debug.

It's OK, thanks. Just more fingers than keys, more keys than brain cells, or so it seems. Would very much like to see if the python 3 problem is fixed but no work now.

It's OK, thanks. Just more fingers than keys, more keys than brain cells, or so it seems. Would very much like to see if the python 3 problem is fixed but no work now.

Just submitted a few more.

OK, thanks, got one with this result.

nsenter: failed to unshare namespaces: Operation not permitted
container_linux.go:336: starting container process caused "process_linux.go:279: running exec setns process for init caused \"exit status 46\""
cranky-0.0.14 ERROR: Container 'runc' failed.

Test Commands

[m@TeeC15 ~]$ unshare -U /bin/bash
[nfsnobody@TeeC15 ~]$

and

[m@TeeC15 ~]$ sudo unshare -U /bin/bash
[nfsnobody@TeeC15 m]$

and

[m@TeeC15 ~]$ grep CONFIG_USER_NS /boot/config-$(uname -r)
CONFIG_USER_NS=y
[m@TeeC15 ~]$ CONFIG_USER_NS=y

Let me know if you figure it out otherwise I will investigate after looking into Opensuse.

I have found a solution. Taken from the Debian mail logs:
"It turns out that the Debian kernel is set up to disable unprivileged users from unsharing the user namespace by default. This can be worked around using":

sudo su -c 'echo 1 > /proc/sys/kernel/unprivileged_userns_clone'

This did the trick and the application ran successfully on Debian Stretch:
https://lhcathomedev.cern.ch/lhcathome-dev/result.php?resultid=2752281

OK, thanks, got one with this result.

nsenter: failed to unshare namespaces: Operation not permitted
container_linux.go:336: starting container process caused "process_linux.go:279: running exec setns process for init caused \"exit status 46\""
cranky-0.0.14 ERROR: Container 'runc' failed.

I
Test Commands

[m@TeeC15 ~]$ unshare -U /bin/bash
[nfsnobody@TeeC15 ~]$

and

[m@TeeC15 ~]$ sudo unshare -U /bin/bash
[nfsnobody@TeeC15 m]$

and

[m@TeeC15 ~]$ grep CONFIG_USER_NS /boot/config-$(uname -r)
CONFIG_USER_NS=y
[m@TeeC15 ~]$ CONFIG_USER_NS=y

User m may be privileged. Try with the boinc user.

Let me know if you figure it out otherwise I will investigate after looking into Opensuse.

I have found a solution. Taken from the Debian mail logs:
"It turns out that the Debian kernel is set up to disable unprivileged users from unsharing the user namespace by default. This can be worked around using":

sudo su -c 'echo 1 > /proc/sys/kernel/unprivileged_userns_clone'

This did the trick and the application ran successfully on Debian Stretch:
https://lhcathomedev.cern.ch/lhcathome-dev/result.php?resultid=2752281

Great! I will add that to the guide.

Let me know if you figure it out otherwise I will investigate after looking into Opensuse.

I have found a solution. Taken from the Debian mail logs:
"It turns out that the Debian kernel is set up to disable unprivileged users from unsharing the user namespace by default. This can be worked around using":

sudo su -c 'echo 1 > /proc/sys/kernel/unprivileged_userns_clone'

This did the trick and the application ran successfully on Debian Stretch:
https://lhcathomedev.cern.ch/lhcathome-dev/result.php?resultid=2752281

There is a similar situation in Centos7 (7.3 in my case) here:-

https://groups.io/g/charliecloud/topic/charliecloud_and_centos7/13269608?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,13269608

I haven't tried it yet.

I created a new VM with OS Ubuntu 18.10 on my Windows 7 host and installed BOINC from the packages into that VM.
Using Laurence's new setup info, I got it to work with correcting some things in the setup text:

sudo wget https://lhcathomedev.cern.ch/lhcathome-dev/download/default.localcvmfs.autofs -O /etc/auto.master.d/cvmfs.autof

should be

sudo wget https://lhcathomedev.cern.ch/lhcathome-dev/download/cvmfs.autofs -O /etc/auto.master.d/cvmfs.autof

and had to reboot before "autofs restart"

Tasks ready so far from this newly host: https://lhcathomedev.cern.ch/lhcathome-dev/results.php?hostid=3717

sudo su -c 'echo 1 > /proc/sys/kernel/unprivileged_userns_clone'

This did the trick and the application ran successfully on Debian Stretch:
https://lhcathomedev.cern.ch/lhcathome-dev/result.php?resultid=2752281

Great! I will add that to the guide.

Unfortunately, it looks like that this fix is only temporarily, since after rebooting the system the value in that file is again 0 and the unshare command does not work anymore.
To fix this for good, I adapted the sysctl.conf file in order to make the value of "/proc/sys/kernel/unprivileged_userns_clone" to be persisant 1.

Open the file with

sudo nano /etc/sysctl.conf

and add the following to the end of that file

kernel.unprivileged_userns_clone = 1

Then apply the changes with

sudo sysctl -p

Using this approach made the value of " /proc/sys/kernel/unprivileged_userns_clone" stay at 1 after reboots and the unshare command also works "out of the box" after rebooting.

There is a similar situation in Centos7 (7.3 in my case) here:-

https://groups.io/g/charliecloud/topic/charliecloud_and_centos7/13269608?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,13269608

I haven't tried it yet.

Well, I now have

$ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-514.10.2.el7.x86_64 root=/dev/mapper/cl_teec15-root ro rd.lvm.lv=cl_teec15/root rd.lvm.lv=cl_teec15/swap rhgb quiet LANG=en_GB.UTF-8 user_namespace.enable=1 namespace unpriv_enable=1

and

$ cat /etc/sysctl.d/51-userns.conf
user.max_user_namespaces = 32767

but unshare still doesn't work.

$ unshare -U /bin/bash
[nfsnobody@TeeC15 ~]$


Short of updating the kernel, I don't know what else to do.

... $ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-514.10.2.el7.x86_64 root=/dev/mapper/cl_teec15-root ro rd.lvm.lv=cl_teec15/root rd.lvm.lv=cl_teec15/swap rhgb quiet LANG=en_GB.UTF-8 user_namespace.enable=1 namespace unpriv_enable=1

There may be a missing ".".
You may try "namespace.unpriv_enable=1" instead of "namespace unpriv_enable=1".

There is a similar situation in Centos7 (7.3 in my case) here:-

Short of updating the kernel, I don't know what else to do.

I think this is what you need to do. From the 7.4 release notes:

User namespace is now fully supported
The user namespace feature, previously available as a Technology Preview, is now fully supported. It provides additional security to servers running Linux containers by providing better isolation between the host and the containers. Administrators of a container are no longer able to perform administrative operations on the host, which increases security. (BZ#1138782)

Upgrade to 7.6 if you can.

I created a new VM with OS Ubuntu 18.10 on my Windows 7 host and installed BOINC from the packages into that VM.
Using Laurence's new setup info, I got it to work with correcting some things in the setup text:

sudo wget https://lhcathomedev.cern.ch/lhcathome-dev/download/default.localcvmfs.autofs -O /etc/auto.master.d/cvmfs.autof

should be

sudo wget https://lhcathomedev.cern.ch/lhcathome-dev/download/cvmfs.autofs -O /etc/auto.master.d/cvmfs.autof

and had to reboot before "autofs restart"

Tasks ready so far from this newly host: https://lhcathomedev.cern.ch/lhcathome-dev/results.php?hostid=3717

Thanks for the feedback. I have updated the guide.

I'm currently running a Theory native task on opensuse 13.1:
https://lhcathomedev.cern.ch/lhcathome-dev/result.php?resultid=2752313

At first I got some errors as runc requested a newer version of libseccomp.so.2.
I copied that lib from opensuse 42.3 and adjusted the library path.

Now it works.

You can find the job log in a subdirectory of the slot directory. Doing a hard link should make it stay available after the job ends. If you let me know what info you would like to see, I can make it available.

I found a method (not a Linux expert) to monitor the job output for a running job.

First line to see what kind of job is running including number of events to process (second last number):

head -n 1 /var/lib/boinc-client/slots/0/cernvm/shared/runRivet.log

and for a continued look at the last n lines:

watch -n 10 tail -n 15 /var/lib/boinc-client/slots/0/cernvm/shared/runRivet.log

where you have/can adjust the slotnumber, number of lines and/or number of seconds watch interval.

Earlier I wrote:

I managed to snag another one of these tasks this morning. It has been running for 58 minutes. Even though it is allocated 2 CPUs, it looks like it is only using 1 CPU.

Please let me know if you need more information.

Laurence responded:

This may be related to it starting two processes.

The latest attempt:

The machine has a 6 core 12 thread CPU. Each thread should show ~8% of total CPU usage according to the System Monitor.

The machine now has a test Theory task running that is allocated 4 CPUs (threads). According to the System Monitor, the processes that I can see that look like they are associated with the test Theory task are rivetvm.exe using 5% of the total CPU and pythia8.exe using 4% of the total CPU. The task has been running for 25+ minutes. Even though it is allocated 4 threads, it looks like it is really only using 1 thread.

Am I missing something?

Am I missing something?

As far as I understand, setting in your preferences # cores is used for running more jobs within a VM (each VM-core 1 job).
Since we are testing Theory native, every BOINC-task will run only 1 job.
If you want to run more jobs select in your preferences the number of tasks you want or set No Limit.

If you have set e.g. 2 cores, BOINC will allocate 2 cores for 1 job and this single job will run a bit faster, but leaving a core partial idle.

The python command does not seem to be consistent between different operating systems. I am relaying on just python being there now but this is not true for Ubuntu where the default seems to be python3.

I think that I might rewrite cranky in bash to be more portable.