Development for LHC@home

New cranky version explained

Legacy version
It's last recent major change introduced suspend/resume which is expected by BOINC users but not supported by the scientific processes.
That suspend/resume method is based on a freeze/thaw request to runc which forwards it to systemd.
The method works fine with cgroups v1 but needs distinct cgroup directories being prepared in advance by the computer's admin (root).

What happens on systems that do not meet the requirements?
New cranky will try to run the task in legacy mode.
Be aware that there is no further development to improve that mode since cgroups v1 will disappear sometime.

I am planning to put v5.18 on the production server tomorrow. Any objections?

(... from the production server). The server will only send Atlas if it can't send Theory, either because of preference settings or because there are no Theory tasks available (as happened a few weeks ago)
This is a " VBox free" host where there have been >50 Theory tasks in a row without the server sending an Atlas task. I thought that the server may be trying to equalise the credit between the sub-projects but this is greater than the difference in credit; unless it's trying to make up the backlog which, for me, will take a long time unless the credit for Theory increases considerably.

The scheduling is a bit of a black box..

Also,
The need to run hosts continuously is a problem for me and, I expect, for many others as well, especially if the project wants to widen it's potential pool of volunteers. I know that with a bit of babysitting and some clever control scripts this can be overcome to some extent, but we're getting a long way from the original ideas of BOINC.

It is possible to suspend the Theory tasks but not the ATLAS tasks.

Some comments

As CMS is also back at the production server I tried to get both (CMS and Theory native) on the same client.
Unfortunately this client only gets Theory native tasks from the server.
Hence it needs manual intervention to get CMS tasks.


The credits the server rewards for Theory native is rather poor.
Seems only 10% of normal credit and it looks like it doesn't normalize over the time.
It should be clarified why.

and Laurence hadn't configured the wrapper to pick up the output messages.

A new version is available. The issue was a missing file in the latest batch of jobs. The code has been updated to be more robust in such situations and in addition the file system is cleaned after the jobs as I noticed that we were collecting slot directories.

With the new version 4.13 (cranky-0.0.20) I get only errors.

Strange, nothing much changed. Will look into it a bit later this evening.

Generator run finished successfully
100000 events processed
dumping histograms...
Rivet.Analysis.Handler: INFO  Finalising analyses
Rivet.Analysis.Handler: INFO  Processed 100000 events

The MCnet usage guidelines apply to Rivet: see http://www.montecarlonet.org/GUIDELINES
Please acknowledge plots made with Rivet analyses, and cite arXiv:1003.0694 (http://arxiv.org/abs/1003.0694)

Processing histograms...
input  = /shared/tmp/tmp.b9ZK1W7cQa/flat
output = /shared
./runRivet.sh: line 742:   205 Killed                  display_service $tmpd_dump "$beam $process $energy $params $generator $version $tune"  (wd: /shared)
mc:  ATLAS_2011_S9131140_d01-x01-y02.dat -> /shared/dat/pp/zinclusive/pT-Z-norm/atlas2011-el-dressed/7000/pythia8/8.235/default-CD.dat
mc:  ATLAS_2011_S9131140_d01-x01-y03.dat -> /shared/dat/pp/zinclusive/pT-Z-norm/atlas2011-el-bare/7000/pythia8/8.235/default-CD.dat
mc:  ATLAS_2011_S9131140_d02-x01-y02.dat -> /shared/dat/pp/zinclusive/pT-Z-norm/atlas2011-mu-dressed/7000/pythia8/8.235/default-CD.dat
mc:  ATLAS_2011_S9131140_d02-x01-y03.dat -> /shared/dat/pp/zinclusive/pT-Z-.....(snip).....ATLAS_2011_S9131140_d02-x01-y03.dat -> /shared/dat/pp/zinclusive/pT-Z-norm/atlas2011-mu-bare/7000/ATLAS_2011_S9131140.dat

Disk usage: 2440 Kb

CPU usage: 12136 s

Clean tmp ...

Run finished successfully

15:35:16 2019-02-18: cranky-0.0.20: [INFO] Running Container 'runc'.
===> [runRivet] Mon Feb 18 15:35:16 UTC 2019 [boinc pp zinclusive 7000 -,-,50,130 - pythia8 8.235 default-CD 100000 19]
19:00:41 2019-02-18: cranky-0.0.20: [INFO] Preparing output.
tar: local.txt: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
19:00:42 (17293): cranky exited; CPU time 12083.959181
19:00:42 (17293): app exit status: 0x2
19:00:42 (17293): called boinc_finish(195)

00:51:44 (5769): wrapper (7.7.26015): starting
00:51:44 (5769): wrapper: running ../../projects/lhcathomedev.cern.ch_lhcathome-dev/cranky-0.0.14 ()
cranky-0.0.14 INFO: Starting
cranky-0.0.14 INFO: Detected Theory App
cranky-0.0.14 INFO: Checking CVMFS.
cranky-0.0.14 INFO: Checking runc.
cranky-0.0.14 INFO: Creating the filesystem.
cranky-0.0.14 INFO: Using /cvmfs/cernvm-prod.cern.ch/cvm3
cranky-0.0.14 INFO: Updating config.json.
cranky-0.0.14 INFO: Running Container 'runc'.
cranky-0.0.14 ERROR: Container 'runc' failed.
00:55:27 (5769): cranky exited; CPU time 20.553370
00:55:27 (5769): app exit status: 0xce
00:55:27 (5769): called boinc_finish(195)

I have rewritten the script in bash to make is more portable between Linux systems.

[m@TeeC15 ~]$ sudo echo 100 > /proc/sys/user/max_user_namespaces
-bash: /proc/sys/user/max_user_namespaces: Permission denied

$ cat /proc/sys/user/max_user_namespaces
32767

{ user]$ ls -l
total 0
-rw-r--r--. 1 root root 0 Feb 13 03:26 max_ipc_namespaces
-rw-r--r--. 1 root root 0 Feb 13 03:26 max_mnt_namespaces
-rw-r--r--. 1 root root 0 Feb 13 03:26 max_net_namespaces
-rw-r--r--. 1 root root 0 Feb 13 03:26 max_pid_namespaces
-rw-r--r--. 1 root root 0 Feb 13 01:30 max_user_namespaces
-rw-r--r--. 1 root root 0 Feb 13 03:26 max_uts_namespaces
[ user]$

00:51:44 (5769): wrapper (7.7.26015): starting
00:51:44 (5769): wrapper: running ../../projects/lhcathomedev.cern.ch_lhcathome-dev/cranky-0.0.14 ()
cranky-0.0.14 INFO: Starting
cranky-0.0.14 INFO: Detected Theory App
cranky-0.0.14 INFO: Checking CVMFS.
cranky-0.0.14 INFO: Checking runc.
cranky-0.0.14 INFO: Creating the filesystem.
cranky-0.0.14 INFO: Using /cvmfs/cernvm-prod.cern.ch/cvm3
cranky-0.0.14 INFO: Updating config.json.
cranky-0.0.14 INFO: Running Container 'runc'.
cranky-0.0.14 ERROR: Container 'runc' failed.
00:55:27 (5769): cranky exited; CPU time 20.553370
00:55:27 (5769): app exit status: 0xce
00:55:27 (5769): called boinc_finish(195)

I think this is what you need to do. From the 7.4 release notes:

User namespace is now fully supported
The user namespace feature, previously available as a Technology Preview, is now fully supported. It provides additional security to servers running Linux containers by providing better isolation between the host and the containers. Administrators of a container are no longer able to perform administrative operations on the host, which increases security. (BZ#1138782)

Upgrade to 7.6 if you can.

Waitng for cheap internet time... hopefully it won't break anything.

[m@TeeC15 ~]$ sudo echo 100 > /proc/sys/user/max_user_namespaces
-bash: /proc/sys/user/max_user_namespaces: Permission denied

$ cat /proc/sys/user/max_user_namespaces
32767

00:51:44 (5769): wrapper (7.7.26015): starting
00:51:44 (5769): wrapper: running ../../projects/lhcathomedev.cern.ch_lhcathome-dev/cranky-0.0.14 ()
cranky-0.0.14 INFO: Starting
cranky-0.0.14 INFO: Detected Theory App
cranky-0.0.14 INFO: Checking CVMFS.
cranky-0.0.14 INFO: Checking runc.
cranky-0.0.14 INFO: Creating the filesystem.
cranky-0.0.14 INFO: Using /cvmfs/cernvm-prod.cern.ch/cvm3
cranky-0.0.14 INFO: Updating config.json.
cranky-0.0.14 INFO: Running Container 'runc'.
cranky-0.0.14 ERROR: Container 'runc' failed.
00:55:27 (5769): cranky exited; CPU time 20.553370
00:55:27 (5769): app exit status: 0xce
00:55:27 (5769): called boinc_finish(195)

... $ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-514.10.2.el7.x86_64 root=/dev/mapper/cl_teec15-root ro rd.lvm.lv=cl_teec15/root rd.lvm.lv=cl_teec15/swap rhgb quiet LANG=en_GB.UTF-8 user_namespace.enable=1 namespace unpriv_enable=1

There may be a missing ".".
You may try "namespace.unpriv_enable=1" instead of "namespace unpriv_enable=1".

I think this is what you need to do. From the 7.4 release notes:

User namespace is now fully supported
The user namespace feature, previously available as a Technology Preview, is now fully supported. It provides additional security to servers running Linux containers by providing better isolation between the host and the containers. Administrators of a container are no longer able to perform administrative operations on the host, which increases security. (BZ#1138782)

Upgrade to 7.6 if you can.

There is a similar situation in Centos7 (7.3 in my case) here:-

https://groups.io/g/charliecloud/topic/charliecloud_and_centos7/13269608?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,13269608

I haven't tried it yet.

Let me know if you figure it out otherwise I will investigate after looking into Opensuse.

I have found a solution. Taken from the Debian mail logs:
"It turns out that the Debian kernel is set up to disable unprivileged users from unsharing the user namespace by default. This can be worked around using":

sudo su -c 'echo 1 > /proc/sys/kernel/unprivileged_userns_clone'

This did the trick and the application ran successfully on Debian Stretch:
https://lhcathomedev.cern.ch/lhcathome-dev/result.php?resultid=2752281

nsenter: failed to unshare namespaces: Operation not permitted
container_linux.go:336: starting container process caused "process_linux.go:279: running exec setns process for init caused \"exit status 46\""
cranky-0.0.14 ERROR: Container 'runc' failed.

[m@TeeC15 ~]$ unshare -U /bin/bash
[nfsnobody@TeeC15 ~]$

[m@TeeC15 ~]$ sudo unshare -U /bin/bash
[nfsnobody@TeeC15 m]$

[m@TeeC15 ~]$ grep CONFIG_USER_NS /boot/config-$(uname -r)
CONFIG_USER_NS=y
[m@TeeC15 ~]$ CONFIG_USER_NS=y

m wrote:

.... trying to get an app_config to get this host to only download one task at a time, but I can't even get that right now..... says "missing start tag"

An app_config will not control the number of tasks that get downloaded, it controls the number of tasks that run concurrently. If you want to limit the number of tasks that get downloaded, use the "Max # Jobs" setting in your project preferences.

If you still want to use an app_config for another purpose, post it here and maybe someone else here can help debug.

]
The python command does not seem to be consistent between different operating systems. I am relaying on just python being there now but this is not true for Ubuntu where the default seems to be python3.

By default the maximum number of namespaces is set to 0. To fix this run:

echo 640 > /proc/sys/user/max_user_namespaces

For background information, see this bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1340238

In short, on RHEL 7.3 and later starting with kernel version kernel-3.10.0-644.el7 and later, (viewable with grubby --default-kernel) if you're using the --userns-remap parameter in docker config to change the container namespace, you must also ensure that value found in /proc/sys/user/max_user_namespaces is set to a value greater than zero. Zero is its default setting

To reset this value, you can use a call like:

sysctl user.max_user_namespaces=15000

 /usr/bin/env: python3: No such file or directory