Development for LHC@home

The machine which got stuck activating the fuse module is still stuck at the same place, but a second machine which was running at the time has picked up the new directory structure without any problems.

I'll restart the stuck one.

Ivan, are any of your work units for your presentation running yet?

Are we out of Jobs again ?

Just added a machine and it Looks, as if it can not get Jobs ?

Ivan, are any of your work units for your presentation running yet?

No, I'm getting an error message in CRAB3 submission that it can't gsissh to the RAL Condor machine -- which is a puzzle as I can log in manually. RAL tells me that nothing has changed at the machine, so I'm at a loss to explain it. I edited out the section that sends to RAL and two jobs ran successfully at Vanderbilt. :-(
We may run out of re-submitted jobs over the weekend, to which I can only say, "OK, relax and run one of your other projects until I can get a handle on this."

---

The machine which was stuck just now is active again (did a VM reset from Oracle, left BOINC running throughout) and seems to be busy (%CPU 86) with

Beginning CMSSW wrapper script
slc6_amd64_gcc491 scramv1 CMSSW
Performing SCRAM setup...
Completed SCRAM setup
Retrieving SCRAM project...
Untarring /home/boinc/CMSRun/glide_RA2Frx/execute/dir_7663/sandbox.tar.gz
Completed SCRAM project
Executing CMSSW
cmsRun -j FrameworkJobReport.xml PSet.py
Begin processing the 1st record. Run 1, Event 167401, LumiSection 1675 at 21-Aug-2015 16:08:35.638 BST
Begin processing the 2nd record. Run 1, Event 167402, LumiSection 1675 at 21-Aug-2015 16:09:16.891 BST
Begin processing the 3rd record. Run 1, Event 167403, LumiSection 1675 at 21-Aug-2015 16:09:16.904 BST
Begin processing the 4th record. Run 1, Event 167404, LumiSection 1675 at 21-Aug-2015 16:09:29.119 BST
Begin processing the 5th record. Run 1, Event 167405, LumiSection 1675 at 21-Aug-2015 16:09:58.715 BST
Begin processing the 6th record. Run 1, Event 167406, LumiSection 1675 at 21-Aug-2015 16:10:13.498 BST
Begin processing the 7th record. Run 1, Event 167407, LumiSection 1675 at 21-Aug-2015 16:10:16.388 BST
Begin processing the 8th record. Run 1, Event 167408, LumiSection 1675 at 21-Aug-2015 16:10:19.684 BST

I've a host stuck at "activating fuse module". Tried rebooting the VM, aborting
the task and rebooting the host... still sticks at the same place. Short of
resetting the project and starting afresh I can't think of anything else to do. Any ideas?

I've a host stuck at "activating fuse module". Tried rebooting the VM, aborting
the task and rebooting the host... still sticks at the same place. Short of
resetting the project and starting afresh I can't think of anything else to do. Any ideas?

IIRC, that's when it's trying to restart the cvmfs distributed file system. You don't have any firewall or network issues, do you?

---

I've a host stuck at "activating fuse module". Tried rebooting the VM, aborting
the task and rebooting the host... still sticks at the same place. Short of
resetting the project and starting afresh I can't think of anything else to do. Any ideas?

IIRC, that's when it's trying to restart the cvmfs distributed file system. You don't have any firewall or network issues, do you?

Not that I know of. The firewall is common to the whole lan and I've just started
another host with no problems. There may have been network problems overnight when this host first started, a different host failed due to not downloading a file from RAL but that was OK this morning.

The screen is as shown here so it's the second time the fuse module is activated.

Looks like each subdirectory:
/logs/run-3/glide_sN6j69/dir_14859/cmsRun-stdout.log
/logs/run-3/glide_sN6j69/dir_19030/cmsRun-stdout.log


contains the results of one lot of 200 runs.

Between run 200 and run 1 of the next lot is a gap of about 7-8 minutes.
I guess, that is the time it takes to reinitialize everything and get new data.

It only generates 3 of these sub-Folders and then starts a new "run-x+1" folder.

/logs/run-4/glide_OmvP3r/dir_4810/
/logs/run-4/glide_OmvP3r/dir_8936/
/logs/run-4/glide_OmvP3r/dir_32714/

During that time the results are also staged-out and this is a 15MB file.

Rasputin42 wrote:

It only generates 3 of these sub-Folders and then starts a new "run-x+1" folder.

This greatly depends on the speed of the CPU of your machine and the % of CPU you have allowed to use in BOINC.

A minor influence has the kind of events (records) you get to do in the job.
There is a large deviation in the used cpu-time for 1 event, varying from almost 0 (zero) seconds up to (on my i7 2600) 69 seconds for 1 single event.
In run-1 my machine did 8 jobs in 5 hours and 37 minutes.

I am running CMS on my desktop OK but on my Windows 7 laptop it does not run.

It sits at 0.00% complete and when I looked at Vbox it had thid message.

Runtime error opening 'C:\ProgramData\BOINC\slots\3\boinc_7b8c6c74b1591434\boinc_7b8c6c74b1591434.vbox' for reading: -103(Path not found.).
D:\tinderbox\win-4.3\src\VBox\Main\src-server\MachineImpl.cpp[731] (long __cdecl Machine::registeredInit(void)).
Result Code:
E_FAIL (0x80004005)
Component:
Machine
Interface:
IMachine {480cf695-2d8d-4256-9c7c-cce4184fa048}



I had a look and the file is not there ?
C:\ProgramData\BOINC\slots\3\boinc_7b8c6c74b1591434\boinc_7b8c6c74b1591434.vbox

This is the host: http://boincai05.cern.ch/CMS-dev/results.php?hostid=641

---

Proud Founder and member of



Have a look at my WebCam

Rasputin42 wrote:

It only generates 3 of these sub-Folders and then starts a new "run-x+1" folder.

This greatly depends on the speed of the CPU of your machine and the % of CPU you have allowed to use in BOINC.

A minor influence has the kind of events (records) you get to do in the job.
There is a large deviation in the used cpu-time for 1 event, varying from almost 0 (zero) seconds up to (on my i7 2600) 69 seconds for 1 single event.
In run-1 my machine did 8 jobs in 5 hours and 37 minutes.

OK, for your information, the current jobs are "minimum-bias" events -- essentially non-(scientifically)-interesting background events from "ordinary" proton-proton collisions. These are then thrown into simulations where we seed the events with "interesting" scenarios, say Higgs->gamma-gamma. The stuff I'm working on at the moment is the ~2025 scenario where we expect, on average, 140 or maybe even 200 such events each time a counter-rotating pair of proton bunches pass through each other (i.e. "collide") in the centre of CMS, with maybe one interesting event every 100 million, or billion, or whatever, collisions.
So these sort of simulations are important since we throw the background events into our simulations of interesting events, to make sure we can still separate the wheat from the chaff.
However, to the best of my knowledge, there is no restriction on the min-bias results -- if the random numbers fall in the right range at the right time, we may even throw a random H->gamma-gamma into the supposed background. So these events are of random complexity, which leads to the variability of simulation times per event.

[Edit] Just to add, the events I'm hoping to use for the "challenge", once I've sorted out the problem that prevents my actually submitting new jobs, are so-called TTbar events, where the proton-proton collision creates a top- and anti-top quark pair (via "pair production", as, e.g. electron-positron production from energetic gamma rays). These top quarks can then decay in a number of ways, so the simulation is necessarily more complex than simple min-bias events. This way I hope to increase the run-time per job by about a factor of three for the same amount of data returned to stage-out. [/Edit]

---

Ivan, are any of your work units for your presentation running yet?

No, I'm getting an error message in CRAB3 submission that it can't gsissh to the RAL Condor machine -- which is a puzzle as I can log in manually. RAL tells me that nothing has changed at the machine, so I'm at a loss to explain it. I edited out the section that sends to RAL and two jobs ran successfully at Vanderbilt. :-(
We may run out of re-submitted jobs over the weekend, to which I can only say, "OK, relax and run one of your other projects until I can get a handle on this."

Ivan, Just noticed these entries in a log:-


CRAB_Dest = "/store/temp/user/ireid.17433470b3faad006e8120ad843d39e3666b08f0/MinBias/CRAB3_tutorial_MC_generation_test4

CRAB_UserWebDir = "/home/cms005/150818_115322:ireid_crab_CMS_at_Home_test7"

Is this one of "your jobs"?

Ivan, are any of your work units for your presentation running yet?

No, I'm getting an error message in CRAB3 submission that it can't gsissh to the RAL Condor machine -- which is a puzzle as I can log in manually. RAL tells me that nothing has changed at the machine, so I'm at a loss to explain it. I edited out the section that sends to RAL and two jobs ran successfully at Vanderbilt. :-(
We may run out of re-submitted jobs over the weekend, to which I can only say, "OK, relax and run one of your other projects until I can get a handle on this."

Ivan, Just noticed these entries in a log:-


CRAB_Dest = "/store/temp/user/ireid.17433470b3faad006e8120ad843d39e3666b08f0/MinBias/CRAB3_tutorial_MC_generation_test4

CRAB_UserWebDir = "/home/cms005/150818_115322:ireid_crab_CMS_at_Home_test7"

Is this one of "your jobs"?

Yeah. In fact everything you're working on at the moment is one of my jobs. The first line is the temporary destination that the results are stored in before being transferred to stage-out. The second line is the actual directory on the RAL Condor server that results are copied to. The mismatch in "test" numbers is because I forgot there are two places in the submission script where you can change an identity -- in fact the results are going into a CRAB3_tutorial_MC_generation_test4 directory, but distinguished by the 150818_115322 bit, which if you squint carefully enough you will see is the date and time of the CRAB task submission.

---

I've a host stuck at "activating fuse module". Tried rebooting the VM, aborting
the task and rebooting the host... still sticks at the same place. Short of
resetting the project and starting afresh I can't think of anything else to do. Any ideas?

IIRC, that's when it's trying to restart the cvmfs distributed file system. You don't have any firewall or network issues, do you?

Not that I know of. The firewall is common to the whole lan and I've just started
another host with no problems. There may have been network problems overnight when this host first started, a different host failed due to not downloading a file from RAL but that was OK this morning.

The screen is as shown here so it's the second time the fuse module is activated.

Running OK now. All 6 working well as I write (that's inviting disaster...)

If this was caused by a transient network problem, it's not good that the system doesn't (or didn't) seem able to recover from it without manual intervention.

After the 2nd Glidein-run, that lasted 5 hours and 48 minutes, Glidein start looping again.
So no real processing anymore of cmsRun's.
Meanwhile 134 run-xxx maps created.
Not sure whether following from run-3 is relevant:

From Glidein-stdout:

Executing /home/boinc/CMSRun/glide_pu0ihT/main/glidein_sitewms_setup.sh
Unsupported GLIDEIN_SiteWMS encountered
Executing /home/boinc/CMSRun/glide_pu0ihT/main/smart_partitionable.sh
=== Last script starting Sat Aug 22 03:15:27 CEST 2015 (1440206127) after validating for 19 ===

=== Condor starting Sat Aug 22 03:15:29 CEST 2015 (1440206129) ===
=== Condor ended Sat Aug 22 03:15:34 CEST 2015 (1440206134) after 5 ===

=== Stats of main ===
=================
Total jobs 0 utilization 0
Total goodZ jobs 0 (0%) utilization 0 (0%)
Total goodNZ jobs 0 (0%) utilization 0 (0%)
Total badSignal jobs 0 (0%) utilization 0 (0%)
Total badOther jobs 0 (0%) utilization 0 (0%)
=== End Stats of main ===

From Glidein-stderr:

CONDOR_OS default
ERROR_GEN_PATH /home/boinc/CMSRun/glide_pu0ihT/main/error_gen.sh
CONSTS_FILE /home/boinc/CMSRun/glide_pu0ihT/main/constants.cfg
CONDOR_VARS_FILE /home/boinc/CMSRun/glide_pu0ihT/main/condor_vars.lst
UNTAR_CFG_FILE /home/boinc/CMSRun/glide_pu0ihT/main/untar.cfg
GRIDMAP /home/boinc/CMSRun/glide_pu0ihT/main/grid-mapfile
#

At least map pu0ihT is not present in the map run-3.

After the 2nd Glidein-run, that lasted 5 hours and 48 minutes, Glidein start looping again.

Requested a new BOINC-task for a fresh start.
No cmsRun is started, but therefore every 7½ minutes a run-x map is created.

In it:

[DIR] glide_l4w0AM/ 22-Aug-2015 08:43 -
[ ] glidein-stderr 22-Aug-2015 08:42 28K
[ ] glidein-stdout 22-Aug-2015 08:42 4.3K

In the glide_xxxxxx-map there are only:

[ ] MasterLog 22-Aug-2015 08:18 2.1K
[ ] ProcLog 22-Aug-2015 08:18 12K
[ ] StartdLog 22-Aug-2015 08:18 5.1K
[ ] StarterLog 22-Aug-2015 08:13 620

No cmsRun is started, but therefore every 7½ minutes a run-x map is created.

We've run out of jobs on the Condor server. You can all stand down until further notice.

---

IT is not working.
It generates run-1, run-2 etc. every 7 minutes. Looks like Condor (whatever it is)is not working.

Here is /logs/run-1/glide_8oUehg/StartdLog

08/26/15 20:59:59 (pid:7614) ******************************************************
08/26/15 20:59:59 (pid:7614) ** condor_startd (CONDOR_STARTD) STARTING UP
08/26/15 20:59:59 (pid:7614) ** /home/boinc/CMSRun/glide_8oUehg/main/condor/sbin/condor_startd
08/26/15 20:59:59 (pid:7614) ** SubsystemInfo: name=STARTD type=STARTD(7) class=DAEMON(1)
08/26/15 20:59:59 (pid:7614) ** Configuration: subsystem:STARTD local:<NONE> class:DAEMON
08/26/15 20:59:59 (pid:7614) ** $CondorVersion: 8.2.3 Sep 30 2014 BuildID: 274619 $
08/26/15 20:59:59 (pid:7614) ** $CondorPlatform: x86_64_RedHat5 $
08/26/15 20:59:59 (pid:7614) ** PID = 7614
08/26/15 20:59:59 (pid:7614) ** Log last touched time unavailable (No such file or directory)
08/26/15 20:59:59 (pid:7614) ******************************************************
08/26/15 20:59:59 (pid:7614) Using config source: /home/boinc/CMSRun/glide_8oUehg/condor_config
08/26/15 20:59:59 (pid:7614) config Macros = 211, Sorted = 211, StringBytes = 10603, TablesBytes = 7636
08/26/15 20:59:59 (pid:7614) CLASSAD_CACHING is ENABLED
08/26/15 20:59:59 (pid:7614) Daemon Log is logging: D_ALWAYS D_ERROR D_JOB
08/26/15 20:59:59 (pid:7614) DaemonCore: command socket at <10.0.2.15:50881?noUDP>
08/26/15 20:59:59 (pid:7614) DaemonCore: private command socket at <10.0.2.15:50881>
08/26/15 21:00:00 (pid:7614) CCBListener: registered with CCB server lcggwms02.gridpp.rl.ac.uk:9622 as ccbid 130.246.180.120:9622#38173
08/26/15 21:00:00 (pid:7614) my_popenv failed
08/26/15 21:00:00 (pid:7614) Failed to run hibernation plugin '/home/boinc/CMSRun/glide_8oUehg/main/condor/libexec/condor_power_state ad'
08/26/15 21:00:00 (pid:7614) VM-gahp server reported an internal error
08/26/15 21:00:00 (pid:7614) VM universe will be tested to check if it is available
08/26/15 21:00:00 (pid:7614) History file rotation is enabled.
08/26/15 21:00:00 (pid:7614) Maximum history file size is: 20971520 bytes
08/26/15 21:00:00 (pid:7614) Number of rotated history files is: 2
08/26/15 21:00:00 (pid:7614) Allocating auto shares for slot type 1: Cpus: 1.000000, Memory: auto, Swap: auto, Disk: auto
slot type 1: Cpus: 1.000000, Memory: 2002, Swap: 100.00%, Disk: 100.00%
08/26/15 21:00:00 (pid:7614) New machine resource of type 1 allocated
08/26/15 21:00:00 (pid:7614) Setting up slot pairings
08/26/15 21:00:00 (pid:7614) my_popenv failed
08/26/15 21:00:00 (pid:7614) Adding 'mips' to the Supplimental ClassAd list
08/26/15 21:00:00 (pid:7614) CronJobList: Adding job 'mips'
08/26/15 21:00:00 (pid:7614) Adding 'kflops' to the Supplimental ClassAd list
08/26/15 21:00:00 (pid:7614) CronJobList: Adding job 'kflops'
08/26/15 21:00:00 (pid:7614) CronJob: Initializing job 'mips' (/home/boinc/CMSRun/glide_8oUehg/main/condor/libexec/condor_mips)
08/26/15 21:00:00 (pid:7614) CronJob: Initializing job 'kflops' (/home/boinc/CMSRun/glide_8oUehg/main/condor/libexec/condor_kflops)
08/26/15 21:00:00 (pid:7614) State change: IS_OWNER is false
08/26/15 21:00:00 (pid:7614) Changing state: Owner -> Unclaimed
08/26/15 21:00:00 (pid:7614) State change: RunBenchmarks is TRUE
08/26/15 21:00:00 (pid:7614) Changing activity: Idle -> Benchmarking
08/26/15 21:00:00 (pid:7614) BenchMgr:StartBenchmarks()
08/26/15 21:00:19 (pid:7614) State change: benchmarks completed
08/26/15 21:00:19 (pid:7614) Changing activity: Benchmarking -> Idle
08/26/15 21:04:33 (pid:7614) No resources have been claimed for 30 seconds
08/26/15 21:04:33 (pid:7614) Shutting down Condor on this machine.
08/26/15 21:04:33 (pid:7614) Got SIGTERM. Performing graceful shutdown.
08/26/15 21:04:33 (pid:7614) shutdown graceful
08/26/15 21:04:33 (pid:7614) Cron: Killing all jobs
08/26/15 21:04:33 (pid:7614) Cron: Killing all jobs
08/26/15 21:04:33 (pid:7614) Killing job mips
08/26/15 21:04:33 (pid:7614) Killing job kflops
08/26/15 21:04:33 (pid:7614) Deleting cron job manager
08/26/15 21:04:33 (pid:7614) Cron: Killing all jobs
08/26/15 21:04:33 (pid:7614) Cron: Killing all jobs
08/26/15 21:04:33 (pid:7614) CronJobList: Deleting all jobs
08/26/15 21:04:33 (pid:7614) Cron: Killing all jobs
08/26/15 21:04:33 (pid:7614) CronJobList: Deleting all jobs
08/26/15 21:04:33 (pid:7614) Deleting benchmark job mgr
08/26/15 21:04:33 (pid:7614) Cron: Killing all jobs
08/26/15 21:04:33 (pid:7614) Killing job mips
08/26/15 21:04:33 (pid:7614) Killing job kflops
08/26/15 21:04:33 (pid:7614) Cron: Killing all jobs
08/26/15 21:04:33 (pid:7614) Killing job mips
08/26/15 21:04:33 (pid:7614) Killing job kflops
08/26/15 21:04:33 (pid:7614) CronJobList: Deleting all jobs
08/26/15 21:04:33 (pid:7614) CronJobList: Deleting job 'mips'
08/26/15 21:04:33 (pid:7614) CronJob: Deleting job 'mips' (/home/boinc/CMSRun/glide_8oUehg/main/condor/libexec/condor_mips), timer -1
08/26/15 21:04:33 (pid:7614) CronJobList: Deleting job 'kflops'
08/26/15 21:04:33 (pid:7614) CronJob: Deleting job 'kflops' (/home/boinc/CMSRun/glide_8oUehg/main/condor/libexec/condor_kflops), timer -1
08/26/15 21:04:33 (pid:7614) Cron: Killing all jobs
08/26/15 21:04:33 (pid:7614) CronJobList: Deleting all jobs
08/26/15 21:04:33 (pid:7614) All resources are free, exiting.