Development for LHC@home

Even with the new version, it still has only one job per run.

Glidein_stderror:

max wall time, 64800
WARNING: job max time is bigger than max_walltime, lowering it.
job max time, 63644
calculated retire time, 666
using default retire spread, 66
Retire time set to 637
Die time set to 64281

I can't see an obvious error but I might be onto it. In any case this will all change once we have one run.

We are waiting on an update to the configuration on the Condor server at RAL. The admins their have to juggle a lot of things so we just have to be patient.

All good things come to those who wait. An update should be available in a few hours.

The intended 12h per task is not working.

A tasks exits after a few minutes:
http://lhcathomedev.cern.ch/vLHCathome-dev/result.php?resultid=124813

Or it has "runs" from 2h10min to 3h47min length (my observation) with a max of 3 runs per task.

In any case: It is not working!

Longest running task: 10.5hours exit reason:
Guest Log: [INFO] No more jobs. Shutting down!

Did you manage to grab the glidein stderr? That looks similar to our little "Oops!" yesterday.

---

No, sorry.

Here is one of the currently running task.

-----------------------------------------------------
used param defined retire time, 43200
using default retire spread, 4320
Retire time set to 41386
Die time set to 170986

Unless you wanted to see something else?

I suggest to double the value for glidein_max_tail and glidein_max_idle.

After starting up, the glidein will wait for Glidein_Max_Idle for its initial matching. If it is Idle for longer than this, it will assume no jobs are available and will shutdown.

EDIT:All tasks, that ended too early had "No more Jobs" as exit reason.
That would match exactly the condition mentioned above.

Here is the relevant line from the StardLog of that task.

03/24/16 10:30:24 (pid:7665) Changing state: Unclaimed -> Claimed
03/24/16 10:30:28 (pid:7665) The DaemonShutdown expression "(DynamicSlot =!= True) && ((((GLIDEIN_ToDie =!= UNDEFINED) && (CurrentTime > GLIDEIN_ToDie)) || ( (Slot1_Activity == "Idle") && (((GLIDEIN_Max_Idle =!= UNDEFINED) && ifThenElse((Slot1_JobStarts =!= UNDEFINED), ((Slot1_JobStarts =!= UNDEFINED) && (Slot1_SelfMonitorAge =!= UNDEFINED) && (GLIDEIN_Max_Idle =!= UNDEFINED) && (Slot1_JobStarts == 0) && (Slot1_SelfMonitorAge > GLIDEIN_Max_Idle)), ((Slot1_TotalTimeUnclaimedIdle =!= UNDEFINED) && (GLIDEIN_Max_Idle =!= UNDEFINED) && ((PartitionableSlot =!= True) || (TotalSlots =?=1)) && (Slot1_TotalTimeUnclaimedIdle > GLIDEIN_Max_Idle)))) || ((GLIDEIN_Max_Tail =!= UNDEFINED) && ifThenElse((Slot1_JobStarts =!= UNDEFINED), ((Slot1_JobStarts =!= UNDEFINED) && (Slot1_ExpectedMachineGracefulDrainingCompletion =!= UNDEFINED) && (GLIDEIN_Max_Tail =!= UNDEFINED) && (Slot1_JobStarts > 0) && ((CurrentTime - Slot1_ExpectedMachineGracefulDrainingCompletion) > GLIDEIN_Max_Tail) ), ((Slot1_TotalTimeUnclaimedIdle =!= UNDEFINED) && (GLIDEIN_Max_Tail =!= UNDEFINED) && (Slot1_TotalTimeClaimedBusy =!= UNDEFINED) && ((PartitionableSlot =!= True) || (TotalSlots =?=1)) && (Slot1_TotalTimeUnclaimedIdle > GLIDEIN_Max_Tail)))) || ((GLIDEIN_ToRetire =!= UNDEFINED) && (CurrentTime > GLIDEIN_ToRetire ))) )))" evaluated to TRUE: starting graceful shutdown

I am going to revert back to the configuration that we had last week when we were running for 6 hours and remove one of the checks that is clearly not working.

We shouldn't really be testing these tuning values like we are here. After 4th April, the new Test4Thoery application will be uploaded here which also uses HTCondor but without the glidein. We can then essentially deploy a standard batch node and experiment with the values.

Thanks.
Well, hard to tell. However, it is worth a shot, as a lot values were doubled to accommodate the change from 6h to 12h.

Unless you have a better idea?

Alternatively, revert back to 6h, as this worked quite well.

OK, I have just reverted the configuration to what we introduced 4th March.

CCB_HEARTBEAT_INTERVAL = 0
NOT_RESPONDING_TIMEOUT = 10800
GLIDEIN_Retire_Time = 21600 (6h, the default)
GLIDEIN_MAX_WALLTIME = unset
CLAIM_WORKLIFE = ifThenElse(DynamicSlot =?= true,3600,-1)

There is a problem with the update.

Somthing is not right: (EDIT:only virtualLHC tasks)

3 runs are done, one after the other. Only the 3rd run is actually doing a job.

Cron_stderr:
touch: cannot touch `/home/boinc/shared/heartbeat': No such file or directory
rm: cannot remove `/home/boinc/CMSRun/glide_Tjhi9N/main': Directory not empty
/cvmfs/cms.cern.ch/CMS@Home/agent/CMSJobAgent.sh: line 244: [: CMSRun/glide_BZZ8yu: binary operator expected
/cvmfs/cms.cern.ch/CMS@Home/agent/CMSJobAgent.sh: line 244: [: CMSRun/glide_BZZ8yu: binary operator expected
/cvmfs/cms.cern.ch/CMS@Home/agent/CMSJobAgent.sh: line 244: [: CMSRun/glide_BZZ8yu: binary operator expected
/cvmfs/cms.cern.ch/CMS@Home/agent/CMSJobAgent.sh: line 244: [: CMSRun/glide_BZZ8yu: binary operator expected
/cvmfs/cms.cern.ch/CMS@Home/agent/CMSJobAgent.sh: line 244: [: CMSRun/glide_BZZ8yu: binary operator expected
/cvmfs/cms.cern.ch/CMS@Home/agent/CMSJobAgent.sh: line 244: [: CMSRun/glide_BZZ8yu: binary operator expected
/cvmfs/cms.cern.ch/CMS@Home/agent/CMSJobAgent.sh: line 244: [: CMSRun/glide_BZZ8yu: binary operator expected
/cvmfs/cms.cern.ch/CMS@Home/agent/CMSJobAgent.sh: line 244: [: CMSRun/glide_BZZ8yu: binary operator expected
/cvmfs/cms.cern.ch/CMS@Home/agent/CMSJobAgent.sh: line 244: [: CMSRun/glide_BZZ8yu: binary operator expected
/cvmfs/cms.cern.ch/CMS@Home/agent/CMSJobAgent.sh: line 244: [: CMSRun/glide_BZZ8yu: binary operator expected
/cvmfs/cms.cern.ch/CMS@Home/agent/CMSJobAgent.sh: line 244: [: CMSRun/glide_BZZ8yu: binary operator expected
/cvmfs/cms.cern.ch/CMS@Home/agent/CMSJobAgent.sh: line 244: [: CMSRun/glide_BZZ8yu: binary operator expected
/cvmfs/cms.cern.ch/CMS@Home/agent/CMSJobAgent.sh: line 244: [: CMSRun/glide_BZZ8yu: binary operator expected
/cvmfs/cms.cern.ch/CMS@Home/agent/CMSJobAgent.sh: line 244: [: CMSRun/glide_BZZ8yu: binary operator expected
/cvmfs/cms.cern.ch/CMS@Home/agent/CMSJobAgent.sh: line 244: [: CMSRun/glide_BZZ8yu: binary operator expected

Is that correct?

glidein_stderr:

GLIDECLIENT_Group_Hold=False
GLIDECLIENT_Group_PREEMPT=False
GLIDECLIENT_Rank=1
GLIDECLIENT_Group_Rank=1
GLIDEIN_CLAIM_WORKLIFE_DYNAMIC=3600
GLIDEIN_CLAIM_WORKLIFE=-1
GLIDEIN_Factory="factory_service"
GLIDEIN_Name="v3_2_7"
GLIDEIN_CredentialIdentifier="GLIDEIN_CredentialIdentifier "


default is 1200 sec

I just hat a task finishing at exactly 18h runtime.
It was in the middle of a job, when it shut down.
No reason for shutdown in the log.

The log does not show the last run jobs.
It was working on the 10th job in total(4th in this run).

http://lhcathomedev.cern.ch/vLHCathome-dev/result.php?resultid=125039

So max wall time of 64800 is functioning faster than vboxwrapper's job_duration of 64800,
else "VM Completion File Detected" would have been written into the result.

In any case it is a termination, not a proper shutdown.