Development for LHC@home

For the Test ATLAS-production (wrapper204) and Theory -dev now also postponed.
Virtualbox 6.0.14 (Boinc-Website)

Does it also happen when you keep the VirtualBox GUI closed?

I closed my eyes last night ;), but left the GUI open.
I'm trying to reproduce something:

1. I still had a normal VM in Virtual Media Manager and removed the multi-attach ATLAS from Media Manager manually after an ATLAS-task finished.
A new resumed Atlas started OK.

2. GUI closed. I still had a normal VM in Virtual Media Manager (a Theory running) and the multi-attach ATLAS (after an ATLAS-task finished) was not removed.
A new resumed Atlas started and got the postponed state. https://lhcathomedev.cern.ch/lhcathome-dev/result.php?resultid=3098220

VBoxSVC.exe keeps the vdi's from Media Manager in memory and therefore the multi attach VM is not removed, so GUI open or closed does not matter.
When you have several VM's running VBoxSVC.exe of course is always running.
From last week I can remember combining multi-attach ATLAS and multi-attach Theory was not a problem, but have to test that to be sure.

2022-07-07 08:40:28 (19672): Adding storage controller(s) to VM.
2022-07-07 08:40:29 (19672): Adding virtual disk drive to VM. (Theory_2020_05_08.vdi)
2022-07-07 08:41:01 (19672): Error in storage attach (fixed disk - multiattach mode) for VM: -2135228409
Command:
VBoxManage -q storageattach "boinc_d2f1d64641adcadc" --storagectl "Hard Disk Controller" --port 0 --device 0 --type hdd --mtype multiattach --medium "C:/ProgramData/BOINC/projects/lhcathomedev.cern.ch_lhcathome-dev/Theory_2020_05_08.vdi"
Output:
VBoxManage.exe: error: Cannot attach medium 'C:\ProgramData\BOINC\projects\lhcathomedev.cern.ch_lhcathome-dev\Theory_2020_05_08.vdi': the media type 'MultiAttach' can only be attached to machines that were created with VirtualBox 4.0 or later
VBoxManage.exe: error: Details: code VBOX_E_INVALID_OBJECT_STATE (0x80bb0007), component SessionMachine, interface IMachine, callee IUnknown
VBoxManage.exe: error: Context: "AttachDevice(Bstr(pszCtl).raw(), port, device, DeviceType_HardDisk, pMedium2Mount)" at line 776 of file VBoxManageStorageController.cpp

Notes:

Another VirtualBox management application has locked the session for
this VM. BOINC cannot properly monitor this VM

There must be a problem with connection -dev (multiattach) and production?
Is this why vboxsvc.exe is not stopped correctly?

What I see from some generic tests is that the VBox GUI - especially the VBox Media Manager - doesn't update the media list when another process - here vboxwrapper via vboxmanage - adds/removes a disk.

The VM description files as well as the (user-)global VirtualBox.xml are also not written.
To make those writes happen you have to close the GUI and reopen it.

That's the situation on Linux. On Windows it may differ.


The test without GUI should ensure that the medium lists are written to the correct destination.
If they appear in VirtualBox.xml this causes the well known error:
"the media type 'MultiAttach' can only be attached to machines that were created with VirtualBox 4.0 or later"


Some vboxmanage subcommands (e.g. showmediuminfo) are known to temporarily write the disk entry to VirtualBox.xml but remove it again immediately.
This "remove" might fail if the GUI is open.



At the end this might be a VirtualBox error but we don't have enough evidence yet to send them a useful error description.

When you shut down Windows (Win11pro or Win10pro) vboxsvc.exe as task is shown as a open task in the Taskmanager and
Windows can not shut down immediatly,
You have to shut down this vboxsvc.exe task yourself, but Boinc-Manager was closed before correct.

I didn't have VirtualBox Manager or any kind of VirtualBox GUI running while "postponed" occurred for v1.14 (with ATLAS v2.00) .

Chapter 9 in the Virtualbox documentation.

From last week I can remember combining multi-attach ATLAS and multi-attach Theory was not a problem, but have to test that to be sure.

First test with a multi attach Theory VM running and an ATLAS finished and tried to start a new ATLAS - no success https://lhcathomedev.cern.ch/lhcathome-dev/result.php?resultid=3098247

lhcathome-dev 07 Jul 09:56:04 Computation for task EvOLDmPqRT1n7Olcko1bjSoqABFKDmABFKDmKFbSDmISHKDmJGP8wn_0 finished
lhcathome-dev 07 Jul 09:56:04 Starting task 9KNKDmhcXT1n7Olcko1bjSoqABFKDmABFKDmKFbSDmMSHKDmMXM2wm_0
lhcathome-dev 07 Jul 09:56:06 Started upload of EvOLDmPqRT1n7Olcko1bjSoqABFKDmABFKDmKFbSDmISHKDmJGP8wn_0_r311759848_ATLAS_result
lhcathome-dev 07 Jul 09:56:06 Started upload of EvOLDmPqRT1n7Olcko1bjSoqABFKDmABFKDmKFbSDmISHKDmJGP8wn_0_r311759848_ATLAS_hits
lhcathome-dev 07 Jul 09:56:08 Finished upload of EvOLDmPqRT1n7Olcko1bjSoqABFKDmABFKDmKFbSDmISHKDmJGP8wn_0_r311759848_ATLAS_result
lhcathome-dev 07 Jul 09:56:09 Finished upload of EvOLDmPqRT1n7Olcko1bjSoqABFKDmABFKDmKFbSDmISHKDmJGP8wn_0_r311759848_ATLAS_hits
lhcathome-dev 07 Jul 09:56:44 Sending scheduler request: To fetch work.
lhcathome-dev 07 Jul 09:56:44 Reporting 1 completed tasks
lhcathome-dev 07 Jul 09:56:44 Requesting new tasks for CPU
lhcathome-dev 07 Jul 09:56:45 Scheduler request completed: got 0 new tasks
lhcathome-dev 07 Jul 09:56:45 No tasks sent
lhcathome-dev 07 Jul 09:56:45 No tasks are available for ATLAS Simulation
lhcathome-dev 07 Jul 09:56:45 Project requested delay of 61 seconds
lhcathome-dev 07 Jul 09:56:50 Task 9KNKDmhcXT1n7Olcko1bjSoqABFKDmABFKDmKFbSDmMSHKDmMXM2wm_0 postponed for 86400 seconds: VM environment needs to be cleaned up.
lhcathome-dev 07 Jul 09:56:50 Starting task Theory_2390-1103695-268_0


What I saw when the last Theory task from production finished was and the GUI open, that the vm was removed from media manager.
The same wrapper, but another type of vdi.

Testing Boinc 7.20.1 with Virtualbox 6.1.34 from Boinc-Website (development) including vbowwrapper 204 for Atlas-Production.
Tomorrow some tests from -dev.

Atlas in -dev with 1.14.vdi running successful (multiattach) with Atlas V2.00 from Production.
Theory get postponed.
Old 1.13.vdi must be deleted manual in Virtualbox-manager.
All are tested with vboxwrapper 204 (-dev AND Production - renamed old vboxwrapper26198ab7 to vboxwrapper204)!

It's nice to know what combinations of projects possibly may run together, although running a few tasks is not the same as deeply tested.
As far as I know:
- Only running tasks with multi attach VM's from 1 single application are running OK.
- Combination of applications with multi attach and 'normal' VM's could lead to postponed state of the multi attach application. GUI open or closed doesn't matter.
- Combination of 2 different multi attach application (e.g. ATLAS - Theory from dev) also sometimes have the postponed problem.

In earlier days postponed means, that the application will resume 24 hours later; most of the time successful,
but this postponed state creates un untouchable VM without a boot medium and runs for ever after a resume or BOINC restart.
I also had a VBox Rosetta-task - resourse share zero - and thought I only had one and that was successful.
Later I found in Virtual Media Manager (I kept it closed as long as possible including VirtualBox Manager) 4 remnants of Rosetta VMs.
The results of Rosetta are purged very quickly, so I could not find the the error. Maybe not related to ATLAS multi attach.

We should not focus on combinations of LHC VBox-applications alone, but have a broader view, before introducing these multi-attach VM's.

The second running multiattach get postponed for Atlas.
So, this is as you wrote very, very difficult to find what is going on.

At the moment I'm testing on my laptop concurrently:

1 dual core ALTAS multi attach together with one Theory multi attach of course both from this dev-server.

On my desktop I'm testing concurrently:

5 dual core ATLAS from dev (multi attach VM) together with 5 Theory's from production server ('normal' vdi; vboxwrapper26198ab7)

The ATLAS queue is running dry:
ATLAS Simulation 0 available 59 in progress

July 12: 12 hours of testing with announced combination to test a new vboxwrapper.

Summary: No errors.

Tasks:
https://lhcathomedev.cern.ch/lhcathome-dev/results.php?hostid=4600&offset=0&show_names=0&state=0&appid=0
https://lhcathomedev.cern.ch/lhcathome-dev/results.php?hostid=4547
https://lhcathome.cern.ch/lhcathome/results.php?hostid=10690380 and
during the last 2 hours added 1 Rosetta VM-task to the mix ending also Valid.

2022-07-12 19:41:55 (8004): Detected: vboxwrapper 26202
2022-07-12 19:41:55 (8004): Detected: BOINC client v7.16.20
2022-07-12 19:41:55 (8004): Detected: VirtualBox VboxManage Interface (Version: 6.1.34)
2022-07-12 19:41:56 (8004): Feature: Checkpoint interval offset (479 seconds)
2022-07-12 19:41:56 (8004): Detected: Minimum checkpoint interval (600.000000 seconds)
2022-07-12 19:41:59 (8004): Create VM. (boinc_7f0e178915cbd935, slot#10)
2022-07-12 19:42:00 (8004): Setting Memory Size for VM. (6144MB)
2022-07-12 19:42:00 (8004): Setting CPU Count for VM. (1)
2022-07-12 19:42:00 (8004): Setting Chipset Options for VM.
2022-07-12 19:42:01 (8004): Setting Boot Options for VM.
2022-07-12 19:42:01 (8004): Setting Network Configuration for NAT.
2022-07-12 19:42:02 (8004): Disabling VM Network Access.
2022-07-12 19:42:02 (8004): Disabling USB Support for VM.
2022-07-12 19:42:02 (8004): Disabling COM Port Support for VM.
2022-07-12 19:42:03 (8004): Disabling LPT Port Support for VM.
2022-07-12 19:42:03 (8004): Disabling Audio Support for VM.
2022-07-12 19:42:04 (8004): Disabling Clipboard Support for VM.
2022-07-12 19:42:04 (8004): Disabling Drag and Drop Support for VM.
2022-07-12 19:42:05 (8004): Adding storage controller(s) to VM.
2022-07-12 19:42:05 (8004): Adding virtual disk drive to VM. (vm_image.vdi)
2022-07-12 19:42:07 (8004): Adding VirtualBox Guest Additions to VM.
2022-07-12 19:42:07 (8004): Adding network bandwidth throttle group to VM. (Defaulting to 1024GB)
2022-07-12 19:42:08 (8004): Enabling shared directory for VM.
2022-07-12 19:42:09 (8004): Starting VM using VBoxManage interface. (boinc_7f0e178915cbd935, slot#10)
2022-07-12 19:42:13 (8004): Successfully started VM. (PID = '3508')
2022-07-12 19:42:13 (8004): Reporting VM Process ID to BOINC.
2022-07-12 19:42:13 (8004): Guest Log: BIOS: VirtualBox 6.1.34
2022-07-12 19:42:13 (8004): Guest Log: CPUID EDX: 0x178bfbff
2022-07-12 19:42:13 (8004): Guest Log: BIOS: ata0-0: PCHS=16383/16/63 LCHS=1024/255/63
2022-07-12 19:42:13 (8004): VM state change detected. (old = 'poweredoff', new = 'running')
2022-07-12 19:42:13 (8004): Preference change detected
2022-07-12 19:42:13 (8004): Setting CPU throttle for VM. (100%)
2022-07-12 19:42:13 (8004): Setting checkpoint interval to 600 seconds. (Higher value of (Preference: 120 seconds) or (Vbox_job.xml: 600 seconds))
2022-07-12 19:42:15 (8004): Guest Log: BIOS: Boot : bseqnr=1, bootseq=0032
2022-07-12 19:42:15 (8004): Guest Log: BIOS: Booting from Hard Disk...
2022-07-12 19:42:55 (8004): Guest Log: vgdrvHeartbeatInit: Setting up heartbeat to trigger every 2000 milliseconds
2022-07-12 19:42:55 (8004): Guest Log: vboxguest: misc device minor 58, IRQ 20, I/O port d020, MMIO at 00000000f0400000 (size 0x400000)
2022-07-12 19:42:57 (8004): Guest Log: VBoxService 5.2.42 r137960 (verbosity: 0) linux.amd64 (May 13 2020 21:45:13) release log
2022-07-12 19:42:57 (8004): Guest Log: 00:00:00.000147 main Log opened 2022-07-12T19:42:54.848659000Z
2022-07-12 19:42:57 (8004): Guest Log: 00:00:00.000203 main OS Product: Linux
2022-07-12 19:42:57 (8004): Guest Log: 00:00:00.000226 main OS Release: 4.19.0-14-amd64
2022-07-12 19:42:57 (8004): Guest Log: 00:00:00.000242 main OS Version: #1 SMP Debian 4.19.171-2 (2021-01-30)
2022-07-12 19:42:57 (8004): Guest Log: 00:00:00.000264 main Executable: /opt/VBoxGuestAdditions-5.2.42/sbin/VBoxService
2022-07-12 19:42:57 (8004): Guest Log: 00:00:00.000265 main Process ID: 542
2022-07-12 19:42:57 (8004): Guest Log: 00:00:00.000266 main Package type: LINUX_64BITS_GENERIC
2022-07-12 19:42:57 (8004): Guest Log: 00:00:00.000996 main 5.2.42 r137960 started. Verbose level = 0
2022-07-12 19:43:07 (8004): Guest Log: 00:00:10.011095 timesync vgsvcTimeSyncWorker: Radical guest time change: -7 188 345 651 000ns (GuestNow=1 657 647 786 513 463 000 ns GuestLast=1 657 654 974 859 114 000 ns fSetTimeLastLoop=true )
2022-07-12 20:00:13 (8004): Creating new snapshot for VM.
2022-07-12 20:00:19 (8004): Checkpoint completed.
2022-07-12 20:10:14 (8004): Creating new snapshot for VM.
2022-07-12 20:10:20 (8004): Deleting stale snapshot.
2022-07-12 20:10:21 (8004): Checkpoint completed.
2022-07-12 20:20:15 (8004): Creating new snapshot for VM.
2022-07-12 20:20:20 (8004): Deleting stale snapshot.
2022-07-12 20:20:21 (8004): Checkpoint completed.
2022-07-12 20:30:16 (8004): Creating new snapshot for VM.
2022-07-12 20:30:21 (8004): Deleting stale snapshot.
2022-07-12 20:30:21 (8004): Checkpoint completed.
2022-07-12 20:40:17 (8004): Creating new snapshot for VM.
2022-07-12 20:40:22 (8004): Deleting stale snapshot.
2022-07-12 20:40:23 (8004): Checkpoint completed.
2022-07-12 20:50:17 (8004): Creating new snapshot for VM.
2022-07-12 20:50:23 (8004): Deleting stale snapshot.
2022-07-12 20:50:24 (8004): Checkpoint completed.
2022-07-12 20:56:51 (8004): Guest Log: 01:13:27.128064 control Guest control service stopped
2022-07-12 20:56:51 (8004): Guest Log: 01:13:27.128143 control Guest control worker returned with rc=VINF_SUCCESS
2022-07-12 20:56:51 (8004): Guest Log: 01:13:27.128729 main Session 0 is about to close ...
2022-07-12 20:56:51 (8004): Guest Log: 01:13:27.128758 main Stopping all guest processes ...
2022-07-12 20:56:51 (8004): Guest Log: 01:13:27.128778 main Closing all guest files ...
2022-07-12 20:56:51 (8004): Guest Log: 01:13:27.313111 main Ended.
2022-07-12 20:58:21 (8004): VM is no longer is a running state. It is in 'poweredoff'.
2022-07-12 20:58:21 (8004): VM state change detected. (old = 'running', new = 'poweredoff')
2022-07-12 20:58:21 (8004): Powering off VM.
2022-07-12 20:58:21 (8004): Deregistering VM. (boinc_7f0e178915cbd935, slot#10)
2022-07-12 20:58:21 (8004): Deleting stale snapshot.
2022-07-12 20:58:21 (8004): Removing network bandwidth throttle group from VM.
2022-07-12 20:58:22 (8004): Removing VM from VirtualBox.
2022-07-12 20:58:27 (8004): Virtual machine exited.
20:58:32 (8004): called boinc_finish(0)

Btw: Rosetta uses a 'normal' VM with a size of 7115MB (almost 7GB)

How many hours is the 1.14 app supposed to run after the progress bar reports 100% and 'time to go' is zero?
I abort after 24 hours in this state.

How many hours is the 1.14 app supposed to run ...

Tasks from the last batch ran about 1h.

A view into your logfiles would be helpful.
Hence, please make your computers visible for other volunteers here:
https://lhcathomedev.cern.ch/lhcathome-dev/prefs.php?subset=project

Not that many hours.
You may have run into: FATAL: Could not read from the boot medium! System halted.
Probably you had a postponed task and waited 24 hours or did a BOINC restart.

Your machine is hidden, so other crunchers can't examine your results.

Edit: I'm typing too slow and/or correcting my text too often.

I have to test further with vboxwrapper_26205.

Last days it seems to be rather stable running dev-ATLAS combined with Prod-Theory until I added dev-Theory to the mix yesterday evening.
So always running 1 dual core ATLAS and 2 Theory's (1 from production and 1 from dev).
I even added the optional element <open_name>vm_image to the dev-Theory app_version.

17-Jul-2022 01:22:48 [lhcathome-dev] Computation for task 0xPKDmV72W1n7Olcko1bjSoqABFKDmABFKDmKFbSDmvwHKDmIpcKim_0 finished
17-Jul-2022 01:22:48 [lhcathome-dev] Starting task 0HeLDmpH3W1n7Olcko1bjSoqABFKDmABFKDmKFbSDmwwHKDmw1xqlm_0
17-Jul-2022 01:22:51 [lhcathome-dev] Started upload of 0xPKDmV72W1n7Olcko1bjSoqABFKDmABFKDmKFbSDmvwHKDmIpcKim_0_r1256053833_ATLAS_result
17-Jul-2022 01:22:51 [lhcathome-dev] Started upload of 0xPKDmV72W1n7Olcko1bjSoqABFKDmABFKDmKFbSDmvwHKDmIpcKim_0_r1256053833_ATLAS_hits
17-Jul-2022 01:22:55 [lhcathome-dev] Finished upload of 0xPKDmV72W1n7Olcko1bjSoqABFKDmABFKDmKFbSDmvwHKDmIpcKim_0_r1256053833_ATLAS_result
17-Jul-2022 01:22:55 [lhcathome-dev] Finished upload of 0xPKDmV72W1n7Olcko1bjSoqABFKDmABFKDmKFbSDmvwHKDmIpcKim_0_r1256053833_ATLAS_hits
17-Jul-2022 01:23:33 [lhcathome-dev] Task 0HeLDmpH3W1n7Olcko1bjSoqABFKDmABFKDmKFbSDmwwHKDmw1xqlm_0 postponed for 86400 seconds: VM environment needs to be cleaned up.
17-Jul-2022 01:23:33 [lhcathome-dev] Starting task rMXNDmPA4W1n7Olcko1bjSoqABFKDmABFKDmKFbSDmzwHKDm3x3Ofn_0
17-Jul-2022 01:24:17 [lhcathome-dev] Task rMXNDmPA4W1n7Olcko1bjSoqABFKDmABFKDmKFbSDmzwHKDm3x3Ofn_0 postponed for 86400 seconds: VM environment needs to be cleaned up.
17-Jul-2022 01:24:17 [lhcathome-dev] Starting task ADfMDmJB4W1n7Olcko1bjSoqABFKDmABFKDmKFbSDm0wHKDmyCnU1m_0
17-Jul-2022 01:25:01 [lhcathome-dev] Task ADfMDmJB4W1n7Olcko1bjSoqABFKDmABFKDmKFbSDm0wHKDmyCnU1m_0 postponed for 86400 seconds: VM environment needs to be cleaned up.
17-Jul-2022 01:25:01 [lhcathome-dev] Starting task gF3MDmPJ4W1n7Olcko1bjSoqABFKDmABFKDmKFbSDm1wHKDmgz2QLo_0
17-Jul-2022 01:25:45 [lhcathome-dev] Task gF3MDmPJ4W1n7Olcko1bjSoqABFKDmABFKDmKFbSDm1wHKDmgz2QLo_0 postponed for 86400 seconds: VM environment needs to be cleaned up.
17-Jul-2022 01:25:45 [lhcathome-dev] Starting task Op0NDmgQ4W1n7Olcko1bjSoqABFKDmABFKDmKFbSDm2wHKDmGU0SLo_0
17-Jul-2022 01:26:29 [lhcathome-dev] Task Op0NDmgQ4W1n7Olcko1bjSoqABFKDmABFKDmKFbSDm2wHKDmGU0SLo_0 postponed for 86400 seconds: VM environment needs to be cleaned up.
17-Jul-2022 01:26:29 [lhcathome-dev] Starting task jthNDmUg4W1n7Olcko1bjSoqABFKDmABFKDmKFbSDm3wHKDm05Nhcm_0
17-Jul-2022 01:27:13 [lhcathome-dev] Task jthNDmUg4W1n7Olcko1bjSoqABFKDmABFKDmKFbSDm3wHKDm05Nhcm_0 postponed for 86400 seconds: VM environment needs to be cleaned up.
17-Jul-2022 01:27:13 [lhcathome-dev] Starting task tGONDm6U5W1n7Olcko1bjSoqABFKDmABFKDmKFbSDm4wHKDmcebYom_0
17-Jul-2022 01:27:57 [lhcathome-dev] Task tGONDm6U5W1n7Olcko1bjSoqABFKDmABFKDmKFbSDm4wHKDmcebYom_0 postponed for 86400 seconds: VM environment needs to be cleaned up.
17-Jul-2022 01:27:57 [lhcathome-dev] Starting task LEYMDm9V5W1n7Olcko1bjSoqABFKDmABFKDmKFbSDm5wHKDmvQlTUo_0
17-Jul-2022 01:28:42 [lhcathome-dev] Task LEYMDm9V5W1n7Olcko1bjSoqABFKDmABFKDmKFbSDm5wHKDmvQlTUo_0 postponed for 86400 seconds: VM environment needs to be cleaned up.
17-Jul-2022 01:28:42 [lhcathome-dev] Starting task FpiKDm6b5W1n7Olcko1bjSoqABFKDmABFKDmKFbSDm6wHKDmubFhPn_0
17-Jul-2022 01:29:26 [lhcathome-dev] Task FpiKDm6b5W1n7Olcko1bjSoqABFKDmABFKDmKFbSDm6wHKDmubFhPn_0 postponed for 86400 seconds: VM environment needs to be cleaned up.
17-Jul-2022 01:29:26 [lhcathome-dev] Starting task ufXNDmFt5W1n7Olcko1bjSoqABFKDmABFKDmKFbSDm7wHKDmpf1fZn_0
17-Jul-2022 01:30:10 [lhcathome-dev] Task ufXNDmFt5W1n7Olcko1bjSoqABFKDmABFKDmKFbSDm7wHKDmpf1fZn_0 postponed for 86400 seconds: VM environment needs to be cleaned up.

====================================================================================================================================================

17-Jul-2022 08:22:53 [lhcathome-dev] Computation for task K18LDm87BX1n7Olcko1bjSoqABFKDmABFKDmKFbSDmMxHKDmVOr3Ao_0 finished
17-Jul-2022 08:22:53 [lhcathome-dev] Starting task QTgMDmtgCX1n7Olcko1bjSoqABFKDmABFKDmKFbSDmNxHKDm29lGBo_0
17-Jul-2022 08:23:37 [lhcathome-dev] Task QTgMDmtgCX1n7Olcko1bjSoqABFKDmABFKDmKFbSDmNxHKDm29lGBo_0 postponed for 86400 seconds: VM environment needs to be cleaned up.
17-Jul-2022 08:23:37 [lhcathome-dev] Starting task 2SHKDm9nCX1n7Olcko1bjSoqABFKDmABFKDmKFbSDmOxHKDmMFDWFm_0
17-Jul-2022 08:24:22 [lhcathome-dev] Task 2SHKDm9nCX1n7Olcko1bjSoqABFKDmABFKDmKFbSDmOxHKDmMFDWFm_0 postponed for 86400 seconds: VM environment needs to be cleaned up.

After the production Theory's have finished, I'll start with ATLAS-dev and Theory-dev only.

Last days it seems to be rather stable running dev-ATLAS combined with Prod-Theory until I added dev-Theory to the mix yesterday evening.

Were all of the postponed tasks dev-Theory?

Did you use vboxwrapper 26204 from the dev server for dev-Theory or did you replace it with the 26205 artifact?
I suggest not to use vboxwrapper 26204 for those tests.