Development for LHC@home

The VirtualBox wrappers for Windows and Linux have been upgraded to 26156. The Mac wrapper has been downgraded to 26105.

Let us know if you have any problems. I will post another general news item soon providing more details about this project.

The VirtualBox wrappers for Windows and Linux have been upgraded to 26156. The Mac wrapper has been downgraded to 26105.

Let us know if you have any problems...

May I let you know when there are no problems with Windows vboxwrapper version 26156 so far.
Just returned the first task:
http://boincai05.cern.ch/CMS-dev/result.php?resultid=33137 - with suspends (LAIM off), resumes and system shutdown without stopping BOINC decent first.

2015-03-20 21:38:14 (4068): Creating new snapshot for VM.
2015-03-20 21:38:22 (4068): Deleting stale snapshot.
2015-03-20 21:38:30 (4068): Checkpoint completed.
.
s h u t down
.
system restart
.
2015-03-20 21:41:18 (3436): vboxwrapper (7.5.26156): starting
2015-03-20 21:41:18 (3436): Feature: Checkpoint interval offset (402 seconds)
2015-03-20 21:41:52 (3436): Detected: VirtualBox COM Interface (Version: 4.3.26)
2015-03-20 21:41:52 (3436): Detected: Anonymous Platform Enabled
2015-03-20 21:41:52 (3436): Detected: Minimum checkpoint interval (600.000000 seconds)
2015-03-20 21:41:52 (3436): Restore from previously saved snapshot.
2015-03-20 21:41:53 (3436): Restore completed.
2015-03-20 21:41:53 (3436): Starting VM. (boinc_da505c21a5c24a94, slot#0)
2015-03-20 21:42:29 (3436): Successfully started VM. (PID = '4648')

The VirtualBox wrappers for Windows and Linux have been upgraded to 26156. The Mac wrapper has been downgraded to 26105.

Let us know if you have any problems. I will post another general news item soon providing more details about this project.

Two ships passing in the night...
I just downgraded Mint at home to VBox 4.3.22 but the latest job seems to have had problems starting up -- SITECONF/local is pointing to SITECONF/T1_CH_CERN! And it can't find CMSJobAgent. I'll try aborting and re-starting BOINC.

---

Hmm, startup says it cant find cvmfs service, nor find an ethernet connection. I tried to get a copy of the boot log but many commands including scp and more fail with an input error. Investigations continue...

---

Upgraded to 4.3.26 again, still the same problem. Seems to be a network problem, perhaps I've screwed up somewhere. Oh, well, I've got Android tablets to update...

---

Upgraded to 4.3.26 again, still the same problem. Seems to be a network problem, perhaps I've screwed up somewhere. Oh, well, I've got Android tablets to update...

That done, I removed VBox, rebooted, and re-installed. Same problem -- it doesn't find the network and asks if the cable is plugged in! The rest stems from that; cvmfs doesn't start so what appears to be /cvmfs is actually the local image's /cvmfs directory.
I think I'm done until Monday when I can investigate again with SLC6 at work. Now, where's that Grauniad crossword?

---

Box running Ubuntu and VBox 4.3.12. CMS 46.06 shows as running in VMmanager
and in BOINC Mgr. BUT...

VM shows error "missing network", but the VM is running OK as is the
network.
There are VM messages:-

"RTNETLINK answers: File exists" ... many of these.
and:-

"tail: /home/boinc/stderr: file truncated" ...many of these, too.

stderr looks OK to me, it's only about 4k and shows normal snapshot
activity.

That's it.. should there be more messages showing some VM activity?

vboxheadless is only using ca. 10%cpu... should be much more?


edit:-
Should the "show graphics" function work? It starts a browser (Firefox36.0.1)
but it doesn't connect to anything.

John.

Box running Ubuntu and VBox 4.3.12. CMS 46.06 shows as running in VMmanager
and in BOINC Mgr. BUT...

VM shows error "missing network", but the VM is running OK as is the
network.
There are VM messages:-

"RTNETLINK answers: File exists" ... many of these.
and:-

"tail: /home/boinc/stderr: file truncated" ...many of these, too.

stderr looks OK to me, it's only about 4k and shows normal snapshot
activity.

That's it.. should there be more messages showing some VM activity?

vboxheadless is only using ca. 10%cpu... should be much more?


edit:-
Should the "show graphics" function work? It starts a browser (Firefox36.0.1)
but it doesn't connect to anything.

John.

John, CPU usage isn't high at the moment -- the present workflow is more dominated by data transfers and internet waits. "Show graphics" should connect to a localhost Web server to display log files.
"Missing network" is suspicious, ties in with my observation tho' I hadn't seen that particular indication. The VM messages you mention sound "normal".

---

Hmm, startup says it cant find cvmfs service, nor find an ethernet connection. I tried to get a copy of the boot log but many commands including scp and more fail with an input error. Investigations continue...

Well, it continues the same. Here's a screenshot that shows some of the error messages I get at the tail end of booting. (The httpd message is "normal" AFAICR.)

---

Ivan,

Ubuntu. On missing networks.

In the vbox_trace.txt file are these entries:-

2015-03-21 00:11:34 (2333):
Command: VBoxManage -q modifyvm "boinc_73e3adbb498f9638" --cableconnected1 on
Exit Code: 0
Output:

and:-


2015-03-21 00:11:35 (2333):
Command: VBoxManage -q modifyvm "boinc_73e3adbb498f9638" --nic1 nat --natdnsproxy1 on --cableconnected1 off
Exit Code: 0
Output:

It looks as though the VM "unplugs it's network cable" Can't imagine why.
I can't find any entry indicating reconnection.

I've now started a Win7 host, the "show graphics" works OK there, and CPU usage is higher, ca50%, which is what I would expect given the other task running.

I'm off to bed, I'll have a good look at the missing graphics later today (nearly wrote tomorrow...)

John.

Ivan,
I've now started a Win7 host, the "show graphics" works OK there, and CPU usage is higher, ca50%, which is what I would expect given the other task running.

I'm off to bed, I'll have a good look at the missing graphics later today (nearly wrote tomorrow...)

John.

John, I should be abed too -- far too late last night, the same tonight but at least it's Saturday tomorrow^Wtoday.
My Win7 host didn't show any problems yesterday, it's the Linux boxen that have problems at the moment. As I said, I should leave it for now and chase differences between the home Mint box and the work SLC6 machine on Monday.

---

Ivan.

Missing network - Linux.

Right, after a lot of typing vboxmanage commands (more fingers than keys),
I find that there is a GUI for this (why didn't I look to start with??)

In the VBox manager, select the VM, go to settings/network/advanced and
there is a little checkbox "cable connected"... not checked.

Checked this, rebooted the VM and after a lot of stuff scrolling past too
fast to see, some errors and "failed" messages, everything starts OK.
Connects to cernvmfs and eventually seems to start proper work, CPU@50%.

After a good delay, the "graphics" button now gives access to the log which
shows "proper" work in progress. Like this:-

[37m[21/03/15 11:03:06] ==============================================================================
[21/03/15 11:03:06] I I
[21/03/15 11:03:06] I PYTHIA will be initialized for a p on p collider I
[21/03/15 11:03:06] I at 8000.000 GeV center-of-mass energy I
[21/03/15 11:03:06] I I

As far as I can tell, all is now working, all it needs is for the setup to make sure that the "cable" is connected - I reckon that this is the default. I'll leave it running for a while... got other things to do.

John.

Ivan.

Missing network - Linux.

In the VBox manager, select the VM, go to settings/network/advanced and
there is a little checkbox "cable connected"... not checked.

As far as I can tell, all is now working, all it needs is for the setup to make sure that the "cable" is connected - I reckon that this is the default. I'll leave it running for a while... got other things to do.

Looks like you're right, I managed to "break in" with the GUI before the boot sequence and enable the cable. Everything ran as before. I suspect this will only affect the running VM, not the image on disk -- I'll find out tomorrow, I guess. If the Windows and Mac machines are running OK, then it must be something in the Linux wrapper that's not enabling the network connexion.

---

... If the Windows and Mac machines are running OK, then it must be something in the Linux wrapper that's not enabling the network connexion.

Cable connected is default ticked for the VM on my Windows machine.

... If the Windows and Mac machines are running OK, then it must be something in the Linux wrapper that's not enabling the network connexion.

Cable connected is default ticked for the VM on my Windows machine.

The other alternative that I realised later is that Oracle accidentally released the Linux version of VirtualBox 4.3.26 with the network disconnected... Ah, no, I still had the same problem when I downgraded versions last night. Finger seems to be pointing at the wrapper still.

---

Finger seems to be pointing at the wrapper still.

I'm inclined to agree.
There doesn't seem to be any accident although it may have been left behind after some test or debug operation. The commands to enable the "cable" and then to disconnect it again are recorded in the trace file. Look a few posts down this thread - message 133.

I can't find a trace file for this project on the Windows box, should it be in the slot directory? The linux and Win wrappers are the same version, 26156, so I would have expected them to be the same, anyway it works OK.

Finger seems to be pointing at the wrapper still.

I'm inclined to agree.
There doesn't seem to be any accident although it may have been left behind after some test or debug operation. The commands to enable the "cable" and then to disconnect it again are recorded in the trace file. Look a few posts down this thread - message 133.

I can't find a trace file for this project on the Windows box, should it be in the slot directory? The linux and Win wrappers are the same version, 26156, so I would have expected them to be the same, anyway it works OK.

OK, I found that in my slot directory here on Linux Mint too. The gun is getting smokier.
No idea where that file lives on Windows though, my only Windows system for this project is my work desktop so I can't dig around until Monday.

[Edit] Actually, I tell a lie. Thanks to the wonders of cygwin and openssl I was able to log on to my Win PC and start a new task (using boinccmd). Then, I found:

admD405@W7-SE-D304-01 /cygdrive/d/ProgramData/BOINC
$ find slots/10 -name \*.log -exec grep -3 -i cable {} \; -print
00:00:04.686511 
00:00:04.686511 [/Devices/e1000/0/Config/] (level 4)
00:00:04.686513   AdapterType    <integer> = 0x0000000000000000 (0)
00:00:04.686514   CableConnected <integer> = 0x0000000000000001 (1)
00:00:04.686515   LineSpeed      <integer> = 0x0000000000000000 (0)
00:00:04.686516   MAC            <bytes>   = "08 00 27 d9 2a e3" (cb=6)
00:00:04.686518 
slots/10/boinc_edb5c7fd0d26a0b2/Logs/VBox.log
00:00:04.686511 
00:00:04.686511 [/Devices/e1000/0/Config/] (level 4)
00:00:04.686513   AdapterType    <integer> = 0x0000000000000000 (0)
00:00:04.686514   CableConnected <integer> = 0x0000000000000001 (1)
00:00:04.686515   LineSpeed      <integer> = 0x0000000000000000 (0)
00:00:04.686516   MAC            <bytes>   = "08 00 27 d9 2a e3" (cb=6)
00:00:04.686518 
slots/10/VBox.log
admD405@W7-SE-D304-01 /cygdrive/d/ProgramData/BOINC
$ ls -l slots/10
total 7326838
drwx------+ 1 Administrators None          0 Mar 21 20:46 boinc_edb5c7fd0d26a0b2
-rwx------+ 1 Administrators None          0 Mar 21 20:46 boinc_lockfile
-rwx------+ 1 Administrators None       9163 Mar 21 20:46 init_data.xml
-rwx------+ 1 Administrators None       3591 Mar 21 20:47 stderr.txt
-rwx------+ 1 Administrators None      58735 Mar 21 20:46 VBox.log
-rwx------+ 1 Administrators None        209 Mar 21 20:46 vbox_checkpoint.xml
-rwx------+ 1 Administrators None         84 Mar 21 20:44 vbox_job.xml
-rwx------+ 1 Administrators None         69 Mar 21 20:46 vbox_remote_desktop.xml
-rwx------+ 1 Administrators None         53 Mar 21 20:46 vbox_webapi.xml
-rwx------+ 1 Administrators None        102 Mar 21 20:44 vboxwrapper_26156_windows_x86_64.exe
-rwx------+ 1 Administrators None        102 Mar 21 20:46 vboxwrapper_26156_windows_x86_64.pdb
-rwx------+ 1 Administrators None 5847908352 Mar 19 14:45 vm_cache.vdi
-rwx------+ 1 Administrators None      28672 Mar 21 20:46 vm_floppy_10.img
-rwx------+ 1 Administrators None 1641021440 Mar 21 20:46 vm_image.vdi

so the VBox.log files are probably what you need to probe.
[/Edit]

---

I can't find a trace file for this project on the Windows box, should it be in the slot directory? The linux and Win wrappers are the same version, 26156, so I would have expected them to be the same, anyway it works OK.

I also can't find the trace and replay files on my Windows machine. IIRC they were not there with vboxwrapper v26155 too.

At least for Linux I found from the wrapper the vboxmanage commands for network settings:

VBoxManage -q modifyvm "VM_NAME" --cableconnected1 on
VBoxManage -q modifyvm "VM_NAME" --nic1 nat --natdnsproxy1 on --cableconnected1 off

The only trace file on Windows I can find is for wrapper 26110 (Atlas). There is one on Linux for 26079, but vLHC no longer uses this version on Windows. In any event, from memory, the trace file shows the commands issued to VB (vboxmanage) by the wrapper. Rom added it specially to facilitate fault finding. It has nothing to do with VM control. This being so, the errant command clearly is being sent by the wrapper so that is the place to look. It's well beyond me, Rom is the expert.

Edit:- The windows wrappers checked are the normal project-supplied ones

26156 (CMS), 26151 (LHCb), 26110 (Atlas) and 26155 (vLHC)

and it looks, from this

At least for Linux I found from the wrapper the vboxmanage commands for network settings:

VBoxManage -q modifyvm "VM_NAME" --cableconnected1 on
VBoxManage -q modifyvm "VM_NAME" --nic1 nat --natdnsproxy1 on --cableconnected1 off

as if CP has found the actual commands in the wrapper, the last of which is responsible for the problem. Thanks CP.

At least for Linux I found from the wrapper the vboxmanage commands for network settings:

VBoxManage -q modifyvm "VM_NAME" --cableconnected1 on
VBoxManage -q modifyvm "VM_NAME" --nic1 nat --natdnsproxy1 on --cableconnected1 off

Did you find them in the actual wrapper? If so, how? (I'm pretty much a novice when it comes to VM management.) Is there a way to modify it? As I expected, when yesterday's run finished just now, a new task started up still with cable off.

---