Message boards : News : CMS Servers up again
Message board moderation

To post messages, you must log in.

AuthorMessage
Profile Magic Quantum Mechanic
Avatar

Send message
Joined: 8 Apr 15
Posts: 459
Credit: 6,596,453
RAC: 1,694
Message 4227 - Posted: 24 Oct 2016, 21:30:36 UTC

https://www.neowin.net/news/dirty-cow-flaw-lets-hackers-gain-control-of-linux-systems-every-single-time

YEP Linux is just the greatest and most secure OS ever 😎


.....I didn't do it.......and I never liked a Dirty Cow

(OK I won't restart the OS war)
ID: 4227 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile ivan
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Avatar

Send message
Joined: 20 Jan 15
Posts: 1093
Credit: 6,328,894
RAC: 4,998
Message 4228 - Posted: 24 Oct 2016, 23:01:24 UTC - in response to Message 4227.  

https://www.neowin.net/news/dirty-cow-flaw-lets-hackers-gain-control-of-linux-systems-every-single-time

YEP Linux is just the greatest and most secure OS ever 😎


.....I didn't do it.......and I never liked a Dirty Cow

(OK I won't restart the OS war)

Tja! OK, there's a slight mitigating factor for Linus; 11 years ago or whenever he spotted the flaw, he didn't see any way to exploit it; I'm not sure from what I've read if he then decided it wasn't worth patching or if he couldn't decide how to patch it. But as I understand it, this didn't become exploitable until copy-on-write (COW) was invented and deployed. There are reports that the original flaw was documented, but it seems that no-one had enough grasp of the overall picture to connect the dots between an old possible problem and an emerging technique.

We need more nexialists (from A.E. van Vogt's, "The Story of the Space Beagle", the novel that made me want to be a scientist, but not be blinkered to my field; e.g. I have a paper on rejuvenating channel-electron-multipliers which is based on a technique for cleaning two-stroke motorcycle exhausts!).
ID: 4228 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
computezrmle
Avatar

Send message
Joined: 28 Jul 16
Posts: 254
Credit: 231,222
RAC: 4,007
Message 4231 - Posted: 25 Oct 2016, 10:29:47 UTC

My hosts got a couple of WUs from the non-dev project although it was clear they would run into an error.
Why can´t you stop sending out WUs until the patches are installed?

The VMs are also linux machines.
If they use a COW filesystem they are also affected by that bug.
I´m sure I am the very first thinking about that :-))
ID: 4231 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Ben Segal
Volunteer moderator
Volunteer developer
Volunteer tester

Send message
Joined: 12 Sep 14
Posts: 65
Credit: 544
RAC: 0
Message 4232 - Posted: 25 Oct 2016, 12:32:33 UTC - in response to Message 4231.  

My hosts got a couple of WUs from the non-dev project although it was clear they would run into an error.
Why can´t you stop sending out WUs until the patches are installed?

The VMs are also linux machines.
If they use a COW filesystem they are also affected by that bug.
I´m sure I am the very first thinking about that :-))

The patch for this bug was issued yesterday and will be applied automagically when your current task expires and your CernVM reboots.
ID: 4232 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
computezrmle
Avatar

Send message
Joined: 28 Jul 16
Posts: 254
Credit: 231,222
RAC: 4,007
Message 4233 - Posted: 25 Oct 2016, 13:27:35 UTC - in response to Message 4232.  

My hosts got a couple of WUs from the non-dev project although it was clear they would run into an error.
Why can´t you stop sending out WUs until the patches are installed?

The VMs are also linux machines.
If they use a COW filesystem they are also affected by that bug.
I´m sure I am the very first thinking about that :-))

The patch for this bug was issued yesterday and will be applied automagically when your current task expires and your CernVM reboots.

As I stated a couple of times in this message board my hosts still do not get the most recent application versions (CMS, CMS-dev).
The older apps download/boot older VM images, e.g. CMS_2016_08_08.vdi in case of CMS-dev, which are not patched.

Resetting the projects or rebooting the hosts do not solve the problem.
ID: 4233 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
computezrmle
Avatar

Send message
Joined: 28 Jul 16
Posts: 254
Credit: 231,222
RAC: 4,007
Message 4234 - Posted: 25 Oct 2016, 18:59:11 UTC

I made some additional tests:

I detached one of my hosts, reattached it and changed the project setting on the dev-webpage so this host asks for Theory Simulation - which I had not used since August.

Result:
My host got Theory v2.04 and not the most recent v2.90.
v2.04 is the last version that I got in August.

Is it a database error due to the consolidation of classical LHC, vLHC, LHC-dev?
Any other ideas?
ID: 4234 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Rasputin42
Volunteer tester

Send message
Joined: 16 Aug 15
Posts: 965
Credit: 1,201,381
RAC: 0
Message 4235 - Posted: 25 Oct 2016, 20:37:04 UTC - in response to Message 4234.  

Did you attach to the latest URL?

http://lhcathomedev.cern.ch/vLHCathome-dev/

or did you use the old one, whatever that was?

(Just a thought...)

Maybe just try a project reset?
ID: 4235 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Ben Segal
Volunteer moderator
Volunteer developer
Volunteer tester

Send message
Joined: 12 Sep 14
Posts: 65
Credit: 544
RAC: 0
Message 4238 - Posted: 26 Oct 2016, 6:33:16 UTC - in response to Message 4233.  

My hosts got a couple of WUs from the non-dev project although it was clear they would run into an error.
Why can´t you stop sending out WUs until the patches are installed?

The VMs are also linux machines.
If they use a COW filesystem they are also affected by that bug.
I´m sure I am the very first thinking about that :-))

The patch for this bug was issued yesterday and will be applied automagically when your current task expires and your CernVM reboots.

As I stated a couple of times in this message board my hosts still do not get the most recent application versions (CMS, CMS-dev).
The older apps download/boot older VM images, e.g. CMS_2016_08_08.vdi in case of CMS-dev, which are not patched.

Resetting the projects or rebooting the hosts do not solve the problem.

Well actually you do get the security patches whatever .vdi version gets loaded. CernVM is connected to its file system CVMFS and it is this which does automagical kernel and library updates right after booting the vdi image.
ID: 4238 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
computezrmle
Avatar

Send message
Joined: 28 Jul 16
Posts: 254
Credit: 231,222
RAC: 4,007
Message 4239 - Posted: 26 Oct 2016, 7:21:29 UTC - in response to Message 4235.  

Did you attach to the latest URL?

http://lhcathomedev.cern.ch/vLHCathome-dev/

or did you use the old one, whatever that was?

(Just a thought...)

Maybe just try a project reset?


I attached to https://lhcathome.cern.ch/vLHCathome-dev/

A detach/reattach includes a project reset. At least in my understanding as it deletes more files/dirs on the local computer than a project reset. Nevertheless I had tried a project reset before.


During the most recent try to attach to the dev project the server sent a get_project_config.xml with 2 special sections.

An error section:
<project_config>
<name>vLHCathome-dev</name>
<master_url>https://lhcathome.cern.ch/vLHCathome-dev/</master_url>
<web_rpc_url_base>https://lhcathome.cern.ch/vLHCathome-dev/</web_rpc_url_base>
<error>
<error_msg>file_get_contents(../../local.revision): failed to open stream: No such file or directory</error_msg>
<type>Warning</type>
<file>/share/data/project-ssl/vLHCathome-dev/html/user/get_project_config.php</file>
<line>66</line>
</error>


A platform section without a plan_class:
<platform>
<platform_name>x86_64-apple-darwin</platform_name>
<user_friendly_name>Intel 64-bit Mac OS 10.5 or later</user_friendly_name>
<plan_class>vbox64_mt_mcore_cms</plan_class>

</platform>
<platform>
<platform_name>x86_64-pc-linux-gnu</platform_name>
<user_friendly_name>Linux running on an AMD x86_64 or Intel EM64T CPU</user_friendly_name>
</platform>
<platform>

<platform_name>x86_64-pc-linux-gnu</platform_name>
<user_friendly_name>Linux running on an AMD x86_64 or Intel EM64T CPU</user_friendly_name>
<plan_class>vbox64</plan_class>

</platform>
ID: 4239 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile ivan
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Avatar

Send message
Joined: 20 Jan 15
Posts: 1093
Credit: 6,328,894
RAC: 4,998
Message 4241 - Posted: 27 Oct 2016, 9:22:40 UTC

Encouraging sign: I now get
ssh: connect to host lcggwms02.gridpp.rl.ac.uk port 9700: Connection refused
instead of the
ssh: connect to host lcggwms02.gridpp.rl.ac.uk port 9700: Connection timed out
I was getting yesterday, so it looks like the VM is up again, if not yet fully operational.
ID: 4241 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile ivan
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Avatar

Send message
Joined: 20 Jan 15
Posts: 1093
Credit: 6,328,894
RAC: 4,998
Message 4242 - Posted: 27 Oct 2016, 13:23:31 UTC - in response to Message 4241.  

Encouraging sign: I now get
ssh: connect to host lcggwms02.gridpp.rl.ac.uk port 9700: Connection refused
instead of the
ssh: connect to host lcggwms02.gridpp.rl.ac.uk port 9700: Connection timed out
I was getting yesterday, so it looks like the VM is up again, if not yet fully operational.


OK, we are operational again.
ID: 4242 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote

Message boards : News : CMS Servers up again


©2019 CERN