Message boards : Theory Application : Prepare a bigger Theory VDI?
Message board moderation

To post messages, you must log in.

AuthorMessage
computezrmle
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 28 Jul 16
Posts: 473
Credit: 389,411
RAC: 62
Message 7576 - Posted: 12 Jul 2022, 8:26:51 UTC

A while ago (starting here: https://lhcathomedev.cern.ch/lhcathome-dev/forum_thread.php?id=563&postid=7329) there was a discussion about whether to prepare a bigger Theory vdi file that includes more data in it's CVMFS cache.
The idea was to avoid the need to download that data again and again, especially if no local proxy is available.

Meanwhile I collected some related numbers:
Within less than 2 weeks the Theory related CVMFS cache grows to roughly 5.7 GB.

This would result in a vdi file >6.4 GB.
Do we really want that (even with differencing images)?

Comments (pros/cons) are welcome.
ID: 7576 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
maeax

Send message
Joined: 22 Apr 16
Posts: 664
Credit: 1,791,620
RAC: 3,116
Message 7577 - Posted: 12 Jul 2022, 8:59:46 UTC - in response to Message 7576.  
Last modified: 12 Jul 2022, 9:02:03 UTC

As we two changed PM years ago, there was also this question.
Is it possible to save CVMFS DATA in a local storage (Pirol?).
Don't remember exactly.
https://cvmfs.readthedocs.io/en/stable/cpt-containers.html
ID: 7577 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 28 Jul 16
Posts: 473
Credit: 389,411
RAC: 62
Message 7578 - Posted: 12 Jul 2022, 9:13:31 UTC - in response to Message 7577.  

That's what ATLAS/Theory native do since they run in a container.
ATLAS/Theory vbox can't. At least not yet.
ID: 7578 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Crystal Pellet
Volunteer tester

Send message
Joined: 13 Feb 15
Posts: 1180
Credit: 815,336
RAC: 431
Message 7579 - Posted: 12 Jul 2022, 9:29:19 UTC - in response to Message 7576.  
Last modified: 12 Jul 2022, 9:54:08 UTC

6.4GB --> as gz-file less than 3GB.
The CVMFS data for Theory will not change very often, so I quess you have to download that zipped-VM once a year at the max.
As I wrote in your mentioned thread one might consider to reduce the size by only load needed CVMFS-data for e.g. Pythia6 and Pythia8,
but for me your measured data for about 2 weeks would reduce the overall data to download for Theory extremely. Specially for the Theory 'only crunchers'.

For every Theory task now necessary data for the differencing VM is downloaded from 1 GB up to 1.5 GB every time for every new task.
Most of the Theory's runtime is less than 2 hours. Go figure when you run several tasks simultaniously.
An extra advantage is that a Theory task will start the event processing earlier.

From the above: PRO
ID: 7579 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
maeax

Send message
Joined: 22 Apr 16
Posts: 664
Credit: 1,791,620
RAC: 3,116
Message 7580 - Posted: 12 Jul 2022, 11:17:56 UTC
Last modified: 12 Jul 2022, 11:36:23 UTC

Parrot is the name HPC, SuperComputer....
https://cvmfs.readthedocs.io/en/stable/cpt-hpc.html
But, for us small User, not useful.
Edit: btw 6.5 TByte Data last month.
The experts of my ISP had made a deeper look in it, really!
ID: 7580 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote

Message boards : Theory Application : Prepare a bigger Theory VDI?


©2024 CERN