Name | yEFMDmO9tYvnShfckohDCDFpABFKDmABFKDmuAgSDmABFKDm5UizSm_1 |
Workunit | 1942527 |
Created | 4 Oct 2019, 4:13:25 UTC |
Sent | 7 Oct 2019, 10:10:43 UTC |
Report deadline | 14 Oct 2019, 10:10:43 UTC |
Received | 7 Oct 2019, 16:19:29 UTC |
Server state | Over |
Outcome | Success |
Client state | Done |
Exit status | 0 (0x00000000) |
Computer ID | 3682 |
Run time | 6 hours 8 min 4 sec |
CPU time | 1 days 0 hours 6 min 16 sec |
Validate state | Valid |
Credit | 911.29 |
Device peak FLOPS | 17.83 GFLOPS |
Application version | ATLAS Simulation v0.74 (native_mt) x86_64-pc-linux-gnu |
Peak working set size | 1.84 GB |
Peak swap size | 2.59 GB |
Peak disk usage | 653.70 MB |
<core_client_version>7.16.1</core_client_version> <![CDATA[ <stderr_txt> 12:10:54 (13015): wrapper (7.7.26015): starting 12:10:54 (13015): wrapper: running run_atlas (--nthreads 4) 2019-10-07 12:10:54,209: singularity image is /cvmfs/atlas.cern.ch/repo/containers/images/singularity/x86_64-centos7.img 2019-10-07 12:10:54,209: sys.argv = ['run_atlas', '--nthreads', '4'] 2019-10-07 12:10:54,209: THREADS=4 2019-10-07 12:10:54,210: Checking for CVMFS 2019-10-07 12:10:54,571: CVMFS is installed 2019-10-07 12:10:54,571: Checking Singularity... 2019-10-07 12:10:54,584: Singularity is installed, version singularity version 3.4.0-1.2.el7 2019-10-07 12:10:54,584: Testing the function of Singularity... 2019-10-07 12:10:54,585: Checking singularity with cmd:singularity exec -B /cvmfs /cvmfs/atlas.cern.ch/repo/containers/images/singularity/x86_64-centos7.img hostname 2019-10-07 12:10:54,764: Singularity Works... 2019-10-07 12:10:54,764: copy /home/dcameron/boinc/slots/1/shared/ATLAS.root_0 2019-10-07 12:10:55,247: copy /home/dcameron/boinc/slots/1/shared/input.tar.gz 2019-10-07 12:10:55,247: copy /home/dcameron/boinc/slots/1/shared/RTE.tar.gz 2019-10-07 12:10:55,248: copy /home/dcameron/boinc/slots/1/shared/start_atlas.sh 2019-10-07 12:10:55,248: export ATHENA_PROC_NUMBER=4; 2019-10-07 12:10:55,260: start atlas job with PandaID=4494828307 2019-10-07 12:10:55,260: cmd = singularity exec --pwd /home/dcameron/boinc/slots/1 -B /cvmfs,/home /cvmfs/atlas.cern.ch/repo/containers/images/singularity/x86_64-centos7.img sh start_atlas.sh > runtime_log 2> runtime_log.err 2019-10-07 18:19:15,539: running cmd return value is 0 2019-10-07 18:19:15,540: Moving ./HITS.19000575._014418.pool.root.1 to shared/HITS.pool.root.1 2019-10-07 18:19:15,540: HITS result file: 2019-10-07 18:19:15,544: -rw-------. 1 dcameron zp 247860015 Oct 7 18:17 shared/HITS.pool.root.1 2019-10-07 18:19:15,544: *****************The last 200 lines of the pilot log****************** 2019-10-07 18:19:15,548: "cpuTime": 46, "wallTime": 81 }, "validation": { "cpuTime": 0, "wallTime": 0 }, "wallTime": 79 } }, "machine": { "cpu_family": "6", "linux_distribution": [ "CentOS Linux", "7.6.1810", "Core" ], "model": "60", "model_name": "Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz", "node": "pcoslo5.cern.ch", "platform": "Linux-3.10.0-1062.1.1.el7.x86_64-x86_64-with-centos-7.6.1810-Core" }, "transform": { "cpuEfficiency": 0.9851, "cpuPWEfficiency": 0.9868, "cpuTime": 5, "cpuTimeTotal": 86733, "externalCpuTime": 145, "processedEvents": 200, "trfPredata": null, "wallTime": 21964 } } } 2019-10-07 16:18:36,830 | DEBUG | queue_monitor | pilot.util.auxiliary.4494828307 | update_server | xml:will send fileinfo 2019-10-07 16:18:36,830 | INFO | queue_monitor | pilot.control.job.4494828307 | send_state | pilot will not update the server (heartbeat message will be written to file) 2019-10-07 16:18:36,830 | INFO | queue_monitor | pilot.control.job.4494828307 | send_state | job 4494828307 has finished - writing final server update 2019-10-07 16:18:36,833 | WARNING | queue_monitor | pilot.api.analytics | get_fitted_data | wrong length of table data, x=[1570443091.0, 1570443152.0, 1570443213.0, 1570443274.0, 1570443335.0, 1570443396.0, 1570443457.0, 1570443518.0, 1570443579.0, 1570443640.0, 1570443701.0, 1570443762.0, 1570443823.0, 1570443884.0, 1570443945.0, 1570444006.0, 1570444067.0, 1570444128.0, 1570444189.0, 1570444250.0, 1570444311.0, 1570444372.0, 1570444433.0, 1570444494.0, 1570444555.0, 1570444616.0, 1570444677.0, 1570444738.0, 1570444799.0, 1570444860.0, 1570444921.0, 1570444982.0, 1570445043.0, 1570445104.0, 1570445165.0, 1570445226.0, 1570445287.0, 1570445348.0, 1570445409.0, 1570445470.0, 1570445531.0, 1570445592.0, 1570445653.0, 1570445714.0, 1570445775.0, 1570445836.0, 1570445897.0, 1570445958.0, 1570446019.0, 1570446080.0, 1570446141.0, 1570446202.0, 1570446263.0, 1570446324.0, 1570446385.0, 1570446446.0, 1570446507.0, 1570446568.0, 1570446629.0, 1570446690.0, 1570446751.0, 1570446812.0, 1570446873.0, 1570446934.0, 1570446995.0, 1570447056.0, 1570447117.0, 1570447178.0, 1570447239.0, 1570447300.0, 1570447361.0, 1570447422.0, 1570447483.0, 1570447544.0, 1570447605.0, 1570447666.0, 1570447727.0, 1570447788.0, 1570447849.0, 1570447910.0, 1570447971.0, 1570448032.0, 1570448093.0, 1570448154.0, 1570448215.0, 1570448276.0, 1570448337.0, 1570448398.0, 1570448459.0, 1570448520.0, 1570448581.0, 1570448642.0, 1570448703.0, 1570448764.0, 1570448825.0, 1570448886.0, 1570448947.0, 1570449008.0, 1570449069.0, 1570449130.0, 1570449191.0, 1570449252.0, 1570449313.0, 1570449374.0, 1570449435.0, 1570449496.0, 1570449557.0, 1570449618.0, 1570449679.0, 1570449740.0, 1570449801.0, 1570449862.0, 1570449923.0, 1570449984.0, 1570450045.0, 1570450106.0, 1570450167.0, 1570450228.0, 1570450289.0, 1570450350.0, 1570450411.0, 1570450472.0, 1570450533.0, 1570450594.0, 1570450655.0, 1570450716.0, 1570450777.0, 1570450838.0, 1570450899.0, 1570450960.0, 1570451021.0, 1570451082.0, 1570451143.0, 1570451204.0, 1570451265.0, 1570451326.0, 1570451387.0, 1570451448.0, 1570451509.0, 1570451570.0, 1570451631.0, 1570451692.0, 1570451753.0, 1570451814.0, 1570451875.0, 1570451936.0, 1570451997.0, 1570452058.0, 1570452119.0, 1570452180.0, 1570452241.0, 1570452302.0, 1570452363.0, 1570452424.0, 1570452485.0, 1570452546.0, 1570452607.0, 1570452668.0, 1570452729.0, 1570452790.0, 1570452851.0, 1570452912.0, 1570452973.0, 1570453034.0, 1570453095.0, 1570453156.0, 1570453217.0, 1570453278.0, 1570453339.0, 1570453400.0, 1570453461.0, 1570453522.0, 1570453583.0, 1570453644.0, 1570453705.0, 1570453766.0, 1570453827.0, 1570453888.0, 1570453949.0, 1570454010.0, 1570454071.0, 1570454132.0, 1570454193.0, 1570454254.0, 1570454315.0, 1570454376.0, 1570454437.0, 1570454498.0, 1570454559.0, 1570454620.0, 1570454681.0, 1570454742.0, 1570454803.0, 1570454864.0, 1570454925.0, 1570454986.0, 1570455047.0, 1570455108.0, 1570455169.0, 1570455230.0, 1570455291.0, 1570455352.0, 1570455413.0, 1570455474.0, 1570455535.0, 1570455596.0, 1570455657.0, 1570455718.0, 1570455779.0, 1570455840.0, 1570455901.0, 1570455962.0, 1570456023.0, 1570456084.0, 1570456145.0, 1570456206.0, 1570456267.0, 1570456328.0, 1570456389.0, 1570456450.0, 1570456511.0, 1570456572.0, 1570456633.0, 1570456694.0, 1570456755.0, 1570456816.0, 1570456877.0, 1570456938.0, 1570456999.0, 1570457060.0, 1570457121.0, 1570457182.0, 1570457243.0, 1570457304.0, 1570457365.0, 1570457426.0, 1570457487.0, 1570457548.0, 1570457609.0, 1570457670.0, 1570457731.0, 1570457792.0, 1570457853.0, 1570457914.0, 1570457975.0, 1570458036.0, 1570458097.0, 1570458158.0, 1570458219.0, 1570458280.0, 1570458341.0, 1570458402.0, 1570458463.0, 1570458524.0, 1570458585.0, 1570458646.0, 1570458707.0, 1570458768.0, 1570458829.0, 1570458890.0, 1570458951.0, 1570459012.0, 1570459073.0, 1570459134.0, 1570459195.0, 1570459256.0, 1570459317.0, 1570459378.0, 1570459439.0, 1570459500.0, 1570459561.0, 1570459622.0, 1570459683.0, 1570459744.0, 1570459805.0, 1570459866.0, 1570459927.0, 1570459988.0, 1570460049.0, 1570460110.0, 1570460171.0, 1570460232.0, 1570460293.0, 1570460354.0, 1570460415.0, 1570460476.0, 1570460537.0, 1570460598.0, 1570460659.0, 1570460720.0, 1570460781.0, 1570460842.0, 1570460903.0, 1570460964.0, 1570461025.0, 1570461086.0, 1570461147.0, 1570461208.0, 1570461269.0, 1570461330.0, 1570461391.0, 1570461452.0, 1570461513.0, 1570461574.0, 1570461635.0, 1570461696.0, 1570461757.0, 1570461818.0, 1570461879.0, 1570461940.0, 1570462001.0, 1570462062.0, 1570462123.0, 1570462184.0, 1570462245.0, 1570462306.0, 1570462367.0, 1570462428.0, 1570462489.0, 1570462550.0, 1570462611.0, 1570462672.0, 1570462733.0, 1570462794.0, 1570462855.0, 1570462916.0, 1570462977.0, 1570463038.0, 1570463099.0, 1570463160.0, 1570463221.0, 1570463282.0, 1570463343.0, 1570463404.0, 1570463465.0, 1570463526.0, 1570463587.0, 1570463648.0, 1570463709.0, 1570463770.0, 1570463831.0, 1570463892.0, 1570463953.0, 1570464014.0, 1570464075.0, 1570464136.0, 1570464197.0, 1570464258.0, 1570464319.0, 1570464380.0, 1570464441.0, 1570464502.0, 1570464563.0, 1570464624.0, 1570464685.0, 1570464746.0, 1570464807.0, 1570464868.0, 1570464929.0, 1570464990.0, 1570465051.0], y=[] (must be same and length>=2) 2019-10-07 16:18:36,833 | DEBUG | queue_monitor | pilot.util.auxiliary.4494828307 | get_job_metrics | job metrics="coreCount=4 actualCoreCount=1 nEvents=200 workDirSize=39782585" 2019-10-07 16:18:36,833 | INFO | queue_monitor | pilot.control.job.4494828307 | get_data_structure | total number of processed events: 200 (read) 2019-10-07 16:18:36,834 | INFO | queue_monitor | pilot.user.atlas.utilities | get_memory_values | using path: /home/dcameron/boinc/slots/1/PanDA_Pilot-4494828307/memory_monitor_summary.json (trf name=prmon) 2019-10-07 16:18:36,834 | DEBUG | queue_monitor | pilot.user.atlas.utilities | get_memory_monitor_info | summary_dictionary={'Max': {'rx_packets': 1436804, 'nprocs': 9, 'nthreads': 1, 'rx_bytes': 3104970624, 'wtime': 21966, 'rss': 9500712, 'write_bytes': 513634304, 'vmem': 14153716, 'read_bytes': 4306150400, 'stime': 135, 'tx_bytes': 2321163014, 'pss': 2886183, 'wchar': 512334844, 'rchar': 4071987292, 'tx_packets': 831156, 'swap': 40, 'utime': 87434}, 'Avg': {'write_bytes': 23382, 'nprocs': 8, 'nthreads': 0, 'rx_bytes': 141351, 'rx_packets': 65, 'vmem': 13856445, 'read_bytes': 196034, 'swap': 12, 'tx_bytes': 105669, 'pss': 2773989, 'wchar': 23323, 'rchar': 185374, 'tx_packets': 37, 'rss': 9266973}} 2019-10-07 16:18:36,834 | INFO | queue_monitor | pilot.user.atlas.utilities | get_memory_monitor_info | extracted standard info from prmon json 2019-10-07 16:18:36,834 | INFO | queue_monitor | pilot.user.atlas.utilities | get_memory_monitor_info | extracted standard memory fields from prmon json 2019-10-07 16:18:36,834 | INFO | queue_monitor | pilot.util.auxiliary.4494828307 | timing_report | .............................. 2019-10-07 16:18:36,834 | INFO | queue_monitor | pilot.util.auxiliary.4494828307 | timing_report | . Timing measurements: 2019-10-07 16:18:36,834 | INFO | queue_monitor | pilot.util.auxiliary.4494828307 | timing_report | . get job = 0 s 2019-10-07 16:18:36,835 | INFO | queue_monitor | pilot.util.auxiliary.4494828307 | timing_report | . initial setup = 0 s 2019-10-07 16:18:36,835 | INFO | queue_monitor | pilot.util.auxiliary.4494828307 | timing_report | . payload setup = 0 s 2019-10-07 16:18:36,835 | INFO | queue_monitor | pilot.util.auxiliary.4494828307 | timing_report | . total setup = 0 s 2019-10-07 16:18:36,835 | INFO | queue_monitor | pilot.util.auxiliary.4494828307 | timing_report | . stage-in = 0 s 2019-10-07 16:18:36,835 | INFO | queue_monitor | pilot.util.auxiliary.4494828307 | timing_report | . payload execution = 22029 s 2019-10-07 16:18:36,835 | INFO | queue_monitor | pilot.util.auxiliary.4494828307 | timing_report | . stage-out = 1 s 2019-10-07 16:18:36,835 | INFO | queue_monitor | pilot.util.auxiliary.4494828307 | timing_report | .............................. 2019-10-07 16:18:36,835 | INFO | queue_monitor | pilot.util.auxiliary.4494828307 | get_log_extracts | building log extracts (sent to the server as 'pilotLog') 2019-10-07 16:18:36,835 | DEBUG | queue_monitor | pilot.util.auxiliary.4494828307 | get_panda_tracer_log | PanDA tracer log does not exist: /home/dcameron/boinc/slots/1/PanDA_Pilot-4494828307/pandatracerlog.txt (ignoring) 2019-10-07 16:18:36,835 | INFO | queue_monitor | pilot.util.container | execute | executing command: tail -n 20 /home/dcameron/boinc/slots/1/PanDA_Pilot-4494828307/pilotlog.txt 2019-10-07 16:18:36,843 | WARNING | queue_monitor | pilot.util.auxiliary.4494828307 | get_log_extracts | detected the following tail of warning/fatal messages in the pilot log: - Log from pilotlog.txt -2019-10-07 16:18:36,833 | WARNING | queue_monitor | pilot.api.analytics | get_fitted_data | wrong length of table data, x=[1570443091.0, 1570443152.0, 1570443213.0, 1570443274.0, 1570443335.0, 1570443396.0, 1570443457.0, 1570443518.0, 1570443579.0, 1570443640.0, 1570443701.0, 1570443762.0, 1570443823.0, 1570443884.0, 1570443945.0, 1570444006.0, 1570444067.0, 1570444128.0, 1570444189.0, 1570444250.0, 1570444311.0, 1570444372.0, 1570444433.0, 1570444494.0, 1570444555.0, 1570444616.0, 1570444677.0, 1570444738.0, 1570444799.0, 1570444860.0, 1570444921.0, 1570444982.0, 1570445043.0, 1570445104.0, 1570445165.0, 1570445226.0, 1570445287.0, 1570445348.0, 1570445409.0, 1570445470.0, 1570445531.0, 1570445592.0, 1570445653.0, 1570445714.0, 1570445775.0, 1570445836.0, 1570445897.0, 1570445958.0, 1570446019.0, 1570446080.0, 1570446141.0, 1570446202.0, 1570446263.0, 1570446324.0, 1570446385.0, 1570446446.0, 1570446507.0, 1570446568.0, 1570446629.0, 1570446690.0, 1570446751.0, 1570446812.0, 1570446873.0, 1570446934.0, 1570446995.0, 1570447056.0, 1570447117.0, 1570447178.0, 1570447239.0, 1570447300.0, 1570447361.0, 1570447422.0, 1570447483.0, 1570447544.0, 1570447605.0, 1570447666.0, 1570447727.0, 1570447788.0, 1570447849.0, 1570447910.0, 1570447971.0, 1570448032.0, 1570448093.0, 1570448154.0, 1570448215.0, 1570448276.0, 1570448337.0, 1570448398.0, 1570448459.0, 1570448520.0, 1570448581.0, 1570448642.0, 1570448703.0, 1570448764.0, 1570448825.0, 1570448886.0, 1570448947.0, 1570449008.0, 1570449069.0, 1570449130.0, 1570449191.0, 1570449252.0, 1570449313.0, 1570449374.0, 1570449435.0, 1570449496.0, 1570449557.0, 1570449618.0, 1570449679.0, 1570449740.0, 1570449801.0, 1570449862.0, 1570449923.0, 1570449984.0, 1570450045.0, 1570450106.0, 1570450167.0, 1570450228.0, 1570450289.0, 1570450350.0, 1570450411.0, 1570450472.0, 1570450533.0, 1570450594.0, 1570450655.0, 1570450716.0, 1570450777.0, 1570450838.0, 1570450899.0, 1570450960.0, 1570451021.0, 1570451082.0, 1570451143.0, 1570451204.0, 1570451265.0, 1570451326.0, 1570451387.0, 1570451448.0, 1570451509.0, 1570451570.0, 1570451631.0, 1570451692.0, 1570451753.0, 1570451814.0, 1570451875.0, 1570451936.0, 1570451997.0, 1570452058.0, 1570452119.0, 1570452180.0, 1570452241.0, 1570452302.0, 1570452363.0, 1570452424.0, 1570452485.0, 1570452546.0, 1570452607.0, 1570452668.0, 1570452729.0, 1570452790.0, 1570452851.0, 1570452912.0, 1570452973.0, 1570453034.0, 1570453095.0, 1570453156.0, 1570453217.0, 1570453278.0, 1570453339.0, 1570453400.0, 1570453461.0, 1570453522.0, 1570453583.0, 1570453644.0, 1570453705.0, 1570453766.0, 1570453827.0, 1570453888.0, 1570453949.0, 1570454010.0, 1570454071.0, 1570454132.0, 1570454193.0, 1570454254.0, 1570454315.0, 1570454376.0, 1570454437.0, 1570454498.0, 1570454559.0, 1570454620.0, 1570454681.0, 1570454742.0, 1570454803.0, 1570454864.0, 1570454925.0, 1570454986.0, 1570455047.0, 1570455108.0, 1570455169.0, 1570455230.0, 1570455291.0, 1570455352.0, 1570455413.0, 1570455474.0, 1570455535.0, 1570455596.0, 1570455657.0, 1570455718.0, 1570455779.0, 1570455840.0, 1570455901.0, 1570455962.0, 1570456023.0, 1570456084.0, 1570456145.0, 1570456206.0, 1570456267.0, 1570456328.0, 1570456389.0, 1570456450.0, 1570456511.0, 1570456572.0, 1570456633.0, 1570456694.0, 1570456755.0, 1570456816.0, 1570456877.0, 1570456938.0, 1570456999.0, 1570457060.0, 1570457121.0, 1570457182.0, 1570457243.0, 1570457304.0, 1570457365.0, 1570457426.0, 1570457487.0, 1570457548.0, 1570457609.0, 1570457670.0, 1570457731.0, 1570457792.0, 1570457853.0, 1570457914.0, 1570457975.0, 1570458036.0, 1570458097.0, 1570458158.0, 1570458219.0, 1570458280.0, 1570458341.0, 1570458402.0, 1570458463.0, 1570458524.0, 1570458585.0, 1570458646.0, 1570458707.0, 1570458768.0, 1570458829.0, 1570458890.0, 1570458951.0, 1570459012.0, 1570459073.0, 1570459134.0, 1570459195.0, 1570459256.0, 1570459317.0, 1570459378.0, 1570459439.0, 1570459500.0, 1570459561.0, 1570459622.0, 1570459683.0, 1570459744.0, 1570459805.0, 1570459866.0, 1570459927.0, 1570459988.0, 1570460049.0, 1570460110.0, 1570460171.0, 1570460232.0, 1570460293.0, 1570460354.0, 1570460415.0, 1570460476.0, 1570460537.0, 1570460598.0, 1570460659.0, 1570460720.0, 1570460781.0, 1570460842.0, 1570460903.0, 1570460964.0, 1570461025.0, 1570461086.0, 1570461147.0, 1570461208.0, 1570461269.0, 1570461330.0, 1570461391.0, 1570461452.0, 1570461513.0, 1570461574.0, 1570461635.0, 1570461696.0, 1570461757.0, 1570461818.0, 1570461879.0, 1570461940.0, 1570462001.0, 1570462062.0, 1570462123.0, 1570462184.0, 1570462245.0, 1570462306.0, 1570462367.0, 1570462428.0, 1570462489.0, 1570462550.0, 1570462611.0, 1570462672.0, 1570462733.0, 1570462794.0, 1570462855.0, 1570462916.0, 1570462977.0, 1570463038.0, 1570463099.0, 1570463160.0, 1570463221.0, 1570463282.0, 1570463343.0, 1570463404.0, 1570463465.0, 1570463526.0, 1570463587.0, 1570463648.0, 1570463709.0, 1570463770.0, 1570463831.0, 1570463892.0, 1570463953.0, 1570464014.0, 1570464075.0, 1570464136.0, 1570464197.0, 1570464258.0, 1570464319.0, 1570464380.0, 1570464441.0, 1570464502.0, 1570464563.0, 1570464624.0, 1570464685.0, 1570464746.0, 1570464807.0, 1570464868.0, 1570464929.0, 1570464990.0, 1570465051.0], y=[] (must be same and length>=2) 2019-10-07 16:18:36,833 | DEBUG | queue_monitor | pilot.util.auxiliary.4494828307 | get_job_metrics | job metrics="coreCount=4 actualCoreCount=1 nEvents=200 workDirSize=39782585" 2019-10-07 16:18:36,833 | INFO | queue_monitor | pilot.control.job.4494828307 | get_data_structure | total number of processed events: 200 (read) 2019-10-07 16:18:36,834 | INFO | queue_monitor | pilot.user.atlas.utilities | get_memory_values | using path: /home/dcameron/boinc/slots/1/PanDA_Pilot-4494828307/memory_monitor_summary.json (trf name=prmon) 2019-10-07 16:18:36,834 | DEBUG | queue_monitor | pilot.user.atlas.utilities | get_memory_monitor_info | summary_dictionary={'Max': {'rx_packets': 1436804, 'nprocs': 9, 'nthreads': 1, 'rx_bytes': 3104970624, 'wtime': 21966, 'rss': 9500712, 'write_bytes': 513634304, 'vmem': 14153716, 'read_bytes': 4306150400, 'stime': 135, 'tx_bytes': 2321163014, 'pss': 2886183, 'wchar': 512334844, 'rchar': 4071987292, 'tx_packets': 831156, 'swap': 40, 'utime': 87434}, 'Avg': {'write_bytes': 23382, 'nprocs': 8, 'nthreads': 0, 'rx_bytes': 141351, 'rx_packets': 65, 'vmem': 13856445, 'read_bytes': 196034, 'swap': 12, 'tx_bytes': 105669, 'pss': 2773989, 'wchar': 23323, 'rchar': 185374, 'tx_packets': 37, 'rss': 9266973}} 2019-10-07 16:18:36,834 | INFO | queue_monitor | pilot.user.atlas.utilities | get_memory_monitor_info | extracted standard info from prmon json 2019-10-07 16:18:36,834 | INFO | queue_monitor | pilot.user.atlas.utilities | get_memory_monitor_info | extracted standard memory fields from prmon json 2019-10-07 16:18:36,834 | INFO | queue_monitor | pilot.util.auxiliary.4494828307 | timing_report | .............................. 2019-10-07 16:18:36,834 | INFO | queue_monitor | pilot.util.auxiliary.4494828307 | timing_report | . Timing measurements: 2019-10-07 16:18:36,834 | INFO | queue_monitor | pilot.util.auxiliary.4494828307 | timing_report | . get job = 0 s 2019-10-07 16:18:36,835 | INFO | queue_monitor | pilot.util.auxiliary.4494828307 | timing_report | . initial setup = 0 s 2019-10-07 16:18:36,835 | INFO | queue_monitor | pilot.util.auxiliary.4494828307 | timing_report | . payload setup = 0 s 2019-10-07 16:18:36,835 | INFO | queue_monitor | pilot.util.auxiliary.4494828307 | timing_report | . total setup = 0 s 2019-10-07 16:18:36,835 | INFO | queue_monitor | pilot.util.auxiliary.4494828307 | timing_report | . stage-in = 0 s 2019-10-07 16:18:36,835 | INFO | queue_monitor | pilot.util.auxiliary.4494828307 | timing_report | . payload execution = 22029 s 2019-10-07 16:18:36,835 | INFO | queue_monitor | pilot.util.auxiliary.4494828307 | timing_report | . stage-out = 1 s 2019-10-07 16:18:36,835 | INFO | queue_monitor | pilot.util.auxiliary.4494828307 | timing_report | .............................. 2019-10-07 16:18:36,835 | INFO | queue_monitor | pilot.util.auxiliary.4494828307 | get_log_extracts | building log extracts (sent to the server as 'pilotLog') 2019-10-07 16:18:36,835 | DEBUG | queue_monitor | pilot.util.auxiliary.4494828307 | get_panda_tracer_log | PanDA tracer log does not exist: /home/dcameron/boinc/slots/1/PanDA_Pilot-4494828307/pandatracerlog.txt (ignoring) 2019-10-07 16:18:36,835 | INFO | queue_monitor | pilot.util.container | execute | executing command: tail -n 20 /home/dcameron/boinc/slots/1/PanDA_Pilot-4494828307/pilotlog.txt 2019-10-07 16:18:36,844 | DEBUG | queue_monitor | pilot.control.job.4494828307 | send_state | wrote heartbeat to file /home/dcameron/boinc/slots/1/heartbeat.json 2019-10-07 16:18:36,844 | INFO | queue_monitor | pilot.control.job | queue_monitor | job 4494828307 was dequeued from the monitored payloads queue 2019-10-07 16:18:37,237 | INFO | retrieve | pilot.util.auxiliary.4494828307 | make_job_report | 2019-10-07 16:18:37,237 | INFO | retrieve | pilot.util.auxiliary.4494828307 | make_job_report | job summary report 2019-10-07 16:18:37,238 | INFO | retrieve | pilot.util.auxiliary.4494828307 | make_job_report | -------------------------------------------------- 2019-10-07 16:18:37,238 | INFO | retrieve | pilot.util.auxiliary.4494828307 | make_job_report | PanDA job id: 4494828307 2019-10-07 16:18:37,238 | INFO | retrieve | pilot.util.auxiliary.4494828307 | make_job_report | task id: 19000575 2019-10-07 16:18:37,238 | INFO | retrieve | pilot.util.auxiliary.4494828307 | make_job_report | errors: (none) 2019-10-07 16:18:37,238 | INFO | retrieve | pilot.util.auxiliary.4494828307 | make_job_report | status: LOG_TRANSFER = DONE 2019-10-07 16:18:37,238 | INFO | retrieve | pilot.util.auxiliary.4494828307 | make_job_report | pilot state: finished 2019-10-07 16:18:37,238 | INFO | retrieve | pilot.util.auxiliary.4494828307 | make_job_report | transexitcode: 0 2019-10-07 16:18:37,238 | INFO | retrieve | pilot.util.auxiliary.4494828307 | make_job_report | exeerrorcode: 0 2019-10-07 16:18:37,238 | INFO | retrieve | pilot.util.auxiliary.4494828307 | make_job_report | exeerrordiag: 2019-10-07 16:18:37,238 | INFO | retrieve | pilot.util.auxiliary.4494828307 | make_job_report | exitcode: 0 2019-10-07 16:18:37,238 | INFO | retrieve | pilot.util.auxiliary.4494828307 | make_job_report | exitmsg: OK 2019-10-07 16:18:37,238 | INFO | retrieve | pilot.util.auxiliary.4494828307 | make_job_report | cpuconsumptiontime: 87084 s 2019-10-07 16:18:37,238 | INFO | retrieve | pilot.util.auxiliary.4494828307 | make_job_report | nevents: 200 2019-10-07 16:18:37,238 | INFO | retrieve | pilot.util.auxiliary.4494828307 | make_job_report | neventsw: 0 2019-10-07 16:18:37,238 | INFO | retrieve | pilot.util.auxiliary.4494828307 | make_job_report | pid: 19822 2019-10-07 16:18:37,238 | INFO | retrieve | pilot.util.auxiliary.4494828307 | make_job_report | pgrp: 19822 2019-10-07 16:18:37,238 | INFO | retrieve | pilot.util.auxiliary.4494828307 | make_job_report | corecount: 4 2019-10-07 16:18:37,238 | INFO | retrieve | pilot.util.auxiliary.4494828307 | make_job_report | event service: False 2019-10-07 16:18:37,238 | INFO | retrieve | pilot.util.auxiliary.4494828307 | make_job_report | -------------------------------------------------- 2019-10-07 16:18:37,238 | INFO | retrieve | pilot.util.auxiliary.4494828307 | make_job_report | 2019-10-07 16:18:37,238 | INFO | retrieve | pilot.control.job.4494828307 | has_job_completed | job 4494828307 has completed 2019-10-07 16:18:37,238 | INFO | retrieve | pilot.util.processes | cleanup | overall cleanup function is called 2019-10-07 16:18:37,242 | DEBUG | retrieve | pilot.util.processes | cleanup | work directory was removed: /home/dcameron/boinc/slots/1/PanDA_Pilot-4494828307 2019-10-07 16:18:38,247 | INFO | retrieve | pilot.info.jobdata | collect_zombies | --- collectZombieJob: --- 10, [19822] 2019-10-07 16:18:38,248 | INFO | retrieve | pilot.info.jobdata | collect_zombies | zombie collector trying to kill pid 19822 2019-10-07 16:18:38,248 | INFO | retrieve | pilot.info.jobdata | collect_zombies | harmless exception when collecting zombies: [Errno 10] No child processes 2019-10-07 16:18:39,253 | INFO | retrieve | pilot.util.processes | cleanup | collected zombie processes 2019-10-07 16:18:39,253 | INFO | retrieve | pilot.util.processes | cleanup | will now attempt to kill all subprocesses of pid=19822 2019-10-07 16:18:39,307 | INFO | retrieve | pilot.util.processes | kill_processes | process IDs to be killed: [19822] (in reverse order) 2019-10-07 16:18:39,335 | WARNING | retrieve | pilot.util.processes | kill_processes | found no corresponding commands to process id(s) 2019-10-07 16:18:39,335 | INFO | retrieve | pilot.util.processes | kill_orphans | Do not look for orphan processes in BOINC jobs 2019-10-07 16:18:39,335 | INFO | retrieve | pilot.control.job | retrieve | ready for new job 2019-10-07 16:18:39,335 | INFO | retrieve | root | retrieve | pilot has finished for previous job - re-establishing logging No handlers could be found for logger "pilot.util.mpi" 2019-10-07 16:18:39,338 | INFO | retrieve | pilot.util.auxiliary | pilot_version_banner | ***************************************** 2019-10-07 16:18:39,338 | INFO | retrieve | pilot.util.auxiliary | pilot_version_banner | *** PanDA Pilot version 2.1.25 (11) *** 2019-10-07 16:18:39,338 | INFO | retrieve | pilot.util.auxiliary | pilot_version_banner | ***************************************** 2019-10-07 16:18:39,338 | INFO | retrieve | pilot.util.auxiliary | pilot_version_banner | 2019-10-07 16:18:39,338 | INFO | retrieve | pilot.util.auxiliary | display_architecture_info | architecture information: 2019-10-07 16:18:39,376 | INFO | retrieve | pilot.util.auxiliary | display_architecture_info | LSB Version: :core-4.1-amd64:core-4.1-noarch Distributor ID: CentOS Description: CentOS Linux release 7.6.1810 (Core) Release: 7.6.1810 Codename: Core 2019-10-07 16:18:39,376 | INFO | retrieve | pilot.util.auxiliary | pilot_version_banner | ***************************************** 2019-10-07 16:18:39,879 | DEBUG | retrieve | pilot.util.monitoring | check_local_space | checking local space on /home/dcameron/boinc/slots/1 2019-10-07 16:18:39,887 | INFO | retrieve | pilot.util.monitoring | check_local_space | sufficient remaining disk space (70588039168 B) 2019-10-07 16:18:39,887 | WARNING | retrieve | pilot.control.job | proceed_with_getjob | since timefloor is set to 0, pilot was only allowed to run one job 2019-10-07 16:18:39,887 | DEBUG | retrieve | pilot.control.job | retrieve | [job] retrieve thread has finished 2019-10-07 16:18:39,910 | INFO | failed_post | pilot.control.payload | failed_post | [payload] failed_post thread has finished 2019-10-07 16:18:39,920 | DEBUG | validate | pilot.control.job | validate | [job] validate thread has finished 2019-10-07 16:18:39,931 | INFO | monitor | pilot.control.monitor | control | [monitor] control thread has ended 2019-10-07 16:18:40,001 | DEBUG | MainThread | pilot.workflow.generic | run | thread count now at 13 threads 2019-10-07 16:18:40,001 | DEBUG | MainThread | pilot.workflow.generic | run | enumerate: [<_MainThread(MainThread, started 139828727146304)>, <ExcThread(job, started 139828565513984)>, <ExcThread(queue_monitoring, started 139828029028096)>, <ExcThread(payload, started 139828523550464)>, <ExcThread(execute_payloads, started 139827483764480)>, <ExcThread(job_monitor, started 139828003849984)>, <ExcThread(create_data_payload, started 139828540335872)>, <ExcThread(copytool_in, started 139827995457280)>, <ExcThread(validate_post, started 139827987064576)>, <ExcThread(data, started 139828557121280)>, <ExcThread(validate_pre, started 139828020635392)>, <ExcThread(queue_monitor, started 139827978671872)>, <ExcThread(copytool_out, started 139828531943168)>] 2019-10-07 16:18:40,156 | INFO | validate_pre | pilot.control.payload | validate_pre | [payload] validate_pre thread has finished 2019-10-07 16:18:40,403 | DEBUG | MainThread | pilot.workflow.generic | run | thread count now at 12 threads 2019-10-07 16:18:40,403 | DEBUG | MainThread | pilot.workflow.generic | run | enumerate: [<_MainThread(MainThread, started 139828727146304)>, <ExcThread(job, started 139828565513984)>, <ExcThread(queue_monitoring, started 139828029028096)>, <ExcThread(payload, started 139828523550464)>, <ExcThread(execute_payloads, started 139827483764480)>, <ExcThread(job_monitor, started 139828003849984)>, <ExcThread(create_data_payload, started 139828540335872)>, <ExcThread(copytool_in, started 139827995457280)>, <ExcThread(validate_post, started 139827987064576)>, <ExcThread(data, started 139828557121280)>, <ExcThread(queue_monitor, started 139827978671872)>, <ExcThread(copytool_out, started 139828531943168)>] 2019-10-07 16:18:40,597 | DEBUG | copytool_in | pilot.control.data | copytool_in | [data] copytool_in thread has finished 2019-10-07 16:18:40,794 | INFO | validate_post | pilot.control.payload | validate_post | [payload] validate_post thread has finished 2019-10-07 16:18:40,804 | DEBUG | MainThread | pilot.workflow.generic | run | thread count now at 10 threads 2019-10-07 16:18:40,804 | DEBUG | MainThread | pilot.workflow.generic | run | enumerate: [<_MainThread(MainThread, started 139828727146304)>, <ExcThread(job, started 139828565513984)>, <ExcThread(queue_monitoring, started 139828029028096)>, <ExcThread(payload, started 139828523550464)>, <ExcThread(execute_payloads, started 139827483764480)>, <ExcThread(job_monitor, started 139828003849984)>, <ExcThread(create_data_payload, started 139828540335872)>, <ExcThread(data, started 139828557121280)>, <ExcThread(queue_monitor, started 139827978671872)>, <ExcThread(copytool_out, started 139828531943168)>] 2019-10-07 16:18:40,831 | DEBUG | payload | pilot.control.payload | control | payload control ending since graceful_stop has been set 2019-10-07 16:18:40,831 | DEBUG | payload | pilot.control.payload | control | [payload] control thread has finished 2019-10-07 16:18:40,846 | DEBUG | data | pilot.control.data | control | data control ending since graceful_stop has been set 2019-10-07 16:18:40,847 | DEBUG | data | pilot.control.data | control | [data] control thread has finished 2019-10-07 16:18:40,882 | DEBUG | create_data_payload | pilot.control.job | create_data_payload | [job] create_data_payload thread has finished 2019-10-07 16:18:40,883 | INFO | execute_payloads | pilot.control.payload | execute_payloads | [payload] execute_payloads thread has finished 2019-10-07 16:18:41,005 | DEBUG | MainThread | pilot.workflow.generic | run | thread count now at 6 threads 2019-10-07 16:18:41,006 | DEBUG | MainThread | pilot.workflow.generic | run | enumerate: [<_MainThread(MainThread, started 139828727146304)>, <ExcThread(job, started 139828565513984)>, <ExcThread(queue_monitoring, started 139828029028096)>, <ExcThread(job_monitor, started 139828003849984)>, <ExcThread(queue_monitor, started 139827978671872)>, <ExcThread(copytool_out, started 139828531943168)>] 2019-10-07 16:18:41,169 | DEBUG | job | pilot.control.job | control | job control ending since graceful_stop has been set 2019-10-07 16:18:41,169 | DEBUG | job | pilot.control.job | control | [job] control thread has finished 2019-10-07 16:18:41,170 | DEBUG | MainThread | pilot.workflow.generic | run | thread count now at 5 threads 2019-10-07 16:18:41,170 | DEBUG | MainThread | pilot.workflow.generic | run | enumerate: [<_MainThread(MainThread, started 139828727146304)>, <ExcThread(queue_monitoring, started 139828029028096)>, <ExcThread(job_monitor, started 139828003849984)>, <ExcThread(queue_monitor, started 139827978671872)>, <ExcThread(copytool_out, started 139828531943168)>] 2019-10-07 16:18:41,209 | WARNING | queue_monitor | pilot.util.common | should_abort | job:queue_monitor:received graceful stop - abort after this iteration 2019-10-07 16:18:41,209 | DEBUG | queue_monitor | pilot.control.job | queue_monitor | [job] queue monitor thread has finished 2019-10-07 16:18:41,241 | WARNING | copytool_out | pilot.util.common | should_abort | data:copytool_out:received graceful stop - abort after this iteration 2019-10-07 16:18:41,271 | DEBUG | MainThread | pilot.workflow.generic | run | thread count now at 4 threads 2019-10-07 16:18:41,271 | DEBUG | MainThread | pilot.workflow.generic | run | enumerate: [<_MainThread(MainThread, started 139828727146304)>, <ExcThread(queue_monitoring, started 139828029028096)>, <ExcThread(job_monitor, started 139828003849984)>, <ExcThread(copytool_out, started 139828531943168)>] 2019-10-07 16:18:42,242 | DEBUG | copytool_out | pilot.control.data | copytool_out | [data] copytool_out thread has finished 2019-10-07 16:18:42,276 | DEBUG | MainThread | pilot.workflow.generic | run | thread count now at 3 threads 2019-10-07 16:18:42,277 | DEBUG | MainThread | pilot.workflow.generic | run | enumerate: [<_MainThread(MainThread, started 139828727146304)>, <ExcThread(queue_monitoring, started 139828029028096)>, <ExcThread(job_monitor, started 139828003849984)>] 2019-10-07 16:18:42,834 | WARNING | queue_monitoring | pilot.util.common | should_abort | data:queue_monitoring:received graceful stop - abort after this iteration 2019-10-07 16:18:45,835 | DEBUG | queue_monitoring | pilot.control.data | queue_monitoring | [data] queue_monitor thread has finished 2019-10-07 16:18:45,899 | DEBUG | MainThread | pilot.workflow.generic | run | thread count now at 2 threads 2019-10-07 16:18:45,900 | DEBUG | MainThread | pilot.workflow.generic | run | enumerate: [<_MainThread(MainThread, started 139828727146304)>, <ExcThread(job_monitor, started 139828003849984)>] 2019-10-07 16:19:15,290 | WARNING | job_monitor | pilot.control.job | check_job_monitor_waiting_time | no jobs in monitored_payloads queue (waited for 72 s) 2019-10-07 16:19:15,290 | DEBUG | job_monitor | pilot.control.job | job_monitor | [job] job monitor thread has finished 2019-10-07 16:19:15,385 | INFO | MainThread | pilot.workflow.generic | run | end of generic workflow (traces error code: 0) 2019-10-07 16:19:15,385 | INFO | MainThread | root | wrap_up | traces error code: 0 2019-10-07 16:19:15,385 | INFO | MainThread | root | wrap_up | pilot has finished 2019-10-07 16:19:15 UTC [wrapper] ==== pilot stdout END ==== 2019-10-07 16:19:15 UTC [wrapper] ==== wrapper stdout RESUME ==== 2019-10-07 16:19:15 UTC [wrapper] Pilot exit status: 0 2019-10-07 16:19:15 UTC [wrapper] STATUSCODE: 0 2019-10-07 16:19:15 UTC [wrapper] apfmon messages muted ---- find pandaIDs.out ---- total 56 -rw-------. 1 dcameron zp 11357 Jul 25 16:38 LICENSE -rw-------. 1 dcameron zp 20 Sep 9 13:04 MANIFEST.IN -rw-------. 1 dcameron zp 11 Oct 7 12:11 pandaIDs.out drwx------. 14 dcameron zp 216 Oct 7 12:11 pilot -rwx------. 1 dcameron zp 20136 Sep 9 13:04 pilot.py -rw-------. 1 dcameron zp 9 Sep 9 13:04 PILOTVERSION -rw-------. 1 dcameron zp 2251 Jul 25 16:38 README.md -rw-------. 1 dcameron zp 760 Aug 22 11:01 setup.py -rw-------. 1 dcameron zp 221 Jul 25 16:38 TODO.md 2019-10-07 16:19:15 UTC [wrapper] pandaIDs.out files: -rw-------. 1 dcameron zp 11 Oct 7 12:11 /home/dcameron/boinc/slots/1/pilot2/pandaIDs.out 2019-10-07 16:19:15 UTC [wrapper] pandaIDs.out content: 4494828307 2019-10-07 16:19:15 UTC [wrapper] Test setup, not cleaning 2019-10-07 16:19:15 UTC [wrapper] ==== wrapper stdout END ==== 2019-10-07 16:19:15 UTC [wrapper] ==== wrapper stderr END ==== 2019-10-07 16:19:15 UTC [wrapper] wrapper wrapperexiting ec=0, duration=22100 2019-10-07 16:19:15 UTC [wrapper] apfmon messages muted 2019-10-07 18:19:15,551: ***************diag file************ 2019-10-07 18:19:15,551: runtimeenvironments=APPS/HEP/ATLAS-SITE; Processors=1 WallTime=22100.04s KernelTime=500.64s UserTime=86775.98s CPUUsage=394% MaxResidentMemory=1960356kB AverageResidentMemory=0kB AverageTotalMemory=0kB AverageUnsharedMemory=0kB AverageUnsharedStack=0kB AverageSharedMemory=0kB PageSize=4096B MajorPageFaults=26449 MinorPageFaults=22728955 Swaps=0 ForcedSwitches=939956 WaitSwitches=12688721 Inputs=8590226 Outputs=1044656 SocketReceived=0 SocketSent=0 Signals=0 nodename=David_Cameron@pcoslo5.cern.ch exitcode=0 2019-10-07 18:19:15,555: ******************************WorkDir*********************** 2019-10-07 18:19:15,555: total 219076 drwxrwx--x. 7 dcameron zp 4096 Oct 7 18:19 . drwxrwx--x. 4 dcameron zp 24 Oct 7 12:10 .. -rw-------. 1 dcameron zp 7586632 Oct 7 12:11 agis_ddmendpoints.json -rw-------. 1 dcameron zp 3957898 Oct 7 12:11 agis_schedconf.cvmfs.json drwx------. 2 dcameron zp 6 Oct 7 12:10 .alrb drwxr-xr-x. 3 dcameron zp 17 Oct 7 12:10 APPS -rw-------. 1 dcameron zp 548 Oct 7 12:10 .asetup -rw-------. 1 dcameron zp 4127 Oct 7 12:11 .asetup.save drwx------. 2 dcameron zp 6 Oct 7 12:11 .asetup-sysbin_19823 -rw-r--r--. 1 dcameron zp 0 Oct 7 12:10 boinc_lockfile -rw-r--r--. 1 dcameron zp 8192 Oct 7 18:19 boinc_mmap_file -rw-r--r--. 1 dcameron zp 537 Oct 7 18:11 boinc_task_state.xml -rw-r--r--. 1 dcameron zp 209848135 Oct 7 12:10 EVNT.18605585._000291.pool.root.1 -rw-------. 1 dcameron zp 66727 Oct 7 18:18 heartbeat.json -rw-r--r--. 1 dcameron zp 6245 Oct 7 12:10 init_data.xml -rw-r--r--. 1 dcameron zp 267504 Oct 7 12:10 input.tar.gz -rw-r--r--. 1 dcameron zp 112 Oct 7 12:10 job.xml -rw-------. 1 dcameron zp 1260730 Oct 7 18:19 log.19000575._014418.job.log.1 -rw-------. 1 dcameron zp 879428 Oct 7 18:18 log.19000575._014418.job.log.tgz.1 -rw-------. 1 dcameron zp 854 Oct 7 18:18 memory_monitor_summary.json -rw-------. 1 dcameron zp 463 Oct 7 18:19 output.list -rw-------. 1 dcameron zp 2887 Sep 29 21:35 pandaJobData.out drwx------. 3 dcameron zp 229 Oct 7 12:11 pilot2 -rw-r--r--. 1 dcameron zp 259319 Sep 29 21:32 pilot2.tar.gz -rw-------. 1 dcameron zp 11195 Oct 7 18:19 pilotlog.txt -rw-r--r--. 1 dcameron zp 4480 Sep 29 21:35 queuedata.json -rw-r--r--. 1 dcameron zp 815 Oct 7 12:10 RTE.tar.gz -rwxr-xr-x. 1 dcameron zp 7950 Oct 7 12:10 run_atlas -rwx------. 1 dcameron zp 12762 Sep 29 21:35 runpilot2-wrapper.sh -rw-r--r--. 1 dcameron zp 692 Oct 7 18:19 runtime_log -rw-r--r--. 1 dcameron zp 8048 Oct 7 18:19 runtime_log.err -rw-------. 1 dcameron zp 240 Oct 7 12:10 setup.sh.local drwxrwx--x. 2 dcameron zp 131 Oct 7 18:19 shared -rw-r--r--. 1 dcameron zp 8688 Oct 7 12:10 start_atlas.sh -rw-r--r--. 1 dcameron zp 40296 Oct 7 18:19 stderr.txt -rw-r--r--. 1 dcameron zp 107 Oct 7 12:10 wrapper_26015_x86_64-pc-linux-gnu -rw-r--r--. 1 dcameron zp 28 Oct 7 18:19 wrapper_checkpoint.txt -rw-------. 1 dcameron zp 513 Oct 7 18:19 yEFMDmO9tYvnShfckohDCDFpABFKDmABFKDmuAgSDmABFKDm5UizSm.diag 2019-10-07 18:19:15,555: running start_atlas return value is 0 2019-10-07 18:19:15,555: Parent exit 0 2019-10-07 18:19:15,556: child process exit 0 18:19:16 (13015): run_atlas exited; CPU time 86776.310581 18:19:16 (13015): called boinc_finish(0) </stderr_txt> ]]>
©2024 CERN