| Name | 3OsMDmx2A2vnShfckohDCDFpABFKDmABFKDmPRANDmABFKDm4oz3Jm_0 |
| Workunit | 1962877 |
| Created | 19 Dec 2019, 13:29:29 UTC |
| Sent | 26 Dec 2019, 9:53:15 UTC |
| Report deadline | 2 Jan 2020, 9:53:15 UTC |
| Received | 26 Dec 2019, 10:21:44 UTC |
| Server state | Over |
| Outcome | Success |
| Client state | Done |
| Exit status | 0 (0x00000000) |
| Computer ID | 4064 |
| Run time | 25 min 46 sec |
| CPU time | 8 min 32 sec |
| Validate state | Valid |
| Credit | 8.97 |
| Device peak FLOPS | 2.51 GFLOPS |
| Application version | ATLAS Simulation v0.98 (native_mt) x86_64-pc-linux-gnu |
| Peak working set size | 153.88 MB |
| Peak swap size | 4.23 GB |
| Peak disk usage | 710.35 MB |
Stderr output
<core_client_version>7.16.1</core_client_version>
<![CDATA[
<stderr_txt>
10:54:45 (16877): wrapper (7.7.26015): starting
10:54:45 (16877): wrapper: running run_atlas (--nthreads 2)
Do 26. Dez 10:54:46 CET 2019: Arguments: --nthreads 2
Do 26. Dez 10:54:46 CET 2019: Threads: 2
Do 26. Dez 10:54:46 CET 2019: Checking for CVMFS
Do 26. Dez 10:54:46 CET 2019: Probing /cvmfs/atlas.cern.ch... OK
Do 26. Dez 10:54:48 CET 2019: Probing /cvmfs/atlas-condb.cern.ch... OK
Do 26. Dez 10:54:50 CET 2019: Probing /cvmfs/grid.cern.ch... OK
Do 26. Dez 10:54:52 CET 2019: Probing /cvmfs/cernvm-prod.cern.ch... OK
Do 26. Dez 10:54:54 CET 2019: Probing /cvmfs/sft.cern.ch... OK
Do 26. Dez 10:54:56 CET 2019: Probing /cvmfs/alice.cern.ch... OK
Do 26. Dez 10:54:57 CET 2019: VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE
Do 26. Dez 10:54:57 CET 2019: 2.7.0.0 952 28 40932 58427 3 76 2425697 4194304 0 65024 0 4295 99.9534 5 10 http://s1cern-cvmfs.openhtc.io/cvmfs/atlas.cern.ch DIRECT 1
Do 26. Dez 10:54:57 CET 2019: CVMFS is ok
Do 26. Dez 10:54:57 CET 2019: Singularity not required
Do 26. Dez 10:54:59 CET 2019: Set ATHENA_PROC_NUMBER=2
Do 26. Dez 10:54:59 CET 2019: Starting ATLAS job with PandaID=4002876565
Do 26. Dez 10:54:59 CET 2019: Running command: sh start_atlas.sh
Do 26. Dez 11:20:30 CET 2019: *** The last 200 lines of the pilot log: ***
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,483 | INFO | queue_monitor | pilot.control.job.4002876565 | send_state | job 4002876565 has failed - writing final server update
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,483 | INFO | queue_monitor | pilot.control.job.4002876565 | verify_error_code | verified error code
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,484 | DEBUG | queue_monitor | pilot.control.job.4002876565 | get_data_structure | building data structure to be sent to server with heartbeat
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,484 | WARNING | queue_monitor | pilot.user.atlas.common | get_db_info | format EVNTtoHITS has no such key: dbData
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,484 | WARNING | queue_monitor | pilot.user.atlas.common | get_db_info | format EVNTtoHITS has no such key: dbTime
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,484 | INFO | queue_monitor | pilot.util.auxiliary.4002876565 | get_job_metrics | will not add max space = -365193805 B to job metrics
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,485 | DEBUG | queue_monitor | pilot.api.analytics | get_fitted_data | removing tails from data to be fitted
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,485 | INFO | queue_monitor | pilot.api.analytics | get_fitted_data | fitting pss+swap vs Time
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,486 | INFO | queue_monitor | pilot.api.analytics | get_fitted_data | current memory leak: 15.47 B/s (using 9 data points, chi2=5796)
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,486 | DEBUG | queue_monitor | pilot.util.auxiliary.4002876565 | get_job_metrics | job metrics="coreCount=2 actualCoreCount=2 leak=15.47 chi2=5796"
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,486 | INFO | queue_monitor | pilot.control.job.4002876565 | get_data_structure | payload/TRF did not report the number of read events
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,487 | INFO | queue_monitor | pilot.user.atlas.utilities | get_memory_values | using path: /var/lib/boinc/slots/0/PanDA_Pilot-4002876565/memory_monitor_summary.json (trf name=prmon)
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,489 | DEBUG | queue_monitor | pilot.user.atlas.utilities | get_memory_monitor_info | summary_dictionary={'Max': {'rx_packets': 11301, 'nprocs': 12, 'nthreads': 0, 'rx_bytes': 3075468, 'wtime': 1108, 'rss': 264012, 'write_bytes': 0, 'vmem': 1451016, 'read_bytes': 0, 'stime': 417, 'tx_bytes': 2827388, 'pss': 251772, 'wchar': 0, 'rchar': 0, 'tx_packets': 7407, 'swap': 0, 'utime': 65}, 'Avg': {'write_bytes': 0, 'nprocs': 6, 'nthreads': 0, 'rx_bytes': 2774, 'rx_packets': 10, 'vmem': 1035101, 'read_bytes': 0, 'swap': 0, 'tx_bytes': 2550, 'pss': 167508, 'wchar': 0, 'rchar': 0, 'tx_packets': 6, 'rss': 175666}}
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,489 | INFO | queue_monitor | pilot.user.atlas.utilities | get_memory_monitor_info | extracted standard info from prmon json
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,490 | INFO | queue_monitor | pilot.user.atlas.utilities | get_memory_monitor_info | extracted standard memory fields from prmon json
11:20:30 (16877): run_atlas exited; CPU time 234.618104
11:20:30 (16877): called boinc_finish(0)
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,490 | INFO | queue_monitor | pilot.util.auxiliary.4002876565 | timing_report | ..............................
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,490 | INFO | queue_monitor | pilot.util.auxiliary.4002876565 | timing_report | . Timing measurements:
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,490 | INFO | queue_monitor | pilot.util.auxiliary.4002876565 | timing_report | . get job = 2 s
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,490 | INFO | queue_monitor | pilot.util.auxiliary.4002876565 | timing_report | . initial setup = 3 s
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,491 | INFO | queue_monitor | pilot.util.auxiliary.4002876565 | timing_report | . payload setup = 0 s
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,491 | INFO | queue_monitor | pilot.util.auxiliary.4002876565 | timing_report | . total setup = 3 s
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,491 | INFO | queue_monitor | pilot.util.auxiliary.4002876565 | timing_report | . stage-in = 2 s
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,491 | INFO | queue_monitor | pilot.util.auxiliary.4002876565 | timing_report | . payload execution = 1286 s
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,491 | INFO | queue_monitor | pilot.util.auxiliary.4002876565 | timing_report | . stage-out = 3 s
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,491 | INFO | queue_monitor | pilot.util.auxiliary.4002876565 | timing_report | ..............................
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,491 | INFO | queue_monitor | pilot.util.auxiliary.4002876565 | get_log_extracts | building log extracts (sent to the server as 'pilotLog')
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,492 | DEBUG | queue_monitor | pilot.util.auxiliary.4002876565 | get_panda_tracer_log | PanDA tracer log does not exist: /var/lib/boinc/slots/0/PanDA_Pilot-4002876565/pandatracerlog.txt (ignoring)
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,492 | INFO | queue_monitor | pilot.util.container | execute | executing command: tail -n 20 /var/lib/boinc/slots/0/PanDA_Pilot-4002876565/pilotlog.txt
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,740 | WARNING | queue_monitor | pilot.util.auxiliary.4002876565 | get_log_extracts | detected the following tail of warning/fatal messages in the pilot log:
Do 26. Dez 11:20:30 CET 2019: - Log from pilotlog.txt -
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,486 | INFO | queue_monitor | pilot.api.analytics | get_fitted_data | current memory leak: 15.47 B/s (using 9 data points, chi2=5796)
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,486 | DEBUG | queue_monitor | pilot.util.auxiliary.4002876565 | get_job_metrics | job metrics="coreCount=2 actualCoreCount=2 leak=15.47 chi2=5796"
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,486 | INFO | queue_monitor | pilot.control.job.4002876565 | get_data_structure | payload/TRF did not report the number of read events
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,487 | INFO | queue_monitor | pilot.user.atlas.utilities | get_memory_values | using path: /var/lib/boinc/slots/0/PanDA_Pilot-4002876565/memory_monitor_summary.json (trf name=prmon)
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,489 | DEBUG | queue_monitor | pilot.user.atlas.utilities | get_memory_monitor_info | summary_dictionary={'Max': {'rx_packets': 11301, 'nprocs': 12, 'nthreads': 0, 'rx_bytes': 3075468, 'wtime': 1108, 'rss': 264012, 'write_bytes': 0, 'vmem': 1451016, 'read_bytes': 0, 'stime': 417, 'tx_bytes': 2827388, 'pss': 251772, 'wchar': 0, 'rchar': 0, 'tx_packets': 7407, 'swap': 0, 'utime': 65}, 'Avg': {'write_bytes': 0, 'nprocs': 6, 'nthreads': 0, 'rx_bytes': 2774, 'rx_packets': 10, 'vmem': 1035101, 'read_bytes': 0, 'swap': 0, 'tx_bytes': 2550, 'pss': 167508, 'wchar': 0, 'rchar': 0, 'tx_packets': 6, 'rss': 175666}}
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,489 | INFO | queue_monitor | pilot.user.atlas.utilities | get_memory_monitor_info | extracted standard info from prmon json
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,490 | INFO | queue_monitor | pilot.user.atlas.utilities | get_memory_monitor_info | extracted standard memory fields from prmon json
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,490 | INFO | queue_monitor | pilot.util.auxiliary.4002876565 | timing_report | ..............................
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,490 | INFO | queue_monitor | pilot.util.auxiliary.4002876565 | timing_report | . Timing measurements:
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,490 | INFO | queue_monitor | pilot.util.auxiliary.4002876565 | timing_report | . get job = 2 s
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,490 | INFO | queue_monitor | pilot.util.auxiliary.4002876565 | timing_report | . initial setup = 3 s
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,491 | INFO | queue_monitor | pilot.util.auxiliary.4002876565 | timing_report | . payload setup = 0 s
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,491 | INFO | queue_monitor | pilot.util.auxiliary.4002876565 | timing_report | . total setup = 3 s
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,491 | INFO | queue_monitor | pilot.util.auxiliary.4002876565 | timing_report | . stage-in = 2 s
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,491 | INFO | queue_monitor | pilot.util.auxiliary.4002876565 | timing_report | . payload execution = 1286 s
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,491 | INFO | queue_monitor | pilot.util.auxiliary.4002876565 | timing_report | . stage-out = 3 s
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,491 | INFO | queue_monitor | pilot.util.auxiliary.4002876565 | timing_report | ..............................
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,491 | INFO | queue_monitor | pilot.util.auxiliary.4002876565 | get_log_extracts | building log extracts (sent to the server as 'pilotLog')
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,492 | DEBUG | queue_monitor | pilot.util.auxiliary.4002876565 | get_panda_tracer_log | PanDA tracer log does not exist: /var/lib/boinc/slots/0/PanDA_Pilot-4002876565/pandatracerlog.txt (ignoring)
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,492 | INFO | queue_monitor | pilot.util.container | execute | executing command: tail -n 20 /var/lib/boinc/slots/0/PanDA_Pilot-4002876565/pilotlog.txt
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,740 | WARNING | queue_monitor | pilot.control.job | add_timing_and_extracts |
Do 26. Dez 11:20:30 CET 2019: XXXXXXXXXXXXXXXXXXXXX[begin log extracts]
Do 26. Dez 11:20:30 CET 2019: - Log from pilotlog.txt -
Do 26. Dez 11:20:30 CET 2019: 2019-12-26 10:19:58,486 | INFO | queue_monitor | pilot.api.analytics | get_fitted_data | current memory leak: 15.47 B/s (using 9 data points, chi2=5796)
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,486 | DEBUG | queue_monitor | pilot.util.auxiliary.4002876565 | get_job_metrics | job metrics="coreCount=2 actualCoreCount=2 leak=15.47 chi2=5796"
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,486 | INFO | queue_monitor | pilot.control.job.4002876565 | get_data_structure | payload/TRF did not report the number of read events
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,487 | INFO | queue_monitor | pilot.user.atlas.utilities | get_memory_values | using path: /var/lib/boinc/slots/0/PanDA_Pilot-4002876565/memory_monitor_summary.json (trf name=prmon)
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,489 | DEBUG | queue_monitor | pilot.user.atlas.utilities | get_memory_monitor_info | summary_dictionary={'Max': {'rx_packets': 11301, 'nprocs': 12, 'nthreads': 0, 'rx_bytes': 3075468, 'wtime': 1108, 'rss': 264012, 'write_bytes': 0, 'vmem': 1451016, 'read_bytes': 0, 'stime': 417, 'tx_bytes': 2827388, 'pss': 251772, 'wchar': 0, 'rchar': 0, 'tx_packets': 7407, 'swap': 0, 'utime': 65}, 'Avg': {'write_bytes': 0, 'nprocs': 6, 'nthreads': 0, 'rx_bytes': 2774, 'rx_packets': 10, 'vmem': 1035101, 'read_bytes': 0, 'swap': 0, 'tx_bytes': 2550, 'pss': 167508, 'wchar': 0, 'rchar': 0, 'tx_packets': 6, 'rss': 175666}}
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,489 | INFO | queue_monitor | pilot.user.atlas.utilities | get_memory_monitor_info | extracted standard info from prmon json
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,490 | INFO | queue_monitor | pilot.user.atlas.utilities | get_memory_monitor_info | extracted standard memory fields from prmon json
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,490 | INFO | queue_monitor | pilot.util.auxiliary.4002876565 | timing_report | ..............................
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,490 | INFO | queue_monitor | pilot.util.auxiliary.4002876565 | timing_report | . Timing measurements:
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,490 | INFO | queue_monitor | pilot.util.auxiliary.4002876565 | timing_report | . get job = 2 s
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,490 | INFO | queue_monitor | pilot.util.auxiliary.4002876565 | timing_report | . initial setup = 3 s
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,491 | INFO | queue_monitor | pilot.util.auxiliary.4002876565 | timing_report | . payload setup = 0 s
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,491 | INFO | queue_monitor | pilot.util.auxiliary.4002876565 | timing_report | . total setup = 3 s
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,491 | INFO | queue_monitor | pilot.util.auxiliary.4002876565 | timing_report | . stage-in = 2 s
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,491 | INFO | queue_monitor | pilot.util.auxiliary.4002876565 | timing_report | . payload execution = 1286 s
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,491 | INFO | queue_monitor | pilot.util.auxiliary.4002876565 | timing_report | . stage-out = 3 s
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,491 | INFO | queue_monitor | pilot.util.auxiliary.4002876565 | timing_report | ..............................
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,491 | INFO | queue_monitor | pilot.util.auxiliary.4002876565 | get_log_extracts | building log extracts (sent to the server as 'pilotLog')
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,492 | DEBUG | queue_monitor | pilot.util.auxiliary.4002876565 | get_panda_tracer_log | PanDA tracer log does not exist: /var/lib/boinc/slots/0/PanDA_Pilot-4002876565/pandatracerlog.txt (ignoring)
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,492 | INFO | queue_monitor | pilot.util.container | execute | executing command: tail -n 20 /var/lib/boinc/slots/0/PanDA_Pilot-4002876565/pilotlog.txt
Do 26. Dez 11:20:31 CET 2019: XXXXXXXXXXXXXXXXXXXXX[end log extracts]
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,741 | DEBUG | queue_monitor | pilot.control.job.4002876565 | send_state | wrote heartbeat to file /var/lib/boinc/slots/0/heartbeat.json
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,741 | DEBUG | queue_monitor | pilot.control.job | queue_monitor | job 4002876565 was dequeued from the monitored payloads queue
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,742 | DEBUG | queue_monitor | pilot.control.job | queue_monitor | tmp job object deleted
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,742 | DEBUG | queue_monitor | pilot.control.job | queue_monitor | [job] queue monitor thread has finished
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,755 | INFO | retrieve | pilot.util.auxiliary.4002876565 | make_job_report |
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,755 | INFO | retrieve | pilot.util.auxiliary.4002876565 | make_job_report | job summary report
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,755 | INFO | retrieve | pilot.util.auxiliary.4002876565 | make_job_report | --------------------------------------------------
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,755 | INFO | retrieve | pilot.util.auxiliary.4002876565 | make_job_report | PanDA job id: 4002876565
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,755 | INFO | retrieve | pilot.util.auxiliary.4002876565 | make_job_report | task id: 000649-2078491-5677
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,755 | INFO | retrieve | pilot.util.auxiliary.4002876565 | make_job_report | errors: (none)
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,756 | INFO | retrieve | pilot.util.auxiliary.4002876565 | make_job_report | status: LOG_TRANSFER = DONE
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,756 | INFO | retrieve | pilot.util.auxiliary.4002876565 | make_job_report | pilot state: failed
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,756 | INFO | retrieve | pilot.util.auxiliary.4002876565 | make_job_report | transexitcode: 65
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,756 | INFO | retrieve | pilot.util.auxiliary.4002876565 | make_job_report | exeerrorcode: 65
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,756 | INFO | retrieve | pilot.util.auxiliary.4002876565 | make_job_report | exeerrordiag: EVNTtoHITS got a SIGBUS signal (exit code 135)
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,756 | INFO | retrieve | pilot.util.auxiliary.4002876565 | make_job_report | exitcode: 65
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,756 | INFO | retrieve | pilot.util.auxiliary.4002876565 | make_job_report | exitmsg: EVNTtoHITS got a SIGBUS signal (exit code 135)
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,757 | INFO | retrieve | pilot.util.auxiliary.4002876565 | make_job_report | cpuconsumptiontime: 652 s
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,757 | INFO | retrieve | pilot.util.auxiliary.4002876565 | make_job_report | nevents: 0
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,757 | INFO | retrieve | pilot.util.auxiliary.4002876565 | make_job_report | neventsw: 0
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,757 | INFO | retrieve | pilot.util.auxiliary.4002876565 | make_job_report | pid: 23797
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,757 | INFO | retrieve | pilot.util.auxiliary.4002876565 | make_job_report | pgrp: 23797
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,757 | INFO | retrieve | pilot.util.auxiliary.4002876565 | make_job_report | corecount: 2
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,757 | INFO | retrieve | pilot.util.auxiliary.4002876565 | make_job_report | event service: False
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,758 | INFO | retrieve | pilot.util.auxiliary.4002876565 | make_job_report | --------------------------------------------------
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,758 | INFO | retrieve | pilot.util.auxiliary.4002876565 | make_job_report |
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,758 | INFO | retrieve | pilot.util.queuehandling | queue_report | queue jobs has 0 job(s)
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,758 | INFO | retrieve | pilot.util.queuehandling | queue_report | queue payloads has 0 job(s)
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,758 | INFO | retrieve | pilot.util.queuehandling | queue_report | queue data_in has 0 job(s)
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,758 | INFO | retrieve | pilot.util.queuehandling | queue_report | queue data_out has 0 job(s)
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,759 | INFO | retrieve | pilot.util.queuehandling | queue_report | queue current_data_in has 0 job(s)
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,759 | INFO | retrieve | pilot.util.queuehandling | queue_report | queue validated_jobs has 0 job(s)
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,759 | INFO | retrieve | pilot.util.queuehandling | queue_report | queue validated_payloads has 0 job(s)
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,759 | INFO | retrieve | pilot.util.queuehandling | queue_report | queue monitored_payloads has 0 job(s)
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,759 | INFO | retrieve | pilot.util.queuehandling | queue_report | queue finished_jobs has 0 job(s)
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,759 | INFO | retrieve | pilot.util.queuehandling | queue_report | queue finished_payloads has 0 job(s)
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,759 | INFO | retrieve | pilot.util.queuehandling | queue_report | queue finished_data_in has 1 job(s)
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,760 | INFO | retrieve | pilot.util.queuehandling | queue_report | queue finished_data_out has 0 job(s)
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,760 | INFO | retrieve | pilot.util.queuehandling | queue_report | queue failed_jobs has 0 job(s)
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,760 | INFO | retrieve | pilot.util.queuehandling | queue_report | queue failed_payloads has 0 job(s)
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,760 | INFO | retrieve | pilot.util.queuehandling | queue_report | queue failed_data_in has 0 job(s)
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,760 | INFO | retrieve | pilot.util.queuehandling | queue_report | queue failed_data_out has 0 job(s)
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,760 | INFO | retrieve | pilot.util.queuehandling | queue_report | queue completed_jobs has 0 job(s)
Do 26. Dez 11:20:31 CET 2019: 2019-12-26 10:19:58,761 | INFO | retrieve | pilot.util.queuehandling | queue_report | queue completed_jobids has 1 job(s)
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:19:58,761 | INFO | retrieve | pilot.control.job.4002876565 | has_job_completed | job 4002876565 has completed (purged errors)
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:19:58,761 | INFO | retrieve | pilot.util.processes | cleanup | overall cleanup function is called
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:19:58,763 | DEBUG | retrieve | pilot.util.processes | cleanup | work directory was removed: /var/lib/boinc/slots/0/PanDA_Pilot-4002876565
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:19:59,565 | DEBUG | MainThread | pilot.workflow.generic | run | thread count now at 16 threads
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:19:59,566 | DEBUG | MainThread | pilot.workflow.generic | run | enumerate: [<_MainThread(MainThread, started 140540791166784)>, <ExcThread(job, started 140540523726592)>, <ExcThread(validate, started 140540515333888)>, <ExcThread(payload, started 140540121184000)>, <ExcThread(queue_monitoring, started 140540104398592)>, <ExcThread(job_monitor, started 140540473370368)>, <ExcThread(copytool_out, started 140540481763072)>, <ExcThread(monitor, started 140540096005888)>, <ExcThread(retrieve, started 140540490155776)>, <ExcThread(copytool_in, started 140540087613184)>, <ExcThread(validate_pre, started 140540079220480)>, <ExcThread(failed_post, started 140539517204224)>, <ExcThread(validate_post, started 140540070827776)>, <ExcThread(data, started 140540506941184)>, <ExcThread(create_data_payload, started 140540498548480)>, <ExcThread(execute_payloads, started 140539508811520)>]
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:19:59,768 | INFO | retrieve | pilot.info.jobdata | collect_zombies | --- collectZombieJob: --- 10, [23797]
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:19:59,769 | INFO | retrieve | pilot.info.jobdata | collect_zombies | zombie collector trying to kill pid 23797
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:19:59,769 | INFO | retrieve | pilot.info.jobdata | collect_zombies | harmless exception when collecting zombies: [Errno 10] No child processes
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:20:00,774 | INFO | retrieve | pilot.util.processes | cleanup | collected zombie processes
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:20:00,775 | INFO | retrieve | pilot.util.processes | cleanup | will now attempt to kill all subprocesses of pid=23797
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:20:01,299 | INFO | retrieve | pilot.util.processes | kill_processes | process IDs to be killed: [23797] (in reverse order)
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:20:01,946 | WARNING | retrieve | pilot.util.processes | kill_processes | found no corresponding commands to process id(s)
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:20:01,950 | INFO | retrieve | pilot.util.processes | kill_orphans | Do not look for orphan processes in BOINC jobs
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:20:01,950 | INFO | retrieve | pilot.control.job | retrieve | ready for new job
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:20:01,950 | INFO | retrieve | root | retrieve | pilot has finished for previous job - re-establishing logging
Do 26. Dez 11:20:32 CET 2019: mpi4py not found
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:20:02,086 | INFO | retrieve | pilot.util.auxiliary | pilot_version_banner | ****************************************
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:20:02,087 | INFO | retrieve | pilot.util.auxiliary | pilot_version_banner | *** PanDA Pilot version 2.3.4 (12) ***
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:20:02,087 | INFO | retrieve | pilot.util.auxiliary | pilot_version_banner | ****************************************
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:20:02,087 | INFO | retrieve | pilot.util.auxiliary | pilot_version_banner |
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:20:02,088 | INFO | retrieve | pilot.util.auxiliary | pilot_version_banner | pilot is running in a VM
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:20:02,088 | INFO | retrieve | pilot.util.auxiliary | display_architecture_info | architecture information:
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:20:02,177 | INFO | retrieve | pilot.util.auxiliary | display_architecture_info |
Do 26. Dez 11:20:32 CET 2019:
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:20:02,178 | INFO | retrieve | pilot.util.auxiliary | pilot_version_banner | ****************************************
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:20:02,688 | DEBUG | retrieve | pilot.util.monitoring | check_local_space | checking local space on /var/lib/boinc/slots/0
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:20:02,833 | INFO | retrieve | pilot.util.monitoring | check_local_space | sufficient remaining disk space (12490637312 B)
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:20:02,833 | WARNING | retrieve | pilot.control.job | proceed_with_getjob | since timefloor is set to 0, pilot was only allowed to run one job
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:20:02,833 | DEBUG | retrieve | pilot.control.job | retrieve | [job] retrieve thread has finished
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:20:02,835 | WARNING | queue_monitoring | pilot.util.common | should_abort | data:queue_monitoring:received graceful stop - abort after this iteration
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:20:02,835 | WARNING | copytool_out | pilot.util.common | should_abort | data:copytool_out:received graceful stop - abort after this iteration
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:20:03,382 | DEBUG | validate | pilot.control.job | validate | [job] validate thread has finished
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:20:03,488 | INFO | validate_pre | pilot.control.payload | validate_pre | [payload] validate_pre thread has finished
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:20:03,590 | INFO | execute_payloads | pilot.control.payload | execute_payloads | [payload] execute_payloads thread has finished
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:20:03,591 | DEBUG | copytool_in | pilot.control.data | copytool_in | [data] copytool_in thread has finished
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:20:03,631 | INFO | monitor | pilot.control.monitor | control | [monitor] control thread has ended
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:20:03,695 | INFO | validate_post | pilot.control.payload | validate_post | [payload] validate_post thread has finished
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:20:03,821 | DEBUG | MainThread | pilot.workflow.generic | run | thread count now at 9 threads
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:20:03,821 | DEBUG | MainThread | pilot.workflow.generic | run | enumerate: [<_MainThread(MainThread, started 140540791166784)>, <ExcThread(job, started 140540523726592)>, <ExcThread(payload, started 140540121184000)>, <ExcThread(queue_monitoring, started 140540104398592)>, <ExcThread(job_monitor, started 140540473370368)>, <ExcThread(copytool_out, started 140540481763072)>, <ExcThread(failed_post, started 140539517204224)>, <ExcThread(data, started 140540506941184)>, <ExcThread(create_data_payload, started 140540498548480)>]
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:20:03,839 | DEBUG | copytool_out | pilot.control.data | copytool_out | [data] copytool_out thread has finished
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:20:03,850 | DEBUG | data | pilot.control.data | control | data control ending since graceful_stop has been set
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:20:03,850 | DEBUG | data | pilot.control.data | control | [data] control thread has finished
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:20:03,958 | DEBUG | job | pilot.control.job | control | job control ending since graceful_stop has been set
Do 26. Dez 11:20:32 CET 2019: 2019-12-26 10:20:03,959 | DEBUG | job | pilot.control.job | control | [job] control thread has finished
</stderr_txt>
]]>
©2024 CERN