Trace number 3692548

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerobjective functionCPU timeWall clock time
SCIP spx E SCIP 2.1.1.4. Exp with SoPlex 1.6.0.3 fixed? (TO) 1802.11 1802.39

General information on the benchmark

Name/OPT-SMALLINT-LIN/sroussel/NG/
normalized-NG.dot_luby_20null10mast_rapportOE1.0_TMAX180_K192_cod2.opb
MD5SUMd26c1c744a6cc6b94bc88bd85095e6b9
Bench CategoryOPT-SMALLINT-LIN (optimisation, small integers, linear constraints)
Best result obtained on this benchmarkOPT
Best value of the objective obtained on this benchmark-36160
Best CPU time to get the best result obtained on this benchmark746.157
Has Objective FunctionYES
Satisfiable
(Un)Satisfiability was proved
Best value of the objective function
Optimality of the best value was proved
Number of variables400896
Total number of constraints402763
Number of constraints which are clauses400785
Number of constraints which are cardinality constraints (but not clauses)194
Number of constraints which are nor clauses,nor cardinality constraints1784
Minimum length of a constraint2
Maximum length of a constraint58023
Number of terms in the objective function 1427
Biggest coefficient in the objective function 1024
Number of bits for the biggest coefficient in the objective function 11
Sum of the numbers in the objective function 187909
Number of bits of the sum of numbers in the objective function 18
Biggest number in a constraint 1024
Number of bits of the biggest number in a constraint 11
Biggest sum of numbers in a constraint 187909
Number of bits of the biggest sum of numbers18
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.00/0.00	c SCIP version 2.1.1.4 [precision: 8 byte] [memory: block] [mode: optimized] [LP solver: SoPlex 1.6.0.3] [GitHash: 947bdb7-dirty]
0.00/0.00	c Copyright (c) 2002-2012 Konrad-Zuse-Zentrum fuer Informationstechnik Berlin (ZIB)
0.00/0.00	c 
0.00/0.00	c user parameter file <scip.set> not found - using default parameters
0.00/0.00	c reading problem <HOME/instance-3692548-1338047205.opb>
2.10/2.13	c original problem has 400896 variables (400896 bin, 0 int, 0 impl, 0 cont) and 402763 constraints
2.10/2.13	c problem read in 2.13
3.98/4.07	c presolving:
7.69/7.79	c (round 1) 0 del vars, 1977 del conss, 0 add conss, 0 chg bounds, 0 chg sides, 0 chg coeffs, 398809 upgd conss, 67806 impls, 1977 clqs
8.79/8.89	c (round 2) 0 del vars, 1977 del conss, 0 add conss, 0 chg bounds, 0 chg sides, 0 chg coeffs, 400786 upgd conss, 67806 impls, 1977 clqs

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3692548-1338047205/watcher-3692548-1338047205 -o /tmp/evaluation-result-3692548-1338047205/solver-3692548-1338047205 -C 1800 -W 1900 -M 15500 HOME/scip-2.1.1.4expfix.linux.x86_64.gnu.opt.spx -f HOME/instance-3692548-1338047205.opb -t 1800 -m 15500 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1900 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 2.01 2.01 2.00 3/176 4315
/proc/meminfo: memFree=26307416/32873648 swapFree=6864/6864
[pid=4315] ppid=4313 vsize=12692 CPUtime=0 cores=1,3,5,7
/proc/4315/stat : 4315 (scip-2.1.1.4exp) R 4313 4315 1798 0 -1 4202496 1930 0 0 0 0 0 0 0 20 0 1 0 270846556 12996608 1884 33554432000 4194304 10219143 140736230415280 140736230414184 8844210 0 0 4096 0 0 0 0 17 5 0 0 0 0 0
/proc/4315/statm: 3173 1884 413 1471 0 1693 0

[startup+0.0830529 s]
/proc/loadavg: 2.01 2.01 2.00 3/176 4315
/proc/meminfo: memFree=26307416/32873648 swapFree=6864/6864
[pid=4315] ppid=4313 vsize=46088 CPUtime=0.07 cores=1,3,5,7
/proc/4315/stat : 4315 (scip-2.1.1.4exp) R 4313 4315 1798 0 -1 4202496 10271 0 0 0 6 1 0 0 20 0 1 0 270846556 47194112 10225 33554432000 4194304 10219143 140736230415280 140736230414184 5345804 0 0 4096 0 0 0 0 17 5 0 0 0 0 0
/proc/4315/statm: 11522 10225 421 1471 0 10042 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 46088

[startup+0.100291 s]
/proc/loadavg: 2.01 2.01 2.00 3/176 4315
/proc/meminfo: memFree=26307416/32873648 swapFree=6864/6864
[pid=4315] ppid=4313 vsize=49668 CPUtime=0.09 cores=1,3,5,7
/proc/4315/stat : 4315 (scip-2.1.1.4exp) R 4313 4315 1798 0 -1 4202496 11155 0 0 0 8 1 0 0 20 0 1 0 270846556 50860032 11109 33554432000 4194304 10219143 140736230415280 140736230414184 5536058 0 0 4096 0 0 0 0 17 5 0 0 0 0 0
/proc/4315/statm: 12417 11110 421 1471 0 10937 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 49668

[startup+0.300239 s]
/proc/loadavg: 2.01 2.01 2.00 3/176 4315
/proc/meminfo: memFree=26307416/32873648 swapFree=6864/6864
[pid=4315] ppid=4313 vsize=94428 CPUtime=0.29 cores=1,3,5,7
/proc/4315/stat : 4315 (scip-2.1.1.4exp) R 4313 4315 1798 0 -1 4202496 22319 0 0 0 27 2 0 0 20 0 1 0 270846556 96694272 22273 33554432000 4194304 10219143 140736230415280 140736230280456 7123594 0 0 4096 0 0 0 0 17 5 0 0 0 0 0
/proc/4315/statm: 23607 22274 421 1471 0 22127 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 94428

[startup+0.700241 s]
/proc/loadavg: 2.01 2.01 2.00 3/176 4315
/proc/meminfo: memFree=26307416/32873648 swapFree=6864/6864
[pid=4315] ppid=4313 vsize=214268 CPUtime=0.69 cores=1,3,5,7
/proc/4315/stat : 4315 (scip-2.1.1.4exp) R 4313 4315 1798 0 -1 4202496 52235 0 0 0 64 5 0 0 20 0 1 0 270846556 219410432 52189 33554432000 4194304 10219143 140736230415280 140736230280456 9029200 0 0 4096 0 0 0 0 17 5 0 0 0 0 0
/proc/4315/statm: 53567 52189 421 1471 0 52087 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 214268

[startup+1.50024 s]
/proc/loadavg: 2.01 2.01 2.00 3/177 4316
/proc/meminfo: memFree=26013744/32873648 swapFree=6864/6864
[pid=4315] ppid=4313 vsize=437700 CPUtime=1.48 cores=1,3,5,7
/proc/4315/stat : 4315 (scip-2.1.1.4exp) R 4313 4315 1798 0 -1 4202496 108023 0 0 0 136 12 0 0 20 0 1 0 270846556 448204800 107977 33554432000 4194304 10219143 140736230415280 140736230280456 7589947 0 0 4096 0 0 0 0 17 5 0 0 0 0 0
/proc/4315/statm: 109425 107977 421 1471 0 107945 0
Current children cumulated CPU time (s) 1.48
Current children cumulated vsize (KiB) 437700

[startup+3.10025 s]
/proc/loadavg: 2.01 2.01 2.00 3/177 4316
/proc/meminfo: memFree=25815592/32873648 swapFree=6864/6864
[pid=4315] ppid=4313 vsize=1095744 CPUtime=3.09 cores=1,3,5,7
/proc/4315/stat : 4315 (scip-2.1.1.4exp) R 4313 4315 1798 0 -1 4202496 272087 0 0 0 274 35 0 0 20 0 1 0 270846556 1122041856 272440 33554432000 4194304 10219143 140736230415280 140736230280424 8930603 0 0 4096 16384 0 0 0 17 5 0 0 0 0 0
/proc/4315/statm: 273969 272441 483 1471 0 272489 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 1095744

[startup+6.30029 s]
/proc/loadavg: 2.01 2.01 2.00 3/177 4316
/proc/meminfo: memFree=24985212/32873648 swapFree=6864/6864
[pid=4315] ppid=4313 vsize=1629856 CPUtime=6.29 cores=1,3,5,7
/proc/4315/stat : 4315 (scip-2.1.1.4exp) R 4313 4315 1798 0 -1 4202496 403028 0 0 0 556 73 0 0 20 0 1 0 270846556 1668972544 404555 33554432000 4194304 10219143 140736230415280 140736230412280 9008087 0 0 4096 16384 0 0 0 17 5 0 0 0 0 0
/proc/4315/statm: 407464 404555 560 1471 0 405984 0
Current children cumulated CPU time (s) 6.29
Current children cumulated vsize (KiB) 1629856

[startup+12.7002 s]
/proc/loadavg: 2.01 2.01 2.00 3/177 4316
/proc/meminfo: memFree=24419812/32873648 swapFree=6864/6864
[pid=4315] ppid=4313 vsize=1904200 CPUtime=12.69 cores=1,3,5,7
/proc/4315/stat : 4315 (scip-2.1.1.4exp) R 4313 4315 1798 0 -1 4202496 493277 0 0 0 1150 119 0 0 20 0 1 0 270846556 1949900800 471776 33554432000 4194304 10219143 140736230415280 140736230410680 8969260 0 0 4096 16384 0 0 0 17 5 0 0 0 0 0
/proc/4315/statm: 476050 471776 612 1471 0 474570 0
Current children cumulated CPU time (s) 12.69
Current children cumulated vsize (KiB) 1904200

[startup+25.5003 s]
/proc/loadavg: 2.01 2.01 2.00 3/177 4316
/proc/meminfo: memFree=24411300/32873648 swapFree=6864/6864
[pid=4315] ppid=4313 vsize=1911624 CPUtime=25.48 cores=1,3,5,7
/proc/4315/stat : 4315 (scip-2.1.1.4exp) R 4313 4315 1798 0 -1 4202496 494492 0 0 0 2429 119 0 0 20 0 1 0 270846556 1957502976 473502 33554432000 4194304 10219143 140736230415280 140736230410680 8969224 0 0 4096 16384 0 0 0 17 5 0 0 0 0 0
/proc/4315/statm: 477906 473502 612 1471 0 476426 0
Current children cumulated CPU time (s) 25.48
Current children cumulated vsize (KiB) 1911624

[startup+51.1003 s]
/proc/loadavg: 2.00 2.01 2.00 3/177 4316
/proc/meminfo: memFree=24407728/32873648 swapFree=6864/6864
[pid=4315] ppid=4313 vsize=1915272 CPUtime=51.08 cores=1,3,5,7
/proc/4315/stat : 4315 (scip-2.1.1.4exp) R 4313 4315 1798 0 -1 4202496 495471 0 0 0 4989 119 0 0 20 0 1 0 270846556 1961238528 474481 33554432000 4194304 10219143 140736230415280 140736230411656 8969234 0 0 4096 16384 0 0 0 17 5 0 0 0 0 0
/proc/4315/statm: 478818 474481 612 1471 0 477338 0
Current children cumulated CPU time (s) 51.08
Current children cumulated vsize (KiB) 1915272

[startup+102.3 s]
/proc/loadavg: 2.00 2.00 2.00 3/177 4317
/proc/meminfo: memFree=23005184/32873648 swapFree=6864/6864
[pid=4315] ppid=4313 vsize=3346764 CPUtime=102.28 cores=1,3,5,7
/proc/4315/stat : 4315 (scip-2.1.1.4exp) R 4313 4315 1798 0 -1 4202496 822553 0 0 0 10066 162 0 0 20 0 1 0 270846556 3427086336 823144 33554432000 4194304 10219143 140736230415280 140736230410392 8969249 0 0 4096 16384 0 0 0 17 5 0 0 0 0 0
/proc/4315/statm: 836691 823144 612 1471 0 835211 0
Current children cumulated CPU time (s) 102.28
Current children cumulated vsize (KiB) 3346764

[startup+162.3 s]
/proc/loadavg: 2.00 2.00 2.00 3/177 4317
/proc/meminfo: memFree=23060164/32873648 swapFree=6864/6864
[pid=4315] ppid=4313 vsize=3348812 CPUtime=162.26 cores=1,3,5,7
/proc/4315/stat : 4315 (scip-2.1.1.4exp) R 4313 4315 1798 0 -1 4202496 823301 0 0 0 16064 162 0 0 20 0 1 0 270846556 3429183488 823892 33554432000 4194304 10219143 140736230415280 140736230410392 8969198 0 0 4096 16384 0 0 0 17 5 0 0 0 0 0
/proc/4315/statm: 837203 823892 612 1471 0 835723 0
Current children cumulated CPU time (s) 162.26
Current children cumulated vsize (KiB) 3348812

[startup+222.307 s]
/proc/loadavg: 2.00 2.00 2.00 3/178 4320
/proc/meminfo: memFree=23058184/32873648 swapFree=6864/6864
[pid=4315] ppid=4313 vsize=3348812 CPUtime=222.26 cores=1,3,5,7
/proc/4315/stat : 4315 (scip-2.1.1.4exp) R 4313 4315 1798 0 -1 4202496 823512 0 0 0 22064 162 0 0 20 0 1 0 270846556 3429183488 824103 33554432000 4194304 10219143 140736230415280 140736230410392 8969254 0 0 4096 16384 0 0 0 17 5 0 0 0 0 0
/proc/4315/statm: 837203 824103 612 1471 0 835723 0
Current children cumulated CPU time (s) 222.26
Current children cumulated vsize (KiB) 3348812

[startup+282.3 s]
/proc/loadavg: 2.04 2.01 2.00 3/177 4320
/proc/meminfo: memFree=23049044/32873648 swapFree=6864/6864
[pid=4315] ppid=4313 vsize=3352716 CPUtime=282.25 cores=1,3,5,7
/proc/4315/stat : 4315 (scip-2.1.1.4exp) R 4313 4315 1798 0 -1 4202496 824705 0 0 0 28062 163 0 0 20 0 1 0 270846556 3433181184 825296 33554432000 4194304 10219143 140736230415280 140736230410392 8969249 0 0 4096 16384 0 0 0 17 5 0 0 0 0 0
/proc/4315/statm: 838179 825296 612 1471 0 836699 0
Current children cumulated CPU time (s) 282.25
Current children cumulated vsize (KiB) 3352716

[startup+342.3 s]
/proc/loadavg: 2.01 2.01 2.00 3/177 4320
/proc/meminfo: memFree=23038008/32873648 swapFree=6864/6864
[pid=4315] ppid=4313 vsize=3352716 CPUtime=342.24 cores=1,3,5,7
/proc/4315/stat : 4315 (scip-2.1.1.4exp) R 4313 4315 1798 0 -1 4202496 824978 0 0 0 34061 163 0 0 20 0 1 0 270846556 3433181184 825569 33554432000 4194304 10219143 140736230415280 140736230410392 8969260 0 0 4096 16384 0 0 0 17 5 0 0 0 0 0
/proc/4315/statm: 838179 825569 612 1471 0 836699 0
Current children cumulated CPU time (s) 342.24
Current children cumulated vsize (KiB) 3352716

[startup+402.3 s]
/proc/loadavg: 2.07 2.02 2.00 3/177 4321
/proc/meminfo: memFree=23034688/32873648 swapFree=6864/6864
[pid=4315] ppid=4313 vsize=3354764 CPUtime=402.23 cores=1,3,5,7
/proc/4315/stat : 4315 (scip-2.1.1.4exp) R 4313 4315 1798 0 -1 4202496 825698 0 0 0 40060 163 0 0 20 0 1 0 270846556 3435278336 826289 33554432000 4194304 10219143 140736230415280 140736230410392 8969254 0 0 4096 16384 0 0 0 17 5 0 0 0 0 0
/proc/4315/statm: 838691 826289 612 1471 0 837211 0
Current children cumulated CPU time (s) 402.23
Current children cumulated vsize (KiB) 3354764

[startup+462.3 s]
/proc/loadavg: 2.02 2.01 2.00 3/177 4359
/proc/meminfo: memFree=24456256/32873648 swapFree=6864/6864
[pid=4315] ppid=4313 vsize=3355596 CPUtime=462.21 cores=1,3,5,7
/proc/4315/stat : 4315 (scip-2.1.1.4exp) R 4313 4315 1798 0 -1 4202496 826122 0 0 0 46058 163 0 0 20 0 1 0 270846556 3436130304 826772 33554432000 4194304 10219143 140736230415280 140736230410392 8969249 0 0 4096 16384 0 0 0 17 5 0 0 0 0 0
/proc/4315/statm: 838899 826772 612 1471 0 837419 0
Current children cumulated CPU time (s) 462.21
Current children cumulated vsize (KiB) 3355596

[startup+522.3 s]
/proc/loadavg: 2.09 2.04 2.01 3/177 4359
/proc/meminfo: memFree=24363620/32873648 swapFree=6864/6864
[pid=4315] ppid=4313 vsize=3355596 CPUtime=522.2 cores=1,3,5,7
/proc/4315/stat : 4315 (scip-2.1.1.4exp) R 4313 4315 1798 0 -1 4202496 826370 0 0 0 52057 163 0 0 20 0 1 0 270846556 3436130304 827020 33554432000 4194304 10219143 140736230415280 140736230410392 8969224 0 0 4096 16384 0 0 0 17 5 0 0 0 0 0
/proc/4315/statm: 838899 827020 612 1471 0 837419 0
Current children cumulated CPU time (s) 522.2
Current children cumulated vsize (KiB) 3355596

[startup+582.3 s]
/proc/loadavg: 2.03 2.03 2.00 3/177 4359
/proc/meminfo: memFree=24346092/32873648 swapFree=6864/6864
[pid=4315] ppid=4313 vsize=3356620 CPUtime=582.2 cores=1,3,5,7
/proc/4315/stat : 4315 (scip-2.1.1.4exp) R 4313 4315 1798 0 -1 4202496 826847 0 0 0 58056 164 0 0 20 0 1 0 270846556 3437178880 827663 33554432000 4194304 10219143 140736230415280 140736230410392 8969249 0 0 4096 16384 0 0 0 17 5 0 0 0 0 0
/proc/4315/statm: 839155 827663 612 1471 0 837675 0
Current children cumulated CPU time (s) 582.2
Current children cumulated vsize (KiB) 3356620

[startup+642.3 s]

################
# More data... #
################

/proc/4315/stat : 4315 (scip-2.1.1.4exp) R 4313 4315 1798 0 -1 4202496 831312 0 0 0 100049 165 0 0 20 0 1 0 270846556 3448844288 835848 33554432000 4194304 10219143 140736230415280 140736230410392 8969229 0 0 4096 16384 0 0 0 17 5 0 0 0 0 0
/proc/4315/statm: 842003 835848 612 1471 0 840523 0
Current children cumulated CPU time (s) 1002.14
Current children cumulated vsize (KiB) 3368012

[startup+1062.3 s]
/proc/loadavg: 2.00 2.03 2.00 3/177 4387
/proc/meminfo: memFree=24250016/32873648 swapFree=6864/6864
[pid=4315] ppid=4313 vsize=3370060 CPUtime=1062.12 cores=1,3,5,7
/proc/4315/stat : 4315 (scip-2.1.1.4exp) R 4313 4315 1798 0 -1 4202496 832082 0 0 0 106047 165 0 0 20 0 1 0 270846556 3450941440 836618 33554432000 4194304 10219143 140736230415280 140736230410392 8969239 0 0 4096 16384 0 0 0 17 5 0 0 0 0 0
/proc/4315/statm: 842515 836618 612 1471 0 841035 0
Current children cumulated CPU time (s) 1062.12
Current children cumulated vsize (KiB) 3370060

[startup+1122.3 s]
/proc/loadavg: 2.00 2.02 2.00 3/177 4387
/proc/meminfo: memFree=24244304/32873648 swapFree=6864/6864
[pid=4315] ppid=4313 vsize=3371084 CPUtime=1122.11 cores=1,3,5,7
/proc/4315/stat : 4315 (scip-2.1.1.4exp) R 4313 4315 1798 0 -1 4202496 832538 0 0 0 112046 165 0 0 20 0 1 0 270846556 3451990016 838052 33554432000 4194304 10219143 140736230415280 140736230410392 8969234 0 0 4096 16384 0 0 0 17 5 0 0 0 0 0
/proc/4315/statm: 842771 838052 612 1471 0 841291 0
Current children cumulated CPU time (s) 1122.11
Current children cumulated vsize (KiB) 3371084

[startup+1182.3 s]
/proc/loadavg: 2.00 2.02 2.00 3/177 4387
/proc/meminfo: memFree=24236236/32873648 swapFree=6864/6864
[pid=4315] ppid=4313 vsize=3372940 CPUtime=1182.1 cores=1,3,5,7
/proc/4315/stat : 4315 (scip-2.1.1.4exp) R 4313 4315 1798 0 -1 4202496 833108 0 0 0 118045 165 0 0 20 0 1 0 270846556 3453890560 839944 33554432000 4194304 10219143 140736230415280 140736230410392 8969234 0 0 4096 16384 0 0 0 17 5 0 0 0 0 0
/proc/4315/statm: 843235 839944 612 1471 0 841755 0
Current children cumulated CPU time (s) 1182.1
Current children cumulated vsize (KiB) 3372940

[startup+1242.3 s]
/proc/loadavg: 2.05 2.03 2.00 3/177 4387
/proc/meminfo: memFree=24234180/32873648 swapFree=6864/6864
[pid=4315] ppid=4313 vsize=3373964 CPUtime=1242.09 cores=1,3,5,7
/proc/4315/stat : 4315 (scip-2.1.1.4exp) R 4313 4315 1798 0 -1 4202496 833393 0 0 0 124044 165 0 0 20 0 1 0 270846556 3454939136 840482 33554432000 4194304 10219143 140736230415280 140736230410392 8969224 0 0 4096 16384 0 0 0 17 5 0 0 0 0 0
/proc/4315/statm: 843491 840482 612 1471 0 842011 0
Current children cumulated CPU time (s) 1242.09
Current children cumulated vsize (KiB) 3373964

[startup+1302.3 s]
/proc/loadavg: 2.02 2.02 2.00 3/177 4388
/proc/meminfo: memFree=24232672/32873648 swapFree=6864/6864
[pid=4315] ppid=4313 vsize=3374988 CPUtime=1302.08 cores=1,3,5,7
/proc/4315/stat : 4315 (scip-2.1.1.4exp) R 4313 4315 1798 0 -1 4202496 833675 0 0 0 130043 165 0 0 20 0 1 0 270846556 3455987712 840764 33554432000 4194304 10219143 140736230415280 140736230410392 8969234 0 0 4096 16384 0 0 0 17 5 0 0 0 0 0
/proc/4315/statm: 843747 840764 612 1471 0 842267 0
Current children cumulated CPU time (s) 1302.08
Current children cumulated vsize (KiB) 3374988

[startup+1362.3 s]
/proc/loadavg: 2.00 2.02 2.00 3/177 4388
/proc/meminfo: memFree=24231092/32873648 swapFree=6864/6864
[pid=4315] ppid=4313 vsize=3376844 CPUtime=1362.08 cores=1,3,5,7
/proc/4315/stat : 4315 (scip-2.1.1.4exp) R 4313 4315 1798 0 -1 4202496 834173 0 0 0 136042 166 0 0 20 0 1 0 270846556 3457888256 841262 33554432000 4194304 10219143 140736230415280 140736230410392 8969229 0 0 4096 16384 0 0 0 17 5 0 0 0 0 0
/proc/4315/statm: 844211 841262 612 1471 0 842731 0
Current children cumulated CPU time (s) 1362.08
Current children cumulated vsize (KiB) 3376844

[startup+1422.3 s]
/proc/loadavg: 2.00 2.01 2.00 3/178 4391
/proc/meminfo: memFree=24228928/32873648 swapFree=6864/6864
[pid=4315] ppid=4313 vsize=3378892 CPUtime=1422.08 cores=1,3,5,7
/proc/4315/stat : 4315 (scip-2.1.1.4exp) R 4313 4315 1798 0 -1 4202496 834711 0 0 0 142042 166 0 0 20 0 1 0 270846556 3459985408 841800 33554432000 4194304 10219143 140736230415280 140736230410392 8969244 0 0 4096 16384 0 0 0 17 5 0 0 0 0 0
/proc/4315/statm: 844723 841800 612 1471 0 843243 0
Current children cumulated CPU time (s) 1422.08
Current children cumulated vsize (KiB) 3378892

[startup+1482.3 s]
/proc/loadavg: 2.00 2.00 2.00 3/177 4391
/proc/meminfo: memFree=24228964/32873648 swapFree=6864/6864
[pid=4315] ppid=4313 vsize=3378892 CPUtime=1482.06 cores=1,3,5,7
/proc/4315/stat : 4315 (scip-2.1.1.4exp) R 4313 4315 1798 0 -1 4202496 834736 0 0 0 148040 166 0 0 20 0 1 0 270846556 3459985408 841825 33554432000 4194304 10219143 140736230415280 140736230410392 8969234 0 0 4096 16384 0 0 0 17 5 0 0 0 0 0
/proc/4315/statm: 844723 841825 612 1471 0 843243 0
Current children cumulated CPU time (s) 1482.06
Current children cumulated vsize (KiB) 3378892

[startup+1542.31 s]
/proc/loadavg: 2.00 2.00 2.00 3/177 4391
/proc/meminfo: memFree=24225596/32873648 swapFree=6864/6864
[pid=4315] ppid=4313 vsize=3381772 CPUtime=1542.06 cores=1,3,5,7
/proc/4315/stat : 4315 (scip-2.1.1.4exp) R 4313 4315 1798 0 -1 4202496 835480 0 0 0 154040 166 0 0 20 0 1 0 270846556 3462934528 842644 33554432000 4194304 10219143 140736230415280 140736230410392 8969224 0 0 4096 16384 0 0 0 17 5 0 0 0 0 0
/proc/4315/statm: 845443 842644 612 1471 0 843963 0
Current children cumulated CPU time (s) 1542.06
Current children cumulated vsize (KiB) 3381772

[startup+1602.3 s]
/proc/loadavg: 2.00 2.00 2.00 3/176 4391
/proc/meminfo: memFree=24224708/32873648 swapFree=6864/6864
[pid=4315] ppid=4313 vsize=3382796 CPUtime=1602.04 cores=1,3,5,7
/proc/4315/stat : 4315 (scip-2.1.1.4exp) R 4313 4315 1798 0 -1 4202496 835762 0 0 0 160038 166 0 0 20 0 1 0 270846556 3463983104 842925 33554432000 4194304 10219143 140736230415280 140736230410392 8969229 0 0 4096 16384 0 0 0 17 5 0 0 0 0 0
/proc/4315/statm: 845699 842925 612 1471 0 844219 0
Current children cumulated CPU time (s) 1602.04
Current children cumulated vsize (KiB) 3382796

[startup+1662.3 s]
/proc/loadavg: 2.00 2.00 2.00 3/177 4392
/proc/meminfo: memFree=24172336/32873648 swapFree=6864/6864
[pid=4315] ppid=4313 vsize=3435740 CPUtime=1662.03 cores=1,3,5,7
/proc/4315/stat : 4315 (scip-2.1.1.4exp) R 4313 4315 1798 0 -1 4202496 848271 0 0 0 166035 168 0 0 20 0 1 0 270846556 3518197760 855946 33554432000 4194304 10219143 140736230415280 140736230410600 8969275 0 0 4096 16384 0 0 0 17 5 0 0 0 0 0
/proc/4315/statm: 858935 855946 612 1471 0 857455 0
Current children cumulated CPU time (s) 1662.03
Current children cumulated vsize (KiB) 3435740

[startup+1722.3 s]
/proc/loadavg: 2.00 2.00 2.00 3/177 4392
/proc/meminfo: memFree=23984368/32873648 swapFree=6864/6864
[pid=4315] ppid=4313 vsize=3629520 CPUtime=1722.02 cores=1,3,5,7
/proc/4315/stat : 4315 (scip-2.1.1.4exp) R 4313 4315 1798 0 -1 4202496 891551 0 0 0 172028 174 0 0 20 0 1 0 270846556 3716628480 902805 33554432000 4194304 10219143 140736230415280 140736230410440 8969260 0 0 4096 16384 0 0 0 17 5 0 0 0 0 0
/proc/4315/statm: 907380 902805 612 1471 0 905900 0
Current children cumulated CPU time (s) 1722.02
Current children cumulated vsize (KiB) 3629520

[startup+1782.3 s]
/proc/loadavg: 2.00 2.00 2.00 3/177 4392
/proc/meminfo: memFree=23980068/32873648 swapFree=6864/6864
[pid=4315] ppid=4313 vsize=3631568 CPUtime=1782.01 cores=1,3,5,7
/proc/4315/stat : 4315 (scip-2.1.1.4exp) R 4313 4315 1798 0 -1 4202496 892280 0 0 0 178027 174 0 0 20 0 1 0 270846556 3718725632 903840 33554432000 4194304 10219143 140736230415280 140736230410392 8969234 0 0 4096 16384 0 0 0 17 5 0 0 0 0 0
/proc/4315/statm: 907892 903840 612 1471 0 906412 0
Current children cumulated CPU time (s) 1782.01
Current children cumulated vsize (KiB) 3631568



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1800.3 s]
/proc/loadavg: 2.00 2.00 2.00 3/177 4392
/proc/meminfo: memFree=23979664/32873648 swapFree=6864/6864
[pid=4315] ppid=4313 vsize=3631568 CPUtime=1800.01 cores=1,3,5,7
/proc/4315/stat : 4315 (scip-2.1.1.4exp) R 4313 4315 1798 0 -1 4202496 892346 0 0 0 179827 174 0 0 20 0 1 0 270846556 3718725632 903906 33554432000 4194304 10219143 140736230415280 140736230410392 8969224 0 0 4096 16384 0 0 0 17 5 0 0 0 0 0
/proc/4315/statm: 907892 903906 612 1471 0 906412 0
Current children cumulated CPU time (s) 1800.01
Current children cumulated vsize (KiB) 3631568

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Sending SIGKILL to process tree (bottom up)

Sending SIGKILL to -4315

Solver just ended. Dumping a history of the last processes samples

[startup+1801.5 s]
/proc/loadavg: 2.00 2.00 2.00 3/178 4393
/proc/meminfo: memFree=23979664/32873648 swapFree=6864/6864
[pid=4315] ppid=4313 vsize=3631568 CPUtime=1801.2 cores=1,3,5,7
/proc/4315/stat : 4315 (scip-2.1.1.4exp) R 4313 4315 1798 0 -1 4202496 892352 0 0 0 179946 174 0 0 20 0 1 0 270846556 3718725632 903912 33554432000 4194304 10219143 140736230415280 140736230409856 8969229 0 0 4096 16384 0 0 0 17 5 0 0 0 0 0
/proc/4315/statm: 907892 903912 612 1471 0 906412 0
Current children cumulated CPU time (s) 1801.2
Current children cumulated vsize (KiB) 3631568

[startup+1801.91 s]
/proc/loadavg: 2.00 2.00 2.00 3/178 4393
/proc/meminfo: memFree=23979656/32873648 swapFree=6864/6864
[pid=4315] ppid=4313 vsize=3631568 CPUtime=1801.61 cores=1,3,5,7
/proc/4315/stat : 4315 (scip-2.1.1.4exp) R 4313 4315 1798 0 -1 4202496 892352 0 0 0 179987 174 0 0 20 0 1 0 270846556 3718725632 903912 33554432000 4194304 10219143 140736230415280 140736230409856 8969270 0 0 4096 16384 0 0 0 17 5 0 0 0 0 0
/proc/4315/statm: 907892 903912 612 1471 0 906412 0
Current children cumulated CPU time (s) 1801.61
Current children cumulated vsize (KiB) 3631568

[startup+1802.3 s]
/proc/loadavg: 2.00 2.00 2.00 3/178 4393
/proc/meminfo: memFree=23979656/32873648 swapFree=6864/6864
[pid=4315] ppid=4313 vsize=3631568 CPUtime=1802 cores=1,3,5,7
/proc/4315/stat : 4315 (scip-2.1.1.4exp) R 4313 4315 1798 0 -1 4202496 892352 0 0 0 180026 174 0 0 20 0 1 0 270846556 3718725632 903912 33554432000 4194304 10219143 140736230415280 140736230409856 8969244 0 0 4096 16384 0 0 0 17 5 0 0 0 0 0
/proc/4315/statm: 907892 903912 612 1471 0 906412 0
Current children cumulated CPU time (s) 1802
Current children cumulated vsize (KiB) 3631568

Child ended because it received signal 9 (SIGKILL)
Real time (s): 1802.39
CPU time (s): 1802.11
CPU user time (s): 1800.27
CPU system time (s): 1.83072
CPU usage (%): 99.9841
Max. virtual memory (cumulated for all children) (KiB): 3631568

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1800.27
system time used= 1.83072
maximum resident set size= 3615648
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 892352
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 22
involuntary context switches= 1922

runsolver used 3.13452 second user time and 8.5487 second system time

The end

Launcher Data 

Begin job on node120 at 2012-05-26 17:46:46
IDJOB=3692548
IDBENCH=91122
IDSOLVER=2271
FILE ID=node120/3692548-1338047205
RUNJOBID= node120-1338029422-2919
PBS_JOBID= 14624351
Free space on /tmp= 71580 MiB

SOLVER NAME= SCIP spx E SCIP 2.1.1.4. Exp with SoPlex 1.6.0.3 fixed
BENCH NAME= PB12/normalized-PB12/OPT-SMALLINT-LIN/sroussel/NG/normalized-NG.dot_luby_20null10mast_rapportOE1.0_TMAX180_K192_cod2.opb
COMMAND LINE= HOME/scip-2.1.1.4expfix.linux.x86_64.gnu.opt.spx -f BENCHNAME -t TIMEOUT -m MEMLIMIT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3692548-1338047205/watcher-3692548-1338047205 -o /tmp/evaluation-result-3692548-1338047205/solver-3692548-1338047205 -C 1800 -W 1900 -M 15500  HOME/scip-2.1.1.4expfix.linux.x86_64.gnu.opt.spx -f HOME/instance-3692548-1338047205.opb -t 1800 -m 15500

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= d26c1c744a6cc6b94bc88bd85095e6b9
RANDOM SEED=1874295519

node120.alineos.net Linux 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.456
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.456
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.456
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.92
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.456
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.88
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.456
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.456
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.456
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.456
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32873648 kB
MemFree:        26307548 kB
Buffers:          230700 kB
Cached:          3498612 kB
SwapCached:          492 kB
Active:          2721496 kB
Inactive:        3072696 kB
Active(anon):    2055144 kB
Inactive(anon):    12388 kB
Active(file):     666352 kB
Inactive(file):  3060308 kB
Unevictable:        6864 kB
Mlocked:            6864 kB
SwapTotal:      67108856 kB
SwapFree:       67107896 kB
Dirty:             41128 kB
Writeback:             0 kB
AnonPages:       2134888 kB
Mapped:            16688 kB
Shmem:               120 kB
Slab:             631632 kB
SReclaimable:      76044 kB
SUnreclaim:       555588 kB
KernelStack:        1448 kB
PageTables:         7976 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83545680 kB
Committed_AS:    2187024 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      346400 kB
VmallocChunk:   34341988052 kB
HardwareCorrupted:     0 kB
AnonHugePages:    997376 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        7680 kB
DirectMap2M:    33538048 kB

Free space on /tmp at the end= 71560 MiB
End job on node120 at 2012-05-26 18:16:48