Trace number 3693718

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerobjective functionCPU timeWall clock time
SCIP spx standard SCIP 2.1.1.4. with SoPlex 1.6.0.3 standard fixed? (TO) 1802.13 1802.41

General information on the benchmark

Name/OPT-SMALLINT-LIN/sroussel/NG/
normalized-NG.dot_luby_50null50mast_rapportOE1.0_TMAX180_K192_cod2.opb
MD5SUM0d3fc2c8f241373f1d8055b65975d8e6
Bench CategoryOPT-SMALLINT-LIN (optimisation, small integers, linear constraints)
Best result obtained on this benchmarkSAT
Best value of the objective obtained on this benchmark-12288
Best CPU time to get the best result obtained on this benchmark1800.71
Has Objective FunctionYES
Satisfiable
(Un)Satisfiability was proved
Best value of the objective function
Optimality of the best value was proved
Number of variables400896
Total number of constraints402763
Number of constraints which are clauses400785
Number of constraints which are cardinality constraints (but not clauses)194
Number of constraints which are nor clauses,nor cardinality constraints1784
Minimum length of a constraint2
Maximum length of a constraint58023
Number of terms in the objective function 892
Biggest coefficient in the objective function 1024
Number of bits for the biggest coefficient in the objective function 11
Sum of the numbers in the objective function 912385
Number of bits of the sum of numbers in the objective function 20
Biggest number in a constraint 1024
Number of bits of the biggest number in a constraint 11
Biggest sum of numbers in a constraint 912385
Number of bits of the biggest sum of numbers20
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.00/0.00	c SCIP version 2.1.1.4 [precision: 8 byte] [memory: block] [mode: optimized] [LP solver: SoPlex 1.6.0.3] [GitHash: a3bf3a4-dirty]
0.00/0.00	c Copyright (c) 2002-2012 Konrad-Zuse-Zentrum fuer Informationstechnik Berlin (ZIB)
0.00/0.00	c 
0.00/0.00	c user parameter file <scip.set> not found - using default parameters
0.00/0.00	c reading problem <HOME/instance-3693718-1338047228.opb>
2.09/2.11	c original problem has 400896 variables (400896 bin, 0 int, 0 impl, 0 cont) and 402763 constraints
2.09/2.11	c problem read in 2.11
4.88/4.91	c presolving:
10.29/10.32	c (round 1) 0 del vars, 1977 del conss, 0 add conss, 0 chg bounds, 0 chg sides, 0 chg coeffs, 398809 upgd conss, 67806 impls, 1977 clqs
11.89/11.96	c (round 2) 0 del vars, 1977 del conss, 0 add conss, 0 chg bounds, 0 chg sides, 0 chg coeffs, 400786 upgd conss, 67806 impls, 1977 clqs

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3693718-1338047228/watcher-3693718-1338047228 -o /tmp/evaluation-result-3693718-1338047228/solver-3693718-1338047228 -C 1800 -W 1900 -M 15500 HOME/scip-2.1.1.4sfix.linux.x86_64.gnu.opt.spx -f HOME/instance-3693718-1338047228.opb -t 1800 -m 15500 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1900 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 2.00 2.00 2.00 3/176 1534
/proc/meminfo: memFree=26197432/32873844 swapFree=7100/7100
[pid=1534] ppid=1532 vsize=10452 CPUtime=0 cores=0,2,4,6
/proc/1534/stat : 1534 (scip-2.1.1.4sfi) R 1532 1534 32069 0 -1 4202496 1345 0 0 0 0 0 0 0 20 0 1 0 270849819 10702848 1300 33554432000 4194304 10169799 140735600677552 140737341919944 7543003 0 0 4096 0 0 0 0 17 4 0 0 0 0 0
/proc/1534/statm: 2613 1300 403 1459 0 1145 0

[startup+0.023954 s]
/proc/loadavg: 2.00 2.00 2.00 3/176 1534
/proc/meminfo: memFree=26197432/32873844 swapFree=7100/7100
[pid=1534] ppid=1532 vsize=19592 CPUtime=0.01 cores=0,2,4,6
/proc/1534/stat : 1534 (scip-2.1.1.4sfi) R 1532 1534 32069 0 -1 4202496 3645 0 0 0 1 0 0 0 20 0 1 0 270849819 20062208 3600 33554432000 4194304 10169799 140735600677552 140737341919944 7139072 0 0 4096 0 0 0 0 17 4 0 0 0 0 0
/proc/1534/statm: 4898 3600 412 1459 0 3430 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 19592

[startup+0.100344 s]
/proc/loadavg: 2.00 2.00 2.00 3/176 1534
/proc/meminfo: memFree=26197432/32873844 swapFree=7100/7100
[pid=1534] ppid=1532 vsize=43772 CPUtime=0.09 cores=0,2,4,6
/proc/1534/stat : 1534 (scip-2.1.1.4sfi) R 1532 1534 32069 0 -1 4202496 9683 0 0 0 8 1 0 0 20 0 1 0 270849819 44822528 9638 33554432000 4194304 10169799 140735600677552 140737341919944 7139415 0 0 4096 0 0 0 0 17 4 0 0 0 0 0
/proc/1534/statm: 10943 9638 412 1459 0 9475 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 43772

[startup+0.300327 s]
/proc/loadavg: 2.00 2.00 2.00 3/176 1534
/proc/meminfo: memFree=26197432/32873844 swapFree=7100/7100
[pid=1534] ppid=1532 vsize=85116 CPUtime=0.29 cores=0,2,4,6
/proc/1534/stat : 1534 (scip-2.1.1.4sfi) R 1532 1534 32069 0 -1 4202496 20010 0 0 0 26 3 0 0 20 0 1 0 270849819 87158784 19965 33554432000 4194304 10169799 140735600677552 140737341919944 7168613 0 0 4096 0 0 0 0 17 4 0 0 0 0 0
/proc/1534/statm: 21279 19965 412 1459 0 19811 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 85116

[startup+0.700331 s]
/proc/loadavg: 2.00 2.00 2.00 3/176 1534
/proc/meminfo: memFree=26197432/32873844 swapFree=7100/7100
[pid=1534] ppid=1532 vsize=190276 CPUtime=0.69 cores=0,2,4,6
/proc/1534/stat : 1534 (scip-2.1.1.4sfi) R 1532 1534 32069 0 -1 4202496 46247 0 0 0 63 6 0 0 20 0 1 0 270849819 194842624 46202 33554432000 4194304 10169799 140735600677552 140735600543880 5282286 0 0 4096 0 0 0 0 17 4 0 0 0 0 0
/proc/1534/statm: 47569 46202 412 1459 0 46101 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 190276

[startup+1.50038 s]
/proc/loadavg: 2.00 2.00 2.00 3/177 1535
/proc/meminfo: memFree=25938892/32873844 swapFree=7100/7100
[pid=1534] ppid=1532 vsize=383796 CPUtime=1.48 cores=0,2,4,6
/proc/1534/stat : 1534 (scip-2.1.1.4sfi) R 1532 1534 32069 0 -1 4202496 94521 0 0 0 135 13 0 0 20 0 1 0 270849819 393007104 94476 33554432000 4194304 10169799 140735600677552 140735600543880 9046144 0 0 4096 0 0 0 0 17 4 0 0 0 0 0
/proc/1534/statm: 95949 94476 412 1459 0 94481 0
Current children cumulated CPU time (s) 1.48
Current children cumulated vsize (KiB) 383796

[startup+3.10032 s]
/proc/loadavg: 2.00 2.00 2.00 3/177 1535
/proc/meminfo: memFree=25765896/32873844 swapFree=7100/7100
[pid=1534] ppid=1532 vsize=935656 CPUtime=3.08 cores=0,2,4,6
/proc/1534/stat : 1534 (scip-2.1.1.4sfi) R 1532 1534 32069 0 -1 4202496 231456 0 0 0 278 30 0 0 20 0 1 0 270849819 958111744 232381 33554432000 4194304 10169799 140735600677552 140735600674536 5757840 0 0 4096 16384 0 0 0 17 4 0 0 0 0 0
/proc/1534/statm: 233914 232381 489 1459 0 232446 0
Current children cumulated CPU time (s) 3.08
Current children cumulated vsize (KiB) 935656

[startup+6.30031 s]
/proc/loadavg: 2.08 2.02 2.01 3/177 1535
/proc/meminfo: memFree=24966996/32873844 swapFree=7100/7100
[pid=1534] ppid=1532 vsize=1243196 CPUtime=6.28 cores=0,2,4,6
/proc/1534/stat : 1534 (scip-2.1.1.4sfi) R 1532 1534 32069 0 -1 4202496 330852 0 0 0 519 109 0 0 20 0 1 0 270849819 1273032704 308080 33554432000 4194304 10169799 140735600677552 140735600675704 8961143 0 0 4096 16384 0 0 0 17 4 0 0 0 0 0
/proc/1534/statm: 310799 308080 545 1459 0 309331 0
Current children cumulated CPU time (s) 6.28
Current children cumulated vsize (KiB) 1243196

[startup+12.7003 s]
/proc/loadavg: 2.07 2.02 2.00 3/177 1535
/proc/meminfo: memFree=24583472/32873844 swapFree=7100/7100
[pid=1534] ppid=1532 vsize=1624348 CPUtime=12.69 cores=0,2,4,6
/proc/1534/stat : 1534 (scip-2.1.1.4sfi) R 1532 1534 32069 0 -1 4202496 462644 0 0 0 951 318 0 0 20 0 1 0 270849819 1663332352 402861 33554432000 4194304 10169799 140735600677552 140735600674760 8961143 0 0 4096 16384 0 0 0 17 4 0 0 0 0 0
/proc/1534/statm: 406087 402861 583 1459 0 404619 0
Current children cumulated CPU time (s) 12.69
Current children cumulated vsize (KiB) 1624348

[startup+25.5004 s]
/proc/loadavg: 2.06 2.01 2.00 3/176 1535
/proc/meminfo: memFree=24307652/32873844 swapFree=7100/7100
[pid=1534] ppid=1532 vsize=1904536 CPUtime=25.48 cores=0,2,4,6
/proc/1534/stat : 1534 (scip-2.1.1.4sfi) R 1532 1534 32069 0 -1 4202496 509749 0 0 0 2199 349 0 0 20 0 1 0 270849819 1950244864 471939 33554432000 4194304 10169799 140735600677552 140735600674200 8922290 0 0 4096 16384 0 0 0 17 4 0 0 0 0 0
/proc/1534/statm: 476134 471939 608 1459 0 474666 0
Current children cumulated CPU time (s) 25.48
Current children cumulated vsize (KiB) 1904536

[startup+51.1003 s]
/proc/loadavg: 2.04 2.01 2.00 3/177 1536
/proc/meminfo: memFree=24297752/32873844 swapFree=7100/7100
[pid=1534] ppid=1532 vsize=1914392 CPUtime=51.09 cores=0,2,4,6
/proc/1534/stat : 1534 (scip-2.1.1.4sfi) R 1532 1534 32069 0 -1 4202496 511688 0 0 0 4759 350 0 0 20 0 1 0 270849819 1960337408 474389 33554432000 4194304 10169799 140735600677552 140735600674104 8922300 0 0 4096 16384 0 0 0 17 4 0 0 0 0 0
/proc/1534/statm: 478598 474389 608 1459 0 477130 0
Current children cumulated CPU time (s) 51.09
Current children cumulated vsize (KiB) 1914392

[startup+102.306 s]
/proc/loadavg: 2.01 2.01 2.00 3/177 1536
/proc/meminfo: memFree=24272500/32873844 swapFree=7100/7100
[pid=1534] ppid=1532 vsize=1940232 CPUtime=102.28 cores=0,2,4,6
/proc/1534/stat : 1534 (scip-2.1.1.4sfi) R 1532 1534 32069 0 -1 4202496 518437 0 0 0 9877 351 0 0 20 0 1 0 270849819 1986797568 480577 33554432000 4194304 10169799 140735600677552 140735600674024 8922290 0 0 4096 16384 0 0 0 17 4 0 0 0 0 0
/proc/1534/statm: 485058 480577 608 1459 0 483590 0
Current children cumulated CPU time (s) 102.28
Current children cumulated vsize (KiB) 1940232

[startup+162.3 s]
/proc/loadavg: 2.00 2.00 2.00 3/177 1536
/proc/meminfo: memFree=22847448/32873844 swapFree=7100/7100
[pid=1534] ppid=1532 vsize=3308036 CPUtime=162.26 cores=0,2,4,6
/proc/1534/stat : 1534 (scip-2.1.1.4sfi) R 1532 1534 32069 0 -1 4202496 833378 0 0 0 15834 392 0 0 20 0 1 0 270849819 3387428864 813419 33554432000 4194304 10169799 140735600677552 140735600673816 8922300 0 0 4096 16384 0 0 0 17 4 0 0 0 0 0
/proc/1534/statm: 827009 813419 608 1459 0 825541 0
Current children cumulated CPU time (s) 162.26
Current children cumulated vsize (KiB) 3308036

[startup+222.3 s]
/proc/loadavg: 2.00 2.00 2.00 3/178 1542
/proc/meminfo: memFree=22845236/32873844 swapFree=7100/7100
[pid=1534] ppid=1532 vsize=3309064 CPUtime=222.26 cores=0,2,4,6
/proc/1534/stat : 1534 (scip-2.1.1.4sfi) R 1532 1534 32069 0 -1 4202496 833842 0 0 0 21833 393 0 0 20 0 1 0 270849819 3388481536 813883 33554432000 4194304 10169799 140735600677552 140735600673816 8922280 0 0 4096 16384 0 0 0 17 4 0 0 0 0 0
/proc/1534/statm: 827266 813883 608 1459 0 825798 0
Current children cumulated CPU time (s) 222.26
Current children cumulated vsize (KiB) 3309064

[startup+282.3 s]
/proc/loadavg: 2.00 2.00 2.00 3/177 1542
/proc/meminfo: memFree=22842296/32873844 swapFree=7100/7100
[pid=1534] ppid=1532 vsize=3310920 CPUtime=282.25 cores=0,2,4,6
/proc/1534/stat : 1534 (scip-2.1.1.4sfi) R 1532 1534 32069 0 -1 4202496 834513 0 0 0 27832 393 0 0 20 0 1 0 270849819 3390382080 814554 33554432000 4194304 10169799 140735600677552 140735600673816 8922290 0 0 4096 16384 0 0 0 17 4 0 0 0 0 0
/proc/1534/statm: 827730 814554 608 1459 0 826262 0
Current children cumulated CPU time (s) 282.25
Current children cumulated vsize (KiB) 3310920

[startup+342.3 s]
/proc/loadavg: 2.00 2.00 2.00 3/177 1543
/proc/meminfo: memFree=22838976/32873844 swapFree=7100/7100
[pid=1534] ppid=1532 vsize=3312964 CPUtime=342.24 cores=0,2,4,6
/proc/1534/stat : 1534 (scip-2.1.1.4sfi) R 1532 1534 32069 0 -1 4202496 835236 0 0 0 33831 393 0 0 20 0 1 0 270849819 3392475136 815277 33554432000 4194304 10169799 140735600677552 140735600673816 8922285 0 0 4096 16384 0 0 0 17 4 0 0 0 0 0
/proc/1534/statm: 828241 815277 608 1459 0 826773 0
Current children cumulated CPU time (s) 342.24
Current children cumulated vsize (KiB) 3312964

[startup+402.3 s]
/proc/loadavg: 2.00 2.00 2.00 3/177 1543
/proc/meminfo: memFree=22838136/32873844 swapFree=7100/7100
[pid=1534] ppid=1532 vsize=3312964 CPUtime=402.22 cores=0,2,4,6
/proc/1534/stat : 1534 (scip-2.1.1.4sfi) R 1532 1534 32069 0 -1 4202496 835454 0 0 0 39829 393 0 0 20 0 1 0 270849819 3392475136 815495 33554432000 4194304 10169799 140735600677552 140735600673816 8922310 0 0 4096 16384 0 0 0 17 4 0 0 0 0 0
/proc/1534/statm: 828241 815495 608 1459 0 826773 0
Current children cumulated CPU time (s) 402.22
Current children cumulated vsize (KiB) 3312964

[startup+462.3 s]
/proc/loadavg: 2.00 2.00 2.00 3/177 1543
/proc/meminfo: memFree=22834568/32873844 swapFree=7100/7100
[pid=1534] ppid=1532 vsize=3314880 CPUtime=462.22 cores=0,2,4,6
/proc/1534/stat : 1534 (scip-2.1.1.4sfi) R 1532 1534 32069 0 -1 4202496 836152 0 0 0 45828 394 0 0 20 0 1 0 270849819 3394437120 816160 33554432000 4194304 10169799 140735600677552 140735600673816 8922290 0 0 4096 16384 0 0 0 17 4 0 0 0 0 0
/proc/1534/statm: 828720 816160 608 1459 0 827252 0
Current children cumulated CPU time (s) 462.22
Current children cumulated vsize (KiB) 3314880

[startup+522.3 s]
/proc/loadavg: 2.04 2.01 2.00 3/177 1543
/proc/meminfo: memFree=22829636/32873844 swapFree=7100/7100
[pid=1534] ppid=1532 vsize=3315712 CPUtime=522.21 cores=0,2,4,6
/proc/1534/stat : 1534 (scip-2.1.1.4sfi) R 1532 1534 32069 0 -1 4202496 836575 0 0 0 51827 394 0 0 20 0 1 0 270849819 3395289088 816583 33554432000 4194304 10169799 140735600677552 140735600673816 8922331 0 0 4096 16384 0 0 0 17 4 0 0 0 0 0
/proc/1534/statm: 828928 816583 608 1459 0 827460 0
Current children cumulated CPU time (s) 522.21
Current children cumulated vsize (KiB) 3315712

[startup+582.3 s]
/proc/loadavg: 2.12 2.04 2.01 3/177 1543
/proc/meminfo: memFree=22823092/32873844 swapFree=7100/7100
[pid=1534] ppid=1532 vsize=3315712 CPUtime=582.2 cores=0,2,4,6
/proc/1534/stat : 1534 (scip-2.1.1.4sfi) R 1532 1534 32069 0 -1 4202496 836791 0 0 0 57826 394 0 0 20 0 1 0 270849819 3395289088 816799 33554432000 4194304 10169799 140735600677552 140735600673816 8922310 0 0 4096 16384 0 0 0 17 4 0 0 0 0 0
/proc/1534/statm: 828928 816799 608 1459 0 827460 0
Current children cumulated CPU time (s) 582.2
Current children cumulated vsize (KiB) 3315712

[startup+642.3 s]

################
# More data... #
################

/proc/1534/stat : 1534 (scip-2.1.1.4sfi) R 1532 1534 32069 0 -1 4202496 840650 0 0 0 99819 395 0 0 20 0 1 0 270849819 3404783616 820589 33554432000 4194304 10169799 140735600677552 140735600673816 8922290 0 0 4096 16384 0 0 0 17 4 0 0 0 0 0
/proc/1534/statm: 831246 820589 608 1459 0 829778 0
Current children cumulated CPU time (s) 1002.14
Current children cumulated vsize (KiB) 3324984

[startup+1062.3 s]
/proc/loadavg: 2.03 2.03 2.00 3/177 1575
/proc/meminfo: memFree=22776864/32873844 swapFree=7100/7100
[pid=1534] ppid=1532 vsize=3326008 CPUtime=1062.13 cores=0,2,4,6
/proc/1534/stat : 1534 (scip-2.1.1.4sfi) R 1532 1534 32069 0 -1 4202496 841134 0 0 0 105818 395 0 0 20 0 1 0 270849819 3405832192 821073 33554432000 4194304 10169799 140735600677552 140735600673816 8922290 0 0 4096 16384 0 0 0 17 4 0 0 0 0 0
/proc/1534/statm: 831502 821073 608 1459 0 830034 0
Current children cumulated CPU time (s) 1062.13
Current children cumulated vsize (KiB) 3326008

[startup+1122.3 s]
/proc/loadavg: 2.05 2.03 2.00 3/177 1575
/proc/meminfo: memFree=22774288/32873844 swapFree=7100/7100
[pid=1534] ppid=1532 vsize=3327864 CPUtime=1122.11 cores=0,2,4,6
/proc/1534/stat : 1534 (scip-2.1.1.4sfi) R 1532 1534 32069 0 -1 4202496 841798 0 0 0 111816 395 0 0 20 0 1 0 270849819 3407732736 821735 33554432000 4194304 10169799 140735600677552 140735600673816 8922305 0 0 4096 16384 0 0 0 17 4 0 0 0 0 0
/proc/1534/statm: 831966 821735 608 1459 0 830498 0
Current children cumulated CPU time (s) 1122.11
Current children cumulated vsize (KiB) 3327864

[startup+1182.3 s]
/proc/loadavg: 2.06 2.04 2.00 3/177 1575
/proc/meminfo: memFree=22771216/32873844 swapFree=7100/7100
[pid=1534] ppid=1532 vsize=3329912 CPUtime=1182.1 cores=0,2,4,6
/proc/1534/stat : 1534 (scip-2.1.1.4sfi) R 1532 1534 32069 0 -1 4202496 842532 0 0 0 117815 395 0 0 20 0 1 0 270849819 3409829888 822469 33554432000 4194304 10169799 140735600677552 140735600673816 8922280 0 0 4096 16384 0 0 0 17 4 0 0 0 0 0
/proc/1534/statm: 832478 822469 608 1459 0 831010 0
Current children cumulated CPU time (s) 1182.1
Current children cumulated vsize (KiB) 3329912

[startup+1242.3 s]
/proc/loadavg: 2.06 2.04 2.00 3/177 1617
/proc/meminfo: memFree=24629184/32873844 swapFree=7100/7100
[pid=1534] ppid=1532 vsize=3330936 CPUtime=1242.09 cores=0,2,4,6
/proc/1534/stat : 1534 (scip-2.1.1.4sfi) R 1532 1534 32069 0 -1 4202496 843014 0 0 0 123814 395 0 0 20 0 1 0 270849819 3410878464 822951 33554432000 4194304 10169799 140735600677552 140735600673816 8922280 0 0 4096 16384 0 0 0 17 4 0 0 0 0 0
/proc/1534/statm: 832734 822951 608 1459 0 831266 0
Current children cumulated CPU time (s) 1242.09
Current children cumulated vsize (KiB) 3330936

[startup+1302.3 s]
/proc/loadavg: 2.02 2.03 2.00 3/177 1618
/proc/meminfo: memFree=24465808/32873844 swapFree=7100/7100
[pid=1534] ppid=1532 vsize=3332792 CPUtime=1302.09 cores=0,2,4,6
/proc/1534/stat : 1534 (scip-2.1.1.4sfi) R 1532 1534 32069 0 -1 4202496 843708 0 0 0 129814 395 0 0 20 0 1 0 270849819 3412779008 823810 33554432000 4194304 10169799 140735600677552 140735600673816 8922290 0 0 4096 16384 0 0 0 17 4 0 0 0 0 0
/proc/1534/statm: 833198 823810 608 1459 0 831730 0
Current children cumulated CPU time (s) 1302.09
Current children cumulated vsize (KiB) 3332792

[startup+1362.3 s]
/proc/loadavg: 2.01 2.02 2.00 3/177 1618
/proc/meminfo: memFree=24444124/32873844 swapFree=7100/7100
[pid=1534] ppid=1532 vsize=3332660 CPUtime=1362.07 cores=0,2,4,6
/proc/1534/stat : 1534 (scip-2.1.1.4sfi) R 1532 1534 32069 0 -1 4202496 843942 0 0 0 135812 395 0 0 20 0 1 0 270849819 3412643840 824011 33554432000 4194304 10169799 140735600677552 140735600674136 8922310 0 0 4096 16384 0 0 0 17 4 0 0 0 0 0
/proc/1534/statm: 833165 824011 608 1459 0 831697 0
Current children cumulated CPU time (s) 1362.07
Current children cumulated vsize (KiB) 3332660

[startup+1422.31 s]
/proc/loadavg: 2.04 2.03 2.00 3/178 1621
/proc/meminfo: memFree=24457364/32873844 swapFree=7100/7100
[pid=1534] ppid=1532 vsize=3334708 CPUtime=1422.07 cores=0,2,4,6
/proc/1534/stat : 1534 (scip-2.1.1.4sfi) R 1532 1534 32069 0 -1 4202496 844694 0 0 0 141812 395 0 0 20 0 1 0 270849819 3414740992 827400 33554432000 4194304 10169799 140735600677552 140735600673816 8922310 0 0 4096 16384 0 0 0 17 4 0 0 0 0 0
/proc/1534/statm: 833677 827400 608 1459 0 832209 0
Current children cumulated CPU time (s) 1422.07
Current children cumulated vsize (KiB) 3334708

[startup+1482.3 s]
/proc/loadavg: 2.01 2.02 2.00 3/177 1621
/proc/meminfo: memFree=24436376/32873844 swapFree=7100/7100
[pid=1534] ppid=1532 vsize=3335732 CPUtime=1482.05 cores=0,2,4,6
/proc/1534/stat : 1534 (scip-2.1.1.4sfi) R 1532 1534 32069 0 -1 4202496 845183 0 0 0 147810 395 0 0 20 0 1 0 270849819 3415789568 828452 33554432000 4194304 10169799 140735600677552 140735600673816 8922316 0 0 4096 16384 0 0 0 17 4 0 0 0 0 0
/proc/1534/statm: 833933 828452 608 1459 0 832465 0
Current children cumulated CPU time (s) 1482.05
Current children cumulated vsize (KiB) 3335732

[startup+1542.3 s]
/proc/loadavg: 2.08 2.05 2.01 3/177 1621
/proc/meminfo: memFree=24425552/32873844 swapFree=7100/7100
[pid=1534] ppid=1532 vsize=3337588 CPUtime=1542.05 cores=0,2,4,6
/proc/1534/stat : 1534 (scip-2.1.1.4sfi) R 1532 1534 32069 0 -1 4202496 845815 0 0 0 153809 396 0 0 20 0 1 0 270849819 3417690112 829398 33554432000 4194304 10169799 140735600677552 140735600673816 8922280 0 0 4096 16384 0 0 0 17 4 0 0 0 0 0
/proc/1534/statm: 834397 829398 608 1459 0 832929 0
Current children cumulated CPU time (s) 1542.05
Current children cumulated vsize (KiB) 3337588

[startup+1602.3 s]
/proc/loadavg: 2.07 2.05 2.01 3/177 1622
/proc/meminfo: memFree=24422728/32873844 swapFree=7100/7100
[pid=1534] ppid=1532 vsize=3338612 CPUtime=1602.04 cores=0,2,4,6
/proc/1534/stat : 1534 (scip-2.1.1.4sfi) R 1532 1534 32069 0 -1 4202496 846311 0 0 0 159808 396 0 0 20 0 1 0 270849819 3418738688 830094 33554432000 4194304 10169799 140735600677552 140735600673816 8922316 0 0 4096 16384 0 0 0 17 4 0 0 0 0 0
/proc/1534/statm: 834653 830094 608 1459 0 833185 0
Current children cumulated CPU time (s) 1602.04
Current children cumulated vsize (KiB) 3338612

[startup+1662.3 s]
/proc/loadavg: 2.02 2.04 2.00 3/177 1622
/proc/meminfo: memFree=24420276/32873844 swapFree=7100/7100
[pid=1534] ppid=1532 vsize=3338612 CPUtime=1662.03 cores=0,2,4,6
/proc/1534/stat : 1534 (scip-2.1.1.4sfi) R 1532 1534 32069 0 -1 4202496 846519 0 0 0 165807 396 0 0 20 0 1 0 270849819 3418738688 830302 33554432000 4194304 10169799 140735600677552 140735600673816 8922310 0 0 4096 16384 0 0 0 17 4 0 0 0 0 0
/proc/1534/statm: 834653 830302 608 1459 0 833185 0
Current children cumulated CPU time (s) 1662.03
Current children cumulated vsize (KiB) 3338612

[startup+1722.3 s]
/proc/loadavg: 2.05 2.04 2.00 3/177 1622
/proc/meminfo: memFree=24416460/32873844 swapFree=7100/7100
[pid=1534] ppid=1532 vsize=3341492 CPUtime=1722.02 cores=0,2,4,6
/proc/1534/stat : 1534 (scip-2.1.1.4sfi) R 1532 1534 32069 0 -1 4202496 847470 0 0 0 171806 396 0 0 20 0 1 0 270849819 3421687808 831253 33554432000 4194304 10169799 140735600677552 140735600673816 8922280 0 0 4096 16384 0 0 0 17 4 0 0 0 0 0
/proc/1534/statm: 835373 831253 608 1459 0 833905 0
Current children cumulated CPU time (s) 1722.02
Current children cumulated vsize (KiB) 3341492

[startup+1782.3 s]
/proc/loadavg: 2.02 2.03 2.00 3/177 1622
/proc/meminfo: memFree=24414140/32873844 swapFree=7100/7100
[pid=1534] ppid=1532 vsize=3342516 CPUtime=1782.01 cores=0,2,4,6
/proc/1534/stat : 1534 (scip-2.1.1.4sfi) R 1532 1534 32069 0 -1 4202496 847966 0 0 0 177805 396 0 0 20 0 1 0 270849819 3422736384 831765 33554432000 4194304 10169799 140735600677552 140735600673816 8922310 0 0 4096 16384 0 0 0 17 4 0 0 0 0 0
/proc/1534/statm: 835629 831765 608 1459 0 834161 0
Current children cumulated CPU time (s) 1782.01
Current children cumulated vsize (KiB) 3342516



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1800.3 s]
/proc/loadavg: 2.01 2.03 2.00 3/177 1622
/proc/meminfo: memFree=24413752/32873844 swapFree=7100/7100
[pid=1534] ppid=1532 vsize=3342516 CPUtime=1800.01 cores=0,2,4,6
/proc/1534/stat : 1534 (scip-2.1.1.4sfi) R 1532 1534 32069 0 -1 4202496 848009 0 0 0 179605 396 0 0 20 0 1 0 270849819 3422736384 831896 33554432000 4194304 10169799 140735600677552 140735600673816 8922310 0 0 4096 16384 0 0 0 17 4 0 0 0 0 0
/proc/1534/statm: 835629 831896 608 1459 0 834161 0
Current children cumulated CPU time (s) 1800.01
Current children cumulated vsize (KiB) 3342516

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Sending SIGKILL to process tree (bottom up)

Sending SIGKILL to -1534

Solver just ended. Dumping a history of the last processes samples

[startup+1801.4 s]
/proc/loadavg: 2.01 2.03 2.00 3/178 1623
/proc/meminfo: memFree=24413744/32873844 swapFree=7100/7100
[pid=1534] ppid=1532 vsize=3342516 CPUtime=1801.1 cores=0,2,4,6
/proc/1534/stat : 1534 (scip-2.1.1.4sfi) R 1532 1534 32069 0 -1 4202496 848009 0 0 0 179714 396 0 0 20 0 1 0 270849819 3422736384 831896 33554432000 4194304 10169799 140735600677552 140735600673280 8922295 0 0 4096 16384 0 0 0 17 4 0 0 0 0 0
/proc/1534/statm: 835629 831896 608 1459 0 834161 0
Current children cumulated CPU time (s) 1801.1
Current children cumulated vsize (KiB) 3342516

[startup+1802.2 s]
/proc/loadavg: 2.01 2.03 2.00 3/178 1623
/proc/meminfo: memFree=24413744/32873844 swapFree=7100/7100
[pid=1534] ppid=1532 vsize=3342516 CPUtime=1801.9 cores=0,2,4,6
/proc/1534/stat : 1534 (scip-2.1.1.4sfi) R 1532 1534 32069 0 -1 4202496 848009 0 0 0 179794 396 0 0 20 0 1 0 270849819 3422736384 831896 33554432000 4194304 10169799 140735600677552 140735600673280 8922295 0 0 4096 16384 0 0 0 17 4 0 0 0 0 0
/proc/1534/statm: 835629 831896 608 1459 0 834161 0
Current children cumulated CPU time (s) 1801.9
Current children cumulated vsize (KiB) 3342516

[startup+1802.4 s]
/proc/loadavg: 2.01 2.03 2.00 3/178 1623
/proc/meminfo: memFree=24413744/32873844 swapFree=7100/7100
[pid=1534] ppid=1532 vsize=0 CPUtime=1802.1 cores=0,2,4,6
/proc/1534/stat : 1534 (scip-2.1.1.4sfi) R 1532 1534 32069 0 -1 4203524 848009 0 0 0 179805 405 0 0 20 0 1 0 270849819 0 0 33554432000 0 0 0 0 0 0 0 4096 16384 18446744073709551615 0 0 17 4 0 0 0 0 0
/proc/1534/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 1802.1
Current children cumulated vsize (KiB) 0

Child ended because it received signal 9 (SIGKILL)
Real time (s): 1802.41
CPU time (s): 1802.13
CPU user time (s): 1798.06
CPU system time (s): 4.07238
CPU usage (%): 99.9842
Max. virtual memory (cumulated for all children) (KiB): 3342516

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1798.06
system time used= 4.07238
maximum resident set size= 3327584
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 848009
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 16
involuntary context switches= 1855

runsolver used 3.2815 second user time and 9.70952 second system time

The end

Launcher Data 

Begin job on node111 at 2012-05-26 17:47:08
IDJOB=3693718
IDBENCH=90929
IDSOLVER=2272
FILE ID=node111/3693718-1338047228
RUNJOBID= node111-1338029237-410
PBS_JOBID= 14624360
Free space on /tmp= 71532 MiB

SOLVER NAME= SCIP spx standard SCIP 2.1.1.4. with SoPlex 1.6.0.3 standard fixed
BENCH NAME= PB12/normalized-PB12/OPT-SMALLINT-LIN/sroussel/NG/normalized-NG.dot_luby_50null50mast_rapportOE1.0_TMAX180_K192_cod2.opb
COMMAND LINE= HOME/scip-2.1.1.4sfix.linux.x86_64.gnu.opt.spx -f BENCHNAME -t TIMEOUT -m MEMLIMIT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3693718-1338047228/watcher-3693718-1338047228 -o /tmp/evaluation-result-3693718-1338047228/solver-3693718-1338047228 -C 1800 -W 1900 -M 15500  HOME/scip-2.1.1.4sfix.linux.x86_64.gnu.opt.spx -f HOME/instance-3693718-1338047228.opb -t 1800 -m 15500

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 0d3fc2c8f241373f1d8055b65975d8e6
RANDOM SEED=542707505

node111.alineos.net Linux 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.152
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.30
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.152
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.152
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.92
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.152
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.152
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.152
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.152
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.152
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.90
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32873844 kB
MemFree:        26197696 kB
Buffers:          248936 kB
Cached:          3396072 kB
SwapCached:            0 kB
Active:          3128360 kB
Inactive:        2767932 kB
Active(anon):    2254040 kB
Inactive(anon):        0 kB
Active(file):     874320 kB
Inactive(file):  2767932 kB
Unevictable:        7100 kB
Mlocked:            7100 kB
SwapTotal:      67108856 kB
SwapFree:       67108856 kB
Dirty:             41228 kB
Writeback:             0 kB
AnonPages:       2262808 kB
Mapped:            17672 kB
Shmem:               200 kB
Slab:             636376 kB
SReclaimable:      78856 kB
SUnreclaim:       557520 kB
KernelStack:        1448 kB
PageTables:         8544 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83545776 kB
Committed_AS:    2410816 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      346368 kB
VmallocChunk:   34341987984 kB
HardwareCorrupted:     0 kB
AnonHugePages:    792576 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        7872 kB
DirectMap2M:    33538048 kB

Free space on /tmp at the end= 71512 MiB
End job on node111 at 2012-05-26 18:17:11