Trace number 3740825

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
PB11: borg pb-dec-11.04.03? (MO) 362.55 502.867

General information on the benchmark

Name/DEC-SMALLINT-LIN/lopes/
normalized-61.opb
MD5SUM242cdc00579036e1487a0255b4e1d84c
Bench CategoryDEC-SMALLINT-LIN (no optimisation, small integers, linear constraints)
Best result obtained on this benchmark
Best value of the objective obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Has Objective FunctionNO
Satisfiable
(Un)Satisfiability was proved
Best value of the objective function
Optimality of the best value was proved
Number of variables10086
Total number of constraints226240
Number of constraints which are clauses203302
Number of constraints which are cardinality constraints (but not clauses)720
Number of constraints which are nor clauses,nor cardinality constraints22218
Minimum length of a constraint2
Maximum length of a constraint231
Number of terms in the objective function 0
Biggest coefficient in the objective function 0
Number of bits for the biggest coefficient in the objective function 0
Sum of the numbers in the objective function 0
Number of bits of the sum of numbers in the objective function 0
Biggest number in a constraint 3
Number of bits of the biggest number in a constraint 2
Biggest sum of numbers in a constraint 232
Number of bits of the biggest sum of numbers8
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.37/0.50	c INFO: loaded portfolio from HOME/etc/borg-mix+class.dec.pickle
0.37/0.50	c INFO: solving HOME/instance-3740825-1338719614.opb
1.87/2.08	c INFO: parsing took 1.54 s
4.67/4.89	c INFO: features took 2.79 CPU seconds
4.77/4.91	c INFO: running wbo1.4b-fixed@0 for 45 with 1795 remaining (b = 0.96)
54.52/54.65	c INFO: running pbct-0.1.2-linear@0 for 45 with 1746 remaining (b = 0.96)
102.49/102.51	c INFO: running wbo1.4a@0 for 45 with 1699 remaining (b = 0.90)
165.32/165.33	c INFO: wrote minion input file to HOME/tmpKNxHxM.minion
165.32/165.33	c INFO: running minion-0.12@0 for 45 with 1650 remaining (b = 0.70)
173.66/316.94	c INFO: running clasp-1.3.7@0 for 45 with 1632 remaining (b = 0.57)
220.82/364.10	c INFO: running bsolo_pb10-l2@0 for 45 with 1585 remaining (b = 0.47)
268.96/412.16	c INFO: running clasp-1.3.7@47 for 45 with 1537 remaining (b = 0.18)
314.23/457.30	c INFO: running sat4j-pb-v20101225-cutting@0 for 45 with 1492 remaining (b = 0.22)
362.45/502.72	c INFO: running sat4j-pb-v20101225@0 for 45 with 1445 remaining (b = 0.16)

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3740825-1338719614/watcher-3740825-1338719614 -o /tmp/evaluation-result-3740825-1338719614/solver-3740825-1338719614 -C 1800 -W 1900 -M 15500 python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3740825-1338719614.opb -seed 438690073 -budget 1800 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1900 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 2.15 2.06 2.04 3/176 7459
/proc/meminfo: memFree=28554260/32873648 swapFree=7628/7628
[pid=7459] ppid=7457 vsize=14840 CPUtime=0 cores=1,3,5,7
/proc/7459/stat : 7459 (python) R 7457 7459 3958 0 -1 4202496 345 0 0 0 0 0 0 0 20 0 1 0 338087485 15196160 268 33554432000 4194304 5683644 140735268569296 140736576013080 4681492 0 0 4096 0 0 0 0 17 5 0 0 0 0 0
/proc/7459/statm: 3710 272 212 364 0 152 0

[startup+0.100255 s]
/proc/loadavg: 2.15 2.06 2.04 3/176 7459
/proc/meminfo: memFree=28554260/32873648 swapFree=7628/7628
[pid=7459] ppid=7457 vsize=92388 CPUtime=0.06 cores=1,3,5,7
/proc/7459/stat : 7459 (python) R 7457 7459 3958 0 -1 4202496 3219 504 1 0 5 1 0 0 20 0 1 0 338087485 94605312 2540 33554432000 4194304 5683644 140735268569296 140735268521768 217410141472 0 0 16781312 2 0 0 0 17 5 0 0 0 0 0
/proc/7459/statm: 23097 2540 816 364 0 1619 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 92388

[startup+0.200412 s]
/proc/loadavg: 2.15 2.06 2.04 3/176 7459
/proc/meminfo: memFree=28554260/32873648 swapFree=7628/7628
[pid=7459] ppid=7457 vsize=121772 CPUtime=0.13 cores=1,3,5,7
/proc/7459/stat : 7459 (python) R 7457 7459 3958 0 -1 4202496 5783 1008 1 0 11 2 0 0 20 0 1 0 338087485 124694528 3727 33554432000 4194304 5683644 140735268569296 140735268487256 4964362 0 0 16781312 2 0 0 0 17 5 0 0 0 0 0
/proc/7459/statm: 30443 3727 1014 364 0 2686 0
Current children cumulated CPU time (s) 0.13
Current children cumulated vsize (KiB) 121772

[startup+0.300805 s]
/proc/loadavg: 2.15 2.06 2.04 3/176 7459
/proc/meminfo: memFree=28554260/32873648 swapFree=7628/7628
[pid=7459] ppid=7457 vsize=166940 CPUtime=0.21 cores=1,3,5,7
/proc/7459/stat : 7459 (python) R 7457 7459 3958 0 -1 4202496 6577 1008 1 0 16 5 0 0 20 0 1 0 338087485 170946560 4367 33554432000 4194304 5683644 140735268569296 140735268473528 140214232294616 0 0 16781312 2 0 0 0 17 5 0 0 0 0 0
/proc/7459/statm: 41735 4367 1243 364 0 3288 0
Current children cumulated CPU time (s) 0.21
Current children cumulated vsize (KiB) 166940

[startup+0.700168 s]
/proc/loadavg: 2.15 2.06 2.04 3/176 7459
/proc/meminfo: memFree=28554260/32873648 swapFree=7628/7628
[pid=7459] ppid=7457 vsize=297212 CPUtime=0.57 cores=1,3,5,7
/proc/7459/stat : 7459 (python) R 7457 7459 3958 0 -1 4202496 12418 1008 1 0 48 9 0 0 20 0 1 0 338087485 304345088 10986 33554432000 4194304 5683644 140735268569296 140735268562792 4445115 0 0 16781312 2 0 0 0 17 5 0 0 0 0 0
/proc/7459/statm: 74303 10986 1811 364 0 9532 0
Current children cumulated CPU time (s) 0.57
Current children cumulated vsize (KiB) 297212

[startup+1.50011 s]
/proc/loadavg: 2.15 2.06 2.04 3/177 7464
/proc/meminfo: memFree=28468188/32873648 swapFree=7628/7628
[pid=7459] ppid=7457 vsize=324236 CPUtime=1.37 cores=1,3,5,7
/proc/7459/stat : 7459 (python) R 7457 7459 3958 0 -1 4202496 19019 1008 1 0 126 11 0 0 20 0 1 0 338087485 332017664 17587 33554432000 4194304 5683644 140735268569296 140735268561912 217410146196 0 0 16781312 2 0 0 0 17 5 0 0 0 0 0
/proc/7459/statm: 81059 17587 1811 364 0 16288 0
Current children cumulated CPU time (s) 1.37
Current children cumulated vsize (KiB) 324236

[startup+3.1001 s]
/proc/loadavg: 2.14 2.06 2.04 3/177 7464
/proc/meminfo: memFree=28388804/32873648 swapFree=7628/7628
[pid=7459] ppid=7457 vsize=341924 CPUtime=2.97 cores=1,3,5,7
/proc/7459/stat : 7459 (python) R 7457 7459 3958 0 -1 4202496 27921 1008 1 0 283 14 0 0 20 0 1 0 338087485 350130176 22233 33554432000 4194304 5683644 140735268569296 140735268556120 140214232604734 0 0 16781312 33554434 0 0 0 17 5 0 0 0 0 0
/proc/7459/statm: 85481 22233 1846 364 0 20710 0
Current children cumulated CPU time (s) 2.97
Current children cumulated vsize (KiB) 341924

[startup+6.30028 s]
/proc/loadavg: 2.14 2.06 2.04 3/180 7467
/proc/meminfo: memFree=28252284/32873648 swapFree=7628/7628
[pid=7459] ppid=7457 vsize=423028 CPUtime=4.79 cores=1,3,5,7
/proc/7459/stat : 7459 (python) S 7457 7459 3958 0 -1 4202496 29536 1008 1 0 465 14 0 0 20 0 2 0 338087485 433180672 23620 33554432000 4194304 5683644 140735268569296 140735268564128 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/7459/statm: 105757 23620 1902 364 0 40980 0
[pid=7465] ppid=7459 vsize=347248 CPUtime=0 cores=1,3,5,7
/proc/7465/stat : 7465 (python) S 7459 7459 3958 0 -1 4202560 756 0 0 0 0 0 0 0 20 0 1 0 338087977 355581952 22052 33554432000 4194304 5683644 140735268569296 140735268560384 217410535848 0 0 16781312 33554946 18446744071580425449 0 0 17 7 0 0 0 0 0
/proc/7465/statm: 86812 22052 340 364 0 22035 0
[pid=7467] ppid=7465 vsize=26788 CPUtime=1.37 cores=1,3,5,7
/proc/7467/stat : 7467 (wbo1.4b-fixed) R 7465 7467 7467 0 -1 4202496 8876 0 0 0 132 5 0 0 20 0 1 0 338087977 27430912 6078 33554432000 134512640 135752315 4292156272 4292154348 135270849 0 0 16781312 3 0 0 0 17 3 0 0 0 0 0
/proc/7467/statm: 6697 6078 146 303 0 6392 0
[pid=7459/tid=7466] ppid=7457 vsize=423028 CPUtime=0 cores=1,3,5,7
/proc/7459/task/7466/stat : 7466 (python) S 7457 7459 3958 0 -1 4202560 27 1008 0 0 0 0 0 0 20 0 2 0 338087977 433180672 23620 33554432000 4194304 5683644 140735268569296 140214059237936 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
Current children cumulated CPU time (s) 6.16
Current children cumulated vsize (KiB) 797064

[startup+12.7004 s]
/proc/loadavg: 2.11 2.06 2.04 3/180 7467
/proc/meminfo: memFree=27987840/32873648 swapFree=7628/7628
[pid=7459] ppid=7457 vsize=423028 CPUtime=4.79 cores=1,3,5,7
/proc/7459/stat : 7459 (python) S 7457 7459 3958 0 -1 4202496 29536 1008 1 0 465 14 0 0 20 0 2 0 338087485 433180672 23620 33554432000 4194304 5683644 140735268569296 140735268564128 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/7459/statm: 105757 23620 1902 364 0 40980 0
[pid=7465] ppid=7459 vsize=347248 CPUtime=0 cores=1,3,5,7
/proc/7465/stat : 7465 (python) S 7459 7459 3958 0 -1 4202560 881 0 0 0 0 0 0 0 20 0 1 0 338087977 355581952 22075 33554432000 4194304 5683644 140735268569296 140735268560384 217410535848 0 0 16781312 33554946 18446744071580425449 0 0 17 1 0 0 0 0 0
/proc/7465/statm: 86812 22075 363 364 0 22035 0
[pid=7467] ppid=7465 vsize=32940 CPUtime=7.77 cores=1,3,5,7
/proc/7467/stat : 7467 (wbo1.4b-fixed) R 7465 7467 7467 0 -1 4202496 10434 0 0 0 747 30 0 0 20 0 1 0 338087977 33730560 7636 33554432000 134512640 135752315 4292156272 4292154348 135270849 0 0 16781312 3 0 0 0 17 3 0 0 0 0 0
/proc/7467/statm: 8235 7636 146 303 0 7930 0
[pid=7459/tid=7466] ppid=7457 vsize=423028 CPUtime=0 cores=1,3,5,7
/proc/7459/task/7466/stat : 7466 (python) S 7457 7459 3958 0 -1 4202560 27 1008 0 0 0 0 0 0 20 0 2 0 338087977 433180672 23620 33554432000 4194304 5683644 140735268569296 140214059237936 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
Current children cumulated CPU time (s) 12.56
Current children cumulated vsize (KiB) 803216

[startup+25.5067 s]
/proc/loadavg: 2.10 2.06 2.04 3/180 7467
/proc/meminfo: memFree=27975784/32873648 swapFree=7628/7628
[pid=7459] ppid=7457 vsize=423028 CPUtime=4.79 cores=1,3,5,7
/proc/7459/stat : 7459 (python) S 7457 7459 3958 0 -1 4202496 29536 1008 1 0 465 14 0 0 20 0 2 0 338087485 433180672 23620 33554432000 4194304 5683644 140735268569296 140735268564128 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/7459/statm: 105757 23620 1902 364 0 40980 0
[pid=7465] ppid=7459 vsize=347248 CPUtime=0.01 cores=1,3,5,7
/proc/7465/stat : 7465 (python) S 7459 7459 3958 0 -1 4202560 895 0 0 0 1 0 0 0 20 0 1 0 338087977 355581952 22075 33554432000 4194304 5683644 140735268569296 140735268560384 217410535848 0 0 16781312 33554946 18446744071580425449 0 0 17 1 0 0 0 0 0
/proc/7465/statm: 86812 22075 363 364 0 22035 0
[pid=7467] ppid=7465 vsize=45068 CPUtime=20.58 cores=1,3,5,7
/proc/7467/stat : 7467 (wbo1.4b-fixed) R 7465 7467 7467 0 -1 4202496 13410 0 0 0 2016 42 0 0 20 0 1 0 338087977 46149632 10612 33554432000 134512640 135752315 4292156272 4292154348 135270849 0 0 16781312 3 0 0 0 17 3 0 0 0 0 0
/proc/7467/statm: 11267 10612 146 303 0 10962 0
[pid=7459/tid=7466] ppid=7457 vsize=423028 CPUtime=0 cores=1,3,5,7
/proc/7459/task/7466/stat : 7466 (python) S 7457 7459 3958 0 -1 4202560 27 1008 0 0 0 0 0 0 20 0 2 0 338087977 433180672 23620 33554432000 4194304 5683644 140735268569296 140214059237936 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
Current children cumulated CPU time (s) 25.38
Current children cumulated vsize (KiB) 815344

[startup+51.1004 s]
/proc/loadavg: 2.12 2.07 2.05 3/180 7467
/proc/meminfo: memFree=27919696/32873648 swapFree=7628/7628
[pid=7459] ppid=7457 vsize=423028 CPUtime=4.79 cores=1,3,5,7
/proc/7459/stat : 7459 (python) S 7457 7459 3958 0 -1 4202496 29536 1008 1 0 465 14 0 0 20 0 2 0 338087485 433180672 23620 33554432000 4194304 5683644 140735268569296 140735268564128 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/7459/statm: 105757 23620 1902 364 0 40980 0
[pid=7465] ppid=7459 vsize=347248 CPUtime=0.06 cores=1,3,5,7
/proc/7465/stat : 7465 (python) S 7459 7459 3958 0 -1 4202560 1005 0 0 0 4 2 0 0 20 0 1 0 338087977 355581952 22080 33554432000 4194304 5683644 140735268569296 140735268560384 217410535848 0 0 16781312 33554946 18446744071580425449 0 0 17 1 0 0 0 0 0
/proc/7465/statm: 86812 22080 368 364 0 22035 0
[pid=7467] ppid=7465 vsize=58908 CPUtime=46.16 cores=1,3,5,7
/proc/7467/stat : 7467 (wbo1.4b-fixed) R 7465 7467 7467 0 -1 4202496 16785 0 0 0 4560 56 0 0 20 0 1 0 338087977 60321792 13987 33554432000 134512640 135752315 4292156272 4292154272 134611576 0 0 16781312 3 0 0 0 17 3 0 0 0 0 0
/proc/7467/statm: 14727 13987 146 303 0 14422 0
[pid=7459/tid=7466] ppid=7457 vsize=423028 CPUtime=0 cores=1,3,5,7
/proc/7459/task/7466/stat : 7466 (python) S 7457 7459 3958 0 -1 4202560 27 1008 0 0 0 0 0 0 20 0 2 0 338087977 433180672 23620 33554432000 4194304 5683644 140735268569296 140214059237936 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
Current children cumulated CPU time (s) 51.01
Current children cumulated vsize (KiB) 829184

[startup+102.3 s]
/proc/loadavg: 2.09 2.07 2.05 3/187 7474
/proc/meminfo: memFree=26848052/32873648 swapFree=7628/7628
[pid=7459] ppid=7457 vsize=498820 CPUtime=4.82 cores=1,3,5,7
/proc/7459/stat : 7459 (python) S 7457 7459 3958 0 -1 4202496 30473 1008 1 0 467 15 0 0 20 0 3 0 338087485 510791680 23629 33554432000 4194304 5683644 140735268569296 140735268564128 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/7459/statm: 124705 23629 1905 364 0 59925 0
[pid=7465] ppid=7459 vsize=423028 CPUtime=0.07 cores=1,3,5,7
/proc/7465/stat : 7465 (python) S 7459 7459 3958 0 -1 4202560 1135 0 0 0 4 3 0 0 20 0 2 0 338087977 433180672 22115 33554432000 4194304 5683644 140735268569296 140735268560944 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/7465/statm: 105757 22115 396 364 0 40980 0
[pid=7467] ppid=7465 vsize=60888 CPUtime=49.7 cores=1,3,5,7
/proc/7467/stat : 7467 (wbo1.4b-fixed) T 7465 7467 7467 0 -1 4202496 17308 0 0 0 4912 58 0 0 20 0 1 0 338087977 62349312 14510 33554432000 134512640 135752315 4292156272 4292154272 134611302 0 0 16781312 3 18446744071579387992 0 0 17 3 0 0 0 0 0
/proc/7467/statm: 15222 14510 146 303 0 14917 0
[pid=7465/tid=7468] ppid=7459 vsize=423028 CPUtime=0 cores=1,3,5,7
/proc/7465/task/7468/stat : 7468 (python) S 7459 7459 3958 0 -1 4202560 43 0 0 0 0 0 0 0 20 0 2 0 338092948 433180672 22115 33554432000 4194304 5683644 140735268569296 140214059237936 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=7469] ppid=7459 vsize=423040 CPUtime=0.06 cores=1,3,5,7
/proc/7469/stat : 7469 (python) S 7459 7459 3958 0 -1 4202560 913 0 0 0 3 3 0 0 20 0 1 0 338092950 433192960 22076 33554432000 4194304 5683644 140735268569296 140735268560384 217410535907 0 0 16781312 33554946 18446744071580425449 0 0 17 1 0 0 0 0 0
/proc/7469/statm: 105760 22076 358 364 0 40980 0
[pid=7471] ppid=7469 vsize=904360 CPUtime=47.63 cores=1,3,5,7
/proc/7471/stat : 7471 (pbct-0.1.2-linu) R 7469 7471 7471 0 -1 4202496 216155 0 0 0 4694 69 0 0 20 0 1 0 338092951 926064640 220196 33554432000 134512640 136422188 4286927856 4286925580 135874785 0 0 16781312 16387 0 0 0 17 3 0 0 0 0 0
/proc/7471/statm: 226090 220196 215 467 0 225621 0
[pid=7459/tid=7466] ppid=7457 vsize=498820 CPUtime=0 cores=1,3,5,7
/proc/7459/task/7466/stat : 7466 (python) S 7457 7459 3958 0 -1 4202560 27 1008 0 0 0 0 0 0 20 0 3 0 338087977 510791680 23629 33554432000 4194304 5683644 140735268569296 140214059237936 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=7459/tid=7470] ppid=7457 vsize=498820 CPUtime=0 cores=1,3,5,7
/proc/7459/task/7470/stat : 7470 (python) S 7457 7459 3958 0 -1 4202560 22 1008 0 0 0 0 0 0 20 0 3 0 338092951 510791680 23629 33554432000 4194304 5683644 140735268569296 140214048748080 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
Current children cumulated CPU time (s) 102.28
Current children cumulated vsize (KiB) 2310136

[startup+162.301 s]
/proc/loadavg: 2.03 2.05 2.04 3/195 7483
/proc/meminfo: memFree=26319328/32873648 swapFree=7628/7628
[pid=7459] ppid=7457 vsize=574616 CPUtime=15.56 cores=1,3,5,7
/proc/7459/stat : 7459 (python) R 7457 7459 3958 0 -1 4202496 31991 1008 1 0 1538 18 0 0 20 0 4 0 338087485 588406784 23638 33554432000 4194304 5683644 140735268569296 140735268563616 4507847 0 0 16781312 33554434 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/7459/statm: 143654 23638 1907 364 0 78871 0
[pid=7465] ppid=7459 vsize=423028 CPUtime=0.07 cores=1,3,5,7
/proc/7465/stat : 7465 (python) S 7459 7459 3958 0 -1 4202560 1135 0 0 0 4 3 0 0 20 0 2 0 338087977 433180672 22115 33554432000 4194304 5683644 140735268569296 140735268560944 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/7465/statm: 105757 22115 396 364 0 40980 0
[pid=7467] ppid=7465 vsize=60888 CPUtime=49.7 cores=1,3,5,7
/proc/7467/stat : 7467 (wbo1.4b-fixed) T 7465 7467 7467 0 -1 4202496 17308 0 0 0 4912 58 0 0 20 0 1 0 338087977 62349312 14510 33554432000 134512640 135752315 4292156272 4292154272 134611302 0 0 16781312 3 18446744071579387992 0 0 17 3 0 0 0 0 0
/proc/7467/statm: 15222 14510 146 303 0 14917 0
[pid=7465/tid=7468] ppid=7459 vsize=423028 CPUtime=0 cores=1,3,5,7
/proc/7465/task/7468/stat : 7468 (python) S 7459 7459 3958 0 -1 4202560 43 0 0 0 0 0 0 0 20 0 2 0 338092948 433180672 22115 33554432000 4194304 5683644 140735268569296 140214059237936 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=7469] ppid=7459 vsize=423040 CPUtime=0.07 cores=1,3,5,7
/proc/7469/stat : 7469 (python) S 7459 7459 3958 0 -1 4202560 1069 0 0 0 4 3 0 0 20 0 2 0 338092950 433192960 22102 33554432000 4194304 5683644 140735268569296 140735268560944 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/7469/statm: 105760 22102 383 364 0 40980 0
[pid=7471] ppid=7469 vsize=904360 CPUtime=47.82 cores=1,3,5,7
/proc/7471/stat : 7471 (pbct-0.1.2-linu) T 7469 7471 7471 0 -1 4202496 216155 0 0 0 4713 69 0 0 20 0 1 0 338092951 926064640 220196 33554432000 134512640 136422188 4286927856 4286924544 135029200 0 0 16781312 16387 18446744071579387992 0 0 17 3 0 0 0 0 0
/proc/7471/statm: 226090 220196 215 467 0 225621 0
[pid=7469/tid=7475] ppid=7459 vsize=423040 CPUtime=0 cores=1,3,5,7
/proc/7469/task/7475/stat : 7475 (python) S 7459 7459 3958 0 -1 4202560 48 0 0 0 0 0 0 0 20 0 2 0 338097734 433192960 22102 33554432000 4194304 5683644 140735268569296 140214059237936 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=7476] ppid=7459 vsize=498832 CPUtime=0.07 cores=1,3,5,7

################
# More data... #
################

Current children cumulated vsize (KiB) 25700328
heavy processes:

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+502.801 s]
/proc/loadavg: 2.18 2.68 2.39 5/247 7603
/proc/meminfo: memFree=18646172/32873648 swapFree=7628/7628
[pid=7459] ppid=7457 vsize=953572 CPUtime=27.06 cores=1,3,5,7
/proc/7459/stat : 7459 (python) S 7457 7459 3958 0 -1 4202496 36008 1039901 1 0 1853 20 449 384 20 0 9 0 338087485 976457728 24185 33554432000 4194304 5683644 140735268569296 140735268564128 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/7459/statm: 238393 24185 1917 364 0 173595 0
[pid=7465] ppid=7459 vsize=423028 CPUtime=0.07 cores=1,3,5,7
/proc/7465/stat : 7465 (python) S 7459 7459 3958 0 -1 4202560 1135 0 0 0 4 3 0 0 20 0 2 0 338087977 433180672 22115 33554432000 4194304 5683644 140735268569296 140735268560944 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/7465/statm: 105757 22115 396 364 0 40980 0
[pid=7467] ppid=7465 vsize=60888 CPUtime=49.7 cores=1,3,5,7
/proc/7467/stat : 7467 (wbo1.4b-fixed) T 7465 7467 7467 0 -1 4202496 17308 0 0 0 4912 58 0 0 20 0 1 0 338087977 62349312 14615 33554432000 134512640 135752315 4292156272 4292154272 134611302 0 0 16781312 3 18446744071579387992 0 0 17 3 0 0 0 0 0
/proc/7467/statm: 15222 14615 146 303 0 14917 0
[pid=7465/tid=7468] ppid=7459 vsize=423028 CPUtime=0 cores=1,3,5,7
/proc/7465/task/7468/stat : 7468 (python) S 7459 7459 3958 0 -1 4202560 43 0 0 0 0 0 0 0 20 0 2 0 338092948 433180672 22115 33554432000 4194304 5683644 140735268569296 140214059237936 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=7469] ppid=7459 vsize=423040 CPUtime=0.07 cores=1,3,5,7
/proc/7469/stat : 7469 (python) S 7459 7459 3958 0 -1 4202560 1069 0 0 0 4 3 0 0 20 0 2 0 338092950 433192960 22102 33554432000 4194304 5683644 140735268569296 140735268560944 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/7469/statm: 105760 22102 383 364 0 40980 0
[pid=7471] ppid=7469 vsize=904360 CPUtime=47.82 cores=1,3,5,7
/proc/7471/stat : 7471 (pbct-0.1.2-linu) T 7469 7471 7471 0 -1 4202496 216155 0 0 0 4713 69 0 0 20 0 1 0 338092951 926064640 220205 33554432000 134512640 136422188 4286927856 4286924544 135029200 0 0 16781312 16387 18446744071579387992 0 0 17 3 0 0 0 0 0
/proc/7471/statm: 226090 220205 215 467 0 225621 0
[pid=7469/tid=7475] ppid=7459 vsize=423040 CPUtime=0 cores=1,3,5,7
/proc/7469/task/7475/stat : 7475 (python) S 7459 7459 3958 0 -1 4202560 48 0 0 0 0 0 0 0 20 0 2 0 338097734 433192960 22102 33554432000 4194304 5683644 140735268569296 140214059237936 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=7476] ppid=7459 vsize=498832 CPUtime=0.07 cores=1,3,5,7
/proc/7476/stat : 7476 (python) S 7459 7459 3958 0 -1 4202560 1094 0 0 0 4 3 0 0 20 0 2 0 338097737 510803968 22108 33554432000 4194304 5683644 140735268569296 140735268560944 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/7476/statm: 124708 22108 383 364 0 59925 0
[pid=7478] ppid=7476 vsize=56724 CPUtime=49.03 cores=1,3,5,7
/proc/7478/stat : 7478 (wbo1.4a) T 7476 7478 7478 0 -1 4202496 16265 0 0 0 4851 52 0 0 20 0 1 0 338097737 58085376 13463 33554432000 134512640 135751879 4292598304 4292596480 134616350 0 0 16781312 3 18446744071579387992 0 0 17 3 0 0 0 0 0
/proc/7478/statm: 14181 13463 143 303 0 13876 0
[pid=7476/tid=7483] ppid=7459 vsize=498832 CPUtime=0 cores=1,3,5,7
/proc/7476/task/7483/stat : 7483 (python) S 7459 7459 3958 0 -1 4202560 44 0 0 0 0 0 0 0 20 0 2 0 338102643 510803968 22108 33554432000 4194304 5683644 140735268569296 140214048748080 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=7543] ppid=7459 vsize=650416 CPUtime=0.14 cores=1,3,5,7
/proc/7543/stat : 7543 (python) S 7459 7459 3958 0 -1 4202560 1140 0 0 0 7 7 0 0 20 0 2 0 338119179 666025984 22120 33554432000 4194304 5683644 140735268569296 140735268560944 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/7543/statm: 162604 22120 383 364 0 97815 0
[pid=7545] ppid=7543 vsize=54456 CPUtime=92.22 cores=1,3,5,7
/proc/7545/stat : 7545 (clasp-1.3.7-x86) T 7543 7545 7545 0 -1 4202496 14229 0 0 0 9219 3 0 0 20 0 1 0 338119180 55762944 13033 33554432000 134512640 136139019 4291162368 4291161232 134608261 0 0 16781312 16386 18446744071579387992 0 0 17 3 0 0 0 0 0
/proc/7545/statm: 13614 13033 183 398 0 13214 0
[pid=7543/tid=7546] ppid=7459 vsize=650416 CPUtime=0 cores=1,3,5,7
/proc/7543/task/7546/stat : 7546 (python) S 7459 7459 3958 0 -1 4202560 44 0 0 0 0 0 0 0 20 0 2 0 338123893 666025984 22120 33554432000 4194304 5683644 140735268569296 140214027768368 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=7547] ppid=7459 vsize=726208 CPUtime=0.07 cores=1,3,5,7
/proc/7547/stat : 7547 (python) S 7459 7459 3958 0 -1 4202560 1105 0 0 0 4 3 0 0 20 0 2 0 338123895 743636992 22126 33554432000 4194304 5683644 140735268569296 140735268560944 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/7547/statm: 181552 22126 383 364 0 116760 0
[pid=7549] ppid=7547 vsize=85632 CPUtime=48 cores=1,3,5,7
/proc/7549/stat : 7549 (bsolo_pb10) T 7547 7549 7549 0 -1 4202496 25006 0 0 0 4745 55 0 0 20 0 1 0 338123896 87687168 18936 33554432000 134512640 137138936 4293800816 4293799744 134542330 0 0 16781312 16384 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/7549/statm: 21408 18936 192 642 0 20764 0
[pid=7547/tid=7553] ppid=7459 vsize=726208 CPUtime=0 cores=1,3,5,7
/proc/7547/task/7553/stat : 7553 (python) S 7459 7459 3958 0 -1 4202560 44 0 0 0 0 0 0 0 20 0 2 0 338128699 743636992 22126 33554432000 4194304 5683644 140735268569296 140214017278512 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=7557] ppid=7459 vsize=802000 CPUtime=0.07 cores=1,3,5,7
/proc/7557/stat : 7557 (python) S 7459 7459 3958 0 -1 4202560 1144 0 0 0 4 3 0 0 20 0 2 0 338133215 821248000 22132 33554432000 4194304 5683644 140735268569296 140735268560944 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/7557/statm: 200500 22132 383 364 0 135705 0
[pid=7559] ppid=7557 vsize=9624972 CPUtime=48.15 cores=1,3,5,7
/proc/7559/stat : 7559 (java) T 7557 7559 7559 0 -1 4202496 24868 0 1 0 4752 63 0 0 20 0 19 0 338133216 9855971328 695098 33554432000 1073741824 1073778376 140734550142624 140734550133760 217413877821 0 0 0 16800975 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/7559/statm: 2406243 695098 2282 9 0 2396937 0
[pid=7559/tid=7560] ppid=7557 vsize=9624972 CPUtime=44.55 cores=1,3,5,7
/proc/7559/task/7560/stat : 7560 (java) T 7557 7559 7559 0 -1 4202560 4157 0 1 0 4412 43 0 0 20 0 19 0 338133217 9855971328 695098 33554432000 1073741824 1073778376 140734550142624 140336833670336 140336664565697 0 4 0 16800975 18446744071579387992 0 0 -1 5 0 0 0 0 0
[pid=7559/tid=7561] ppid=7557 vsize=9624972 CPUtime=0.14 cores=1,3,5,7
/proc/7559/task/7561/stat : 7561 (java) T 7557 7559 7559 0 -1 4202560 63 0 0 0 11 3 0 0 20 0 19 0 338133218 9855971328 695098 33554432000 1073741824 1073778376 140734550142624 140336795162792 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 1 0 0 0 0 0
[pid=7559/tid=7562] ppid=7557 vsize=9624972 CPUtime=0.09 cores=1,3,5,7
/proc/7559/task/7562/stat : 7562 (java) T 7557 7559 7559 0 -1 4202560 44 0 0 0 7 2 0 0 20 0 19 0 338133218 9855971328 695098 33554432000 1073741824 1073778376 140734550142624 140336794110248 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 5 0 0 0 0 0
[pid=7559/tid=7563] ppid=7557 vsize=9624972 CPUtime=0.06 cores=1,3,5,7
/proc/7559/task/7563/stat : 7563 (java) T 7557 7559 7559 0 -1 4202560 18 0 0 0 6 0 0 0 20 0 19 0 338133218 9855971328 695098 33554432000 1073741824 1073778376 140734550142624 140336793057704 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 7 0 0 0 0 0
[pid=7559/tid=7564] ppid=7557 vsize=9624972 CPUtime=0.13 cores=1,3,5,7
/proc/7559/task/7564/stat : 7564 (java) T 7557 7559 7559 0 -1 4202560 57 0 0 0 10 3 0 0 20 0 19 0 338133218 9855971328 695098 33554432000 1073741824 1073778376 140734550142624 140336792005160 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=7559/tid=7565] ppid=7557 vsize=9624972 CPUtime=0.09 cores=1,3,5,7
/proc/7559/task/7565/stat : 7565 (java) T 7557 7559 7559 0 -1 4202560 32 0 0 0 7 2 0 0 20 0 19 0 338133218 9855971328 695098 33554432000 1073741824 1073778376 140734550142624 140336790952616 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 7 0 0 0 0 0
[pid=7559/tid=7566] ppid=7557 vsize=9624972 CPUtime=0.13 cores=1,3,5,7
/proc/7559/task/7566/stat : 7566 (java) T 7557 7559 7559 0 -1 4202560 64 0 0 0 10 3 0 0 20 0 19 0 338133218 9855971328 695098 33554432000 1073741824 1073778376 140734550142624 140336789900072 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=7559/tid=7567] ppid=7557 vsize=9624972 CPUtime=0.12 cores=1,3,5,7
/proc/7559/task/7567/stat : 7567 (java) T 7557 7559 7559 0 -1 4202560 52 0 0 0 10 2 0 0 20 0 19 0 338133218 9855971328 695098 33554432000 1073741824 1073778376 140734550142624 140336788847528 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 7 0 0 0 0 0
[pid=7559/tid=7568] ppid=7557 vsize=9624972 CPUtime=0.12 cores=1,3,5,7
/proc/7559/task/7568/stat : 7568 (java) T 7557 7559 7559 0 -1 4202560 45 0 0 0 9 3 0 0 20 0 19 0 338133218 9855971328 695098 33554432000 1073741824 1073778376 140734550142624 140336787794984 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 5 0 0 0 0 0
[pid=7559/tid=7569] ppid=7557 vsize=9624972 CPUtime=0.53 cores=1,3,5,7
/proc/7559/task/7569/stat : 7569 (java) T 7557 7559 7559 0 -1 4202560 1310 0 0 0 52 1 0 0 20 0 19 0 338133219 9855971328 695098 33554432000 1073741824 1073778376 140734550142624 140336657565800 217413892009 0 0 0 16800975 18446744071579387992 0 0 -1 1 0 0 0 0 0
[pid=7559/tid=7570] ppid=7557 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/7559/task/7570/stat : 7570 (java) T 7557 7559 7559 0 -1 4202560 17 0 0 0 0 0 0 0 20 0 19 0 338133220 9855971328 695098 33554432000 1073741824 1073778376 140734550142624 140336656512120 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 7 0 0 0 0 0
[pid=7559/tid=7571] ppid=7557 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/7559/task/7571/stat : 7571 (java) T 7557 7559 7559 0 -1 4202560 8 0 0 0 0 0 0 0 20 0 19 0 338133220 9855971328 695098 33554432000 1073741824 1073778376 140734550142624 140336655459464 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=7559/tid=7572] ppid=7557 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/7559/task/7572/stat : 7572 (java) T 7557 7559 7559 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338133222 9855971328 695098 33554432000 1073741824 1073778376 140734550142624 140336654408208 217413900064 0 4 0 16800975 18446744071579387992 0 0 -1 7 0 0 0 0 0
[pid=7559/tid=7573] ppid=7557 vsize=9624972 CPUtime=0.95 cores=1,3,5,7
/proc/7559/task/7573/stat : 7573 (java) T 7557 7559 7559 0 -1 4202560 7969 0 0 0 95 0 0 0 20 0 19 0 338133222 9855971328 695098 33554432000 1073741824 1073778376 140734550142624 140336653355576 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 1 0 0 0 0 0
[pid=7559/tid=7574] ppid=7557 vsize=9624972 CPUtime=1.07 cores=1,3,5,7
/proc/7559/task/7574/stat : 7574 (java) T 7557 7559 7559 0 -1 4202560 9633 0 0 0 106 1 0 0 20 0 19 0 338133222 9855971328 695098 33554432000 1073741824 1073778376 140734550142624 140336652303032 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=7559/tid=7575] ppid=7557 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/7559/task/7575/stat : 7575 (java) T 7557 7559 7559 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338133222 9855971328 695098 33554432000 1073741824 1073778376 140734550142624 140336651250680 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 1 0 0 0 0 0
[pid=7559/tid=7576] ppid=7557 vsize=9624972 CPUtime=0.01 cores=1,3,5,7
/proc/7559/task/7576/stat : 7576 (java) T 7557 7559 7559 0 -1 4202560 3 0 0 0 0 1 0 0 20 0 19 0 338133222 9855971328 695098 33554432000 1073741824 1073778376 140734550142624 140336650198344 217413892009 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=7559/tid=7577] ppid=7557 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/7559/task/7577/stat : 7577 (java) T 7557 7559 7559 0 -1 4202560 12 0 0 0 0 0 0 0 20 0 19 0 338133384 9855971328 695098 33554432000 1073741824 1073778376 140734550142624 140336649143080 217413892009 0 4 0 16800975 18446744071579387992 0 0 -1 7 0 0 0 0 0
[pid=7557/tid=7583] ppid=7459 vsize=802000 CPUtime=0 cores=1,3,5,7
/proc/7557/task/7583/stat : 7583 (python) S 7459 7459 3958 0 -1 4202560 47 0 0 0 0 0 0 0 20 0 2 0 338137755 821248000 22132 33554432000 4194304 5683644 140735268569296 140213993464368 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=7584] ppid=7459 vsize=877792 CPUtime=0 cores=1,3,5,7
/proc/7584/stat : 7584 (python) S 7459 7459 3958 0 -1 4202560 848 0 0 0 0 0 0 0 20 0 1 0 338137758 898859008 22089 33554432000 4194304 5683644 140735268569296 140735268560384 217410535907 0 0 16781312 33554946 18446744071580425449 0 0 17 1 0 0 0 0 0
/proc/7584/statm: 219448 22089 335 364 0 154650 0
[pid=7586] ppid=7584 vsize=9558408 CPUtime=0.08 cores=1,3,5,7
/proc/7586/stat : 7586 (java) S 7584 7586 7586 0 -1 4202496 4487 0 1 0 8 0 0 0 20 0 18 0 338137758 9787809792 5423 33554432000 1073741824 1073778376 140734702894928 140734702886064 217413877821 0 0 0 16800975 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/7586/statm: 2389602 5423 2107 9 0 2380296 0
[pid=7586/tid=7587] ppid=7584 vsize=9558408 CPUtime=0.05 cores=1,3,5,7
/proc/7586/task/7587/stat : 7587 (java) R 7584 7586 7586 0 -1 4202560 2266 0 1 0 5 0 0 0 20 0 18 0 338137759 9787809792 5429 33554432000 1073741824 1073778376 140734702894928 139854109035440 217413903629 0 4 0 16800975 0 0 0 -1 5 0 0 0 0 0
[pid=7586/tid=7588] ppid=7584 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/7586/task/7588/stat : 7588 (java) S 7584 7586 7586 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338137760 9787809792 5430 33554432000 1073741824 1073778376 140734702894928 139854070528808 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=7586/tid=7589] ppid=7584 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/7586/task/7589/stat : 7589 (java) S 7584 7586 7586 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338137760 9787809792 5430 33554432000 1073741824 1073778376 140734702894928 139854069476008 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=7586/tid=7590] ppid=7584 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/7586/task/7590/stat : 7590 (java) S 7584 7586 7586 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338137760 9787809792 5430 33554432000 1073741824 1073778376 140734702894928 139854068423720 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=7586/tid=7591] ppid=7584 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/7586/task/7591/stat : 7591 (java) S 7584 7586 7586 0 -1 4202560 2 0 0 0 0 0 0 0 20 0 18 0 338137760 9787809792 5430 33554432000 1073741824 1073778376 140734702894928 139853949823912 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=7586/tid=7592] ppid=7584 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/7586/task/7592/stat : 7592 (java) S 7584 7586 7586 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338137760 9787809792 5430 33554432000 1073741824 1073778376 140734702894928 139853948770600 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=7586/tid=7593] ppid=7584 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/7586/task/7593/stat : 7593 (java) S 7584 7586 7586 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338137760 9787809792 5430 33554432000 1073741824 1073778376 140734702894928 139853947717800 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=7586/tid=7594] ppid=7584 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/7586/task/7594/stat : 7594 (java) S 7584 7586 7586 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338137760 9787809792 5430 33554432000 1073741824 1073778376 140734702894928 139853946665512 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=7586/tid=7595] ppid=7584 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/7586/task/7595/stat : 7595 (java) S 7584 7586 7586 0 -1 4202560 3 0 0 0 0 0 0 0 20 0 18 0 338137760 9787809792 5430 33554432000 1073741824 1073778376 140734702894928 139853945612712 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=7586/tid=7596] ppid=7584 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/7586/task/7596/stat : 7596 (java) S 7584 7586 7586 0 -1 4202560 2 0 0 0 0 0 0 0 20 0 18 0 338137761 9787809792 5430 33554432000 1073741824 1073778376 140734702894928 139853463284456 217413892009 0 0 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=7586/tid=7597] ppid=7584 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/7586/task/7597/stat : 7597 (java) S 7584 7586 7586 0 -1 4202560 16 0 0 0 0 0 0 0 20 0 18 0 338137761 9787809792 5430 33554432000 1073741824 1073778376 140734702894928 139853462230520 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=7586/tid=7598] ppid=7584 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/7586/task/7598/stat : 7598 (java) S 7584 7586 7586 0 -1 4202560 7 0 0 0 0 0 0 0 20 0 18 0 338137761 9787809792 5431 33554432000 1073741824 1073778376 140734702894928 139853461178120 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=7586/tid=7599] ppid=7584 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/7586/task/7599/stat : 7599 (java) S 7584 7586 7586 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338137763 9787809792 5431 33554432000 1073741824 1073778376 140734702894928 139853460126608 217413900064 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=7586/tid=7600] ppid=7584 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/7586/task/7600/stat : 7600 (java) S 7584 7586 7586 0 -1 4202560 294 0 0 0 0 0 0 0 20 0 18 0 338137763 9787809792 5432 33554432000 1073741824 1073778376 140734702894928 139853459073208 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=7586/tid=7601] ppid=7584 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/7586/task/7601/stat : 7601 (java) S 7584 7586 7586 0 -1 4202560 518 0 0 0 0 0 0 0 20 0 18 0 338137763 9787809792 5432 33554432000 1073741824 1073778376 140734702894928 139853458020408 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=7586/tid=7602] ppid=7584 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/7586/task/7602/stat : 7602 (java) S 7584 7586 7586 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338137763 9787809792 5432 33554432000 1073741824 1073778376 140734702894928 139853456968312 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=7586/tid=7603] ppid=7584 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/7586/task/7603/stat : 7603 (java) S 7584 7586 7586 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338137763 9787809792 5432 33554432000 1073741824 1073778376 140734702894928 139853455915720 217413892009 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=7459/tid=7466] ppid=7457 vsize=953572 CPUtime=8.33 cores=1,3,5,7
/proc/7459/task/7466/stat : 7466 (python) S 7457 7459 3958 0 -1 4202560 27 1039901 0 0 0 0 449 384 20 0 9 0 338087977 976457728 24185 33554432000 4194304 5683644 140735268569296 140214059237936 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=7459/tid=7470] ppid=7457 vsize=953572 CPUtime=8.33 cores=1,3,5,7
/proc/7459/task/7470/stat : 7470 (python) S 7457 7459 3958 0 -1 4202560 22 1039901 0 0 0 0 449 384 20 0 9 0 338092951 976457728 24185 33554432000 4194304 5683644 140735268569296 140214048748080 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=7459/tid=7477] ppid=7457 vsize=953572 CPUtime=8.33 cores=1,3,5,7
/proc/7459/task/7477/stat : 7477 (python) S 7457 7459 3958 0 -1 4202560 22 1039901 0 0 0 0 449 384 20 0 9 0 338097737 976457728 24185 33554432000 4194304 5683644 140735268569296 140214038258224 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=7459/tid=7486] ppid=7457 vsize=953572 CPUtime=8.33 cores=1,3,5,7
/proc/7459/task/7486/stat : 7486 (python) S 7457 7459 3958 0 -1 4202560 22 1039901 0 0 0 0 449 384 20 0 9 0 338104019 976457728 24185 33554432000 4194304 5683644 140735268569296 140214027768368 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=7459/tid=7544] ppid=7457 vsize=953572 CPUtime=8.33 cores=1,3,5,7
/proc/7459/task/7544/stat : 7544 (python) S 7457 7459 3958 0 -1 4202560 24 1039901 0 0 0 0 449 384 20 0 9 0 338119180 976457728 24185 33554432000 4194304 5683644 140735268569296 140214017278512 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=7459/tid=7548] ppid=7457 vsize=953572 CPUtime=8.33 cores=1,3,5,7
/proc/7459/task/7548/stat : 7548 (python) S 7457 7459 3958 0 -1 4202560 21 1039901 0 0 0 0 449 384 20 0 9 0 338123895 976457728 24185 33554432000 4194304 5683644 140735268569296 140213993464368 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=7459/tid=7558] ppid=7457 vsize=953572 CPUtime=8.33 cores=1,3,5,7
/proc/7459/task/7558/stat : 7558 (python) S 7457 7459 3958 0 -1 4202560 22 1039901 0 0 0 0 449 384 20 0 9 0 338133216 976457728 24185 33554432000 4194304 5683644 140735268569296 140213982974512 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=7459/tid=7585] ppid=7457 vsize=953572 CPUtime=8.33 cores=1,3,5,7
/proc/7459/task/7585/stat : 7585 (python) S 7457 7459 3958 0 -1 4202560 19 1039901 0 0 0 0 449 384 20 0 9 0 338137758 976457728 24185 33554432000 4194304 5683644 140735268569296 140213502337584 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
Current children cumulated CPU time (s) 362.55
Current children cumulated vsize (KiB) 25700328

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 7459 and gives
#  childrusage.ru_utime.tv_sec=23
#  childrusage.ru_utime.tv_usec=37497
#  childrusage.ru_stime.tv_sec=4
#  childrusage.ru_stime.tv_usec=53383
# CPU time returned by wait4() is 27.0909
# while last known CPU time is 362.55
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 502.867
CPU time (s): 362.55
CPU user time (s): 355.29
CPU system time (s): 7.26
CPU usage (%): 72.0966
Max. virtual memory (cumulated for all children) (KiB): 25700328

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 23.0375
system time used= 4.05338
maximum resident set size= 4160132
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1075909
page faults= 1
swaps= 0
block input operations= 0
block output operations= 28608
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 1031049
involuntary context switches= 87

runsolver used 1.73174 second user time and 4.39333 second system time

The end

Launcher Data 

Begin job on node122 at 2012-06-03 12:33:34
IDJOB=3740825
IDBENCH=89634
IDSOLVER=2327
FILE ID=node122/3740825-1338719614
RUNJOBID= node122-1338713392-3975
PBS_JOBID= 14636732
Free space on /tmp= 70820 MiB

SOLVER NAME= PB11: borg pb-dec-11.04.03
BENCH NAME= PB11/normalized-PB11/DEC-SMALLINT-LIN/lopes/normalized-61.opb
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.dec.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3740825-1338719614/watcher-3740825-1338719614 -o /tmp/evaluation-result-3740825-1338719614/solver-3740825-1338719614 -C 1800 -W 1900 -M 15500  python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3740825-1338719614.opb -seed 438690073 -budget 1800

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 242cdc00579036e1487a0255b4e1d84c
RANDOM SEED=438690073

node122.alineos.net Linux 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5438.54
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32873648 kB
MemFree:        28555020 kB
Buffers:          267580 kB
Cached:          2170052 kB
SwapCached:         1804 kB
Active:          2120908 kB
Inactive:        1377512 kB
Active(anon):    1061184 kB
Inactive(anon):     1988 kB
Active(file):    1059724 kB
Inactive(file):  1375524 kB
Unevictable:        7628 kB
Mlocked:            7628 kB
SwapTotal:      67108856 kB
SwapFree:       67100164 kB
Dirty:            477588 kB
Writeback:            16 kB
AnonPages:       1064892 kB
Mapped:            14652 kB
Shmem:                 0 kB
Slab:             678444 kB
SReclaimable:     120248 kB
SUnreclaim:       558196 kB
KernelStack:        1440 kB
PageTables:         6564 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83545680 kB
Committed_AS:    1210844 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      346400 kB
VmallocChunk:   34341988052 kB
HardwareCorrupted:     0 kB
AnonHugePages:    272384 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        7680 kB
DirectMap2M:    33538048 kB

Free space on /tmp at the end= 66628 MiB
End job on node122 at 2012-06-03 12:42:00