Trace number 3712792

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
pwbo 2.01MSAT (TO) 1800.06 900.318

General information on the benchmark

Name/SOFT-SMALLINT-LIN/PB06/submitted-PB06/roussel/
normalized-pigeon-cardinality-31-30--soft-0-100-0.wbo
MD5SUM168e81c26eea0b6e6155adb2f3f4d195
Bench CategorySOFT-SMALLINT-LIN (only soft constraints, small integers, linear constraints)
Best result obtained on this benchmarkMOPT
Best cost obtained on this benchmark1
Best CPU time to get the best result obtained on this benchmark0.037993
Max-Satisfiable
Max-(Un)Satisfiability was proved
Best value of the cost
Optimality of the best cost was proved
Number of variables930
Total number of constraints61
Number of soft constraints61
Number of constraints which are clauses31
Number of constraints which are cardinality constraints (but not clauses)30
Number of constraints which are nor clauses,nor cardinality constraints0
Minimum length of a constraint30
Maximum length of a constraint31
Top cost 3358
Min constraint cost 1
Max constraint cost 97
Sum of constraints costs 3357
Biggest number in a constraint 1
Number of bits of the biggest number in a constraint 1
Biggest sum of numbers in a constraint 32
Number of bits of the biggest sum of numbers6
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Quality of the solution as a function of time


Download the above graph as a PDF file / as an EPS file

Solver Data 

0.00/0.00	c ==============================[ Solver Version ]===============================
0.00/0.00	c |                                                                             |
0.00/0.00	c |  pwbo:                    2.01 beta - May, 2012                            |
0.00/0.00	c |                                                                             |
0.00/0.00	c Parsing wbo file format. File HOME/instance-3712792-1338200655.wbo.
0.00/0.00	c Instance file HOME/instance-3712792-1338200655.wbo
0.00/0.00	c File size is 15679 bytes.
0.00/0.00	c Highest Coefficient sum: 32
0.00/0.00	c ============================[ Problem Statistics ]=============================
0.00/0.00	c |                                                                             |
0.00/0.00	c |  Parsing time:          0.00         s                                      |
0.00/0.00	c |  Number Variables:      991                                                 |
0.00/0.00	c |  Number Clauses:        31                                                  |
0.00/0.00	c |  Number Cardinality:    0                                                   |
0.00/0.00	c |  Number PB Constraints: 30                                                  |
0.00/0.00	c |                                                                             |
0.00/0.00	c |  Problem Type:          Optimization                                        |
0.00/0.00	c |  Objective Function:    PB Constraint                                       |
0.00/0.00	c |                                                                             |
0.00/0.00	c ===============================================================================
1800.06/900.31	o 91
1800.06/900.31	o 39
1800.06/900.31	o 1
1800.06/900.31	c ===============================================================================
1800.06/900.31	c Wall Clock Time: 900.31 s
1800.06/900.31	c Best Solution: 1
1800.06/900.31	s SATISFIABLE
1800.06/900.31	v -x1 -x2 -x3 -x4 -x5 -x6 -x7 -x8 -x9 -x10 -x11 -x12 -x13 -x14 -x15 -x16 -x17 -x18 -x19 -x20 -x21 -x22 -x23 -x24 -x25 x26 -x27 -x28 -x29 -x30 -x31 -x32 -x33 -x34 -x35 -x36 -x37 -x38 -x39 -x40 -x41 -x42 -x43 -x44 -x45 -x46 -x47 -x48 -x49 -x50 -x51 -x52 -x53 -x54 x55 -x56 -x57 -x58 -x59 -x60 -x61 -x62 -x63 -x64 -x65 -x66 -x67 -x68 -x69 -x70 -x71 -x72 -x73 -x74 -x75 -x76 -x77 -x78 -x79 -x80 -x81 -x82 -x83 -x84 -x85 -x86 -x87 x88 -x89 -x90 -x91 -x92 -x93 -x94 -x95 -x96 -x97 -x98 -x99 -x100 -x101 -x102 -x103 -x104 -x105 -x106 -x107 -x108 -x109 -x110 -x111 -x112 -x113 -x114 -x115 -x116 x117 -x118 -x119 -x120 -x121 -x122 -x123 -x124 -x125 -x126 -x127 -x128 -x129 -x130 -x131 -x132 -x133 -x134 -x135 -x136 -x137 -x138 -x139 -x140 -x141 -x142 -x143 -x144 -x145 -x146 -x147 -x148 x149 -x150 -x151 -x152 -x153 -x154 -x155 -x156 -x157 -x158 -x159 -x160 -x161 -x162 -x163 -x164 -x165 -x166 -x167 -x168 -x169 -x170 -x171 -x172 -x173 -x174 -x175 -x176 -x177 -x178 -x179 x180 -x181 -x182 -x183 -x184 -x185 -x186 -x187 -x188 -x189 -x190 -x191 -x192 -x193 -x194 -x195 -x196 -x197 -x198 -x199 -x200 -x201 -x202 -x203 -x204 -x205 -x206 -x207 -x208 -x209 -x210 -x211 -x212 -x213 -x214 -x215 -x216 -x217 -x218 -x219 -x220 -x221 -x222 -x223 -x224 -x225 -x226 -x227 -x228 -x229 -x230 -x231 -x232 -x233 x234 -x235 -x236 -x237 -x238 -x239 -x240 -x241 -x242 -x243 -x244 -x245 -x246 -x247 -x248 -x249 -x250 -x251 -x252 -x253 -x254 -x255 -x256 -x257 -x258 -x259 -x260 -x261 -x262 x263 -x264 -x265 -x266 -x267 -x268 -x269 -x270 -x271 -x272 -x273 -x274 -x275 -x276 -x277 -x278 -x279 -x280 -x281 -x282 -x283 -x284 -x285 -x286 -x287 -x288 -x289 -x290 -x291 x292 -x293 -x294 -x295 -x296 -x297 -x298 -x299 -x300 -x301 -x302 -x303 -x304 -x305 -x306 -x307 -x308 -x309 -x310 -x311 -x312 -x313 -x314 -x315 -x316 -x317 -x318 -x319 -x320 x321 -x322 -x323 -x324 -x325 -x326 -x327 -x328 -x329 -x330 -x331 -x332 -x333 -x334 -x335 -x336 -x337 -x338 -x339 -x340 -x341 -x342 -x343 -x344 -x345 -x346 -x347 -x348 -x349 x350 -x351 -x352 -x353 -x354 -x355 -x356 -x357 -x358 -x359 -x360 -x361 -x362 -x363 -x364 -x365 -x366 -x367 -x368 -x369 -x370 -x371 -x372 -x373 -x374 -x375 -x376 -x377 -x378 x379 -x380 -x381 -x382 -x383 -x384 -x385 -x386 -x387 -x388 -x389 -x390 -x391 -x392 -x393 -x394 -x395 -x396 -x397 -x398 -x399 -x400 -x401 -x402 -x403 -x404 -x405 -x406 -x407 x408 -x409 -x410 -x411 -x412 -x413 -x414 -x415 -x416 -x417 -x418 -x419 -x420 -x421 -x422 -x423 -x424 -x425 -x426 -x427 -x428 -x429 -x430 -x431 -x432 -x433 -x434 -x435 -x436 x437 -x438 -x439 -x440 -x441 -x442 -x443 -x444 -x445 -x446 -x447 -x448 -x449 -x450 -x451 -x452 -x453 -x454 -x455 -x456 -x457 -x458 -x459 -x460 -x461 -x462 -x463 -x464 -x465 x466 -x467 -x468 -x469 -x470 -x471 -x472 -x473 -x474 -x475 -x476 -x477 -x478 -x479 -x480 -x481 -x482 -x483 -x484 -x485 -x486 -x487 -x488 -x489 -x490 -x491 -x492 -x493 -x494 x495 -x496 -x497 -x498 -x499 -x500 -x501 -x502 -x503 -x504 -x505 -x506 -x507 -x508 -x509 -x510 -x511 -x512 -x513 -x514 -x515 -x516 -x517 -x518 -x519 -x520 -x521 -x522 -x523 x524 -x525 -x526 -x527 -x528 -x529 -x530 -x531 -x532 -x533 -x534 -x535 -x536 -x537 -x538 -x539 -x540 -x541 -x542 -x543 -x544 -x545 -x546 -x547 -x548 -x549 -x550 -x551 -x552 x553 -x554 -x555 -x556 -x557 -x558 -x559 -x560 -x561 -x562 -x563 -x564 -x565 -x566 -x567 -x568 -x569 -x570 -x571 -x572 -x573 -x574 -x575 -x576 -x577 -x578 -x579 -x580 -x581 x582 -x583 -x584 -x585 -x586 -x587 -x588 -x589 -x590 -x591 -x592 -x593 -x594 -x595 -x596 -x597 -x598 -x599 -x600 -x601 -x602 -x603 -x604 -x605 -x606 -x607 -x608 -x609 -x610 x611 -x612 -x613 -x614 -x615 -x616 -x617 -x618 -x619 -x620 -x621 -x622 -x623 -x624 -x625 -x626 -x627 -x628 -x629 -x630 -x631 -x632 -x633 -x634 -x635 -x636 -x637 -x638 -x639 x640 -x641 -x642 -x643 -x644 -x645 -x646 -x647 -x648 -x649 -x650 -x651 -x652 -x653 -x654 -x655 -x656 -x657 -x658 -x659 -x660 -x661 -x662 -x663 -x664 -x665 -x666 -x667 -x668 x669 -x670 -x671 -x672 -x673 -x674 -x675 -x676 -x677 -x678 -x679 -x680 -x681 -x682 -x683 -x684 -x685 -x686 -x687 -x688 -x689 -x690 -x691 -x692 -x693 -x694 -x695 -x696 -x697 x698 -x699 -x700 -x701 -x702 -x703 -x704 -x705 -x706 -x707 -x708 -x709 -x710 -x711 -x712 -x713 -x714 -x715 -x716 -x717 -x718 -x719 -x720 -x721 -x722 -x723 -x724 -x725 -x726 x727 -x728 -x729 -x730 -x731 -x732 -x733 -x734 -x735 -x736 -x737 -x738 -x739 -x740 -x741 -x742 -x743 -x744 -x745 -x746 -x747 -x748 -x749 -x750 -x751 -x752 -x753 -x754 -x755 x756 -x757 -x758 -x759 -x760 -x761 -x762 -x763 -x764 -x765 -x766 -x767 -x768 -x769 -x770 -x771 -x772 -x773 -x774 -x775 -x776 -x777 -x778 -x779 -x780 -x781 -x782 -x783 -x784 x785 -x786 -x787 -x788 -x789 -x790 -x791 -x792 -x793 -x794 -x795 -x796 -x797 -x798 -x799 -x800 -x801 -x802 -x803 -x804 -x805 -x806 -x807 -x808 -x809 -x810 -x811 -x812 -x813 x814 -x815 -x816 -x817 -x818 -x819 -x820 -x821 -x822 -x823 -x824 -x825 -x826 -x827 -x828 -x829 -x830 -x831 -x832 -x833 -x834 -x835 -x836 -x837 -x838 -x839 -x840 -x841 -x842 x843 -x844 -x845 -x846 -x847 -x848 -x849 -x850 -x851 -x852 -x853 -x854 -x855 -x856 -x857 -x858 -x859 -x860 -x861 -x862 -x863 -x864 -x865 -x866 -x867 -x868 -x869 -x870 -x871 x872 -x873 -x874 -x875 -x876 -x877 -x878 -x879 -x880 -x881 -x882 -x883 -x884 -x885 -x886 -x887 -x888 -x889 -x890 -x891 -x892 -x893 -x894 -x895 -x896 -x897 -x898 -x899 -x900 x901 -x902 -x903 -x904 -x905 -x906 -x907 -x908 -x909 -x910 -x911 -x912 -x913 -x914 -x915 -x916 -x917 -x918 -x919 -x920 -x921 -x922 -x923 -x924 -x925 -x926 -x927 -x928 -x929 -x930

Verifier Data 

OK	1

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3712792-1338200655/watcher-3712792-1338200655 -o /tmp/evaluation-result-3712792-1338200655/solver-3712792-1338200655 -C 1800 -W 1900 -M 15500 pwbo2.01 -file-format=wbo -verbosity=0 HOME/instance-3712792-1338200655.wbo 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1900 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 3.99 3.98 3.97 4/178 21542
/proc/meminfo: memFree=30727008/32873844 swapFree=6080/6080
[pid=21542] ppid=21540 vsize=162744 CPUtime=0 cores=0,2,4,6
/proc/21542/stat : 21542 (pwbo2.01) S 21540 21542 20249 0 -1 4202496 1093 0 0 0 0 0 0 0 20 0 3 0 286190054 166649856 1041 33554432000 4194304 5964478 140735379027424 140735379026944 4546669 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0 0 0
/proc/21542/statm: 40686 1041 196 433 0 40250 0
[pid=21542/tid=21543] ppid=21540 vsize=162744 CPUtime=0 cores=0,2,4,6
/proc/21542/task/21543/stat : 21543 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 125 0 0 0 0 0 0 0 20 0 3 0 286190055 166649856 1041 33554432000 4194304 5964478 140735379027424 139866200251080 5136404 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=21542/tid=21544] ppid=21540 vsize=162744 CPUtime=0 cores=0,2,4,6
/proc/21542/task/21544/stat : 21544 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 99 0 0 0 0 0 0 0 20 0 3 0 286190055 166649856 1041 33554432000 4194304 5964478 140735379027424 139866189760760 4381599 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0

[startup+0.0306021 s]
/proc/loadavg: 3.99 3.98 3.97 4/178 21542
/proc/meminfo: memFree=30727008/32873844 swapFree=6080/6080
[pid=21542] ppid=21540 vsize=162744 CPUtime=0.04 cores=0,2,4,6
/proc/21542/stat : 21542 (pwbo2.01) S 21540 21542 20249 0 -1 4202496 1273 0 0 0 4 0 0 0 20 0 3 0 286190054 166649856 1221 33554432000 4194304 5964478 140735379027424 140735379026944 4546669 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0 0 0
/proc/21542/statm: 40686 1221 198 433 0 40250 0
[pid=21542/tid=21543] ppid=21540 vsize=162744 CPUtime=0.01 cores=0,2,4,6
/proc/21542/task/21543/stat : 21543 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 224 0 0 0 1 0 0 0 20 0 3 0 286190055 166649856 1221 33554432000 4194304 5964478 140735379027424 139866200250360 5136404 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=21542/tid=21544] ppid=21540 vsize=162744 CPUtime=0.01 cores=0,2,4,6
/proc/21542/task/21544/stat : 21544 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 180 0 0 0 1 0 0 0 20 0 3 0 286190055 166649856 1221 33554432000 4194304 5964478 140735379027424 139866189760872 4400071 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 162744

[startup+0.100298 s]
/proc/loadavg: 3.99 3.98 3.97 4/178 21542
/proc/meminfo: memFree=30727008/32873844 swapFree=6080/6080
[pid=21542] ppid=21540 vsize=162744 CPUtime=0.18 cores=0,2,4,6
/proc/21542/stat : 21542 (pwbo2.01) S 21540 21542 20249 0 -1 4202496 1687 0 0 0 17 1 0 0 20 0 3 0 286190054 166649856 1635 33554432000 4194304 5964478 140735379027424 140735379026944 4546669 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0 0 0
/proc/21542/statm: 40686 1635 198 433 0 40250 0
[pid=21542/tid=21543] ppid=21540 vsize=162744 CPUtime=0.08 cores=0,2,4,6
/proc/21542/task/21543/stat : 21543 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 446 0 0 0 8 0 0 0 20 0 3 0 286190055 166649856 1635 33554432000 4194304 5964478 140735379027424 139866200250360 5136404 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=21542/tid=21544] ppid=21540 vsize=162744 CPUtime=0.08 cores=0,2,4,6
/proc/21542/task/21544/stat : 21544 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 372 0 0 0 8 0 0 0 20 0 3 0 286190055 166649856 1635 33554432000 4194304 5964478 140735379027424 139866189760872 4400062 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 0.18
Current children cumulated vsize (KiB) 162744

[startup+0.300318 s]
/proc/loadavg: 3.99 3.98 3.97 4/178 21542
/proc/meminfo: memFree=30727008/32873844 swapFree=6080/6080
[pid=21542] ppid=21540 vsize=162744 CPUtime=0.59 cores=0,2,4,6
/proc/21542/stat : 21542 (pwbo2.01) S 21540 21542 20249 0 -1 4202496 2705 0 0 0 53 6 0 0 20 0 3 0 286190054 166649856 2653 33554432000 4194304 5964478 140735379027424 140735379026944 4546669 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0 0 0
/proc/21542/statm: 40686 2653 198 433 0 40250 0
[pid=21542/tid=21543] ppid=21540 vsize=162744 CPUtime=0.28 cores=0,2,4,6
/proc/21542/task/21543/stat : 21543 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 985 0 0 0 25 3 0 0 20 0 3 0 286190055 166649856 2653 33554432000 4194304 5964478 140735379027424 139866200250360 5136404 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=21542/tid=21544] ppid=21540 vsize=162744 CPUtime=0.28 cores=0,2,4,6
/proc/21542/task/21544/stat : 21544 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 851 0 0 0 25 3 0 0 20 0 3 0 286190055 166649856 2653 33554432000 4194304 5964478 140735379027424 139866189760824 4388212 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 0.59
Current children cumulated vsize (KiB) 162744

[startup+0.700255 s]
/proc/loadavg: 3.99 3.98 3.97 4/178 21542
/proc/meminfo: memFree=30727008/32873844 swapFree=6080/6080
[pid=21542] ppid=21540 vsize=162744 CPUtime=1.38 cores=0,2,4,6
/proc/21542/stat : 21542 (pwbo2.01) S 21540 21542 20249 0 -1 4202496 4479 0 0 0 127 11 0 0 20 0 3 0 286190054 166649856 4427 33554432000 4194304 5964478 140735379027424 140735379026944 4546669 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0 0 0
/proc/21542/statm: 40686 4428 198 433 0 40250 0
[pid=21542/tid=21543] ppid=21540 vsize=162744 CPUtime=0.68 cores=0,2,4,6
/proc/21542/task/21543/stat : 21543 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 1958 0 0 0 63 5 0 0 20 0 3 0 286190055 166649856 4428 33554432000 4194304 5964478 140735379027424 139866200250360 4381977 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=21542/tid=21544] ppid=21540 vsize=162744 CPUtime=0.68 cores=0,2,4,6
/proc/21542/task/21544/stat : 21544 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 1653 0 0 0 61 7 0 0 20 0 3 0 286190055 166649856 4428 33554432000 4194304 5964478 140735379027424 139866189760824 5136404 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 1.38
Current children cumulated vsize (KiB) 162744

[startup+1.50027 s]
/proc/loadavg: 3.99 3.98 3.97 5/181 21545
/proc/meminfo: memFree=30702184/32873844 swapFree=6080/6080
[pid=21542] ppid=21540 vsize=162888 CPUtime=2.98 cores=0,2,4,6
/proc/21542/stat : 21542 (pwbo2.01) S 21540 21542 20249 0 -1 4202496 7418 0 0 0 285 13 0 0 20 0 3 0 286190054 166797312 7366 33554432000 4194304 5964478 140735379027424 140735379026944 4546669 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0 0 0
/proc/21542/statm: 40722 7366 198 433 0 40286 0
[pid=21542/tid=21543] ppid=21540 vsize=162888 CPUtime=1.48 cores=0,2,4,6
/proc/21542/task/21543/stat : 21543 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 3586 0 0 0 142 6 0 0 20 0 3 0 286190055 166797312 7366 33554432000 4194304 5964478 140735379027424 139866200250360 5136404 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=21542/tid=21544] ppid=21540 vsize=162888 CPUtime=1.48 cores=0,2,4,6
/proc/21542/task/21544/stat : 21544 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 2963 0 0 0 141 7 0 0 20 0 3 0 286190055 166797312 7366 33554432000 4194304 5964478 140735379027424 139866189760824 4220672 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 2.98
Current children cumulated vsize (KiB) 162888

[startup+3.10031 s]
/proc/loadavg: 3.99 3.98 3.97 5/181 21545
/proc/meminfo: memFree=30690280/32873844 swapFree=6080/6080
[pid=21542] ppid=21540 vsize=163164 CPUtime=6.18 cores=0,2,4,6
/proc/21542/stat : 21542 (pwbo2.01) S 21540 21542 20249 0 -1 4202496 9131 0 0 0 604 14 0 0 20 0 3 0 286190054 167079936 9079 33554432000 4194304 5964478 140735379027424 140735379026944 4546669 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0 0 0
/proc/21542/statm: 40791 9079 198 433 0 40355 0
[pid=21542/tid=21543] ppid=21540 vsize=163164 CPUtime=3.07 cores=0,2,4,6
/proc/21542/task/21543/stat : 21543 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 4494 0 0 0 300 7 0 0 20 0 3 0 286190055 167079936 9079 33554432000 4194304 5964478 140735379027424 139866200250168 4392757 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=21542/tid=21544] ppid=21540 vsize=163164 CPUtime=3.07 cores=0,2,4,6
/proc/21542/task/21544/stat : 21544 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 3768 0 0 0 300 7 0 0 20 0 3 0 286190055 167079936 9079 33554432000 4194304 5964478 140735379027424 139866189760824 4221120 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 6.18
Current children cumulated vsize (KiB) 163164

[startup+6.30032 s]
/proc/loadavg: 3.99 3.98 3.97 5/181 21545
/proc/meminfo: memFree=30668952/32873844 swapFree=6080/6080
[pid=21542] ppid=21540 vsize=163308 CPUtime=12.57 cores=0,2,4,6
/proc/21542/stat : 21542 (pwbo2.01) S 21540 21542 20249 0 -1 4202496 12413 0 0 0 1232 25 0 0 20 0 3 0 286190054 167227392 12361 33554432000 4194304 5964478 140735379027424 140735379026944 4546669 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0 0 0
/proc/21542/statm: 40827 12361 198 433 0 40391 0
[pid=21542/tid=21543] ppid=21540 vsize=163308 CPUtime=6.26 cores=0,2,4,6
/proc/21542/task/21543/stat : 21543 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 6249 0 0 0 614 12 0 0 20 0 3 0 286190055 167227392 12361 33554432000 4194304 5964478 140735379027424 139866200250360 4400071 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=21542/tid=21544] ppid=21540 vsize=163308 CPUtime=6.26 cores=0,2,4,6
/proc/21542/task/21544/stat : 21544 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 5295 0 0 0 613 13 0 0 20 0 3 0 286190055 167227392 12361 33554432000 4194304 5964478 140735379027424 139866189760824 4382255 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 12.57
Current children cumulated vsize (KiB) 163308

[startup+12.7003 s]
/proc/loadavg: 3.99 3.98 3.97 5/181 21545
/proc/meminfo: memFree=30629264/32873844 swapFree=6080/6080
[pid=21542] ppid=21540 vsize=163308 CPUtime=25.37 cores=0,2,4,6
/proc/21542/stat : 21542 (pwbo2.01) S 21540 21542 20249 0 -1 4202496 19438 0 0 0 2490 47 0 0 20 0 3 0 286190054 167227392 19386 33554432000 4194304 5964478 140735379027424 140735379026944 4546669 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0 0 0
/proc/21542/statm: 40827 19386 198 433 0 40391 0
[pid=21542/tid=21543] ppid=21540 vsize=163308 CPUtime=12.65 cores=0,2,4,6
/proc/21542/task/21543/stat : 21543 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 8787 0 0 0 1247 18 0 0 20 0 3 0 286190055 167227392 19386 33554432000 4194304 5964478 140735379027424 139866200250168 4220672 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=21542/tid=21544] ppid=21540 vsize=163308 CPUtime=12.65 cores=0,2,4,6
/proc/21542/task/21544/stat : 21544 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 9782 0 0 0 1237 28 0 0 20 0 3 0 286190055 167227392 19386 33554432000 4194304 5964478 140735379027424 139866189760824 4220672 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 25.37
Current children cumulated vsize (KiB) 163308

[startup+25.5003 s]
/proc/loadavg: 3.99 3.98 3.97 5/181 21545
/proc/meminfo: memFree=30573332/32873844 swapFree=6080/6080
[pid=21542] ppid=21540 vsize=229324 CPUtime=50.97 cores=0,2,4,6
/proc/21542/stat : 21542 (pwbo2.01) S 21540 21542 20249 0 -1 4202496 31427 0 0 0 5040 57 0 0 20 0 3 0 286190054 234827776 31375 33554432000 4194304 5964478 140735379027424 140735379026944 4546669 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0 0 0
/proc/21542/statm: 57331 31375 198 433 0 56895 0
[pid=21542/tid=21543] ppid=21540 vsize=229324 CPUtime=25.41 cores=0,2,4,6
/proc/21542/task/21543/stat : 21543 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 17377 0 0 0 2520 21 0 0 20 0 3 0 286190055 234827776 31375 33554432000 4194304 5964478 140735379027424 139866200250360 4221120 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=21542/tid=21544] ppid=21540 vsize=229324 CPUtime=25.41 cores=0,2,4,6
/proc/21542/task/21544/stat : 21544 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 13181 0 0 0 2506 35 0 0 20 0 3 0 286190055 234827776 31375 33554432000 4194304 5964478 140735379027424 139866189760584 4400570 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 50.97
Current children cumulated vsize (KiB) 229324

[startup+51.1003 s]
/proc/loadavg: 3.99 3.98 3.97 5/181 21545
/proc/meminfo: memFree=30521716/32873844 swapFree=6080/6080
[pid=21542] ppid=21540 vsize=295736 CPUtime=102.16 cores=0,2,4,6
/proc/21542/stat : 21542 (pwbo2.01) S 21540 21542 20249 0 -1 4202496 44603 0 0 0 10150 66 0 0 20 0 3 0 286190054 302833664 44551 33554432000 4194304 5964478 140735379027424 140735379026944 4546669 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0 0 0
/proc/21542/statm: 73934 44551 198 433 0 73498 0
[pid=21542/tid=21543] ppid=21540 vsize=295736 CPUtime=50.95 cores=0,2,4,6
/proc/21542/task/21543/stat : 21543 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 24513 0 0 0 5070 25 0 0 20 0 3 0 286190055 302833664 44551 33554432000 4194304 5964478 140735379027424 139866200250360 4400454 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=21542/tid=21544] ppid=21540 vsize=295736 CPUtime=50.95 cores=0,2,4,6
/proc/21542/task/21544/stat : 21544 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 19221 0 0 0 5055 40 0 0 20 0 3 0 286190055 302833664 44551 33554432000 4194304 5964478 140735379027424 139866189760824 4400058 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 102.16
Current children cumulated vsize (KiB) 295736

[startup+102.307 s]
/proc/loadavg: 3.99 3.98 3.97 5/181 21545
/proc/meminfo: memFree=30439596/32873844 swapFree=6080/6080
[pid=21542] ppid=21540 vsize=361792 CPUtime=204.55 cores=0,2,4,6
/proc/21542/stat : 21542 (pwbo2.01) S 21540 21542 20249 0 -1 4202496 63664 0 0 0 20346 109 0 0 20 0 3 0 286190054 370475008 63612 33554432000 4194304 5964478 140735379027424 140735379026944 4546669 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0 0 0
/proc/21542/statm: 90448 63612 198 433 0 90012 0
[pid=21542/tid=21543] ppid=21540 vsize=361792 CPUtime=102.02 cores=0,2,4,6
/proc/21542/task/21543/stat : 21543 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 35600 0 0 0 10152 50 0 0 20 0 3 0 286190055 370475008 63612 33554432000 4194304 5964478 140735379027424 139866200250360 4400071 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=21542/tid=21544] ppid=21540 vsize=361792 CPUtime=102.02 cores=0,2,4,6
/proc/21542/task/21544/stat : 21544 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 27195 0 0 0 10141 61 0 0 20 0 3 0 286190055 370475008 63612 33554432000 4194304 5964478 140735379027424 139866189760824 4220672 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 204.55
Current children cumulated vsize (KiB) 361792

[startup+162.3 s]
/proc/loadavg: 3.99 3.98 3.97 5/181 21546
/proc/meminfo: memFree=30226128/32873844 swapFree=6080/6080
[pid=21542] ppid=21540 vsize=427568 CPUtime=324.49 cores=0,2,4,6
/proc/21542/stat : 21542 (pwbo2.01) S 21540 21542 20249 0 -1 4202496 83255 0 0 0 32286 163 0 0 20 0 3 0 286190054 437829632 83566 33554432000 4194304 5964478 140735379027424 140735379026944 4546669 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0 0 0
/proc/21542/statm: 106892 83566 198 433 0 106456 0
[pid=21542/tid=21543] ppid=21540 vsize=427568 CPUtime=161.85 cores=0,2,4,6
/proc/21542/task/21543/stat : 21543 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 43049 0 0 0 16120 65 0 0 20 0 3 0 286190055 437829632 83566 33554432000 4194304 5964478 140735379027424 139866200250168 4400040 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=21542/tid=21544] ppid=21540 vsize=427568 CPUtime=161.85 cores=0,2,4,6
/proc/21542/task/21544/stat : 21544 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 39337 0 0 0 16084 101 0 0 20 0 3 0 286190055 437829632 83566 33554432000 4194304 5964478 140735379027424 139866189760824 4220672 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 324.49
Current children cumulated vsize (KiB) 427568

[startup+222.3 s]
/proc/loadavg: 3.99 3.98 3.97 5/181 21546
/proc/meminfo: memFree=30204708/32873844 swapFree=6080/6080
[pid=21542] ppid=21540 vsize=427568 CPUtime=444.44 cores=0,2,4,6
/proc/21542/stat : 21542 (pwbo2.01) S 21540 21542 20249 0 -1 4202496 87129 0 0 0 44275 169 0 0 20 0 3 0 286190054 437829632 87497 33554432000 4194304 5964478 140735379027424 140735379026944 4546669 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0 0 0
/proc/21542/statm: 106892 87497 198 433 0 106456 0
[pid=21542/tid=21543] ppid=21540 vsize=427568 CPUtime=221.66 cores=0,2,4,6
/proc/21542/task/21543/stat : 21543 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 44743 0 0 0 22098 68 0 0 20 0 3 0 286190055 437829632 87497 33554432000 4194304 5964478 140735379027424 139866200250360 4400062 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=21542/tid=21544] ppid=21540 vsize=427568 CPUtime=221.68 cores=0,2,4,6
/proc/21542/task/21544/stat : 21544 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 41517 0 0 0 22062 106 0 0 20 0 3 0 286190055 437829632 87497 33554432000 4194304 5964478 140735379027424 139866189760824 4221120 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 444.44
Current children cumulated vsize (KiB) 427568

################
# More data... #
################

[pid=21542/tid=21544] ppid=21540 vsize=494328 CPUtime=341.34 cores=0,2,4,6
/proc/21542/task/21544/stat : 21544 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 41558 0 0 0 34026 108 0 0 20 0 3 0 286190055 506191872 98536 33554432000 4194304 5964478 140735379027424 139866189760824 4220672 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 684.31
Current children cumulated vsize (KiB) 494328

[startup+402.3 s]
/proc/loadavg: 3.99 3.98 3.97 5/182 21549
/proc/meminfo: memFree=30035556/32873844 swapFree=6080/6080
[pid=21542] ppid=21540 vsize=496188 CPUtime=804.3 cores=0,2,4,6
/proc/21542/stat : 21542 (pwbo2.01) S 21540 21542 20249 0 -1 4202496 98803 0 0 0 80244 186 0 0 20 0 3 0 286190054 508096512 100235 33554432000 4194304 5964478 140735379027424 140735379026944 4546669 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0 0 0
/proc/21542/statm: 124047 100235 198 433 0 123611 0
[pid=21542/tid=21543] ppid=21540 vsize=496188 CPUtime=401.13 cores=0,2,4,6
/proc/21542/task/21543/stat : 21543 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 56213 0 0 0 40025 88 0 0 20 0 3 0 286190055 508096512 100235 33554432000 4194304 5964478 140735379027424 139866200250168 4400067 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=21542/tid=21544] ppid=21540 vsize=496188 CPUtime=401.19 cores=0,2,4,6
/proc/21542/task/21544/stat : 21544 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 41721 0 0 0 40010 109 0 0 20 0 3 0 286190055 508096512 100235 33554432000 4194304 5964478 140735379027424 139866189760856 4220672 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 804.3
Current children cumulated vsize (KiB) 496188

[startup+462.3 s]
/proc/loadavg: 3.99 3.98 3.97 5/181 21550
/proc/meminfo: memFree=30006204/32873844 swapFree=6080/6080
[pid=21542] ppid=21540 vsize=497768 CPUtime=924.27 cores=0,2,4,6
/proc/21542/stat : 21542 (pwbo2.01) S 21540 21542 20249 0 -1 4202496 105450 0 0 0 92230 197 0 0 20 0 3 0 286190054 509714432 106882 33554432000 4194304 5964478 140735379027424 140735379026944 4546669 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0 0 0
/proc/21542/statm: 124442 106882 198 433 0 124006 0
[pid=21542/tid=21543] ppid=21540 vsize=497768 CPUtime=460.98 cores=0,2,4,6
/proc/21542/task/21543/stat : 21543 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 57279 0 0 0 46010 88 0 0 20 0 3 0 286190055 509714432 106882 33554432000 4194304 5964478 140735379027424 139866200250168 4221120 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=21542/tid=21544] ppid=21540 vsize=497768 CPUtime=461.03 cores=0,2,4,6
/proc/21542/task/21544/stat : 21544 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 47302 0 0 0 45983 120 0 0 20 0 3 0 286190055 509714432 106882 33554432000 4194304 5964478 140735379027424 139866189760824 4220672 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 924.27
Current children cumulated vsize (KiB) 497768

[startup+522.3 s]
/proc/loadavg: 3.99 3.98 3.97 5/181 21550
/proc/meminfo: memFree=29983664/32873844 swapFree=6080/6080
[pid=21542] ppid=21540 vsize=501556 CPUtime=1044.26 cores=0,2,4,6
/proc/21542/stat : 21542 (pwbo2.01) S 21540 21542 20249 0 -1 4202496 110914 0 0 0 104221 205 0 0 20 0 3 0 286190054 513593344 112295 33554432000 4194304 5964478 140735379027424 140735379026944 4546669 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0 0 0
/proc/21542/statm: 125389 112295 198 433 0 124953 0
[pid=21542/tid=21543] ppid=21540 vsize=501556 CPUtime=520.82 cores=0,2,4,6
/proc/21542/task/21543/stat : 21543 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 62599 0 0 0 51985 97 0 0 20 0 3 0 286190055 513593344 112295 33554432000 4194304 5964478 140735379027424 139866200250168 4399904 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=21542/tid=21544] ppid=21540 vsize=501556 CPUtime=520.88 cores=0,2,4,6
/proc/21542/task/21544/stat : 21544 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 47446 0 0 0 51967 121 0 0 20 0 3 0 286190055 513593344 112295 33554432000 4194304 5964478 140735379027424 139866189760648 4220672 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 1044.26
Current children cumulated vsize (KiB) 501556

[startup+582.3 s]
/proc/loadavg: 3.99 3.98 3.97 5/181 21550
/proc/meminfo: memFree=29948028/32873844 swapFree=6080/6080
[pid=21542] ppid=21540 vsize=571856 CPUtime=1164.24 cores=0,2,4,6
/proc/21542/stat : 21542 (pwbo2.01) S 21540 21542 20249 0 -1 4202496 119796 0 0 0 116202 222 0 0 20 0 3 0 286190054 585580544 121177 33554432000 4194304 5964478 140735379027424 140735379026944 4546669 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0 0 0
/proc/21542/statm: 142964 121177 198 433 0 142528 0
[pid=21542/tid=21543] ppid=21540 vsize=571856 CPUtime=580.66 cores=0,2,4,6
/proc/21542/task/21543/stat : 21543 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 71441 0 0 0 57951 115 0 0 20 0 3 0 286190055 585580544 121177 33554432000 4194304 5964478 140735379027424 139866200250168 4400062 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=21542/tid=21544] ppid=21540 vsize=571856 CPUtime=580.73 cores=0,2,4,6
/proc/21542/task/21544/stat : 21544 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 47486 0 0 0 57951 122 0 0 20 0 3 0 286190055 585580544 121177 33554432000 4194304 5964478 140735379027424 139866189761000 4400071 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 1164.24
Current children cumulated vsize (KiB) 571856

[startup+642.3 s]
/proc/loadavg: 3.99 3.98 3.97 5/181 21550
/proc/meminfo: memFree=29838536/32873844 swapFree=6080/6080
[pid=21542] ppid=21540 vsize=639280 CPUtime=1284.23 cores=0,2,4,6
/proc/21542/stat : 21542 (pwbo2.01) S 21540 21542 20249 0 -1 4202496 135127 0 0 0 128175 248 0 0 20 0 3 0 286190054 654622720 136508 33554432000 4194304 5964478 140735379027424 140735379026944 4546669 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0 0 0
/proc/21542/statm: 159820 136508 198 433 0 159384 0
[pid=21542/tid=21543] ppid=21540 vsize=639280 CPUtime=640.51 cores=0,2,4,6
/proc/21542/task/21543/stat : 21543 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 86704 0 0 0 63909 142 0 0 20 0 3 0 286190055 654622720 136508 33554432000 4194304 5964478 140735379027424 139866200250360 4400040 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=21542/tid=21544] ppid=21540 vsize=639280 CPUtime=640.58 cores=0,2,4,6
/proc/21542/task/21544/stat : 21544 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 47554 0 0 0 63935 123 0 0 20 0 3 0 286190055 654622720 136508 33554432000 4194304 5964478 140735379027424 139866189760584 4400040 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 1284.23
Current children cumulated vsize (KiB) 639280

[startup+702.3 s]
/proc/loadavg: 3.99 3.98 3.97 5/182 21592
/proc/meminfo: memFree=30409668/32873844 swapFree=6080/6080
[pid=21542] ppid=21540 vsize=705360 CPUtime=1404.2 cores=0,2,4,6
/proc/21542/stat : 21542 (pwbo2.01) S 21540 21542 20249 0 -1 4202496 150673 0 0 0 140149 271 0 0 20 0 3 0 286190054 722288640 152092 33554432000 4194304 5964478 140735379027424 140735379026944 4546669 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0 0 0
/proc/21542/statm: 176340 152092 198 433 0 175904 0
[pid=21542/tid=21543] ppid=21540 vsize=705360 CPUtime=700.34 cores=0,2,4,6
/proc/21542/task/21543/stat : 21543 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 102153 0 0 0 69868 166 0 0 20 0 3 0 286190055 722288640 152092 33554432000 4194304 5964478 140735379027424 139866200250360 4221120 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=21542/tid=21544] ppid=21540 vsize=705360 CPUtime=700.42 cores=0,2,4,6
/proc/21542/task/21544/stat : 21544 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 47651 0 0 0 69917 125 0 0 20 0 3 0 286190055 722288640 152092 33554432000 4194304 5964478 140735379027424 139866189760584 4400075 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 1404.2
Current children cumulated vsize (KiB) 705360

[startup+762.307 s]
/proc/loadavg: 3.99 3.98 3.97 5/181 21593
/proc/meminfo: memFree=30186008/32873844 swapFree=6080/6080
[pid=21542] ppid=21540 vsize=705932 CPUtime=1524.16 cores=0,2,4,6
/proc/21542/stat : 21542 (pwbo2.01) S 21540 21542 20249 0 -1 4202496 151778 0 0 0 152144 272 0 0 20 0 3 0 286190054 722874368 153304 33554432000 4194304 5964478 140735379027424 140735379026944 4546669 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0 0 0
/proc/21542/statm: 176483 153304 198 433 0 176047 0
[pid=21542/tid=21543] ppid=21540 vsize=705932 CPUtime=760.16 cores=0,2,4,6
/proc/21542/task/21543/stat : 21543 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 102185 0 0 0 75849 167 0 0 20 0 3 0 286190055 722874368 153304 33554432000 4194304 5964478 140735379027424 139866200250360 4400060 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=21542/tid=21544] ppid=21540 vsize=705932 CPUtime=760.27 cores=0,2,4,6
/proc/21542/task/21544/stat : 21544 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 48724 0 0 0 75900 127 0 0 20 0 3 0 286190055 722874368 153304 33554432000 4194304 5964478 140735379027424 139866189760824 4400616 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 1524.16
Current children cumulated vsize (KiB) 705932

[startup+822.3 s]
/proc/loadavg: 3.99 3.98 3.97 5/181 21593
/proc/meminfo: memFree=30105132/32873844 swapFree=6080/6080
[pid=21542] ppid=21540 vsize=705932 CPUtime=1644.09 cores=0,2,4,6
/proc/21542/stat : 21542 (pwbo2.01) S 21540 21542 20249 0 -1 4202496 151849 0 0 0 164137 272 0 0 20 0 3 0 286190054 722874368 153391 33554432000 4194304 5964478 140735379027424 140735379026944 4546669 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0 0 0
/proc/21542/statm: 176483 153391 198 433 0 176047 0
[pid=21542/tid=21543] ppid=21540 vsize=705932 CPUtime=819.99 cores=0,2,4,6
/proc/21542/task/21543/stat : 21543 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 102211 0 0 0 81831 168 0 0 20 0 3 0 286190055 722874368 153391 33554432000 4194304 5964478 140735379027424 139866200250360 4400087 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=21542/tid=21544] ppid=21540 vsize=705932 CPUtime=820.09 cores=0,2,4,6
/proc/21542/task/21544/stat : 21544 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 48769 0 0 0 81881 128 0 0 20 0 3 0 286190055 722874368 153391 33554432000 4194304 5964478 140735379027424 139866189760856 4400071 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 1644.09
Current children cumulated vsize (KiB) 705932

[startup+882.3 s]
/proc/loadavg: 3.99 3.98 3.97 5/181 21593
/proc/meminfo: memFree=29949544/32873844 swapFree=6080/6080
[pid=21542] ppid=21540 vsize=771656 CPUtime=1764.06 cores=0,2,4,6
/proc/21542/stat : 21542 (pwbo2.01) S 21540 21542 20249 0 -1 4202496 157327 0 0 0 176129 277 0 0 20 0 3 0 286190054 790175744 158981 33554432000 4194304 5964478 140735379027424 140735379026944 4546669 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0 0 0
/proc/21542/statm: 192914 158981 198 433 0 192478 0
[pid=21542/tid=21543] ppid=21540 vsize=771656 CPUtime=879.82 cores=0,2,4,6
/proc/21542/task/21543/stat : 21543 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 102225 0 0 0 87813 169 0 0 20 0 3 0 286190055 790175744 158981 33554432000 4194304 5964478 140735379027424 139866200250360 4220672 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=21542/tid=21544] ppid=21540 vsize=771656 CPUtime=879.94 cores=0,2,4,6
/proc/21542/task/21544/stat : 21544 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 54233 0 0 0 87860 134 0 0 20 0 3 0 286190055 790175744 158981 33554432000 4194304 5964478 140735379027424 139866189760824 4400051 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 1764.06
Current children cumulated vsize (KiB) 771656



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+900.3 s]
/proc/loadavg: 3.99 3.98 3.97 5/181 21593
/proc/meminfo: memFree=29949420/32873844 swapFree=6080/6080
[pid=21542] ppid=21540 vsize=771656 CPUtime=1800.06 cores=0,2,4,6
/proc/21542/stat : 21542 (pwbo2.01) S 21540 21542 20249 0 -1 4202496 157331 0 0 0 179729 277 0 0 20 0 3 0 286190054 790175744 158985 33554432000 4194304 5964478 140735379027424 140735379026944 4546669 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0 0 0
/proc/21542/statm: 192914 158985 198 433 0 192478 0
[pid=21542/tid=21543] ppid=21540 vsize=771656 CPUtime=897.78 cores=0,2,4,6
/proc/21542/task/21543/stat : 21543 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 102226 0 0 0 89609 169 0 0 20 0 3 0 286190055 790175744 158985 33554432000 4194304 5964478 140735379027424 139866200250360 4400067 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=21542/tid=21544] ppid=21540 vsize=771656 CPUtime=897.9 cores=0,2,4,6
/proc/21542/task/21544/stat : 21544 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 54236 0 0 0 89656 134 0 0 20 0 3 0 286190055 790175744 158985 33554432000 4194304 5964478 140735379027424 139866189760824 4399904 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 1800.06
Current children cumulated vsize (KiB) 771656

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+900.3 s]
/proc/loadavg: 3.99 3.98 3.97 5/181 21593
/proc/meminfo: memFree=29949420/32873844 swapFree=6080/6080
[pid=21542] ppid=21540 vsize=771656 CPUtime=1800.06 cores=0,2,4,6
/proc/21542/stat : 21542 (pwbo2.01) S 21540 21542 20249 0 -1 4202496 157331 0 0 0 179729 277 0 0 20 0 3 0 286190054 790175744 158985 33554432000 4194304 5964478 140735379027424 140735379026944 4546669 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0 0 0
/proc/21542/statm: 192914 158985 198 433 0 192478 0
[pid=21542/tid=21543] ppid=21540 vsize=771656 CPUtime=897.78 cores=0,2,4,6
/proc/21542/task/21543/stat : 21543 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 102226 0 0 0 89609 169 0 0 20 0 3 0 286190055 790175744 158985 33554432000 4194304 5964478 140735379027424 139866200250360 4400067 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=21542/tid=21544] ppid=21540 vsize=771656 CPUtime=897.9 cores=0,2,4,6
/proc/21542/task/21544/stat : 21544 (pwbo2.01) R 21540 21542 20249 0 -1 4202560 54236 0 0 0 89656 134 0 0 20 0 3 0 286190055 790175744 158985 33554432000 4194304 5964478 140735379027424 139866189760824 4399904 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 1800.06
Current children cumulated vsize (KiB) 771656

Child status: 0

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 21542 and gives
#  childrusage.ru_utime.tv_sec=1792
#  childrusage.ru_utime.tv_usec=668472
#  childrusage.ru_stime.tv_sec=3
#  childrusage.ru_stime.tv_usec=61534
# CPU time returned by wait4() is 1795.73
# while last known CPU time is 1800.06
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 900.318
CPU time (s): 1800.06
CPU user time (s): 1797.29
CPU system time (s): 2.77
CPU usage (%): 199.936
Max. virtual memory (cumulated for all children) (KiB): 771656

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1792.67
system time used= 3.06153
maximum resident set size= 635948
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 157333
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 549
involuntary context switches= 1863

runsolver used 1.66975 second user time and 5.08523 second system time

The end

Launcher Data 

Begin job on node139 at 2012-05-28 12:24:15
IDJOB=3712792
IDBENCH=79172
IDSOLVER=2292
FILE ID=node139/3712792-1338200655
RUNJOBID= node139-1338192730-20265
PBS_JOBID= 14624580
Free space on /tmp= 71648 MiB

SOLVER NAME= pwbo 2.01
BENCH NAME= PB10/normalized-PB10/SOFT-SMALLINT-LIN/PB06/submitted-PB06/roussel/normalized-pigeon-cardinality-31-30--soft-0-100-0.wbo
COMMAND LINE= pwbo2.01 -file-format=wbo -verbosity=0 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3712792-1338200655/watcher-3712792-1338200655 -o /tmp/evaluation-result-3712792-1338200655/solver-3712792-1338200655 -C 1800 -W 1900 -M 15500  pwbo2.01 -file-format=wbo -verbosity=0 HOME/instance-3712792-1338200655.wbo

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 168e81c26eea0b6e6155adb2f3f4d195
RANDOM SEED=818072906

node139.alineos.net Linux 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.750
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.50
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.750
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.750
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.90
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.750
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.750
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.92
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.750
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.750
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.750
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32873844 kB
MemFree:        30727644 kB
Buffers:          426684 kB
Cached:           451912 kB
SwapCached:          828 kB
Active:          1086720 kB
Inactive:         203516 kB
Active(anon):     405124 kB
Inactive(anon):     7968 kB
Active(file):     681596 kB
Inactive(file):   195548 kB
Unevictable:        6080 kB
Mlocked:            6080 kB
SwapTotal:      67108856 kB
SwapFree:       67104692 kB
Dirty:              5480 kB
Writeback:             0 kB
AnonPages:        418048 kB
Mapped:            12648 kB
Shmem:                 8 kB
Slab:             716760 kB
SReclaimable:     159164 kB
SUnreclaim:       557596 kB
KernelStack:        1472 kB
PageTables:         4420 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83545776 kB
Committed_AS:     160372 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      346368 kB
VmallocChunk:   34341987984 kB
HardwareCorrupted:     0 kB
AnonHugePages:    135168 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        7872 kB
DirectMap2M:    33538048 kB

Free space on /tmp at the end= 71640 MiB
End job on node139 at 2012-05-28 12:39:17