Trace number 3262046

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.54 1202.44

General information on the benchmark

Namecrafted/kullmann/VanDerWaerden/VanderWaerden_pd_3k/
VanDerWaerden_pd_2-3-23_507.cnf
MD5SUM0c4fc551e2fe598a8074df6204d87f7d
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables254
Number of clauses34289
Sum of the clauses size152236
Maximum clause length23
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2337
Number of clauses of size 331448
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 52504

Solver Data 

0.00/0.04	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3262046-1304090906.cnf 
0.00/0.04	c Using SATTIMEOUT of 4800
0.00/0.04	c Using SATRAM of 15500
0.00/0.04	c Size of subset node = 32
0.00/0.04	c Variables = 254
0.00/0.04	c Clauses = 34289
0.00/0.04	c Literals = 152236
0.00/0.04	c Reduced size:
0.00/0.04	c Variables = 254
0.00/0.04	c Clauses = 34289
0.00/0.04	c Literals = 152236
0.00/0.04	c a2
0.19/0.22	c a1
0.49/0.56	c Independent sub-problems = 1
0.49/0.56	c Sub-problem with 254 variables and 34289 clauses
0.49/0.56	c using permutation method 4
0.49/0.56	c count = 1
0.49/0.56	c Ordering finished 0.550 seconds
0.49/0.56	c mapped 0.560 seconds
0.49/0.56	c permuted 0.560 seconds
0.49/0.56	c about to sort
0.49/0.57	c sorted
0.59/0.68	c clauses now 34289
0.59/0.68	c init finished 0.570 seconds
0.59/0.68	c tree built 0.670 seconds
0.59/0.68	c 
0.59/0.68	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.59/0.68	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.59/0.68	c ................................ | ..........                    |                                  0.7         0.0             
0.59/0.68	c ................................ | ..........                  1 |                                  0.7         1.5             
0.59/0.68	c ................................ | ..........                  2 |                                  0.7         3.0             
0.59/0.68	c ................................ | ..........                  3 |                                  0.7         4.5             
0.59/0.68	c ................................ | ..........                  4 |                                  0.7         6.0             
0.59/0.68	c ................................ | ..........                  5 |                                  0.7         7.5             
0.59/0.72	c ................................ | ..........               1000 | 65           355                 0.8      1298.7        29723
95.78/24.59	c ................................ | ..........             320000 | 30534        158160             96.2      3327.4     20335176
192.06/48.68	c ................................ | ..........             469000 | 46715        229057            192.4      2438.0     32277454
287.93/72.66	c ................................ | ..........             574000 | 59382        280872            288.2      1991.7     40646407
384.20/96.70	c ................................ | ..........             669000 | 69929        335776            384.2      1741.2     48110495
479.69/120.69	c ................................ | *.........             752000 | 79223        382435            480.0      1566.5     55719167
576.36/144.89	c ................................ | *.........             828000 | 88491        420283            576.7      1435.7     61318537
672.63/168.95	c ................................ | *.........             899000 | 96716        453596            672.8      1336.1     67118730
768.88/193.02	c ................................ | *.........             971000 | 105502       483146            769.0      1262.7     71297418
864.35/216.94	c ................................ | *.........            1031000 | 113948       511786            864.5      1192.6     75533077
961.42/241.29	c ................................ | **........            1094000 | 122619       538286            961.8      1137.5     80864102
1056.09/264.97	c ................................ | **........            1147000 | 128569       563220           1056.4      1085.8     85485903
1153.56/289.34	c ................................ | **........            1196000 | 135099       588466           1153.8      1036.6     90542236
1249.03/313.20	c ................................ | **........            1250000 | 142320       611851           1249.1      1000.8     94292804
1344.93/337.23	c ................................ | **........            1303000 | 150116       637603           1345.0       968.8     98056989
1441.99/361.50	c ................................ | ***.......            1351000 | 157096       659679           1442.0       936.9    102958373
1537.45/385.46	c ................................ | ***.......            1396000 | 163008       680416           1537.7       907.8    106733874
1632.12/409.18	c ................................ | ***.......            1439000 | 168626       699796           1632.4       881.5    110231274
1728.39/433.23	c ................................ | ***.......            1485000 | 174334       723521           1728.5       859.1    114380035
1825.86/457.64	c ................................ | ***.......            1524000 | 180140       743443           1826.0       834.6    117285718
1921.33/481.59	c ................................ | ****......            1561000 | 184968       759869           1921.7       812.3    119842253
2018.00/505.73	c ................................ | ****......            1610000 | 190511       779016           2018.1       797.8    123125480
2112.67/529.47	c ................................ | ****......            1652000 | 195355       795170           2113.0       781.8    125903460
2208.54/553.47	c ................................ | ****......            1697000 | 200705       814764           2208.8       768.3    128640070
2304.81/577.54	c ................................ | ****......            1737000 | 206247       829145           2305.0       753.6    131520165
2399.91/601.37	c ................................ | *****.....            1780000 | 212050       846258           2400.2       741.6    134447705
2496.95/625.64	c ................................ | *****.....            1821000 | 217885       862257           2497.1       729.2    137117721
2592.03/649.50	c ................................ | *****.....            1862000 | 224051       879692           2592.4       718.2    139925882
2689.91/673.93	c ................................ | *****.....            1900000 | 229889       894907           2690.0       706.3    142549237
2786.18/698.03	c ................................ | *****.....            1935000 | 235898       913069           2786.3       694.5    145022064
2882.44/722.15	c ................................ | ******....            1971000 | 240748       933143           2882.7       683.7    147532261
2977.51/745.94	c ................................ | ******....            2010000 | 246051       947719           2977.7       675.0    150449919
3072.97/769.81	c ................................ | ******....            2057000 | 251364       961919           3073.0       669.4    153066906
3169.24/793.90	c ................................ | ******....            2100000 | 257472       977033           3169.3       662.6    155916990
3263.92/817.66	c ................................ | ******....            2137000 | 264026       989564           3264.2       654.7    158659169
3360.59/841.84	c ................................ | *******...            2177000 | 271253       1004110          3360.8       647.8    161156668
3456.06/865.75	c ................................ | *******...            2211000 | 276456       1020642          3456.3       639.7    163556938
3553.54/890.14	c ................................ | *******...            2249000 | 281588       1038743          3553.7       632.9    166315676
3649.40/914.14	c ................................ | *******...            2283000 | 286874       1052163          3649.6       625.6    168706133
3744.10/937.89	c ................................ | *******...            2318000 | 293405       1063078          3744.4       619.1    171119053
3841.55/962.25	c ................................ | ********..            2354000 | 298197       1076195          3841.8       612.7    173403808
3935.83/985.87	c ................................ | ********..            2387000 | 303727       1087926          3936.1       606.4    175929868
4034.10/1010.48	c ................................ | ********..            2418000 | 308481       1100521          4034.4       599.3    177874531
4128.37/1034.06	c ................................ | ********..            2451000 | 313580       1114236          4128.6       593.7    179524841
4225.43/1058.33	c ................................ | ********..            2482000 | 317795       1129257          4225.6       587.4    181749596
4322.10/1082.51	c ................................ | *********.            2513000 | 322783       1141975          4322.1       581.4    184215827
4419.17/1106.84	c ................................ | *********.            2542000 | 327372       1152540          4419.3       575.2    186241696
4513.06/1130.34	c ................................ | *********.            2571000 | 331435       1164090          4513.2       569.7    188278702
4608.93/1154.39	c ................................ | *********.            2606000 | 336037       1180991          4609.3       565.4    190133644
4704.79/1178.34	c ................................ | *********.            2642000 | 340359       1197660          4705.0       561.5    192153294

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3262046-1304090906/watcher-3262046-1304090906 -o /tmp/evaluation-result-3262046-1304090906/solver-3262046-1304090906 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3262046-1304090906.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.73 8.02 7.98 6/181 21100
/proc/meminfo: memFree=27599696/32951124 swapFree=67111464/67111528
[pid=21100] ppid=21098 vsize=9244 CPUtime=0 cores=1,3,5,7
/proc/21100/stat : 21100 (ttsth-5-4) R 21098 21100 20765 0 -1 4202496 378 0 0 0 0 0 0 0 25 0 1 0 599321222 9465856 320 33554432000 4194304 4338100 140735711484480 18446744073709551615 260020056773 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21100/statm: 2311 320 146 36 0 216 0

[startup+0.057929 s]
/proc/loadavg: 7.73 8.02 7.98 6/181 21100
/proc/meminfo: memFree=27599696/32951124 swapFree=67111464/67111528
[pid=21100] ppid=21098 vsize=26464 CPUtime=0.05 cores=1,3,5,7
/proc/21100/stat : 21100 (ttsth-5-4) R 21098 21100 20765 0 -1 4202496 3975 0 0 0 5 0 0 0 25 0 1 0 599321222 27099136 3839 33554432000 4194304 4338100 140735711484480 18446744073709551615 260020093120 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21100/statm: 6664 3858 358 36 0 4569 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 26464

[startup+0.10092 s]
/proc/loadavg: 7.73 8.02 7.98 6/181 21100
/proc/meminfo: memFree=27599696/32951124 swapFree=67111464/67111528
[pid=21100] ppid=21098 vsize=36808 CPUtime=0.09 cores=1,3,5,7
/proc/21100/stat : 21100 (ttsth-5-4) R 21098 21100 20765 0 -1 4202496 5829 0 0 0 9 0 0 0 25 0 1 0 599321222 37691392 5568 33554432000 4194304 4338100 140735711484480 18446744073709551615 260019791042 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21100/statm: 9202 5568 569 36 0 7107 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 36808

[startup+0.300882 s]
/proc/loadavg: 7.73 8.02 7.98 6/181 21100
/proc/meminfo: memFree=27599696/32951124 swapFree=67111464/67111528
[pid=21100] ppid=21098 vsize=16576 CPUtime=0.29 cores=1,3,5,7
/proc/21100/stat : 21100 (ttsth-5-4) R 21098 21100 20765 0 -1 4202496 6618 0 0 0 29 0 0 0 25 0 1 0 599321222 16973824 2171 33554432000 4194304 4338100 140735711484480 18446744073709551615 4245205 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21100/statm: 4144 2171 155 36 0 2049 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 16576

[startup+0.701 s]
/proc/loadavg: 7.73 8.02 7.98 6/181 21100
/proc/meminfo: memFree=27599696/32951124 swapFree=67111464/67111528
[pid=21100] ppid=21098 vsize=264136 CPUtime=3.9 cores=1,3,5,7
/proc/21100/stat : 21100 (ttsth-5-4) S 21098 21100 20765 0 -1 4202496 23393 0 0 0 387 3 0 0 25 0 5 0 599321222 270475264 18878 33554432000 4194304 4338100 140735711484480 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21100/statm: 66034 18879 176 36 0 63939 0
Current children cumulated CPU time (s) 3.9
Current children cumulated vsize (KiB) 264136

[startup+1.59861 s]
/proc/loadavg: 7.73 8.02 7.98 6/181 21100
/proc/meminfo: memFree=27599696/32951124 swapFree=67111464/67111528
[pid=21100] ppid=21098 vsize=267264 CPUtime=4.3 cores=1,3,5,7
/proc/21100/stat : 21100 (ttsth-5-4) S 21098 21100 20765 0 -1 4202496 24556 0 0 0 427 3 0 0 25 0 5 0 599321222 273678336 20041 33554432000 4194304 4338100 140735711484480 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21100/statm: 66816 20041 176 36 0 64721 0
Current children cumulated CPU time (s) 4.3
Current children cumulated vsize (KiB) 267264

[startup+3.10033 s]
/proc/loadavg: 7.73 8.02 7.98 9/186 21105
/proc/meminfo: memFree=27451116/32951124 swapFree=67111464/67111528
[pid=21100] ppid=21098 vsize=333744 CPUtime=10.3 cores=1,3,5,7
/proc/21100/stat : 21100 (ttsth-5-4) S 21098 21100 20765 0 -1 4202496 40761 0 0 0 1023 7 0 0 25 0 5 0 599321222 341753856 36246 33554432000 4194304 4338100 140735711484480 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21100/statm: 83436 36247 176 36 0 81341 0
[pid=21100/tid=21102] ppid=21098 vsize=333744 CPUtime=2.38 cores=1,3,5,7
/proc/21100/task/21102/stat : 21102 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 6964 0 0 0 237 1 0 0 22 0 5 0 599321291 341753856 36247 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21100/tid=21103] ppid=21098 vsize=333744 CPUtime=2.4 cores=1,3,5,7
/proc/21100/task/21103/stat : 21103 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 6015 0 0 0 240 0 0 0 22 0 5 0 599321291 341753856 36247 33554432000 4194304 4338100 140735711484480 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21100/tid=21104] ppid=21098 vsize=333744 CPUtime=2.4 cores=1,3,5,7
/proc/21100/task/21104/stat : 21104 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 7341 0 0 0 239 1 0 0 22 0 5 0 599321291 341753856 36248 33554432000 4194304 4338100 140735711484480 18446744073709551615 4256403 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21100/tid=21105] ppid=21098 vsize=333744 CPUtime=2.39 cores=1,3,5,7
/proc/21100/task/21105/stat : 21105 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 7365 0 0 0 238 1 0 0 22 0 5 0 599321291 341753856 36248 33554432000 4194304 4338100 140735711484480 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 10.3
Current children cumulated vsize (KiB) 333744

[startup+6.30074 s]
/proc/loadavg: 7.75 8.02 7.98 9/186 21105
/proc/meminfo: memFree=27308808/32951124 swapFree=67111464/67111528
[pid=21100] ppid=21098 vsize=447144 CPUtime=23.09 cores=1,3,5,7
/proc/21100/stat : 21100 (ttsth-5-4) S 21098 21100 20765 0 -1 4202496 70101 0 0 0 2297 12 0 0 25 0 5 0 599321222 457875456 65586 33554432000 4194304 4338100 140735711484480 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21100/statm: 111786 65586 176 36 0 109691 0
[pid=21100/tid=21102] ppid=21098 vsize=447144 CPUtime=5.57 cores=1,3,5,7
/proc/21100/task/21102/stat : 21102 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 15056 0 0 0 554 3 0 0 25 0 5 0 599321291 457875456 65587 33554432000 4194304 4338100 140735711484480 18446744073709551615 260032208590 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21100/tid=21103] ppid=21098 vsize=447144 CPUtime=5.6 cores=1,3,5,7
/proc/21100/task/21103/stat : 21103 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 13297 0 0 0 558 2 0 0 25 0 5 0 599321291 457875456 65587 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21100/tid=21104] ppid=21098 vsize=447144 CPUtime=5.6 cores=1,3,5,7
/proc/21100/task/21104/stat : 21104 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 15225 0 0 0 557 3 0 0 25 0 5 0 599321291 457875456 65588 33554432000 4194304 4338100 140735711484480 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21100/tid=21105] ppid=21098 vsize=447144 CPUtime=5.6 cores=1,3,5,7
/proc/21100/task/21105/stat : 21105 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 13448 0 0 0 558 2 0 0 25 0 5 0 599321291 457875456 65588 33554432000 4194304 4338100 140735711484480 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 23.09
Current children cumulated vsize (KiB) 447144

[startup+12.7006 s]
/proc/loadavg: 7.77 8.02 7.98 9/186 21106
/proc/meminfo: memFree=27090080/32951124 swapFree=67111464/67111528
[pid=21100] ppid=21098 vsize=632620 CPUtime=48.67 cores=1,3,5,7
/proc/21100/stat : 21100 (ttsth-5-4) S 21098 21100 20765 0 -1 4202496 116093 0 0 0 4845 22 0 0 25 0 5 0 599321222 647802880 111578 33554432000 4194304 4338100 140735711484480 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21100/statm: 158155 111578 176 36 0 156060 0
[pid=21100/tid=21102] ppid=21098 vsize=632620 CPUtime=11.94 cores=1,3,5,7
/proc/21100/task/21102/stat : 21102 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 25622 0 0 0 1189 5 0 0 25 0 5 0 599321291 647802880 111579 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21100/tid=21103] ppid=21098 vsize=632620 CPUtime=12 cores=1,3,5,7
/proc/21100/task/21103/stat : 21103 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 25702 0 0 0 1196 4 0 0 25 0 5 0 599321291 647802880 111580 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21100/tid=21104] ppid=21098 vsize=632620 CPUtime=12.01 cores=1,3,5,7
/proc/21100/task/21104/stat : 21104 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 25681 0 0 0 1196 5 0 0 25 0 5 0 599321291 647802880 111581 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21100/tid=21105] ppid=21098 vsize=632620 CPUtime=12 cores=1,3,5,7
/proc/21100/task/21105/stat : 21105 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 26014 0 0 0 1195 5 0 0 25 0 5 0 599321291 647802880 111581 33554432000 4194304 4338100 140735711484480 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 48.67
Current children cumulated vsize (KiB) 632620

[startup+25.5012 s]
/proc/loadavg: 7.97 8.05 7.99 9/186 21106
/proc/meminfo: memFree=26806988/32951124 swapFree=67111464/67111528
[pid=21100] ppid=21098 vsize=880788 CPUtime=99.78 cores=1,3,5,7
/proc/21100/stat : 21100 (ttsth-5-4) S 21098 21100 20765 0 -1 4202496 179154 0 0 0 9943 35 0 0 25 0 5 0 599321222 901926912 174639 33554432000 4194304 4338100 140735711484480 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21100/statm: 220197 174639 176 36 0 218102 0
[pid=21100/tid=21102] ppid=21098 vsize=880788 CPUtime=24.67 cores=1,3,5,7
/proc/21100/task/21102/stat : 21102 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 41128 0 0 0 2459 8 0 0 25 0 5 0 599321291 901926912 174639 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21100/tid=21103] ppid=21098 vsize=880788 CPUtime=24.81 cores=1,3,5,7
/proc/21100/task/21103/stat : 21103 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 42958 0 0 0 2472 9 0 0 25 0 5 0 599321291 901926912 174639 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21100/tid=21104] ppid=21098 vsize=880788 CPUtime=24.8 cores=1,3,5,7
/proc/21100/task/21104/stat : 21104 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 41137 0 0 0 2472 8 0 0 25 0 5 0 599321291 901926912 174639 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21100/tid=21105] ppid=21098 vsize=880788 CPUtime=24.79 cores=1,3,5,7
/proc/21100/task/21105/stat : 21105 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 40854 0 0 0 2471 8 0 0 25 0 5 0 599321291 901926912 174640 33554432000 4194304 4338100 140735711484480 18446744073709551615 4256139 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 99.78
Current children cumulated vsize (KiB) 880788

[startup+51.1005 s]
/proc/loadavg: 7.98 8.05 7.99 9/186 21106
/proc/meminfo: memFree=26289128/32951124 swapFree=67111464/67111528
[pid=21100] ppid=21098 vsize=1263856 CPUtime=202.04 cores=1,3,5,7
/proc/21100/stat : 21100 (ttsth-5-4) S 21098 21100 20765 0 -1 4202496 275121 0 0 0 20150 54 0 0 25 0 5 0 599321222 1294188544 270606 33554432000 4194304 4338100 140735711484480 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21100/statm: 315964 270607 176 36 0 313869 0
[pid=21100/tid=21102] ppid=21098 vsize=1263856 CPUtime=50.13 cores=1,3,5,7
/proc/21100/task/21102/stat : 21102 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 62862 0 0 0 5002 11 0 0 25 0 5 0 599321291 1294188544 270607 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254275 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21100/tid=21103] ppid=21098 vsize=1263856 CPUtime=50.4 cores=1,3,5,7
/proc/21100/task/21103/stat : 21103 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 68068 0 0 0 5027 13 0 0 25 0 5 0 599321291 1294188544 270608 33554432000 4194304 4338100 140735711484480 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21100/tid=21104] ppid=21098 vsize=1263856 CPUtime=50.4 cores=1,3,5,7
/proc/21100/task/21104/stat : 21104 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 67161 0 0 0 5028 12 0 0 25 0 5 0 599321291 1294188544 270608 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21100/tid=21105] ppid=21098 vsize=1263856 CPUtime=50.39 cores=1,3,5,7
/proc/21100/task/21105/stat : 21105 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 63955 0 0 0 5025 14 0 0 25 0 5 0 599321291 1294188544 270609 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 202.04
Current children cumulated vsize (KiB) 1263856

[startup+102.301 s]
/proc/loadavg: 8.10 8.07 8.00 9/186 21108
/proc/meminfo: memFree=25550292/32951124 swapFree=67111464/67111528
[pid=21100] ppid=21098 vsize=1785104 CPUtime=406.57 cores=1,3,5,7
/proc/21100/stat : 21100 (ttsth-5-4) S 21098 21100 20765 0 -1 4202496 405164 0 0 0 40574 83 0 0 25 0 5 0 599321222 1827946496 400649 33554432000 4194304 4338100 140735711484480 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21100/statm: 446276 400649 176 36 0 444181 0
[pid=21100/tid=21102] ppid=21098 vsize=1785104 CPUtime=101.05 cores=1,3,5,7
/proc/21100/task/21102/stat : 21102 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 90049 0 0 0 10088 17 0 0 25 0 5 0 599321291 1827946496 400649 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21100/tid=21103] ppid=21098 vsize=1785104 CPUtime=101.61 cores=1,3,5,7
/proc/21100/task/21103/stat : 21103 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 107160 0 0 0 10139 22 0 0 25 0 5 0 599321291 1827946496 400649 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21100/tid=21104] ppid=21098 vsize=1785104 CPUtime=101.6 cores=1,3,5,7
/proc/21100/task/21104/stat : 21104 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 99213 0 0 0 10141 19 0 0 25 0 5 0 599321291 1827946496 400649 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21100/tid=21105] ppid=21098 vsize=1785104 CPUtime=101.59 cores=1,3,5,7
/proc/21100/task/21105/stat : 21105 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 95665 0 0 0 10137 22 0 0 25 0 5 0 599321291 1827946496 400649 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 406.57
Current children cumulated vsize (KiB) 1785104

[startup+162.301 s]
/proc/loadavg: 8.14 8.08 8.01 9/186 21110
/proc/meminfo: memFree=24805980/32951124 swapFree=67111464/67111528
[pid=21100] ppid=21098 vsize=2284040 CPUtime=646.26 cores=1,3,5,7
/proc/21100/stat : 21100 (ttsth-5-4) S 21098 21100 20765 0 -1 4202496 530945 0 0 0 64513 113 0 0 25 0 5 0 599321222 2338856960 526430 33554432000 4194304 4338100 140735711484480 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21100/statm: 571010 526430 176 36 0 568915 0
[pid=21100/tid=21102] ppid=21098 vsize=2284040 CPUtime=160.73 cores=1,3,5,7
/proc/21100/task/21102/stat : 21102 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 120550 0 0 0 16049 24 0 0 25 0 5 0 599321291 2338856960 526430 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21100/tid=21103] ppid=21098 vsize=2284040 CPUtime=161.61 cores=1,3,5,7
/proc/21100/task/21103/stat : 21103 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 143891 0 0 0 16131 30 0 0 25 0 5 0 599321291 2338856960 526430 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21100/tid=21104] ppid=21098 vsize=2284040 CPUtime=161.61 cores=1,3,5,7
/proc/21100/task/21104/stat : 21104 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 131251 0 0 0 16134 27 0 0 25 0 5 0 599321291 2338856960 526430 33554432000 4194304 4338100 140735711484480 18446744073709551615 260032208590 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21100/tid=21105] ppid=21098 vsize=2284040 CPUtime=161.6 cores=1,3,5,7
/proc/21100/task/21105/stat : 21105 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 122176 0 0 0 16130 30 0 0 25 0 5 0 599321291 2338856960 526430 33554432000 4194304 4338100 140735711484480 18446744073709551615 4256032 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 646.26
Current children cumulated vsize (KiB) 2284040

[startup+222.301 s]
/proc/loadavg: 8.14 8.09 8.01 9/186 21111
/proc/meminfo: memFree=24245196/32951124 swapFree=67111464/67111528
[pid=21100] ppid=21098 vsize=2633852 CPUtime=885.93 cores=1,3,5,7
/proc/21100/stat : 21100 (ttsth-5-4) S 21098 21100 20765 0 -1 4202496 617200 0 0 0 88458 135 0 0 25 0 5 0 599321222 2697064448 612685 33554432000 4194304 4338100 140735711484480 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21100/statm: 658463 612685 176 36 0 656368 0
[pid=21100/tid=21102] ppid=21098 vsize=2633852 CPUtime=220.4 cores=1,3,5,7
/proc/21100/task/21102/stat : 21102 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 145866 0 0 0 22010 30 0 0 25 0 5 0 599321291 2697064448 612685 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 7 0 0 0

################
# More data... #
################

/proc/21100/task/21105/stat : 21105 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 288880 0 0 0 76009 74 0 0 25 0 5 0 599321291 5161840640 1213255 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3043.01
Current children cumulated vsize (KiB) 5040860

[startup+822.3 s]
/proc/loadavg: 5.89 7.55 7.86 5/176 21148
/proc/meminfo: memFree=25807068/32951124 swapFree=67111464/67111528
[pid=21100] ppid=21098 vsize=5260216 CPUtime=3282.7 cores=1,3,5,7
/proc/21100/stat : 21100 (ttsth-5-4) S 21098 21100 20765 0 -1 4202496 1272729 0 0 0 327968 302 0 0 25 0 5 0 599321222 5386461184 1268214 33554432000 4194304 4338100 140735711484480 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21100/statm: 1315054 1268214 176 36 0 1312959 0
[pid=21100/tid=21102] ppid=21098 vsize=5260216 CPUtime=820.14 cores=1,3,5,7
/proc/21100/task/21102/stat : 21102 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 303830 0 0 0 81944 70 0 0 25 0 5 0 599321291 5386461184 1268214 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254357 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21100/tid=21103] ppid=21098 vsize=5260216 CPUtime=821.63 cores=1,3,5,7
/proc/21100/task/21103/stat : 21103 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 312075 0 0 0 82085 78 0 0 25 0 5 0 599321291 5386461184 1268214 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21100/tid=21104] ppid=21098 vsize=5260216 CPUtime=819.71 cores=1,3,5,7
/proc/21100/task/21104/stat : 21104 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 343412 0 0 0 81896 75 0 0 25 0 5 0 599321291 5386461184 1268214 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21100/tid=21105] ppid=21098 vsize=5260216 CPUtime=820.51 cores=1,3,5,7
/proc/21100/task/21105/stat : 21105 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 300335 0 0 0 81974 77 0 0 25 0 5 0 599321291 5386461184 1268214 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254239 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3282.7
Current children cumulated vsize (KiB) 5260216

[startup+882.3 s]
/proc/loadavg: 4.74 6.92 7.62 5/176 21150
/proc/meminfo: memFree=25609468/32951124 swapFree=67111464/67111528
[pid=21100] ppid=21098 vsize=5460804 CPUtime=3522.37 cores=1,3,5,7
/proc/21100/stat : 21100 (ttsth-5-4) S 21098 21100 20765 0 -1 4202496 1322684 0 0 0 351921 316 0 0 25 0 5 0 599321222 5591863296 1318169 33554432000 4194304 4338100 140735711484480 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21100/statm: 1365201 1318169 176 36 0 1363106 0
[pid=21100/tid=21102] ppid=21098 vsize=5460804 CPUtime=880.14 cores=1,3,5,7
/proc/21100/task/21102/stat : 21102 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 315928 0 0 0 87941 73 0 0 25 0 5 0 599321291 5591863296 1318169 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21100/tid=21103] ppid=21098 vsize=5460804 CPUtime=881.62 cores=1,3,5,7
/proc/21100/task/21103/stat : 21103 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 327144 0 0 0 88080 82 0 0 25 0 5 0 599321291 5591863296 1318169 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254337 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21100/tid=21104] ppid=21098 vsize=5460804 CPUtime=879.71 cores=1,3,5,7
/proc/21100/task/21104/stat : 21104 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 355505 0 0 0 87892 79 0 0 25 0 5 0 599321291 5591863296 1318169 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254221 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21100/tid=21105] ppid=21098 vsize=5460804 CPUtime=880.18 cores=1,3,5,7
/proc/21100/task/21105/stat : 21105 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 311030 0 0 0 87939 79 0 0 25 0 5 0 599321291 5591863296 1318169 33554432000 4194304 4338100 140735711484480 18446744073709551615 260019790990 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3522.37
Current children cumulated vsize (KiB) 5460804

[startup+942.306 s]
/proc/loadavg: 4.32 6.40 7.40 5/176 21151
/proc/meminfo: memFree=25415352/32951124 swapFree=67111464/67111528
[pid=21100] ppid=21098 vsize=5655136 CPUtime=3762.07 cores=1,3,5,7
/proc/21100/stat : 21100 (ttsth-5-4) S 21098 21100 20765 0 -1 4202496 1370544 0 0 0 375877 330 0 0 25 0 5 0 599321222 5790859264 1366029 33554432000 4194304 4338100 140735711484480 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21100/statm: 1413784 1366029 176 36 0 1411689 0
[pid=21100/tid=21102] ppid=21098 vsize=5655136 CPUtime=940.15 cores=1,3,5,7
/proc/21100/task/21102/stat : 21102 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 329927 0 0 0 93939 76 0 0 25 0 5 0 599321291 5790859264 1366029 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21100/tid=21103] ppid=21098 vsize=5655136 CPUtime=941.63 cores=1,3,5,7
/proc/21100/task/21103/stat : 21103 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 338477 0 0 0 94077 86 0 0 25 0 5 0 599321291 5790859264 1366029 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21100/tid=21104] ppid=21098 vsize=5655136 CPUtime=939.72 cores=1,3,5,7
/proc/21100/task/21104/stat : 21104 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 365434 0 0 0 93890 82 0 0 25 0 5 0 599321291 5790859264 1366029 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21100/tid=21105] ppid=21098 vsize=5655136 CPUtime=939.86 cores=1,3,5,7
/proc/21100/task/21105/stat : 21105 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 323629 0 0 0 93903 83 0 0 25 0 5 0 599321291 5790859264 1366029 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3762.07
Current children cumulated vsize (KiB) 5655136

[startup+1002.3 s]
/proc/loadavg: 4.12 5.96 7.18 5/176 21153
/proc/meminfo: memFree=25228940/32951124 swapFree=67111464/67111528
[pid=21100] ppid=21098 vsize=5839952 CPUtime=4001.74 cores=1,3,5,7
/proc/21100/stat : 21100 (ttsth-5-4) S 21098 21100 20765 0 -1 4202496 1417561 0 0 0 399828 346 0 0 25 0 5 0 599321222 5980110848 1413046 33554432000 4194304 4338100 140735711484480 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21100/statm: 1459988 1413046 176 36 0 1457893 0
[pid=21100/tid=21102] ppid=21098 vsize=5839952 CPUtime=1000.15 cores=1,3,5,7
/proc/21100/task/21102/stat : 21102 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 344334 0 0 0 99935 80 0 0 25 0 5 0 599321291 5980110848 1413046 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21100/tid=21103] ppid=21098 vsize=5839952 CPUtime=1001.63 cores=1,3,5,7
/proc/21100/task/21103/stat : 21103 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 350004 0 0 0 100073 90 0 0 25 0 5 0 599321291 5980110848 1413046 33554432000 4194304 4338100 140735711484480 18446744073709551615 4286315 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21100/tid=21104] ppid=21098 vsize=5839952 CPUtime=999.71 cores=1,3,5,7
/proc/21100/task/21104/stat : 21104 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 374928 0 0 0 99885 86 0 0 25 0 5 0 599321291 5980110848 1413046 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21100/tid=21105] ppid=21098 vsize=5839952 CPUtime=999.53 cores=1,3,5,7
/proc/21100/task/21105/stat : 21105 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 335218 0 0 0 99866 87 0 0 25 0 5 0 599321291 5980110848 1413046 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4001.74
Current children cumulated vsize (KiB) 5839952

[startup+1062.3 s]
/proc/loadavg: 4.08 5.62 6.99 5/176 21155
/proc/meminfo: memFree=25080940/32951124 swapFree=67111464/67111528
[pid=21100] ppid=21098 vsize=5981108 CPUtime=4241.41 cores=1,3,5,7
/proc/21100/stat : 21100 (ttsth-5-4) S 21098 21100 20765 0 -1 4202496 1454042 0 0 0 423784 357 0 0 25 0 5 0 599321222 6124654592 1449527 33554432000 4194304 4338100 140735711484480 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21100/statm: 1495277 1449527 176 36 0 1493182 0
[pid=21100/tid=21102] ppid=21098 vsize=5981108 CPUtime=1060.14 cores=1,3,5,7
/proc/21100/task/21102/stat : 21102 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 355555 0 0 0 105931 83 0 0 25 0 5 0 599321291 6124654592 1449527 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254357 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21100/tid=21103] ppid=21098 vsize=5981108 CPUtime=1061.63 cores=1,3,5,7
/proc/21100/task/21103/stat : 21103 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 360087 0 0 0 106070 93 0 0 25 0 5 0 599321291 6124654592 1449527 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254221 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21100/tid=21104] ppid=21098 vsize=5981108 CPUtime=1059.71 cores=1,3,5,7
/proc/21100/task/21104/stat : 21104 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 384695 0 0 0 105882 89 0 0 25 0 5 0 599321291 6124654592 1449527 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254429 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21100/tid=21105] ppid=21098 vsize=5981108 CPUtime=1059.21 cores=1,3,5,7
/proc/21100/task/21105/stat : 21105 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 340628 0 0 0 105831 90 0 0 25 0 5 0 599321291 6124654592 1449527 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4241.41
Current children cumulated vsize (KiB) 5981108

[startup+1122.3 s]
/proc/loadavg: 4.03 5.32 6.80 5/176 21156
/proc/meminfo: memFree=24911236/32951124 swapFree=67111464/67111528
[pid=21100] ppid=21098 vsize=6153412 CPUtime=4481.09 cores=1,3,5,7
/proc/21100/stat : 21100 (ttsth-5-4) S 21098 21100 20765 0 -1 4202496 1496876 0 0 0 447739 370 0 0 25 0 5 0 599321222 6301093888 1492361 33554432000 4194304 4338100 140735711484480 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21100/statm: 1538353 1492361 176 36 0 1536258 0
[pid=21100/tid=21102] ppid=21098 vsize=6153412 CPUtime=1120.15 cores=1,3,5,7
/proc/21100/task/21102/stat : 21102 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 367076 0 0 0 111928 87 0 0 25 0 5 0 599321291 6301093888 1492361 33554432000 4194304 4338100 140735711484480 18446744073709551615 260032208678 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21100/tid=21103] ppid=21098 vsize=6153412 CPUtime=1121.63 cores=1,3,5,7
/proc/21100/task/21103/stat : 21103 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 370668 0 0 0 112068 95 0 0 25 0 5 0 599321291 6301093888 1492361 33554432000 4194304 4338100 140735711484480 18446744073709551615 260032210075 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21100/tid=21104] ppid=21098 vsize=6153412 CPUtime=1119.71 cores=1,3,5,7
/proc/21100/task/21104/stat : 21104 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 395069 0 0 0 111879 92 0 0 25 0 5 0 599321291 6301093888 1492361 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21100/tid=21105] ppid=21098 vsize=6153412 CPUtime=1118.88 cores=1,3,5,7
/proc/21100/task/21105/stat : 21105 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 350986 0 0 0 111795 93 0 0 25 0 5 0 599321291 6301093888 1492361 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4481.09
Current children cumulated vsize (KiB) 6153412

[startup+1182.3 s]
/proc/loadavg: 4.01 5.08 6.62 5/176 21158
/proc/meminfo: memFree=24752076/32951124 swapFree=67111464/67111528
[pid=21100] ppid=21098 vsize=6313468 CPUtime=4720.77 cores=1,3,5,7
/proc/21100/stat : 21100 (ttsth-5-4) S 21098 21100 20765 0 -1 4202496 1536253 0 0 0 471696 381 0 0 25 0 5 0 599321222 6464991232 1531738 33554432000 4194304 4338100 140735711484480 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21100/statm: 1578367 1531738 176 36 0 1576272 0
[pid=21100/tid=21102] ppid=21098 vsize=6313468 CPUtime=1180.15 cores=1,3,5,7
/proc/21100/task/21102/stat : 21102 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 377942 0 0 0 117925 90 0 0 25 0 5 0 599321291 6464991232 1531739 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21100/tid=21103] ppid=21098 vsize=6313468 CPUtime=1181.64 cores=1,3,5,7
/proc/21100/task/21103/stat : 21103 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 377792 0 0 0 118066 98 0 0 25 0 5 0 599321291 6464991232 1531739 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21100/tid=21104] ppid=21098 vsize=6313468 CPUtime=1179.71 cores=1,3,5,7
/proc/21100/task/21104/stat : 21104 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 405770 0 0 0 117877 94 0 0 25 0 5 0 599321291 6464991232 1531739 33554432000 4194304 4338100 140735711484480 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21100/tid=21105] ppid=21098 vsize=6313468 CPUtime=1178.56 cores=1,3,5,7
/proc/21100/task/21105/stat : 21105 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 361673 0 0 0 117760 96 0 0 25 0 5 0 599321291 6464991232 1531739 33554432000 4194304 4338100 140735711484480 18446744073709551615 4285854 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4720.77
Current children cumulated vsize (KiB) 6313468



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.2 s]
/proc/loadavg: 4.00 5.01 6.56 5/176 21158
/proc/meminfo: memFree=24688812/32951124 swapFree=67111464/67111528
[pid=21100] ppid=21098 vsize=6372900 CPUtime=4800.28 cores=1,3,5,7
/proc/21100/stat : 21100 (ttsth-5-4) S 21098 21100 20765 0 -1 4202496 1551811 0 0 0 479640 388 0 0 25 0 5 0 599321222 6525849600 1547296 33554432000 4194304 4338100 140735711484480 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21100/statm: 1593225 1547296 176 36 0 1591130 0
[pid=21100/tid=21102] ppid=21098 vsize=6372900 CPUtime=1200.05 cores=1,3,5,7
/proc/21100/task/21102/stat : 21102 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 381892 0 0 0 119914 91 0 0 25 0 5 0 599321291 6525849600 1547296 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21100/tid=21103] ppid=21098 vsize=6372900 CPUtime=1201.54 cores=1,3,5,7
/proc/21100/task/21103/stat : 21103 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 382336 0 0 0 120054 100 0 0 25 0 5 0 599321291 6525849600 1547296 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21100/tid=21104] ppid=21098 vsize=6372900 CPUtime=1199.62 cores=1,3,5,7
/proc/21100/task/21104/stat : 21104 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 409432 0 0 0 119866 96 0 0 25 0 5 0 599321291 6525849600 1547296 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21100/tid=21105] ppid=21098 vsize=6372900 CPUtime=1198.35 cores=1,3,5,7
/proc/21100/task/21105/stat : 21105 (ttsth-5-4) R 21098 21100 20765 0 -1 4202560 365074 0 0 0 119737 98 0 0 25 0 5 0 599321291 6525849600 1547296 33554432000 4194304 4338100 140735711484480 18446744073709551615 4254321 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4800.28
Current children cumulated vsize (KiB) 6372900

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.31 s]
/proc/loadavg: 4.00 5.01 6.56 5/176 21158
/proc/meminfo: memFree=24688812/32951124 swapFree=67111464/67111528
[pid=21100] ppid=21098 vsize=0 CPUtime=4800.4 cores=1,3,5,7
/proc/21100/stat : 21100 (ttsth-5-4) Z 21098 21100 20765 0 -1 4203548 1551817 0 0 0 479642 398 0 0 18 0 2 0 599321222 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21100/statm: 0 0 0 0 0 0 0
[pid=21100/tid=21105] ppid=21098 vsize=0 CPUtime=1198.45 cores=1,3,5,7
/proc/21100/task/21105/stat : 21105 (ttsth-5-4) R 21098 21100 20765 0 -1 4203588 365080 0 0 0 119737 108 0 0 25 0 2 0 599321291 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.4
Current children cumulated vsize (KiB) 0

[startup+1202.4 s]
/proc/loadavg: 4.00 5.01 6.56 5/176 21158
/proc/meminfo: memFree=24688812/32951124 swapFree=67111464/67111528
[pid=21100] ppid=21098 vsize=0 CPUtime=4800.49 cores=1,3,5,7
/proc/21100/stat : 21100 (ttsth-5-4) Z 21098 21100 20765 0 -1 4203548 1551817 0 0 0 479642 407 0 0 18 0 2 0 599321222 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21100/statm: 0 0 0 0 0 0 0
[pid=21100/tid=21105] ppid=21098 vsize=0 CPUtime=1198.54 cores=1,3,5,7
/proc/21100/task/21105/stat : 21105 (ttsth-5-4) R 21098 21100 20765 0 -1 4203588 365080 0 0 0 119737 117 0 0 25 0 2 0 599321291 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.49
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1202.44
CPU time (s): 4800.54
CPU user time (s): 4796.43
CPU system time (s): 4.10738
CPU usage (%): 399.234
Max. virtual memory (cumulated for all children) (KiB): 6372900

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4796.43
system time used= 4.10738
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1551817
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 672
involuntary context switches= 33283

runsolver used 1.87671 second user time and 4.67729 second system time

The end

Launcher Data 

Begin job on node110 at 2011-04-29 17:28:26
IDJOB=3262046
IDBENCH=83269
IDSOLVER=1654
FILE ID=node110/3262046-1304090906
RUNJOBID= node110-1304088491-20784
PBS_JOBID= 13170568
Free space on /tmp= 72840 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/kullmann/VanDerWaerden/VanderWaerden_pd_3k/VanDerWaerden_pd_2-3-23_507.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3262046-1304090906/watcher-3262046-1304090906 -o /tmp/evaluation-result-3262046-1304090906/solver-3262046-1304090906 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3262046-1304090906.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 0c4fc551e2fe598a8074df6204d87f7d
RANDOM SEED=150088227

node110.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      27599852 kB
Buffers:        897476 kB
Cached:         693640 kB
SwapCached:         64 kB
Active:        3402996 kB
Inactive:      1537032 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      27599852 kB
SwapTotal:    67111528 kB
SwapFree:     67111464 kB
Dirty:             840 kB
Writeback:          20 kB
AnonPages:     3348560 kB
Mapped:          15268 kB
Slab:           341820 kB
PageTables:      10876 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:  3572176 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72840 MiB
End job on node110 at 2011-04-29 17:48:31