Trace number 3266842

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.4 1204.08

General information on the benchmark

Namecrafted/anton/SRHD-SGI/
srhd-sgi-m32-q326.25-n30-p30-s48700942.cnf
MD5SUM1bb1eb2a3bb0b22f560ab6712b727e98
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.385941
Satisfiable
(Un)Satisfiability was proved
Number of variables943
Number of clauses93431
Sum of the clauses size187745
Maximum clause length32
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 293401
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 530

Solver Data 

0.02/0.06	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3266842-1304106566.cnf 
0.02/0.06	c Using SATTIMEOUT of 4800
0.02/0.06	c Using SATRAM of 15500
0.02/0.06	c Size of subset node = 32
0.02/0.06	c Variables = 943
0.02/0.06	c Clauses = 93431
0.02/0.06	c Literals = 187745
0.02/0.06	c Reduced size:
0.02/0.06	c Variables = 943
0.02/0.06	c Clauses = 93431
0.02/0.06	c Literals = 187745
0.02/0.06	c a2
0.09/0.11	c a1
0.19/0.24	c Independent sub-problems = 1
0.19/0.24	c Sub-problem with 943 variables and 93431 clauses
0.19/0.24	c using permutation method 4
0.19/0.24	c count = 1
0.19/0.24	c Ordering finished 0.220 seconds
0.19/0.24	c mapped 0.230 seconds
0.19/0.24	c permuted 0.230 seconds
0.19/0.24	c about to sort
0.19/0.24	c sorted
0.39/0.46	c clauses now 93431
0.39/0.46	c init finished 0.260 seconds
0.39/0.46	c tree built 0.450 seconds
0.39/0.46	c 
0.39/0.46	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.39/0.46	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.39/0.46	c ................................ | ..........                    |                                  0.5         0.0             
0.39/0.46	c ................................ | ..........                  1 |                                  0.5         2.2             
0.39/0.46	c ................................ | ..........                  2 |                                  0.5         4.4             
0.39/0.46	c ................................ | ..........                  3 |                                  0.5         6.7             
0.39/0.46	c ................................ | ..........                  4 |                                  0.5         8.9             
0.39/0.46	c ................................ | ..........                  5 |                                  0.5        11.1             
0.39/0.46	c ................................ | ..........               1000 | 1            94                  0.5      2173.9         4393
95.64/24.39	c ................................ | ..........            6243000 | 11408        815045             96.0     65024.5     28894555
191.92/48.43	c ................................ | ..........           11950000 | 20017        1524248           192.0     62236.3     52597840
287.72/72.47	c ................................ | ..........           17216000 | 29636        2193864           288.0     59777.8     69525596
383.95/96.51	c ................................ | ..........           22244000 | 36572        2826761           384.0     57927.1     85275290
479.76/120.55	c ................................ | *.........           27223000 | 45678        3444390           480.0     56714.6    103984616
575.94/144.62	c ................................ | *.........           31845000 | 53991        4040034           576.0     55285.5    123185164
671.77/168.66	c ................................ | *.........           36392000 | 63810        4602764           672.0     54154.0    132359407
767.99/192.70	c ................................ | *.........           40685000 | 72625        5142756           768.0     52974.6    143867402
863.82/216.75	c ................................ | *.........           44995000 | 80155        5665015           864.0     52077.5    159401486
959.61/240.80	c ................................ | **........           49067000 | 86497        6147929           960.0     51110.9    163793240
1056.22/264.91	c ................................ | **........           52860000 | 92687        6619497          1056.3     50044.0    169317143
1151.65/288.89	c ................................ | **........           56547000 | 99193        7101254          1152.0     49085.9    175586706
1247.84/312.94	c ................................ | **........           60382000 | 104451       7594671          1248.0     48383.0    188910476
1343.65/336.99	c ................................ | **........           63998000 | 110704       8068097          1344.0     47616.9    199850425
1439.89/361.02	c ................................ | ***.......           67455000 | 115685       8522603          1440.0     46843.8    208839516
1535.69/385.08	c ................................ | ***.......           70776000 | 121867       8949942          1536.0     46077.5    216842040
1631.89/409.13	c ................................ | ***.......           75142000 | 130225       9411245          1632.0     46042.6    226287522
1727.65/433.19	c ................................ | ***.......           79166000 | 138523       9843918          1728.0     45813.4    231234958
1823.81/457.25	c ................................ | ***.......           83423000 | 148136       10334943         1824.0     45735.8    234165713
1919.93/481.32	c ................................ | ****......           87742000 | 158056       10859046         1920.0     45698.7    236022301
2015.70/505.38	c ................................ | ****......           91884000 | 166897       11376687         2016.0     45577.4    241295530
2111.81/529.45	c ................................ | ****......           96211000 | 176627       11907356         2112.0     45554.2    250894463
2207.98/553.50	c ................................ | ****......          100143000 | 185615       12376984         2208.0     45354.4    253882358
2303.73/577.57	c ................................ | ****......          104037000 | 195869       12855285         2304.0     45154.8    256280175
2399.89/601.64	c ................................ | *****.....          108144000 | 207427       13365314         2400.0     45059.6    264414012
2495.91/625.72	c ................................ | *****.....          112095000 | 216272       13867124         2496.0     44909.9    266386378
2591.65/649.79	c ................................ | *****.....          115899000 | 226753       14329616         2592.0     44713.9    268502160
2687.81/673.84	c ................................ | *****.....          119822000 | 233966       14806822         2688.0     44576.6    271152286
2783.91/697.92	c ................................ | *****.....          123793000 | 242316       15289025         2784.0     44465.7    280863805
2879.61/722.00	c ................................ | ******....          127803000 | 250362       15756914         2880.0     44375.9    291214079
2975.75/746.06	c ................................ | ******....          131467000 | 258113       16191658         2976.0     44175.7    295606670
3071.92/770.12	c ................................ | ******....          135179000 | 267403       16631917         3072.0     44003.6    299681117
3167.64/794.20	c ................................ | ******....          138704000 | 275803       17098152         3168.0     43782.6    302851292
3263.66/818.29	c ................................ | ******....          142191000 | 283210       17556579         3264.0     43563.2    308459276
3359.86/842.33	c ................................ | *******...          145703000 | 292613       18002139         3360.0     43364.0    316971524
3456.01/866.40	c ................................ | *******...          149376000 | 299857       18471828         3456.0     43222.0    327456794
3551.72/890.47	c ................................ | *******...          152737000 | 306777       18894511         3552.0     43000.2    329775532
3647.93/914.71	c ................................ | *******...          155890000 | 314539       19294079         3648.0     42732.9    330908361
3743.96/938.81	c ................................ | *******...          159082000 | 320430       19687090         3744.0     42489.5    331216428
3839.75/962.86	c ................................ | ********..          162712000 | 329504       20111908         3840.0     42372.9    332302029
3935.69/986.98	c ................................ | ********..          166434000 | 338890       20583240         3936.0     42284.8    333701230
4031.81/1011.05	c ................................ | ********..          170325000 | 350829       21062773         4032.0     42243.3    334552008
4127.81/1035.16	c ................................ | ********..          174263000 | 366206       21529047         4128.0     42214.9    336872490
4223.87/1059.24	c ................................ | ********..          177975000 | 377175       21979246         4224.0     42134.1    339066844
4319.88/1083.33	c ................................ | *********.          181971000 | 390358       22445976         4320.0     42122.7    343642622
4415.60/1107.40	c ................................ | *********.          185836000 | 404045       22895079         4416.0     42082.4    349442778
4511.69/1131.47	c ................................ | *********.          189680000 | 417660       23355809         4512.0     42039.0    355988844
4607.85/1155.53	c ................................ | *********.          193540000 | 431701       23810699         4608.0     42000.8    365910808
4703.96/1179.61	c ................................ | *********.          197242000 | 445430       24262945         4704.0     41930.6    367843707
4799.64/1203.69	c CPU limit (4800) reached
4799.64/1203.69	c Time = 4800.00 seconds
4799.64/1203.69	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3266842-1304106566/watcher-3266842-1304106566 -o /tmp/evaluation-result-3266842-1304106566/solver-3266842-1304106566 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3266842-1304106566.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.43 7.80 7.98 2/181 11486
/proc/meminfo: memFree=30131692/32951124 swapFree=67102692/67111528
[pid=11486] ppid=11484 vsize=9760 CPUtime=0 cores=1,3,5,7
/proc/11486/stat : 11486 (ttsth-5-4) R 11484 11486 11448 0 -1 4202496 499 0 0 0 0 0 0 0 24 0 1 0 522254813 9994240 441 33554432000 4194304 4338100 140734803076272 18446744073709551615 246516495045 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/11486/statm: 2440 442 147 36 0 345 0

[startup+0.0269839 s]
/proc/loadavg: 6.43 7.80 7.98 2/181 11486
/proc/meminfo: memFree=30131692/32951124 swapFree=67102692/67111528
[pid=11486] ppid=11484 vsize=14072 CPUtime=0.02 cores=1,3,5,7
/proc/11486/stat : 11486 (ttsth-5-4) R 11484 11486 11448 0 -1 4202496 1563 0 0 0 2 0 0 0 24 0 1 0 522254813 14409728 1505 33554432000 4194304 4338100 140734803076272 18446744073709551615 246516495045 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/11486/statm: 3518 1507 148 36 0 1423 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 14072

[startup+0.100963 s]
/proc/loadavg: 6.43 7.80 7.98 2/181 11486
/proc/meminfo: memFree=30131692/32951124 swapFree=67102692/67111528
[pid=11486] ppid=11484 vsize=27428 CPUtime=0.09 cores=1,3,5,7
/proc/11486/stat : 11486 (ttsth-5-4) R 11484 11486 11448 0 -1 4202496 4859 0 0 0 9 0 0 0 24 0 1 0 522254813 28086272 4800 33554432000 4194304 4338100 140734803076272 18446744073709551615 246516229305 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/11486/statm: 6857 4800 153 36 0 4762 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 27428

[startup+0.300924 s]
/proc/loadavg: 6.43 7.80 7.98 2/181 11486
/proc/meminfo: memFree=30131692/32951124 swapFree=67102692/67111528
[pid=11486] ppid=11484 vsize=40516 CPUtime=0.29 cores=1,3,5,7
/proc/11486/stat : 11486 (ttsth-5-4) R 11484 11486 11448 0 -1 4202496 9839 0 0 0 28 1 0 0 25 0 1 0 522254813 41488384 8043 33554432000 4194304 4338100 140734803076272 18446744073709551615 4278319 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/11486/statm: 10129 8043 164 36 0 8034 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 40516

[startup+0.700816 s]
/proc/loadavg: 6.43 7.80 7.98 2/181 11486
/proc/meminfo: memFree=30131692/32951124 swapFree=67102692/67111528
[pid=11486] ppid=11484 vsize=255696 CPUtime=1.4 cores=1,3,5,7
/proc/11486/stat : 11486 (ttsth-5-4) S 11484 11486 11448 0 -1 4202496 19654 0 0 0 135 5 0 0 25 0 5 0 522254813 261832704 17675 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/11486/statm: 63924 17676 176 36 0 61829 0
Current children cumulated CPU time (s) 1.4
Current children cumulated vsize (KiB) 255696

[startup+1.50067 s]
/proc/loadavg: 6.43 7.80 7.98 9/190 11495
/proc/meminfo: memFree=29980004/32951124 swapFree=67102692/67111528
[pid=11486] ppid=11484 vsize=293892 CPUtime=4.58 cores=1,3,5,7
/proc/11486/stat : 11486 (ttsth-5-4) S 11484 11486 11448 0 -1 4202496 29667 0 0 0 447 11 0 0 25 0 5 0 522254813 300945408 27688 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/11486/statm: 73473 27688 176 36 0 71378 0
[pid=11486/tid=11492] ppid=11484 vsize=293892 CPUtime=1.03 cores=1,3,5,7
/proc/11486/task/11492/stat : 11492 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 4701 0 0 0 102 1 0 0 25 0 5 0 522254859 300945408 27688 33554432000 4194304 4338100 140734803076272 18446744073709551615 246516229217 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=11486/tid=11493] ppid=11484 vsize=293892 CPUtime=1.02 cores=1,3,5,7
/proc/11486/task/11493/stat : 11493 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 2406 0 0 0 100 2 0 0 19 0 5 0 522254859 300945408 27688 33554432000 4194304 4338100 140734803076272 18446744073709551615 246516229601 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=11486/tid=11494] ppid=11484 vsize=293892 CPUtime=1.02 cores=1,3,5,7
/proc/11486/task/11494/stat : 11494 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 4429 0 0 0 101 1 0 0 25 0 5 0 522254859 300945408 27689 33554432000 4194304 4338100 140734803076272 18446744073709551615 4263469 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=11486/tid=11495] ppid=11484 vsize=293892 CPUtime=1.02 cores=1,3,5,7
/proc/11486/task/11495/stat : 11495 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 2181 0 0 0 99 3 0 0 25 0 5 0 522254859 300945408 27689 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4.58
Current children cumulated vsize (KiB) 293892

[startup+3.10038 s]
/proc/loadavg: 6.63 7.82 7.98 9/190 11495
/proc/meminfo: memFree=29871752/32951124 swapFree=67102692/67111528
[pid=11486] ppid=11484 vsize=375616 CPUtime=10.98 cores=1,3,5,7
/proc/11486/stat : 11486 (ttsth-5-4) S 11484 11486 11448 0 -1 4202496 49596 0 0 0 1070 28 0 0 25 0 5 0 522254813 384630784 47617 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/11486/statm: 93904 47618 176 36 0 91809 0
[pid=11486/tid=11492] ppid=11484 vsize=375616 CPUtime=2.63 cores=1,3,5,7
/proc/11486/task/11492/stat : 11492 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 11609 0 0 0 261 2 0 0 25 0 5 0 522254859 384630784 47619 33554432000 4194304 4338100 140734803076272 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=11486/tid=11493] ppid=11484 vsize=375616 CPUtime=2.63 cores=1,3,5,7
/proc/11486/task/11493/stat : 11493 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 5814 0 0 0 253 10 0 0 22 0 5 0 522254859 384630784 47620 33554432000 4194304 4338100 140734803076272 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=11486/tid=11494] ppid=11484 vsize=375616 CPUtime=2.61 cores=1,3,5,7
/proc/11486/task/11494/stat : 11494 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 11052 0 0 0 259 2 0 0 25 0 5 0 522254859 384630784 47620 33554432000 4194304 4338100 140734803076272 18446744073709551615 4241507 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=11486/tid=11495] ppid=11484 vsize=375616 CPUtime=2.62 cores=1,3,5,7
/proc/11486/task/11495/stat : 11495 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 5174 0 0 0 251 11 0 0 25 0 5 0 522254859 384630784 47620 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 10.98
Current children cumulated vsize (KiB) 375616

[startup+6.30079 s]
/proc/loadavg: 6.63 7.82 7.98 9/190 11495
/proc/meminfo: memFree=29556932/32951124 swapFree=67102692/67111528
[pid=11486] ppid=11484 vsize=528112 CPUtime=23.76 cores=1,3,5,7
/proc/11486/stat : 11486 (ttsth-5-4) S 11484 11486 11448 0 -1 4202496 87600 0 0 0 2317 59 0 0 25 0 5 0 522254813 540786688 85621 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/11486/statm: 132028 85621 176 36 0 129933 0
[pid=11486/tid=11492] ppid=11484 vsize=528112 CPUtime=5.83 cores=1,3,5,7
/proc/11486/task/11492/stat : 11492 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 24890 0 0 0 579 4 0 0 25 0 5 0 522254859 540786688 85621 33554432000 4194304 4338100 140734803076272 18446744073709551615 4230384 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=11486/tid=11493] ppid=11484 vsize=528112 CPUtime=5.83 cores=1,3,5,7
/proc/11486/task/11493/stat : 11493 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 12460 0 0 0 560 23 0 0 25 0 5 0 522254859 540786688 85621 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=11486/tid=11494] ppid=11484 vsize=528112 CPUtime=5.8 cores=1,3,5,7
/proc/11486/task/11494/stat : 11494 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 23968 0 0 0 575 5 0 0 25 0 5 0 522254859 540786688 85621 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528648347 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=11486/tid=11495] ppid=11484 vsize=528112 CPUtime=5.82 cores=1,3,5,7
/proc/11486/task/11495/stat : 11495 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 10332 0 0 0 559 23 0 0 25 0 5 0 522254859 540786688 85621 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 23.76
Current children cumulated vsize (KiB) 528112

[startup+12.7006 s]
/proc/loadavg: 6.84 7.83 7.98 9/190 11495
/proc/meminfo: memFree=28997624/32951124 swapFree=67102692/67111528
[pid=11486] ppid=11484 vsize=778724 CPUtime=49.32 cores=1,3,5,7
/proc/11486/stat : 11486 (ttsth-5-4) S 11484 11486 11448 0 -1 4202496 149417 0 0 0 4810 122 0 0 25 0 5 0 522254813 797413376 147438 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/11486/statm: 194681 147438 176 36 0 192586 0
[pid=11486/tid=11492] ppid=11484 vsize=778724 CPUtime=12.23 cores=1,3,5,7
/proc/11486/task/11492/stat : 11492 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 48085 0 0 0 1214 9 0 0 25 0 5 0 522254859 797413376 147438 33554432000 4194304 4338100 140734803076272 18446744073709551615 4283433 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=11486/tid=11493] ppid=11484 vsize=778724 CPUtime=12.23 cores=1,3,5,7
/proc/11486/task/11493/stat : 11493 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 23038 0 0 0 1172 51 0 0 25 0 5 0 522254859 797413376 147438 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=11486/tid=11494] ppid=11484 vsize=778724 CPUtime=12.16 cores=1,3,5,7
/proc/11486/task/11494/stat : 11494 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 43636 0 0 0 1208 8 0 0 25 0 5 0 522254859 797413376 147438 33554432000 4194304 4338100 140734803076272 18446744073709551615 246516230142 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=11486/tid=11495] ppid=11484 vsize=778724 CPUtime=12.22 cores=1,3,5,7
/proc/11486/task/11495/stat : 11495 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 18708 0 0 0 1172 50 0 0 25 0 5 0 522254859 797413376 147438 33554432000 4194304 4338100 140734803076272 18446744073709551615 4263456 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 49.32
Current children cumulated vsize (KiB) 778724

[startup+25.5003 s]
/proc/loadavg: 7.02 7.83 7.98 9/190 11495
/proc/meminfo: memFree=28052828/32951124 swapFree=67102692/67111528
[pid=11486] ppid=11484 vsize=1205560 CPUtime=100.43 cores=1,3,5,7
/proc/11486/stat : 11486 (ttsth-5-4) S 11484 11486 11448 0 -1 4202496 255082 0 0 0 9784 259 0 0 25 0 5 0 522254813 1234493440 253103 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/11486/statm: 301390 253103 176 36 0 299295 0
[pid=11486/tid=11492] ppid=11484 vsize=1205560 CPUtime=25.02 cores=1,3,5,7
/proc/11486/task/11492/stat : 11492 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 88583 0 0 0 2484 18 0 0 25 0 5 0 522254859 1234493440 253103 33554432000 4194304 4338100 140734803076272 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=11486/tid=11493] ppid=11484 vsize=1205560 CPUtime=25.01 cores=1,3,5,7
/proc/11486/task/11493/stat : 11493 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 39456 0 0 0 2388 113 0 0 25 0 5 0 522254859 1234493440 253103 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=11486/tid=11494] ppid=11484 vsize=1205560 CPUtime=24.88 cores=1,3,5,7
/proc/11486/task/11494/stat : 11494 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 80231 0 0 0 2474 14 0 0 25 0 5 0 522254859 1234493440 253104 33554432000 4194304 4338100 140734803076272 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=11486/tid=11495] ppid=11484 vsize=1205560 CPUtime=25.02 cores=1,3,5,7
/proc/11486/task/11495/stat : 11495 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 30862 0 0 0 2393 109 0 0 25 0 5 0 522254859 1234493440 253104 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 100.43
Current children cumulated vsize (KiB) 1205560

[startup+51.1006 s]
/proc/loadavg: 7.35 7.85 7.98 9/190 11496
/proc/meminfo: memFree=26564540/32951124 swapFree=67102692/67111528
[pid=11486] ppid=11484 vsize=2028224 CPUtime=202.68 cores=1,3,5,7
/proc/11486/stat : 11486 (ttsth-5-4) S 11484 11486 11448 0 -1 4202496 461321 0 0 0 19725 543 0 0 25 0 5 0 522254813 2076901376 459342 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/11486/statm: 507056 459343 176 36 0 504961 0
[pid=11486/tid=11492] ppid=11484 vsize=2028224 CPUtime=50.62 cores=1,3,5,7
/proc/11486/task/11492/stat : 11492 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 172729 0 0 0 5023 39 0 0 25 0 5 0 522254859 2076901376 459343 33554432000 4194304 4338100 140734803076272 18446744073709551615 4285757 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=11486/tid=11493] ppid=11484 vsize=2028224 CPUtime=50.61 cores=1,3,5,7
/proc/11486/task/11493/stat : 11493 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 69055 0 0 0 4823 238 0 0 25 0 5 0 522254859 2076901376 459344 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=11486/tid=11494] ppid=11484 vsize=2028224 CPUtime=50.34 cores=1,3,5,7
/proc/11486/task/11494/stat : 11494 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 146149 0 0 0 5010 24 0 0 25 0 5 0 522254859 2076901376 459344 33554432000 4194304 4338100 140734803076272 18446744073709551615 246516229267 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=11486/tid=11495] ppid=11484 vsize=2028224 CPUtime=50.61 cores=1,3,5,7
/proc/11486/task/11495/stat : 11495 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 57439 0 0 0 4824 237 0 0 25 0 5 0 522254859 2076901376 459344 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 202.68
Current children cumulated vsize (KiB) 2028224

[startup+102.306 s]
/proc/loadavg: 7.84 7.90 7.99 9/190 11528
/proc/meminfo: memFree=26282936/32951124 swapFree=67102692/67111528
[pid=11486] ppid=11484 vsize=3175640 CPUtime=407.12 cores=1,3,5,7
/proc/11486/stat : 11486 (ttsth-5-4) S 11484 11486 11448 0 -1 4202496 746064 0 0 0 39641 1071 0 0 25 0 5 0 522254813 3251855360 744085 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/11486/statm: 793910 744086 176 36 0 791815 0
[pid=11486/tid=11492] ppid=11484 vsize=3175640 CPUtime=101.8 cores=1,3,5,7
/proc/11486/task/11492/stat : 11492 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 302024 0 0 0 10109 71 0 0 25 0 5 0 522254859 3251855360 744086 33554432000 4194304 4338100 140734803076272 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=11486/tid=11493] ppid=11484 vsize=3175640 CPUtime=101.79 cores=1,3,5,7
/proc/11486/task/11493/stat : 11493 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 104837 0 0 0 9699 480 0 0 25 0 5 0 522254859 3251855360 744086 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=11486/tid=11494] ppid=11484 vsize=3175640 CPUtime=101.26 cores=1,3,5,7
/proc/11486/task/11494/stat : 11494 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 239918 0 0 0 10085 41 0 0 25 0 5 0 522254859 3251855360 744086 33554432000 4194304 4338100 140734803076272 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=11486/tid=11495] ppid=11484 vsize=3175640 CPUtime=101.78 cores=1,3,5,7
/proc/11486/task/11495/stat : 11495 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 83336 0 0 0 9703 475 0 0 25 0 5 0 522254859 3251855360 744086 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 407.12
Current children cumulated vsize (KiB) 3175640

[startup+162.3 s]
/proc/loadavg: 8.07 7.96 8.01 9/190 11530
/proc/meminfo: memFree=23925888/32951124 swapFree=67102692/67111528
[pid=11486] ppid=11484 vsize=4448472 CPUtime=646.6 cores=1,3,5,7
/proc/11486/stat : 11486 (ttsth-5-4) S 11484 11486 11448 0 -1 4202496 1055595 0 0 0 62989 1671 0 0 25 0 5 0 522254813 4555235328 1053616 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/11486/statm: 1112118 1053617 176 36 0 1110023 0
[pid=11486/tid=11492] ppid=11484 vsize=4448472 CPUtime=161.76 cores=1,3,5,7
/proc/11486/task/11492/stat : 11492 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 436056 0 0 0 16069 107 0 0 25 0 5 0 522254859 4555235328 1053617 33554432000 4194304 4338100 140734803076272 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=11486/tid=11493] ppid=11484 vsize=4448472 CPUtime=161.72 cores=1,3,5,7
/proc/11486/task/11493/stat : 11493 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 139063 0 0 0 15422 750 0 0 25 0 5 0 522254859 4555235328 1053617 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=11486/tid=11494] ppid=11484 vsize=4448472 CPUtime=160.9 cores=1,3,5,7
/proc/11486/task/11494/stat : 11494 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 348179 0 0 0 16030 60 0 0 25 0 5 0 522254859 4555235328 1053617 33554432000 4194304 4338100 140734803076272 18446744073709551615 4283332 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=11486/tid=11495] ppid=11484 vsize=4448472 CPUtime=161.72 cores=1,3,5,7
/proc/11486/task/11495/stat : 11495 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 116348 0 0 0 15423 749 0 0 25 0 5 0 522254859 4555235328 1053617 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 646.6
Current children cumulated vsize (KiB) 4448472


################
# More data... #
################

/proc/11486/task/11492/stat : 11492 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 1150593 0 0 0 74318 1649 0 0 25 0 5 0 522254859 10101350400 2393966 33554432000 4194304 4338100 140734803076272 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=11486/tid=11493] ppid=11484 vsize=9864600 CPUtime=760.49 cores=1,3,5,7
/proc/11486/task/11493/stat : 11493 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 258158 0 0 0 72903 3146 0 0 25 0 5 0 522254859 10101350400 2393966 33554432000 4194304 4338100 140734803076272 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=11486/tid=11494] ppid=11484 vsize=9864600 CPUtime=759.44 cores=1,3,5,7
/proc/11486/task/11494/stat : 11494 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 748658 0 0 0 74588 1356 0 0 25 0 5 0 522254859 10101350400 2393966 33554432000 4194304 4338100 140734803076272 18446744073709551615 246516229136 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=11486/tid=11495] ppid=11484 vsize=9864600 CPUtime=760.72 cores=1,3,5,7
/proc/11486/task/11495/stat : 11495 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 222586 0 0 0 72969 3103 0 0 25 0 5 0 522254859 10101350400 2393966 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3040.82
Current children cumulated vsize (KiB) 9864600

[startup+822.301 s]
/proc/loadavg: 8.06 8.10 8.07 9/190 11587
/proc/meminfo: memFree=19702068/32951124 swapFree=67102692/67111528
[pid=11486] ppid=11484 vsize=10158632 CPUtime=3280.03 cores=1,3,5,7
/proc/11486/stat : 11486 (ttsth-5-4) S 11484 11486 11448 0 -1 4202496 2468215 0 0 0 317805 10198 0 0 25 0 5 0 522254813 10402439168 2466236 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/11486/statm: 2539658 2466236 176 36 0 2537563 0
[pid=11486/tid=11492] ppid=11484 vsize=10158632 CPUtime=819.26 cores=1,3,5,7
/proc/11486/task/11492/stat : 11492 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 1192691 0 0 0 80036 1890 0 0 25 0 5 0 522254859 10402439168 2466236 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=11486/tid=11493] ppid=11484 vsize=10158632 CPUtime=820.34 cores=1,3,5,7
/proc/11486/task/11493/stat : 11493 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 265854 0 0 0 78666 3368 0 0 25 0 5 0 522254859 10402439168 2466237 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=11486/tid=11494] ppid=11484 vsize=10158632 CPUtime=819.38 cores=1,3,5,7
/proc/11486/task/11494/stat : 11494 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 759058 0 0 0 80344 1594 0 0 25 0 5 0 522254859 10402439168 2466237 33554432000 4194304 4338100 140734803076272 18446744073709551615 4241252 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=11486/tid=11495] ppid=11484 vsize=10158632 CPUtime=820.56 cores=1,3,5,7
/proc/11486/task/11495/stat : 11495 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 234662 0 0 0 78715 3341 0 0 25 0 5 0 522254859 10402439168 2466237 33554432000 4194304 4338100 140734803076272 18446744073709551615 246516229203 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3280.03
Current children cumulated vsize (KiB) 10158632

[startup+882.301 s]
/proc/loadavg: 8.21 8.13 8.08 9/190 11589
/proc/meminfo: memFree=17509172/32951124 swapFree=67102692/67111528
[pid=11486] ppid=11484 vsize=10815512 CPUtime=3519.42 cores=1,3,5,7
/proc/11486/stat : 11486 (ttsth-5-4) S 11484 11486 11448 0 -1 4202496 2633076 0 0 0 340847 11095 0 0 25 0 5 0 522254813 11075084288 2631097 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/11486/statm: 2703878 2631097 176 36 0 2701783 0
[pid=11486/tid=11492] ppid=11484 vsize=10815512 CPUtime=879.14 cores=1,3,5,7
/proc/11486/task/11492/stat : 11492 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 1262822 0 0 0 85783 2131 0 0 25 0 5 0 522254859 11075084288 2631097 33554432000 4194304 4338100 140734803076272 18446744073709551615 246516229275 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=11486/tid=11493] ppid=11484 vsize=10815512 CPUtime=880.16 cores=1,3,5,7
/proc/11486/task/11493/stat : 11493 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 298353 0 0 0 84421 3595 0 0 25 0 5 0 522254859 11075084288 2631097 33554432000 4194304 4338100 140734803076272 18446744073709551615 246516229279 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=11486/tid=11494] ppid=11484 vsize=10815512 CPUtime=879.21 cores=1,3,5,7
/proc/11486/task/11494/stat : 11494 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 790550 0 0 0 86114 1807 0 0 25 0 5 0 522254859 11075084288 2631097 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=11486/tid=11495] ppid=11484 vsize=10815512 CPUtime=880.41 cores=1,3,5,7
/proc/11486/task/11495/stat : 11495 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 265401 0 0 0 84483 3558 0 0 25 0 5 0 522254859 11075084288 2631097 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3519.42
Current children cumulated vsize (KiB) 10815512

[startup+942.301 s]
/proc/loadavg: 8.27 8.17 8.10 9/190 11594
/proc/meminfo: memFree=16487160/32951124 swapFree=67102692/67111528
[pid=11486] ppid=11484 vsize=10862432 CPUtime=3757.95 cores=1,3,5,7
/proc/11486/stat : 11486 (ttsth-5-4) S 11484 11486 11448 0 -1 4202496 2646033 0 0 0 363866 11929 0 0 25 0 5 0 522254813 11123130368 2644054 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/11486/statm: 2715608 2644054 176 36 0 2713513 0
[pid=11486/tid=11492] ppid=11484 vsize=10862432 CPUtime=938.72 cores=1,3,5,7
/proc/11486/task/11492/stat : 11492 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 1275011 0 0 0 91526 2346 0 0 25 0 5 0 522254859 11123130368 2644054 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=11486/tid=11493] ppid=11484 vsize=10862432 CPUtime=939.58 cores=1,3,5,7
/proc/11486/task/11493/stat : 11493 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 298456 0 0 0 90157 3801 0 0 25 0 5 0 522254859 11123130368 2644054 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=11486/tid=11494] ppid=11484 vsize=10862432 CPUtime=938.92 cores=1,3,5,7
/proc/11486/task/11494/stat : 11494 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 791213 0 0 0 91882 2010 0 0 25 0 5 0 522254859 11123130368 2644054 33554432000 4194304 4338100 140734803076272 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=11486/tid=11495] ppid=11484 vsize=10862432 CPUtime=940.23 cores=1,3,5,7
/proc/11486/task/11495/stat : 11495 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 265403 0 0 0 90255 3768 0 0 25 0 5 0 522254859 11123130368 2644054 33554432000 4194304 4338100 140734803076272 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3757.95
Current children cumulated vsize (KiB) 10862432

[startup+1002.3 s]
/proc/loadavg: 8.23 8.18 8.10 9/190 11595
/proc/meminfo: memFree=15509740/32951124 swapFree=67102692/67111528
[pid=11486] ppid=11484 vsize=10962528 CPUtime=3997.1 cores=1,3,5,7
/proc/11486/stat : 11486 (ttsth-5-4) S 11484 11486 11448 0 -1 4202496 2671699 0 0 0 386880 12830 0 0 25 0 5 0 522254813 11225628672 2669720 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/11486/statm: 2740632 2669720 176 36 0 2738537 0
[pid=11486/tid=11492] ppid=11484 vsize=10962528 CPUtime=998.52 cores=1,3,5,7
/proc/11486/task/11492/stat : 11492 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 1299899 0 0 0 97259 2593 0 0 25 0 5 0 522254859 11225628672 2669720 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=11486/tid=11493] ppid=11484 vsize=10962528 CPUtime=999.47 cores=1,3,5,7
/proc/11486/task/11493/stat : 11493 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 298457 0 0 0 95917 4030 0 0 25 0 5 0 522254859 11225628672 2669720 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=11486/tid=11494] ppid=11484 vsize=10962528 CPUtime=998.62 cores=1,3,5,7
/proc/11486/task/11494/stat : 11494 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 791987 0 0 0 97642 2220 0 0 25 0 5 0 522254859 11225628672 2669720 33554432000 4194304 4338100 140734803076272 18446744073709551615 4283433 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=11486/tid=11495] ppid=11484 vsize=10962528 CPUtime=1000 cores=1,3,5,7
/proc/11486/task/11495/stat : 11495 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 265406 0 0 0 96018 3982 0 0 25 0 5 0 522254859 11225628672 2669720 33554432000 4194304 4338100 140734803076272 18446744073709551615 4285386 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3997.1
Current children cumulated vsize (KiB) 10962528

[startup+1062.3 s]
/proc/loadavg: 8.09 8.15 8.09 9/190 11597
/proc/meminfo: memFree=14413392/32951124 swapFree=67102692/67111528
[pid=11486] ppid=11484 vsize=11122056 CPUtime=4236.23 cores=1,3,5,7
/proc/11486/stat : 11486 (ttsth-5-4) S 11484 11486 11448 0 -1 4202496 2711394 0 0 0 409912 13711 0 0 25 0 5 0 522254813 11388985344 2709415 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/11486/statm: 2780514 2709415 176 36 0 2778419 0
[pid=11486/tid=11492] ppid=11484 vsize=11122056 CPUtime=1058.24 cores=1,3,5,7
/proc/11486/task/11492/stat : 11492 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 1339534 0 0 0 102988 2836 0 0 25 0 5 0 522254859 11388985344 2709415 33554432000 4194304 4338100 140734803076272 18446744073709551615 4229521 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=11486/tid=11493] ppid=11484 vsize=11122056 CPUtime=1059.34 cores=1,3,5,7
/proc/11486/task/11493/stat : 11493 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 298459 0 0 0 101684 4250 0 0 25 0 5 0 522254859 11388985344 2709415 33554432000 4194304 4338100 140734803076272 18446744073709551615 4229521 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=11486/tid=11494] ppid=11484 vsize=11122056 CPUtime=1058.52 cores=1,3,5,7
/proc/11486/task/11494/stat : 11494 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 792038 0 0 0 103421 2431 0 0 25 0 5 0 522254859 11388985344 2709415 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=11486/tid=11495] ppid=11484 vsize=11122056 CPUtime=1059.64 cores=1,3,5,7
/proc/11486/task/11495/stat : 11495 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 265413 0 0 0 101775 4189 0 0 25 0 5 0 522254859 11388985344 2709415 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4236.23
Current children cumulated vsize (KiB) 11122056

[startup+1122.3 s]
/proc/loadavg: 8.20 8.16 8.10 8/190 11598
/proc/meminfo: memFree=13331936/32951124 swapFree=67102692/67111528
[pid=11486] ppid=11484 vsize=11550592 CPUtime=4475.37 cores=1,3,5,7
/proc/11486/stat : 11486 (ttsth-5-4) S 11484 11486 11448 0 -1 4202496 2818497 0 0 0 432936 14601 0 0 25 0 5 0 522254813 11827806208 2816518 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/11486/statm: 2887648 2816519 176 36 0 2885553 0
[pid=11486/tid=11492] ppid=11484 vsize=11550592 CPUtime=1118.06 cores=1,3,5,7
/proc/11486/task/11492/stat : 11492 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 1404539 0 0 0 108728 3078 0 0 25 0 5 0 522254859 11827806208 2816519 33554432000 4194304 4338100 140734803076272 18446744073709551615 246516229566 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=11486/tid=11493] ppid=11484 vsize=11550592 CPUtime=1119 cores=1,3,5,7
/proc/11486/task/11493/stat : 11493 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 325561 0 0 0 107430 4470 0 0 25 0 5 0 522254859 11827806208 2816519 33554432000 4194304 4338100 140734803076272 18446744073709551615 246516530741 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=11486/tid=11494] ppid=11484 vsize=11550592 CPUtime=1118.37 cores=1,3,5,7
/proc/11486/task/11494/stat : 11494 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 801793 0 0 0 109191 2646 0 0 25 0 5 0 522254859 11827806208 2816519 33554432000 4194304 4338100 140734803076272 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=11486/tid=11495] ppid=11484 vsize=11550592 CPUtime=1119.45 cores=1,3,5,7
/proc/11486/task/11495/stat : 11495 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 270655 0 0 0 107542 4403 0 0 21 0 5 0 522254859 11827806208 2816519 33554432000 4194304 4338100 140734803076272 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4475.37
Current children cumulated vsize (KiB) 11550592

[startup+1182.3 s]
/proc/loadavg: 8.08 8.13 8.09 9/190 11600
/proc/meminfo: memFree=11748544/32951124 swapFree=67102692/67111528
[pid=11486] ppid=11484 vsize=12010408 CPUtime=4714.72 cores=1,3,5,7
/proc/11486/stat : 11486 (ttsth-5-4) S 11484 11486 11448 0 -1 4202496 2933780 0 0 0 455952 15520 0 0 25 0 5 0 522254813 12298657792 2931801 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/11486/statm: 3002602 2931801 176 36 0 3000507 0
[pid=11486/tid=11492] ppid=11484 vsize=12010408 CPUtime=1177.86 cores=1,3,5,7
/proc/11486/task/11492/stat : 11492 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 1455440 0 0 0 114458 3328 0 0 25 0 5 0 522254859 12298657792 2931801 33554432000 4194304 4338100 140734803076272 18446744073709551615 4239484 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=11486/tid=11493] ppid=11484 vsize=12010408 CPUtime=1178.77 cores=1,3,5,7
/proc/11486/task/11493/stat : 11493 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 352570 0 0 0 113172 4705 0 0 25 0 5 0 522254859 12298657792 2931801 33554432000 4194304 4338100 140734803076272 18446744073709551615 4286366 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=11486/tid=11494] ppid=11484 vsize=12010408 CPUtime=1178.29 cores=1,3,5,7
/proc/11486/task/11494/stat : 11494 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 823661 0 0 0 114967 2862 0 0 25 0 5 0 522254859 12298657792 2931801 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=11486/tid=11495] ppid=11484 vsize=12010408 CPUtime=1179.29 cores=1,3,5,7
/proc/11486/task/11495/stat : 11495 (ttsth-5-4) R 11484 11486 11448 0 -1 4202560 286159 0 0 0 113309 4620 0 0 25 0 5 0 522254859 12298657792 2931801 33554432000 4194304 4338100 140734803076272 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4714.72
Current children cumulated vsize (KiB) 12010408



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1203.7 s]
/proc/loadavg: 8.20 8.15 8.10 6/187 11601
/proc/meminfo: memFree=11473504/32951124 swapFree=67102692/67111528
[pid=11486] ppid=11484 vsize=0 CPUtime=4800.01 cores=1,3,5,7
/proc/11486/stat : 11486 (ttsth-5-4) Z 11484 11486 11448 0 -1 4203548 2963142 0 0 0 464168 15833 0 0 18 0 2 0 522254813 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/11486/statm: 0 0 0 0 0 0 0
[pid=11486/tid=11494] ppid=11484 vsize=0 CPUtime=1199.67 cores=1,3,5,7
/proc/11486/task/11494/stat : 11494 (ttsth-5-4) R 11484 11486 11448 0 -1 4203588 827751 0 0 0 117027 2940 0 0 25 0 2 0 522254859 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.01
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1203.81 s]
/proc/loadavg: 8.20 8.15 8.10 6/187 11601
/proc/meminfo: memFree=11473504/32951124 swapFree=67102692/67111528
[pid=11486] ppid=11484 vsize=0 CPUtime=4800.12 cores=1,3,5,7
/proc/11486/stat : 11486 (ttsth-5-4) Z 11484 11486 11448 0 -1 4203548 2963142 0 0 0 464168 15844 0 0 18 0 2 0 522254813 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/11486/statm: 0 0 0 0 0 0 0
[pid=11486/tid=11494] ppid=11484 vsize=0 CPUtime=1199.77 cores=1,3,5,7
/proc/11486/task/11494/stat : 11494 (ttsth-5-4) R 11484 11486 11448 0 -1 4203588 827751 0 0 0 117027 2950 0 0 25 0 2 0 522254859 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.12
Current children cumulated vsize (KiB) 0

[startup+1204 s]
/proc/loadavg: 8.20 8.15 8.10 6/187 11601
/proc/meminfo: memFree=11473504/32951124 swapFree=67102692/67111528
[pid=11486] ppid=11484 vsize=0 CPUtime=4800.31 cores=1,3,5,7
/proc/11486/stat : 11486 (ttsth-5-4) Z 11484 11486 11448 0 -1 4203548 2963142 0 0 0 464168 15863 0 0 18 0 2 0 522254813 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/11486/statm: 0 0 0 0 0 0 0
[pid=11486/tid=11494] ppid=11484 vsize=0 CPUtime=1199.96 cores=1,3,5,7
/proc/11486/task/11494/stat : 11494 (ttsth-5-4) R 11484 11486 11448 0 -1 4203588 827751 0 0 0 117027 2969 0 0 25 0 2 0 522254859 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.31
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1204.08
CPU time (s): 4800.4
CPU user time (s): 4641.69
CPU system time (s): 158.713
CPU usage (%): 398.678
Max. virtual memory (cumulated for all children) (KiB): 12129272

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4641.69
system time used= 158.713
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 2963142
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 22902
involuntary context switches= 22373

runsolver used 2.12568 second user time and 4.45232 second system time

The end

Launcher Data 

Begin job on node134 at 2011-04-29 21:49:26
IDJOB=3266842
IDBENCH=88214
IDSOLVER=1654
FILE ID=node134/3266842-1304106566
RUNJOBID= node134-1304106566-11467
PBS_JOBID= 13170663
Free space on /tmp= 71584 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/anton/SRHD-SGI/srhd-sgi-m32-q326.25-n30-p30-s48700942.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3266842-1304106566/watcher-3266842-1304106566 -o /tmp/evaluation-result-3266842-1304106566/solver-3266842-1304106566 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3266842-1304106566.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 1bb1eb2a3bb0b22f560ab6712b727e98
RANDOM SEED=961282260

node134.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 7186.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      30133956 kB
Buffers:        988440 kB
Cached:        1357952 kB
SwapCached:       4400 kB
Active:         118152 kB
Inactive:      2297792 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      30133956 kB
SwapTotal:    67111528 kB
SwapFree:     67102692 kB
Dirty:            1752 kB
Writeback:           0 kB
AnonPages:       66968 kB
Mapped:          15208 kB
Slab:           334656 kB
PageTables:       5004 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   217216 kB
VmallocTotal: 34359738367 kB
VmallocUsed:     33868 kB
VmallocChunk: 34359703231 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71580 MiB
End job on node134 at 2011-04-29 22:09:32