Trace number 3257470

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.16 1203.54

General information on the benchmark

NameCRAFTED/edgematching/
fbcolors/em_8_4_5_fbc.cnf
MD5SUM6fdce6b69c397ad71b0f25957edff1fa
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark5.41518
Satisfiable
(Un)Satisfiability was proved
Number of variables2420
Number of clauses33572
Sum of the clauses size97252
Maximum clause length36
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 214396
Number of clauses of size 313968
Number of clauses of size 42628
Number of clauses of size 52460
Number of clauses of size over 5120

Solver Data 

0.00/0.03	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3257470-1304046354.cnf 
0.00/0.03	c Using SATTIMEOUT of 4800
0.00/0.03	c Using SATRAM of 15500
0.00/0.03	c Size of subset node = 32
0.00/0.03	c Variables = 2420
0.00/0.03	c Clauses = 33572
0.00/0.03	c Literals = 97252
0.00/0.03	c Reduced size:
0.00/0.03	c Variables = 2420
0.00/0.03	c Clauses = 33572
0.00/0.03	c Literals = 97252
0.00/0.03	c a2
0.03/0.07	c a1
1.59/1.60	c Independent sub-problems = 1
1.59/1.60	c Sub-problem with 2420 variables and 33572 clauses
1.59/1.60	c using permutation method 4
1.59/1.60	c count = 2
1.59/1.60	c Ordering finished 1.590 seconds
1.59/1.60	c mapped 1.590 seconds
1.59/1.60	c permuted 1.590 seconds
1.59/1.60	c about to sort
1.59/1.60	c sorted
1.59/1.68	c clauses now 33572
1.59/1.68	c init finished 1.620 seconds
1.59/1.68	c tree built 1.670 seconds
1.59/1.68	c 
1.59/1.68	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
1.59/1.68	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
1.59/1.68	c ................................ | ..........                    |                                  1.7         0.0             
1.59/1.68	c ................................ | ..........                  1 |                                  1.7         0.6             
1.59/1.68	c ................................ | ..........                  2 |                                  1.7         1.2             
1.59/1.68	c ................................ | ..........                  3 |                                  1.7         1.8             
1.59/1.68	c ................................ | ..........                  4 |                                  1.7         2.4             
1.59/1.68	c ................................ | ..........                  5 |                                  1.7         3.0             
1.59/1.68	c ................................ | ..........               1000 | 3            65                  1.7       595.2          262
95.62/25.30	c ................................ | ..........            3024000 | 119742       1909706            96.0     31496.7      2773247
191.85/49.33	c ................................ | ..........            5499000 | 219810       3384658           192.0     28639.1      4504057
287.71/73.37	c ................................ | ..........            7655000 | 309459       4688737           288.0     26578.0      6065078
383.93/97.41	c ................................ | ..........            9059000 | 393951       5888714           384.0     23591.1      7902739
479.78/121.46	c ................................ | *.........           10572000 | 466921       7068471           480.0     22024.1      9532037
576.03/145.50	c ................................ | *.........           11732000 | 541000       8056826           576.0     20367.0     10890435
671.87/169.54	c ................................ | *.........           13003000 | 612089       9165363           672.0     19348.6     11758032
767.72/193.58	c ................................ | *.........           14249000 | 676630       10199065          768.0     18552.7     12919711
863.96/217.61	c ................................ | *.........           15591000 | 744637       11276232          864.0     18044.9     13861444
959.79/241.65	c ................................ | **........           16755000 | 801438       12222242          960.0     17452.9     15283078
1056.02/265.70	c ................................ | **........           18131000 | 863779       13268777         1056.0     17168.9     16031379
1151.84/289.75	c ................................ | **........           19478000 | 933029       14245572         1152.0     16907.5     16375602
1247.67/313.79	c ................................ | **........           20478000 | 983371       14894820         1248.0     16408.0     16545576
1343.90/337.82	c ................................ | **........           21113000 | 1025891      15440031         1344.0     15709.1     16545576
1439.76/361.87	c ................................ | ***.......           21867000 | 1072381      16054621         1440.0     15185.2     16618835
1536.00/385.90	c ................................ | ***.......           22819000 | 1133763      16817480         1536.0     14856.0     16708328
1631.83/409.94	c ................................ | ***.......           23852000 | 1191351      17632722         1632.0     14615.1     16980493
1728.06/434.00	c ................................ | ***.......           25035000 | 1254621      18520306         1728.1     14487.3     17233708
1823.89/458.05	c ................................ | ***.......           26321000 | 1317011      19437364         1824.1     14429.6     17447620
1920.12/482.10	c ................................ | ****......           27482000 | 1370634      20298915         1920.2     14312.4     17913696
2015.96/506.11	c ................................ | ****......           28685000 | 1426219      21152222         2016.0     14228.5     18538946
2111.81/530.16	c ................................ | ****......           29946000 | 1486499      22045131         2112.0     14178.8     19481919
2207.98/554.22	c ................................ | ****......           30895000 | 1533028      22796473         2208.1     13991.9     19626212
2303.81/578.27	c ................................ | ****......           31787000 | 1577691      23436944         2304.1     13795.9     20106957
2400.05/602.30	c ................................ | *****.....           32542000 | 1615988      23926092         2400.1     13558.8     20335665
2495.89/626.38	c ................................ | *****.....           33445000 | 1664204      24552663         2496.2     13398.3     21046728
2591.71/650.37	c ................................ | *****.....           34258000 | 1711158      25247595         2592.0     13216.8     21129552
2687.92/674.43	c ................................ | *****.....           35340000 | 1769017      26106925         2688.1     13147.0     21394084
2783.77/698.46	c ................................ | *****.....           36359000 | 1821495      26883455         2784.0     13059.9     21436222
2879.98/722.51	c ................................ | ******....           37302000 | 1870622      27647066         2880.0     12951.9     21883552
2975.81/746.55	c ................................ | ******....           38307000 | 1921229      28439298         2976.0     12871.8     22190869
3071.66/770.59	c ................................ | ******....           39066000 | 1968022      29096399         3072.1     12716.5     22577326
3167.91/794.67	c ................................ | ******....           39756000 | 2006070      29688374         3168.2     12548.5     22658310
3263.67/818.69	c ................................ | ******....           40617000 | 2051400      30385764         3264.0     12443.9     22735776
3359.88/842.74	c ................................ | *******...           41632000 | 2105940      31215047         3360.0     12390.3     22763225
3455.70/866.78	c ................................ | *******...           42504000 | 2149914      31912043         3456.0     12298.6     23067204
3551.92/890.82	c ................................ | *******...           43408000 | 2199099      32635492         3552.0     12220.7     23191932
3647.71/914.88	c ................................ | *******...           44347000 | 2245409      33349838         3648.0     12156.4     23361288
3743.95/938.92	c ................................ | *******...           45193000 | 2288523      33951637         3744.1     12070.6     23907219
3839.77/962.98	c ................................ | ********..           45900000 | 2328318      34529794         3840.1     11952.8     24404751
3935.97/987.01	c ................................ | ********..           46739000 | 2372063      35195780         3936.0     11874.7     25222700
4031.80/1011.06	c ................................ | ********..           47545000 | 2416328      35851197         4032.1     11791.8     25742811
4127.65/1035.10	c ................................ | ********..           48318000 | 2454241      36480370         4128.1     11704.8     26459965
4223.88/1059.13	c ................................ | ********..           49105000 | 2494613      37115775         4224.0     11625.2     26812137
4319.67/1083.19	c ................................ | *********.           49965000 | 2539391      37813736         4320.1     11565.8     27191420
4415.90/1107.23	c ................................ | *********.           50910000 | 2587994      38533617         4416.0     11528.5     27191420
4511.72/1131.27	c ................................ | *********.           51914000 | 2631924      39217815         4512.0     11505.7     27281085
4607.88/1155.33	c ................................ | *********.           52812000 | 2675070      39872580         4608.0     11460.9     27637375
4703.71/1179.39	c ................................ | *********.           53669000 | 2716381      40510518         4704.1     11409.0     27719165
4799.93/1203.43	c CPU limit (4800) reached
4799.93/1203.43	c Time = 4800.05 seconds
4799.93/1203.43	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3257470-1304046354/watcher-3257470-1304046354 -o /tmp/evaluation-result-3257470-1304046354/solver-3257470-1304046354 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3257470-1304046354.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.51 8.34 8.12 5/185 18360
/proc/meminfo: memFree=24967340/32951124 swapFree=67111524/67111528
[pid=18360] ppid=18358 vsize=9500 CPUtime=0 cores=1,3,5,7
/proc/18360/stat : 18360 (ttsth-5-4) R 18358 18360 18211 0 -1 4202496 429 0 0 0 0 0 0 0 25 0 1 0 516233532 9728000 372 33554432000 4194304 4338100 140733361405904 18446744073709551615 232335553221 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/18360/statm: 2375 372 147 36 0 280 0

[startup+0.034047 s]
/proc/loadavg: 8.51 8.34 8.12 5/185 18360
/proc/meminfo: memFree=24967340/32951124 swapFree=67111524/67111528
[pid=18360] ppid=18358 vsize=14988 CPUtime=0.03 cores=1,3,5,7
/proc/18360/stat : 18360 (ttsth-5-4) R 18358 18360 18211 0 -1 4202496 1790 0 0 0 3 0 0 0 25 0 1 0 516233532 15347712 1733 33554432000 4194304 4338100 140733361405904 18446744073709551615 232335553221 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/18360/statm: 3747 1737 151 36 0 1652 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 14988

[startup+0.101039 s]
/proc/loadavg: 8.51 8.34 8.12 5/185 18360
/proc/meminfo: memFree=24967340/32951124 swapFree=67111524/67111528
[pid=18360] ppid=18358 vsize=18924 CPUtime=0.09 cores=1,3,5,7
/proc/18360/stat : 18360 (ttsth-5-4) R 18358 18360 18211 0 -1 4202496 2941 0 0 0 9 0 0 0 25 0 1 0 516233532 19378176 2751 33554432000 4194304 4338100 140733361405904 18446744073709551615 4245186 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/18360/statm: 4731 2751 155 36 0 2636 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 18924

[startup+0.301013 s]
/proc/loadavg: 8.51 8.34 8.12 5/185 18360
/proc/meminfo: memFree=24967340/32951124 swapFree=67111524/67111528
[pid=18360] ppid=18358 vsize=18924 CPUtime=0.29 cores=1,3,5,7
/proc/18360/stat : 18360 (ttsth-5-4) R 18358 18360 18211 0 -1 4202496 2942 0 0 0 29 0 0 0 25 0 1 0 516233532 19378176 2752 33554432000 4194304 4338100 140733361405904 18446744073709551615 4295986 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/18360/statm: 4731 2752 156 36 0 2636 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 18924

[startup+0.70094 s]
/proc/loadavg: 8.51 8.34 8.12 5/185 18360
/proc/meminfo: memFree=24967340/32951124 swapFree=67111524/67111528
[pid=18360] ppid=18358 vsize=18924 CPUtime=0.69 cores=1,3,5,7
/proc/18360/stat : 18360 (ttsth-5-4) R 18358 18360 18211 0 -1 4202496 2942 0 0 0 69 0 0 0 25 0 1 0 516233532 19378176 2752 33554432000 4194304 4338100 140733361405904 18446744073709551615 4245205 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/18360/statm: 4731 2752 156 36 0 2636 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 18924

[startup+1.5008 s]
/proc/loadavg: 8.22 8.29 8.10 6/186 18361
/proc/meminfo: memFree=24956376/32951124 swapFree=67111524/67111528
[pid=18360] ppid=18358 vsize=18924 CPUtime=1.49 cores=1,3,5,7
/proc/18360/stat : 18360 (ttsth-5-4) R 18358 18360 18211 0 -1 4202496 2942 0 0 0 149 0 0 0 25 0 1 0 516233532 19378176 2752 33554432000 4194304 4338100 140733361405904 18446744073709551615 4245192 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/18360/statm: 4731 2752 156 36 0 2636 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 18924

[startup+3.10047 s]
/proc/loadavg: 8.22 8.29 8.10 10/190 18365
/proc/meminfo: memFree=24917780/32951124 swapFree=67111524/67111528
[pid=18360] ppid=18358 vsize=309360 CPUtime=7.33 cores=1,3,5,7
/proc/18360/stat : 18360 (ttsth-5-4) S 18358 18360 18211 0 -1 4202496 14835 0 0 0 729 4 0 0 25 0 5 0 516233532 316784640 14579 33554432000 4194304 4338100 140733361405904 18446744073709551615 232347695077 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18360/statm: 77340 14579 176 36 0 75245 0
[pid=18360/tid=18362] ppid=18358 vsize=309360 CPUtime=1.4 cores=1,3,5,7
/proc/18360/task/18362/stat : 18362 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 700 0 0 0 140 0 0 0 20 0 5 0 516233700 316784640 14579 33554432000 4194304 4338100 140733361405904 18446744073709551615 232335287330 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18360/tid=18363] ppid=18358 vsize=309360 CPUtime=1.41 cores=1,3,5,7
/proc/18360/task/18363/stat : 18363 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 710 0 0 0 141 0 0 0 20 0 5 0 516233700 316784640 14579 33554432000 4194304 4338100 140733361405904 18446744073709551615 232335287521 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=18360/tid=18364] ppid=18358 vsize=309360 CPUtime=1.4 cores=1,3,5,7
/proc/18360/task/18364/stat : 18364 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 514 0 0 0 140 0 0 0 20 0 5 0 516233700 316784640 14579 33554432000 4194304 4338100 140733361405904 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=18360/tid=18365] ppid=18358 vsize=309360 CPUtime=1.4 cores=1,3,5,7
/proc/18360/task/18365/stat : 18365 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 1637 0 0 0 140 0 0 0 25 0 5 0 516233700 316784640 14580 33554432000 4194304 4338100 140733361405904 18446744073709551615 4254291 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 7.33
Current children cumulated vsize (KiB) 309360

[startup+6.3009 s]
/proc/loadavg: 8.22 8.29 8.10 9/190 18365
/proc/meminfo: memFree=24894096/32951124 swapFree=67111524/67111528
[pid=18360] ppid=18358 vsize=322176 CPUtime=20.12 cores=1,3,5,7
/proc/18360/stat : 18360 (ttsth-5-4) S 18358 18360 18211 0 -1 4202496 19700 0 0 0 2005 7 0 0 25 0 5 0 516233532 329908224 19444 33554432000 4194304 4338100 140733361405904 18446744073709551615 232347695077 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18360/statm: 80544 19444 176 36 0 78449 0
[pid=18360/tid=18362] ppid=18358 vsize=322176 CPUtime=4.61 cores=1,3,5,7
/proc/18360/task/18362/stat : 18362 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 1333 0 0 0 460 1 0 0 25 0 5 0 516233700 329908224 19444 33554432000 4194304 4338100 140733361405904 18446744073709551615 232335287451 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18360/tid=18363] ppid=18358 vsize=322176 CPUtime=4.61 cores=1,3,5,7
/proc/18360/task/18363/stat : 18363 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 1432 0 0 0 460 1 0 0 25 0 5 0 516233700 329908224 19444 33554432000 4194304 4338100 140733361405904 18446744073709551615 4286690 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=18360/tid=18364] ppid=18358 vsize=322176 CPUtime=4.58 cores=1,3,5,7
/proc/18360/task/18364/stat : 18364 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 1132 0 0 0 457 1 0 0 25 0 5 0 516233700 329908224 19444 33554432000 4194304 4338100 140733361405904 18446744073709551615 4256087 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=18360/tid=18365] ppid=18358 vsize=322176 CPUtime=4.61 cores=1,3,5,7
/proc/18360/task/18365/stat : 18365 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 4529 0 0 0 460 1 0 0 25 0 5 0 516233700 329908224 19444 33554432000 4194304 4338100 140733361405904 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 20.12
Current children cumulated vsize (KiB) 322176

[startup+12.7008 s]
/proc/loadavg: 8.19 8.28 8.10 9/190 18365
/proc/meminfo: memFree=24856000/32951124 swapFree=67111524/67111528
[pid=18360] ppid=18358 vsize=348560 CPUtime=45.67 cores=1,3,5,7
/proc/18360/stat : 18360 (ttsth-5-4) S 18358 18360 18211 0 -1 4202496 29192 0 0 0 4555 12 0 0 25 0 5 0 516233532 356925440 28936 33554432000 4194304 4338100 140733361405904 18446744073709551615 232347695077 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18360/statm: 87140 28936 176 36 0 85045 0
[pid=18360/tid=18362] ppid=18358 vsize=348560 CPUtime=11 cores=1,3,5,7
/proc/18360/task/18362/stat : 18362 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 2896 0 0 0 1098 2 0 0 25 0 5 0 516233700 356925440 28936 33554432000 4194304 4338100 140733361405904 18446744073709551615 232335287764 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18360/tid=18363] ppid=18358 vsize=348560 CPUtime=11.01 cores=1,3,5,7
/proc/18360/task/18363/stat : 18363 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 2723 0 0 0 1099 2 0 0 25 0 5 0 516233700 356925440 28936 33554432000 4194304 4338100 140733361405904 18446744073709551615 4287092 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=18360/tid=18364] ppid=18358 vsize=348560 CPUtime=10.94 cores=1,3,5,7
/proc/18360/task/18364/stat : 18364 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 2330 0 0 0 1092 2 0 0 25 0 5 0 516233700 356925440 28936 33554432000 4194304 4338100 140733361405904 18446744073709551615 4258054 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=18360/tid=18365] ppid=18358 vsize=348560 CPUtime=11.01 cores=1,3,5,7
/proc/18360/task/18365/stat : 18365 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 9969 0 0 0 1098 3 0 0 25 0 5 0 516233700 356925440 28936 33554432000 4194304 4338100 140733361405904 18446744073709551615 232335287764 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 45.67
Current children cumulated vsize (KiB) 348560

[startup+25.5005 s]
/proc/loadavg: 8.16 8.27 8.10 9/190 18365
/proc/meminfo: memFree=24776548/32951124 swapFree=67111524/67111528
[pid=18360] ppid=18358 vsize=403884 CPUtime=96.8 cores=1,3,5,7
/proc/18360/stat : 18360 (ttsth-5-4) S 18358 18360 18211 0 -1 4202496 48224 0 0 0 9643 37 0 0 25 0 5 0 516233532 413577216 47968 33554432000 4194304 4338100 140733361405904 18446744073709551615 232347695077 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18360/statm: 100971 47968 176 36 0 98876 0
[pid=18360/tid=18362] ppid=18358 vsize=403884 CPUtime=23.8 cores=1,3,5,7
/proc/18360/task/18362/stat : 18362 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 7856 0 0 0 2370 10 0 0 25 0 5 0 516233700 413577216 47968 33554432000 4194304 4338100 140733361405904 18446744073709551615 232335287379 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18360/tid=18363] ppid=18358 vsize=403884 CPUtime=23.81 cores=1,3,5,7
/proc/18360/task/18363/stat : 18363 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 5791 0 0 0 2374 7 0 0 25 0 5 0 516233700 413577216 47968 33554432000 4194304 4338100 140733361405904 18446744073709551615 4286296 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=18360/tid=18364] ppid=18358 vsize=403884 CPUtime=23.68 cores=1,3,5,7
/proc/18360/task/18364/stat : 18364 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 4810 0 0 0 2360 8 0 0 25 0 5 0 516233700 413577216 47968 33554432000 4194304 4338100 140733361405904 18446744073709551615 232335588890 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=18360/tid=18365] ppid=18358 vsize=403884 CPUtime=23.8 cores=1,3,5,7
/proc/18360/task/18365/stat : 18365 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 18493 0 0 0 2371 9 0 0 25 0 5 0 516233700 413577216 47968 33554432000 4194304 4338100 140733361405904 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 96.8
Current children cumulated vsize (KiB) 403884

[startup+51.1009 s]
/proc/loadavg: 8.28 8.29 8.11 9/190 18366
/proc/meminfo: memFree=24666272/32951124 swapFree=67111524/67111528
[pid=18360] ppid=18358 vsize=479508 CPUtime=199.04 cores=1,3,5,7
/proc/18360/stat : 18360 (ttsth-5-4) S 18358 18360 18211 0 -1 4202496 75724 0 0 0 19808 96 0 0 25 0 5 0 516233532 491016192 75468 33554432000 4194304 4338100 140733361405904 18446744073709551615 232347695077 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18360/statm: 119877 75468 176 36 0 117782 0
[pid=18360/tid=18362] ppid=18358 vsize=479508 CPUtime=49.4 cores=1,3,5,7
/proc/18360/task/18362/stat : 18362 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 13839 0 0 0 4913 27 0 0 25 0 5 0 516233700 491016192 75468 33554432000 4194304 4338100 140733361405904 18446744073709551615 232347718325 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18360/tid=18363] ppid=18358 vsize=479508 CPUtime=49.4 cores=1,3,5,7
/proc/18360/task/18363/stat : 18363 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 9643 0 0 0 4917 23 0 0 25 0 5 0 516233700 491016192 75468 33554432000 4194304 4338100 140733361405904 18446744073709551615 232347718325 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=18360/tid=18364] ppid=18358 vsize=479508 CPUtime=49.13 cores=1,3,5,7
/proc/18360/task/18364/stat : 18364 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 8477 0 0 0 4893 20 0 0 25 0 5 0 516233700 491016192 75468 33554432000 4194304 4338100 140733361405904 18446744073709551615 232335588809 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=18360/tid=18365] ppid=18358 vsize=479508 CPUtime=49.4 cores=1,3,5,7
/proc/18360/task/18365/stat : 18365 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 32491 0 0 0 4917 23 0 0 25 0 5 0 516233700 491016192 75468 33554432000 4194304 4338100 140733361405904 18446744073709551615 232347718325 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 199.04
Current children cumulated vsize (KiB) 479508

[startup+102.307 s]
/proc/loadavg: 7.83 8.17 8.08 9/190 18429
/proc/meminfo: memFree=24691084/32951124 swapFree=67111524/67111528
[pid=18360] ppid=18358 vsize=612576 CPUtime=403.53 cores=1,3,5,7
/proc/18360/stat : 18360 (ttsth-5-4) S 18358 18360 18211 0 -1 4202496 124615 0 0 0 40082 271 0 0 25 0 5 0 516233532 627277824 124359 33554432000 4194304 4338100 140733361405904 18446744073709551615 232347695077 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18360/statm: 153144 124359 176 36 0 151049 0
[pid=18360/tid=18362] ppid=18358 vsize=612576 CPUtime=100.59 cores=1,3,5,7
/proc/18360/task/18362/stat : 18362 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 24405 0 0 0 9988 71 0 0 25 0 5 0 516233700 627277824 124359 33554432000 4194304 4338100 140733361405904 18446744073709551615 232347718325 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18360/tid=18363] ppid=18358 vsize=612576 CPUtime=100.58 cores=1,3,5,7
/proc/18360/task/18363/stat : 18363 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 20336 0 0 0 9988 70 0 0 20 0 5 0 516233700 627277824 124359 33554432000 4194304 4338100 140733361405904 18446744073709551615 232347718325 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=18360/tid=18364] ppid=18358 vsize=612576 CPUtime=100.11 cores=1,3,5,7
/proc/18360/task/18364/stat : 18364 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 16828 0 0 0 9948 63 0 0 25 0 5 0 516233700 627277824 124359 33554432000 4194304 4338100 140733361405904 18446744073709551615 232347718325 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=18360/tid=18365] ppid=18358 vsize=612576 CPUtime=100.53 cores=1,3,5,7
/proc/18360/task/18365/stat : 18365 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 51772 0 0 0 9989 64 0 0 25 0 5 0 516233700 627277824 124359 33554432000 4194304 4338100 140733361405904 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 403.53
Current children cumulated vsize (KiB) 612576

[startup+162.3 s]
/proc/loadavg: 8.21 8.22 8.10 10/190 18431
/proc/meminfo: memFree=24063276/32951124 swapFree=67111524/67111528
[pid=18360] ppid=18358 vsize=862376 CPUtime=643.12 cores=1,3,5,7
/proc/18360/stat : 18360 (ttsth-5-4) S 18358 18360 18211 0 -1 4202496 169394 0 0 0 63824 488 0 0 25 0 5 0 516233532 883073024 169138 33554432000 4194304 4338100 140733361405904 18446744073709551615 232347695077 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18360/statm: 215594 169138 176 36 0 213499 0
[pid=18360/tid=18362] ppid=18358 vsize=862376 CPUtime=160.56 cores=1,3,5,7
/proc/18360/task/18362/stat : 18362 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 37599 0 0 0 15926 130 0 0 25 0 5 0 516233700 883073024 169138 33554432000 4194304 4338100 140733361405904 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18360/tid=18363] ppid=18358 vsize=862376 CPUtime=160.56 cores=1,3,5,7
/proc/18360/task/18363/stat : 18363 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 29854 0 0 0 15933 123 0 0 25 0 5 0 516233700 883073024 169138 33554432000 4194304 4338100 140733361405904 18446744073709551615 232347718325 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=18360/tid=18364] ppid=18358 vsize=862376 CPUtime=160.09 cores=1,3,5,7
/proc/18360/task/18364/stat : 18364 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 26162 0 0 0 15892 117 0 0 25 0 5 0 516233700 883073024 169138 33554432000 4194304 4338100 140733361405904 18446744073709551615 232347718325 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=18360/tid=18365] ppid=18358 vsize=862376 CPUtime=160.19 cores=1,3,5,7
/proc/18360/task/18365/stat : 18365 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 64505 0 0 0 15905 114 0 0 25 0 5 0 516233700 883073024 169138 33554432000 4194304 4338100 140733361405904 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 643.12
Current children cumulated vsize (KiB) 862376

[startup+222.301 s]
/proc/loadavg: 8.21 8.21 8.10 9/190 18433
/proc/meminfo: memFree=23610788/32951124 swapFree=67111524/67111528
[pid=18360] ppid=18358 vsize=1044924 CPUtime=882.72 cores=1,3,5,7
/proc/18360/stat : 18360 (ttsth-5-4) S 18358 18360 18211 0 -1 4202496 213334 0 0 0 87554 718 0 0 25 0 5 0 516233532 1070002176 213078 33554432000 4194304 4338100 140733361405904 18446744073709551615 232347695077 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18360/statm: 261231 213078 176 36 0 259136 0
[pid=18360/tid=18362] ppid=18358 vsize=1044924 CPUtime=220.55 cores=1,3,5,7
/proc/18360/task/18362/stat : 18362 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 51791 0 0 0 21866 189 0 0 25 0 5 0 516233700 1070002176 213078 33554432000 4194304 4338100 140733361405904 18446744073709551615 232347718325 0 0 4096 0 0 0 0 -1 1 0 0 0

################
# More data... #
################

/proc/18360/task/18365/stat : 18365 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 131280 0 0 0 69313 648 0 0 25 0 5 0 516233700 1857765376 397400 33554432000 4194304 4338100 140733361405904 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 2799.3
Current children cumulated vsize (KiB) 1814224

[startup+762.307 s]
/proc/loadavg: 8.16 8.22 8.17 9/190 18447
/proc/meminfo: memFree=21655964/32951124 swapFree=67111524/67111528
[pid=18360] ppid=18358 vsize=1857160 CPUtime=3038.93 cores=1,3,5,7
/proc/18360/stat : 18360 (ttsth-5-4) S 18358 18360 18211 0 -1 4202496 419563 0 0 0 300982 2911 0 0 25 0 5 0 516233532 1901731840 419307 33554432000 4194304 4338100 140733361405904 18446744073709551615 232347695077 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18360/statm: 464290 419307 176 36 0 462195 0
[pid=18360/tid=18362] ppid=18358 vsize=1857160 CPUtime=758.76 cores=1,3,5,7
/proc/18360/task/18362/stat : 18362 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 108554 0 0 0 75132 744 0 0 25 0 5 0 516233700 1901731840 419307 33554432000 4194304 4338100 140733361405904 18446744073709551615 4286005 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=18360/tid=18363] ppid=18358 vsize=1857160 CPUtime=758.98 cores=1,3,5,7
/proc/18360/task/18363/stat : 18363 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 80378 0 0 0 75163 735 0 0 25 0 5 0 516233700 1901731840 419307 33554432000 4194304 4338100 140733361405904 18446744073709551615 232335588868 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18360/tid=18364] ppid=18358 vsize=1857160 CPUtime=759.87 cores=1,3,5,7
/proc/18360/task/18364/stat : 18364 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 80173 0 0 0 75268 719 0 0 24 0 5 0 516233700 1901731840 419307 33554432000 4194304 4338100 140733361405904 18446744073709551615 232335287730 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=18360/tid=18365] ppid=18358 vsize=1857160 CPUtime=759.6 cores=1,3,5,7
/proc/18360/task/18365/stat : 18365 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 139184 0 0 0 75251 709 0 0 25 0 5 0 516233700 1901731840 419307 33554432000 4194304 4338100 140733361405904 18446744073709551615 4253962 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3038.93
Current children cumulated vsize (KiB) 1857160

[startup+822.301 s]
/proc/loadavg: 8.24 8.22 8.17 10/190 18449
/proc/meminfo: memFree=21554900/32951124 swapFree=67111524/67111528
[pid=18360] ppid=18358 vsize=1888388 CPUtime=3278.46 cores=1,3,5,7
/proc/18360/stat : 18360 (ttsth-5-4) S 18358 18360 18211 0 -1 4202496 437026 0 0 0 324698 3148 0 0 25 0 5 0 516233532 1933709312 436770 33554432000 4194304 4338100 140733361405904 18446744073709551615 232347695077 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18360/statm: 472097 436770 176 36 0 470002 0
[pid=18360/tid=18362] ppid=18358 vsize=1888388 CPUtime=818.41 cores=1,3,5,7
/proc/18360/task/18362/stat : 18362 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 113354 0 0 0 81036 805 0 0 25 0 5 0 516233700 1933709312 436770 33554432000 4194304 4338100 140733361405904 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=18360/tid=18363] ppid=18358 vsize=1888388 CPUtime=818.95 cores=1,3,5,7
/proc/18360/task/18363/stat : 18363 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 83773 0 0 0 81102 793 0 0 25 0 5 0 516233700 1933709312 436770 33554432000 4194304 4338100 140733361405904 18446744073709551615 232347718325 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18360/tid=18364] ppid=18358 vsize=1888388 CPUtime=819.82 cores=1,3,5,7
/proc/18360/task/18364/stat : 18364 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 83831 0 0 0 81202 780 0 0 25 0 5 0 516233700 1933709312 436770 33554432000 4194304 4338100 140733361405904 18446744073709551615 232347718325 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=18360/tid=18365] ppid=18358 vsize=1888388 CPUtime=819.56 cores=1,3,5,7
/proc/18360/task/18365/stat : 18365 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 144794 0 0 0 81190 766 0 0 25 0 5 0 516233700 1933709312 436770 33554432000 4194304 4338100 140733361405904 18446744073709551615 232347718325 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3278.46
Current children cumulated vsize (KiB) 1888388

[startup+882.3 s]
/proc/loadavg: 8.09 8.18 8.16 9/190 18450
/proc/meminfo: memFree=21418856/32951124 swapFree=67111524/67111528
[pid=18360] ppid=18358 vsize=1916476 CPUtime=3517.97 cores=1,3,5,7
/proc/18360/stat : 18360 (ttsth-5-4) S 18358 18360 18211 0 -1 4202496 455769 0 0 0 348381 3416 0 0 25 0 5 0 516233532 1962471424 455513 33554432000 4194304 4338100 140733361405904 18446744073709551615 232347695077 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18360/statm: 479119 455513 176 36 0 477024 0
[pid=18360/tid=18362] ppid=18358 vsize=1916476 CPUtime=878.03 cores=1,3,5,7
/proc/18360/task/18362/stat : 18362 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 120459 0 0 0 86930 873 0 0 25 0 5 0 516233700 1962471424 455513 33554432000 4194304 4338100 140733361405904 18446744073709551615 4254239 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=18360/tid=18363] ppid=18358 vsize=1916476 CPUtime=878.91 cores=1,3,5,7
/proc/18360/task/18363/stat : 18363 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 87650 0 0 0 87026 865 0 0 25 0 5 0 516233700 1962471424 455513 33554432000 4194304 4338100 140733361405904 18446744073709551615 232347718325 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18360/tid=18364] ppid=18358 vsize=1916476 CPUtime=879.79 cores=1,3,5,7
/proc/18360/task/18364/stat : 18364 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 87712 0 0 0 87136 843 0 0 25 0 5 0 516233700 1962471424 455513 33554432000 4194304 4338100 140733361405904 18446744073709551615 232347718325 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=18360/tid=18365] ppid=18358 vsize=1916476 CPUtime=879.53 cores=1,3,5,7
/proc/18360/task/18365/stat : 18365 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 148674 0 0 0 87120 833 0 0 25 0 5 0 516233700 1962471424 455513 33554432000 4194304 4338100 140733361405904 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3517.97
Current children cumulated vsize (KiB) 1916476

[startup+942.301 s]
/proc/loadavg: 8.14 8.17 8.16 9/190 18452
/proc/meminfo: memFree=21251684/32951124 swapFree=67111524/67111528
[pid=18360] ppid=18358 vsize=2148180 CPUtime=3757.52 cores=1,3,5,7
/proc/18360/stat : 18360 (ttsth-5-4) S 18358 18360 18211 0 -1 4202496 475149 0 0 0 372094 3658 0 0 25 0 5 0 516233532 2199736320 474893 33554432000 4194304 4338100 140733361405904 18446744073709551615 232347695077 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18360/statm: 537045 474893 176 36 0 534950 0
[pid=18360/tid=18362] ppid=18358 vsize=2148180 CPUtime=937.66 cores=1,3,5,7
/proc/18360/task/18362/stat : 18362 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 126140 0 0 0 92834 932 0 0 25 0 5 0 516233700 2199736320 474893 33554432000 4194304 4338100 140733361405904 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=18360/tid=18363] ppid=18358 vsize=2148180 CPUtime=938.88 cores=1,3,5,7
/proc/18360/task/18363/stat : 18363 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 94756 0 0 0 92962 926 0 0 25 0 5 0 516233700 2199736320 474893 33554432000 4194304 4338100 140733361405904 18446744073709551615 4254429 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18360/tid=18364] ppid=18358 vsize=2148180 CPUtime=939.77 cores=1,3,5,7
/proc/18360/task/18364/stat : 18364 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 91007 0 0 0 93075 902 0 0 25 0 5 0 516233700 2199736320 474893 33554432000 4194304 4338100 140733361405904 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=18360/tid=18365] ppid=18358 vsize=2148180 CPUtime=939.51 cores=1,3,5,7
/proc/18360/task/18365/stat : 18365 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 151972 0 0 0 93056 895 0 0 25 0 5 0 516233700 2199736320 474893 33554432000 4194304 4338100 140733361405904 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3757.52
Current children cumulated vsize (KiB) 2148180

[startup+1002.3 s]
/proc/loadavg: 8.11 8.16 8.15 9/190 18453
/proc/meminfo: memFree=21072116/32951124 swapFree=67111524/67111528
[pid=18360] ppid=18358 vsize=2214292 CPUtime=3997.08 cores=1,3,5,7
/proc/18360/stat : 18360 (ttsth-5-4) S 18358 18360 18211 0 -1 4202496 500619 0 0 0 395802 3906 0 0 25 0 5 0 516233532 2267435008 500363 33554432000 4194304 4338100 140733361405904 18446744073709551615 232347695077 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18360/statm: 553573 500363 176 36 0 551478 0
[pid=18360/tid=18362] ppid=18358 vsize=2214292 CPUtime=997.31 cores=1,3,5,7
/proc/18360/task/18362/stat : 18362 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 134314 0 0 0 98739 992 0 0 25 0 5 0 516233700 2267435008 500363 33554432000 4194304 4338100 140733361405904 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=18360/tid=18363] ppid=18358 vsize=2214292 CPUtime=998.85 cores=1,3,5,7
/proc/18360/task/18363/stat : 18363 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 101411 0 0 0 98897 988 0 0 25 0 5 0 516233700 2267435008 500363 33554432000 4194304 4338100 140733361405904 18446744073709551615 232347718325 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18360/tid=18364] ppid=18358 vsize=2214292 CPUtime=999.72 cores=1,3,5,7
/proc/18360/task/18364/stat : 18364 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 96429 0 0 0 99009 963 0 0 25 0 5 0 516233700 2267435008 500363 33554432000 4194304 4338100 140733361405904 18446744073709551615 232347718325 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=18360/tid=18365] ppid=18358 vsize=2214292 CPUtime=999.48 cores=1,3,5,7
/proc/18360/task/18365/stat : 18365 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 157191 0 0 0 98988 960 0 0 18 0 5 0 516233700 2267435008 500363 33554432000 4194304 4338100 140733361405904 18446744073709551615 232347718325 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3997.08
Current children cumulated vsize (KiB) 2214292

[startup+1062.3 s]
/proc/loadavg: 8.20 8.17 8.16 9/190 18455
/proc/meminfo: memFree=20932104/32951124 swapFree=67111524/67111528
[pid=18360] ppid=18358 vsize=2268288 CPUtime=4236.67 cores=1,3,5,7
/proc/18360/stat : 18360 (ttsth-5-4) S 18358 18360 18211 0 -1 4202496 524534 0 0 0 419518 4149 0 0 25 0 5 0 516233532 2322726912 524278 33554432000 4194304 4338100 140733361405904 18446744073709551615 232347695077 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18360/statm: 567072 524278 176 36 0 564977 0
[pid=18360/tid=18362] ppid=18358 vsize=2268288 CPUtime=1056.96 cores=1,3,5,7
/proc/18360/task/18362/stat : 18362 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 140896 0 0 0 104645 1051 0 0 25 0 5 0 516233700 2322726912 524278 33554432000 4194304 4338100 140733361405904 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=18360/tid=18363] ppid=18358 vsize=2268288 CPUtime=1058.83 cores=1,3,5,7
/proc/18360/task/18363/stat : 18363 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 107521 0 0 0 104834 1049 0 0 25 0 5 0 516233700 2322726912 524278 33554432000 4194304 4338100 140733361405904 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18360/tid=18364] ppid=18358 vsize=2268288 CPUtime=1059.7 cores=1,3,5,7
/proc/18360/task/18364/stat : 18364 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 102693 0 0 0 104945 1025 0 0 25 0 5 0 516233700 2322726912 524278 33554432000 4194304 4338100 140733361405904 18446744073709551615 232347718325 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=18360/tid=18365] ppid=18358 vsize=2268288 CPUtime=1059.46 cores=1,3,5,7
/proc/18360/task/18365/stat : 18365 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 162150 0 0 0 104926 1020 0 0 25 0 5 0 516233700 2322726912 524278 33554432000 4194304 4338100 140733361405904 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4236.67
Current children cumulated vsize (KiB) 2268288

[startup+1122.3 s]
/proc/loadavg: 8.25 8.18 8.16 10/190 18457
/proc/meminfo: memFree=20796060/32951124 swapFree=67111524/67111528
[pid=18360] ppid=18358 vsize=2295056 CPUtime=4476.18 cores=1,3,5,7
/proc/18360/stat : 18360 (ttsth-5-4) S 18358 18360 18211 0 -1 4202496 541122 0 0 0 443227 4391 0 0 25 0 5 0 516233532 2350137344 540866 33554432000 4194304 4338100 140733361405904 18446744073709551615 232347695077 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18360/statm: 573764 540866 176 36 0 571669 0
[pid=18360/tid=18362] ppid=18358 vsize=2295056 CPUtime=1116.83 cores=1,3,5,7
/proc/18360/task/18362/stat : 18362 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 145495 0 0 0 110571 1112 0 0 25 0 5 0 516233700 2350137344 540866 33554432000 4194304 4338100 140733361405904 18446744073709551615 232347718325 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=18360/tid=18363] ppid=18358 vsize=2295056 CPUtime=1118.77 cores=1,3,5,7
/proc/18360/task/18363/stat : 18363 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 111695 0 0 0 110769 1108 0 0 25 0 5 0 516233700 2350137344 540867 33554432000 4194304 4338100 140733361405904 18446744073709551615 4206915 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18360/tid=18364] ppid=18358 vsize=2295056 CPUtime=1119.44 cores=1,3,5,7
/proc/18360/task/18364/stat : 18364 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 106962 0 0 0 110857 1087 0 0 25 0 5 0 516233700 2350137344 540867 33554432000 4194304 4338100 140733361405904 18446744073709551615 4254347 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=18360/tid=18365] ppid=18358 vsize=2295056 CPUtime=1119.41 cores=1,3,5,7
/proc/18360/task/18365/stat : 18365 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 165696 0 0 0 110861 1080 0 0 25 0 5 0 516233700 2350137344 540867 33554432000 4194304 4338100 140733361405904 18446744073709551615 232335588854 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4476.18
Current children cumulated vsize (KiB) 2295056

[startup+1182.3 s]
/proc/loadavg: 8.34 8.22 8.18 9/190 18458
/proc/meminfo: memFree=20686800/32951124 swapFree=67111524/67111528
[pid=18360] ppid=18358 vsize=2323896 CPUtime=4715.67 cores=1,3,5,7
/proc/18360/stat : 18360 (ttsth-5-4) S 18358 18360 18211 0 -1 4202496 558875 0 0 0 466931 4636 0 0 25 0 5 0 516233532 2379669504 558619 33554432000 4194304 4338100 140733361405904 18446744073709551615 232347695077 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18360/statm: 580974 558619 176 36 0 578879 0
[pid=18360/tid=18362] ppid=18358 vsize=2323896 CPUtime=1176.72 cores=1,3,5,7
/proc/18360/task/18362/stat : 18362 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 152027 0 0 0 116498 1174 0 0 25 0 5 0 516233700 2379669504 558619 33554432000 4194304 4338100 140733361405904 18446744073709551615 232347718325 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=18360/tid=18363] ppid=18358 vsize=2323896 CPUtime=1178.75 cores=1,3,5,7
/proc/18360/task/18363/stat : 18363 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 115421 0 0 0 116703 1172 0 0 25 0 5 0 516233700 2379669504 558619 33554432000 4194304 4338100 140733361405904 18446744073709551615 232347718325 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18360/tid=18364] ppid=18358 vsize=2323896 CPUtime=1179.34 cores=1,3,5,7
/proc/18360/task/18364/stat : 18364 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 110288 0 0 0 116786 1148 0 0 25 0 5 0 516233700 2379669504 558619 33554432000 4194304 4338100 140733361405904 18446744073709551615 232347718325 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=18360/tid=18365] ppid=18358 vsize=2323896 CPUtime=1179.16 cores=1,3,5,7
/proc/18360/task/18365/stat : 18365 (ttsth-5-4) R 18358 18360 18211 0 -1 4202560 169865 0 0 0 116776 1140 0 0 25 0 5 0 516233700 2379669504 558619 33554432000 4194304 4338100 140733361405904 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4715.67
Current children cumulated vsize (KiB) 2323896



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1203.5 s]
/proc/loadavg: 8.38 8.24 8.18 6/187 18459
/proc/meminfo: memFree=22041276/32951124 swapFree=67111524/67111528
[pid=18360] ppid=18358 vsize=0 CPUtime=4800.12 cores=1,3,5,7
/proc/18360/stat : 18360 (ttsth-5-4) Z 18358 18360 18211 0 -1 4203548 564419 0 0 0 475293 4719 0 0 18 0 2 0 516233532 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18360/statm: 0 0 0 0 0 0 0
[pid=18360/tid=18365] ppid=18358 vsize=0 CPUtime=1200.22 cores=1,3,5,7
/proc/18360/task/18365/stat : 18365 (ttsth-5-4) R 18358 18360 18211 0 -1 4203588 170820 0 0 0 118856 1166 0 0 18 0 2 0 516233700 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.12
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1203.5 s]
/proc/loadavg: 8.38 8.24 8.18 6/187 18459
/proc/meminfo: memFree=22041276/32951124 swapFree=67111524/67111528
[pid=18360] ppid=18358 vsize=0 CPUtime=4800.12 cores=1,3,5,7
/proc/18360/stat : 18360 (ttsth-5-4) Z 18358 18360 18211 0 -1 4203548 564419 0 0 0 475293 4719 0 0 18 0 2 0 516233532 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18360/statm: 0 0 0 0 0 0 0
[pid=18360/tid=18365] ppid=18358 vsize=0 CPUtime=1200.22 cores=1,3,5,7
/proc/18360/task/18365/stat : 18365 (ttsth-5-4) R 18358 18360 18211 0 -1 4203588 170820 0 0 0 118856 1166 0 0 18 0 2 0 516233700 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.12
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1203.54
CPU time (s): 4800.16
CPU user time (s): 4752.93
CPU system time (s): 47.2268
CPU usage (%): 398.838
Max. virtual memory (cumulated for all children) (KiB): 2333980

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4752.93
system time used= 47.2268
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 564419
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 206111
involuntary context switches= 20780

runsolver used 1.9487 second user time and 4.6323 second system time

The end

Launcher Data 

Begin job on node132 at 2011-04-29 05:05:54
IDJOB=3257470
IDBENCH=70930
IDSOLVER=1654
FILE ID=node132/3257470-1304046354
RUNJOBID= node132-1304045149-18230
PBS_JOBID= 13170396
Free space on /tmp= 71868 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT09/CRAFTED/edgematching/fbcolors/em_8_4_5_fbc.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3257470-1304046354/watcher-3257470-1304046354 -o /tmp/evaluation-result-3257470-1304046354/solver-3257470-1304046354 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3257470-1304046354.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 6fdce6b69c397ad71b0f25957edff1fa
RANDOM SEED=1584513301

node132.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      24967620 kB
Buffers:        760332 kB
Cached:        6211196 kB
SwapCached:          4 kB
Active:        2311496 kB
Inactive:      5174480 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      24967620 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:             428 kB
Writeback:           0 kB
AnonPages:      513896 kB
Mapped:          15232 kB
Slab:           434896 kB
PageTables:       5748 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   696320 kB
VmallocTotal: 34359738367 kB
VmallocUsed:     33868 kB
VmallocChunk: 34359703231 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71868 MiB
End job on node132 at 2011-04-29 05:25:59