Trace number 3260110

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.28 1204.61

General information on the benchmark

Namecrafted/mosoi/
289/289-sat-6x20.cnf
MD5SUMd7291dc9e56649ed6dcce35d6dd461ae
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.010997
Satisfiable
(Un)Satisfiability was proved
Number of variables480
Number of clauses12240
Sum of the clauses size47520
Maximum clause length4
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2720
Number of clauses of size 30
Number of clauses of size 411520
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.01	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3260110-1304062369.cnf 
0.00/0.01	c Using SATTIMEOUT of 4800
0.00/0.01	c Using SATRAM of 15500
0.00/0.01	c Size of subset node = 32
0.00/0.01	c Variables = 480
0.00/0.01	c Clauses = 12240
0.00/0.01	c Literals = 47520
0.00/0.01	c Reduced size:
0.00/0.01	c Variables = 480
0.00/0.01	c Clauses = 12240
0.00/0.01	c Literals = 47520
0.00/0.01	c a2
0.00/0.03	c a1
0.59/0.61	c Independent sub-problems = 1
0.59/0.61	c Sub-problem with 480 variables and 12240 clauses
0.59/0.61	c using permutation method 4
0.59/0.61	c count = 2
0.59/0.61	c Ordering finished 0.610 seconds
0.59/0.61	c mapped 0.610 seconds
0.59/0.61	c permuted 0.610 seconds
0.59/0.61	c about to sort
0.59/0.61	c sorted
0.59/0.63	c clauses now 12240
0.59/0.63	c init finished 0.610 seconds
0.59/0.63	c tree built 0.630 seconds
0.59/0.63	c 
0.59/0.63	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.59/0.63	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.59/0.63	c ................................ | ..........                    |                                  0.6         0.0             
0.59/0.63	c ................................ | ..........                  1 |                                  0.6         1.6             
0.59/0.63	c ................................ | ..........                  2 |                                  0.6         3.2             
0.59/0.63	c ................................ | ..........                  3 |                                  0.6         4.8             
0.59/0.63	c ................................ | ..........                  4 |                                  0.6         6.3             
0.59/0.63	c ................................ | ..........                  5 |                                  0.6         7.9             
0.59/0.64	c ................................ | ..........               1000 | 15           2012                0.7      1538.5        12278
95.69/24.59	c ................................ | ..........             794000 | 169784       1476405            96.1      8264.8      8033125
191.87/48.75	c ................................ | ..........            1543000 | 334623       2716827           192.1      8033.5     13604101
288.06/72.93	c ................................ | ..........            2129000 | 470507       3619406           288.2      7387.2     17917232
384.03/97.01	c ................................ | ..........            2616000 | 589844       4351929           384.1      6811.1     21006129
479.99/121.13	c ................................ | *.........            3082000 | 703318       5036508           480.1      6419.2     23595775
575.90/145.26	c ................................ | *.........            3567000 | 821148       5693084           576.1      6191.4     25742084
671.87/169.38	c ................................ | *.........            3998000 | 942221       6254927           672.2      5947.8     30338327
767.78/193.48	c ................................ | *.........            4398000 | 1050663      6785715           768.1      5725.6     32540791
864.00/217.63	c ................................ | *.........            4820000 | 1160751      7325595           864.1      5577.7     36030187
960.16/241.83	c ................................ | **........            5171000 | 1256115      7768226           960.3      5384.9     36987236
1056.17/265.90	c ................................ | **........            5513000 | 1350789      8183874          1056.2      5219.8     36987236
1151.74/289.99	c ................................ | **........            5856000 | 1443640      8611057          1152.1      5082.9     38375330
1248.07/314.13	c ................................ | **........            6257000 | 1558677      9040214          1248.2      5012.8     39385996
1343.96/338.28	c ................................ | **........            6595000 | 1656832      9397152          1344.3      4906.0     42338706
1440.22/362.38	c ................................ | ***.......            6642000 | 1670594      9458264          1440.6      4610.7     43060477
1536.04/386.31	c ................................ | ***.......            6698000 | 1686513      9522769          1536.1      4360.4     43263387
1631.94/410.33	c ................................ | ***.......            6745000 | 1700060      9590721          1632.1      4132.8     44078310
1727.75/434.38	c ................................ | ***.......            6807000 | 1717539      9664045          1728.1      3939.1     44170182
1823.94/458.44	c ................................ | ***.......            6858000 | 1732681      9729292          1824.1      3759.7     44214574
1919.71/482.49	c ................................ | ****......            6922000 | 1751742      9814918          1920.1      3605.1     44415657
2015.93/506.58	c ................................ | ****......            7008000 | 1774440      9940009          2016.2      3475.8     44577686
2112.44/530.74	c ................................ | ****......            7113000 | 1801785      10077397         2112.6      3366.9     44754185
2208.24/554.75	c ................................ | ****......            7216000 | 1829634      10216932         2208.4      3267.5     44811962
2304.05/578.72	c ................................ | ****......            7317000 | 1858707      10347532         2304.1      3175.6     45009128
2400.71/602.92	c ................................ | *****.....            7405000 | 1882963      10472500         2400.8      3084.3     45612267
2496.52/626.93	c ................................ | *****.....            7511000 | 1910584      10606955         2496.6      3008.4     45763974
2592.34/650.98	c ................................ | *****.....            7603000 | 1935372      10733356         2592.7      2932.5     46362400
2688.13/674.97	c ................................ | *****.....            7705000 | 1963493      10863454         2688.4      2866.0     46714018
2784.36/699.06	c ................................ | *****.....            7825000 | 1996707      11034750         2784.6      2810.1     47074246
2880.05/723.04	c ................................ | ******....            8000000 | 2041113      11257945         2880.2      2777.6     47571962
2976.43/747.23	c ................................ | ******....            8153000 | 2079734      11471560         2976.6      2739.1     47571962
3072.19/771.26	c ................................ | ******....            8320000 | 2123033      11695273         3072.4      2707.9     47786058
3167.99/795.30	c ................................ | ******....            8469000 | 2162281      11899130         3168.4      2673.0     47881017
3264.55/819.43	c ................................ | ******....            8568000 | 2190526      12046734         3264.7      2624.5     49117446
3360.69/843.51	c ................................ | *******...            8666000 | 2218277      12166944         3360.7      2578.6     50004745
3456.79/867.67	c ................................ | *******...            8762000 | 2246382      12281743         3457.1      2534.5     50004745
3552.54/891.61	c ................................ | *******...            8866000 | 2276765      12420210         3552.6      2495.6     50471574
3648.32/915.66	c ................................ | *******...            9022000 | 2318768      12613452         3648.6      2472.7     51837502
3744.04/939.61	c ................................ | *******...            9178000 | 2359274      12817690         3744.1      2451.3     52829138
3840.83/963.91	c ................................ | ********..            9320000 | 2397106      13012456         3840.9      2426.5     53931798
3936.15/987.84	c ................................ | ********..            9460000 | 2436195      13190382         3936.3      2403.3     54180058
4032.20/1011.91	c ................................ | ********..            9595000 | 2472451      13368055         4032.2      2379.6     54449102
4128.29/1036.04	c ................................ | ********..            9740000 | 2510673      13543809         4128.5      2359.2     54449102
4224.43/1060.10	c ................................ | ********..            9891000 | 2552354      13751324         4224.4      2341.4     55206897
4320.00/1084.12	c ................................ | *********.           10112000 | 2608646      14011228         4320.1      2340.7     55798848
4416.06/1108.28	c ................................ | *********.           10315000 | 2659490      14276652         4416.4      2335.6     56243174
4511.72/1132.27	c ................................ | *********.           10504000 | 2706819      14524808         4512.0      2328.0     57161259
4608.20/1156.41	c ................................ | *********.           10634000 | 2743956      14695295         4608.2      2307.6     58030636
4704.33/1180.54	c ................................ | *********.           10740000 | 2777578      14808496         4704.5      2282.9     58437956

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3260110-1304062369/watcher-3260110-1304062369 -o /tmp/evaluation-result-3260110-1304062369/solver-3260110-1304062369 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3260110-1304062369.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 3.70 4.62 6.21 3/176 31590
/proc/meminfo: memFree=15118068/32950928 swapFree=67111528/67111528
[pid=31590] ppid=31588 vsize=9408 CPUtime=0 cores=0,2,4,6
/proc/31590/stat : 31590 (ttsth-5-4) R 31588 31590 31098 0 -1 4202496 407 0 0 0 0 0 0 0 23 0 1 0 517836533 9633792 349 33554432000 4194304 4338100 140737338431584 18446744073709551615 274473628357 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/31590/statm: 2352 350 147 36 0 257 0

[startup+0.056651 s]
/proc/loadavg: 3.70 4.62 6.21 3/176 31590
/proc/meminfo: memFree=15118068/32950928 swapFree=67111528/67111528
[pid=31590] ppid=31588 vsize=12320 CPUtime=0.05 cores=0,2,4,6
/proc/31590/stat : 31590 (ttsth-5-4) R 31588 31590 31098 0 -1 4202496 1170 0 0 0 5 0 0 0 23 0 1 0 517836533 12615680 1087 33554432000 4194304 4338100 140737338431584 18446744073709551615 4295973 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/31590/statm: 3080 1087 155 36 0 985 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 12320

[startup+0.100712 s]
/proc/loadavg: 3.70 4.62 6.21 3/176 31590
/proc/meminfo: memFree=15118068/32950928 swapFree=67111528/67111528
[pid=31590] ppid=31588 vsize=12320 CPUtime=0.09 cores=0,2,4,6
/proc/31590/stat : 31590 (ttsth-5-4) R 31588 31590 31098 0 -1 4202496 1170 0 0 0 9 0 0 0 23 0 1 0 517836533 12615680 1087 33554432000 4194304 4338100 140737338431584 18446744073709551615 4296000 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/31590/statm: 3080 1087 155 36 0 985 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 12320

[startup+0.300672 s]
/proc/loadavg: 3.70 4.62 6.21 3/176 31590
/proc/meminfo: memFree=15118068/32950928 swapFree=67111528/67111528
[pid=31590] ppid=31588 vsize=12320 CPUtime=0.29 cores=0,2,4,6
/proc/31590/stat : 31590 (ttsth-5-4) R 31588 31590 31098 0 -1 4202496 1170 0 0 0 29 0 0 0 25 0 1 0 517836533 12615680 1087 33554432000 4194304 4338100 140737338431584 18446744073709551615 4295986 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/31590/statm: 3080 1087 155 36 0 985 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 12320

[startup+0.700491 s]
/proc/loadavg: 3.70 4.62 6.21 3/176 31590
/proc/meminfo: memFree=15118068/32950928 swapFree=67111528/67111528
[pid=31590] ppid=31588 vsize=135952 CPUtime=0.88 cores=0,2,4,6
/proc/31590/stat : 31590 (ttsth-5-4) S 31588 31590 31098 0 -1 4202496 3554 0 0 0 88 0 0 0 25 0 5 0 517836533 139214848 3471 33554432000 4194304 4338100 140737338431584 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/31590/statm: 33988 3471 175 36 0 31893 0
Current children cumulated CPU time (s) 0.88
Current children cumulated vsize (KiB) 135952

[startup+1.50035 s]
/proc/loadavg: 3.70 4.62 6.21 10/186 31601
/proc/meminfo: memFree=15070016/32950928 swapFree=67111528/67111528
[pid=31590] ppid=31588 vsize=227568 CPUtime=4.08 cores=0,2,4,6
/proc/31590/stat : 31590 (ttsth-5-4) S 31588 31590 31098 0 -1 4202496 10191 0 0 0 406 2 0 0 25 0 5 0 517836533 233029632 10108 33554432000 4194304 4338100 140737338431584 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/31590/statm: 56892 10108 175 36 0 54797 0
[pid=31590/tid=31598] ppid=31588 vsize=227568 CPUtime=0.86 cores=0,2,4,6
/proc/31590/task/31598/stat : 31598 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 2020 0 0 0 86 0 0 0 25 0 5 0 517836597 233029632 10108 33554432000 4194304 4338100 140737338431584 18446744073709551615 274473664004 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=31590/tid=31599] ppid=31588 vsize=227568 CPUtime=0.85 cores=0,2,4,6
/proc/31590/task/31599/stat : 31599 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 1924 0 0 0 85 0 0 0 19 0 5 0 517836597 233029632 10108 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=31590/tid=31600] ppid=31588 vsize=227568 CPUtime=0.84 cores=0,2,4,6
/proc/31590/task/31600/stat : 31600 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 1911 0 0 0 84 0 0 0 25 0 5 0 517836597 233029632 10108 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=31590/tid=31601] ppid=31588 vsize=227568 CPUtime=0.86 cores=0,2,4,6
/proc/31590/task/31601/stat : 31601 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 2005 0 0 0 86 0 0 0 19 0 5 0 517836597 233029632 10109 33554432000 4194304 4338100 140737338431584 18446744073709551615 274473362461 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4.08
Current children cumulated vsize (KiB) 227568

[startup+3.10107 s]
/proc/loadavg: 3.70 4.62 6.21 8/186 31601
/proc/meminfo: memFree=15011784/32950928 swapFree=67111528/67111528
[pid=31590] ppid=31588 vsize=250400 CPUtime=10.47 cores=0,2,4,6
/proc/31590/stat : 31590 (ttsth-5-4) S 31588 31590 31098 0 -1 4202496 17144 0 0 0 1043 4 0 0 25 0 5 0 517836533 256409600 17061 33554432000 4194304 4338100 140737338431584 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/31590/statm: 62600 17061 175 36 0 60505 0
[pid=31590/tid=31598] ppid=31588 vsize=250400 CPUtime=2.45 cores=0,2,4,6
/proc/31590/task/31598/stat : 31598 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 3368 0 0 0 245 0 0 0 25 0 5 0 517836597 256409600 17061 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=31590/tid=31599] ppid=31588 vsize=250400 CPUtime=2.45 cores=0,2,4,6
/proc/31590/task/31599/stat : 31599 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 3690 0 0 0 245 0 0 0 22 0 5 0 517836597 256409600 17061 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254244 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=31590/tid=31600] ppid=31588 vsize=250400 CPUtime=2.44 cores=0,2,4,6
/proc/31590/task/31600/stat : 31600 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 3360 0 0 0 243 1 0 0 18 0 5 0 517836597 256409600 17061 33554432000 4194304 4338100 140737338431584 18446744073709551615 4284064 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=31590/tid=31601] ppid=31588 vsize=250400 CPUtime=2.45 cores=0,2,4,6
/proc/31590/task/31601/stat : 31601 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 4395 0 0 0 245 0 0 0 22 0 5 0 517836597 256409600 17061 33554432000 4194304 4338100 140737338431584 18446744073709551615 274473362681 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 10.47
Current children cumulated vsize (KiB) 250400

[startup+6.30051 s]
/proc/loadavg: 4.05 4.67 6.21 10/186 31601
/proc/meminfo: memFree=14926972/32950928 swapFree=67111528/67111528
[pid=31590] ppid=31588 vsize=343676 CPUtime=23.2 cores=0,2,4,6
/proc/31590/stat : 31590 (ttsth-5-4) S 31588 31590 31098 0 -1 4202496 24774 0 0 0 2307 13 0 0 25 0 5 0 517836533 351924224 24691 33554432000 4194304 4338100 140737338431584 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/31590/statm: 85919 24691 176 36 0 83824 0
[pid=31590/tid=31598] ppid=31588 vsize=343676 CPUtime=5.65 cores=0,2,4,6
/proc/31590/task/31598/stat : 31598 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 4969 0 0 0 562 3 0 0 25 0 5 0 517836597 351924224 24691 33554432000 4194304 4338100 140737338431584 18446744073709551615 4242601 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=31590/tid=31599] ppid=31588 vsize=343676 CPUtime=5.64 cores=0,2,4,6
/proc/31590/task/31599/stat : 31599 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 5788 0 0 0 561 3 0 0 25 0 5 0 517836597 351924224 24691 33554432000 4194304 4338100 140737338431584 18446744073709551615 4229545 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=31590/tid=31600] ppid=31588 vsize=343676 CPUtime=5.61 cores=0,2,4,6
/proc/31590/task/31600/stat : 31600 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 5254 0 0 0 558 3 0 0 25 0 5 0 517836597 351924224 24691 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=31590/tid=31601] ppid=31588 vsize=343676 CPUtime=5.63 cores=0,2,4,6
/proc/31590/task/31601/stat : 31601 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 6432 0 0 0 561 2 0 0 20 0 5 0 517836597 351924224 24691 33554432000 4194304 4338100 140737338431584 18446744073709551615 274473362662 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 23.2
Current children cumulated vsize (KiB) 343676

[startup+12.7004 s]
/proc/loadavg: 4.36 4.73 6.22 10/186 31601
/proc/meminfo: memFree=14780244/32950928 swapFree=67111528/67111528
[pid=31590] ppid=31588 vsize=407556 CPUtime=48.73 cores=0,2,4,6
/proc/31590/stat : 31590 (ttsth-5-4) S 31588 31590 31098 0 -1 4202496 41507 0 0 0 4850 23 0 0 25 0 5 0 517836533 417337344 41424 33554432000 4194304 4338100 140737338431584 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/31590/statm: 101889 41424 176 36 0 99794 0
[pid=31590/tid=31598] ppid=31588 vsize=407556 CPUtime=12.04 cores=0,2,4,6
/proc/31590/task/31598/stat : 31598 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 8587 0 0 0 1199 5 0 0 25 0 5 0 517836597 417337344 41424 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254259 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=31590/tid=31599] ppid=31588 vsize=407556 CPUtime=12.03 cores=0,2,4,6
/proc/31590/task/31599/stat : 31599 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 10120 0 0 0 1198 5 0 0 25 0 5 0 517836597 417337344 41424 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=31590/tid=31600] ppid=31588 vsize=407556 CPUtime=11.95 cores=0,2,4,6
/proc/31590/task/31600/stat : 31600 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 9227 0 0 0 1190 5 0 0 18 0 5 0 517836597 417337344 41424 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254321 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=31590/tid=31601] ppid=31588 vsize=407556 CPUtime=12.03 cores=0,2,4,6
/proc/31590/task/31601/stat : 31601 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 11242 0 0 0 1198 5 0 0 25 0 5 0 517836597 417337344 41424 33554432000 4194304 4338100 140737338431584 18446744073709551615 274473362458 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 48.73
Current children cumulated vsize (KiB) 407556

[startup+25.5011 s]
/proc/loadavg: 5.17 4.89 6.25 10/186 31601
/proc/meminfo: memFree=14563888/32950928 swapFree=67111528/67111528
[pid=31590] ppid=31588 vsize=530492 CPUtime=99.68 cores=0,2,4,6
/proc/31590/stat : 31590 (ttsth-5-4) S 31588 31590 31098 0 -1 4202496 73509 0 0 0 9917 51 0 0 25 0 5 0 517836533 543223808 73426 33554432000 4194304 4338100 140737338431584 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/31590/statm: 132623 73426 176 36 0 130528 0
[pid=31590/tid=31598] ppid=31588 vsize=530492 CPUtime=24.8 cores=0,2,4,6
/proc/31590/task/31598/stat : 31598 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 15796 0 0 0 2468 12 0 0 25 0 5 0 517836597 543223808 73426 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=31590/tid=31599] ppid=31588 vsize=530492 CPUtime=24.81 cores=0,2,4,6
/proc/31590/task/31599/stat : 31599 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 17625 0 0 0 2468 13 0 0 25 0 5 0 517836597 543223808 73426 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=31590/tid=31600] ppid=31588 vsize=530492 CPUtime=24.61 cores=0,2,4,6
/proc/31590/task/31600/stat : 31600 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 17137 0 0 0 2449 12 0 0 18 0 5 0 517836597 543223808 73426 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=31590/tid=31601] ppid=31588 vsize=530492 CPUtime=24.81 cores=0,2,4,6
/proc/31590/task/31601/stat : 31601 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 20620 0 0 0 2468 13 0 0 25 0 5 0 517836597 543223808 73426 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 99.68
Current children cumulated vsize (KiB) 530492

[startup+51.1006 s]
/proc/loadavg: 6.13 5.14 6.30 10/186 31602
/proc/meminfo: memFree=14247060/32950928 swapFree=67111528/67111528
[pid=31590] ppid=31588 vsize=714140 CPUtime=201.4 cores=0,2,4,6
/proc/31590/stat : 31590 (ttsth-5-4) S 31588 31590 31098 0 -1 4202496 122958 0 0 0 20018 122 0 0 25 0 5 0 517836533 731279360 122875 33554432000 4194304 4338100 140737338431584 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/31590/statm: 178535 122875 176 36 0 176440 0
[pid=31590/tid=31598] ppid=31588 vsize=714140 CPUtime=50.17 cores=0,2,4,6
/proc/31590/task/31598/stat : 31598 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 28999 0 0 0 4986 31 0 0 19 0 5 0 517836597 731279360 122875 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254352 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=31590/tid=31599] ppid=31588 vsize=714140 CPUtime=50.32 cores=0,2,4,6
/proc/31590/task/31599/stat : 31599 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 26045 0 0 0 5003 29 0 0 25 0 5 0 517836597 731279360 122875 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254451 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=31590/tid=31600] ppid=31588 vsize=714140 CPUtime=49.97 cores=0,2,4,6
/proc/31590/task/31600/stat : 31600 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 29546 0 0 0 4970 27 0 0 25 0 5 0 517836597 731279360 122875 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=31590/tid=31601] ppid=31588 vsize=714140 CPUtime=50.27 cores=0,2,4,6
/proc/31590/task/31601/stat : 31601 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 36037 0 0 0 4994 33 0 0 25 0 5 0 517836597 731279360 122875 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 201.4
Current children cumulated vsize (KiB) 714140

[startup+102.307 s]
/proc/loadavg: 7.27 5.59 6.39 10/186 31603
/proc/meminfo: memFree=13719568/32950928 swapFree=67111528/67111528
[pid=31590] ppid=31588 vsize=993192 CPUtime=405.18 cores=0,2,4,6
/proc/31590/stat : 31590 (ttsth-5-4) S 31588 31590 31098 0 -1 4202496 197353 0 0 0 40315 203 0 0 25 0 5 0 517836533 1017028608 197270 33554432000 4194304 4338100 140737338431584 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/31590/statm: 248298 197270 176 36 0 246203 0
[pid=31590/tid=31598] ppid=31588 vsize=993192 CPUtime=100.78 cores=0,2,4,6
/proc/31590/task/31598/stat : 31598 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 50351 0 0 0 10030 48 0 0 25 0 5 0 517836597 1017028608 197270 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=31590/tid=31599] ppid=31588 vsize=993192 CPUtime=101.39 cores=0,2,4,6
/proc/31590/task/31599/stat : 31599 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 41368 0 0 0 10089 50 0 0 25 0 5 0 517836597 1017028608 197270 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=31590/tid=31600] ppid=31588 vsize=993192 CPUtime=101.01 cores=0,2,4,6
/proc/31590/task/31600/stat : 31600 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 49773 0 0 0 10053 48 0 0 25 0 5 0 517836597 1017028608 197270 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254357 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=31590/tid=31601] ppid=31588 vsize=993192 CPUtime=101.33 cores=0,2,4,6
/proc/31590/task/31601/stat : 31601 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 53530 0 0 0 10078 55 0 0 25 0 5 0 517836597 1017028608 197270 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254433 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 405.18
Current children cumulated vsize (KiB) 993192

[startup+162.301 s]
/proc/loadavg: 7.84 6.06 6.50 10/186 31605
/proc/meminfo: memFree=13182276/32950928 swapFree=67111528/67111528
[pid=31590] ppid=31588 vsize=1237888 CPUtime=644.03 cores=0,2,4,6
/proc/31590/stat : 31590 (ttsth-5-4) S 31588 31590 31098 0 -1 4202496 262012 0 0 0 64084 319 0 0 25 0 5 0 517836533 1267597312 261929 33554432000 4194304 4338100 140737338431584 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/31590/statm: 309472 261929 176 36 0 307377 0
[pid=31590/tid=31598] ppid=31588 vsize=1237888 CPUtime=160.49 cores=0,2,4,6
/proc/31590/task/31598/stat : 31598 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 68928 0 0 0 15970 79 0 0 25 0 5 0 517836597 1267597312 261929 33554432000 4194304 4338100 140737338431584 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=31590/tid=31599] ppid=31588 vsize=1237888 CPUtime=161.06 cores=0,2,4,6
/proc/31590/task/31599/stat : 31599 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 55503 0 0 0 16030 76 0 0 25 0 5 0 517836597 1267597312 261929 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=31590/tid=31600] ppid=31588 vsize=1237888 CPUtime=160.64 cores=0,2,4,6
/proc/31590/task/31600/stat : 31600 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 65742 0 0 0 15986 78 0 0 25 0 5 0 517836597 1267597312 261929 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254352 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=31590/tid=31601] ppid=31588 vsize=1237888 CPUtime=161.19 cores=0,2,4,6
/proc/31590/task/31601/stat : 31601 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 69509 0 0 0 16034 85 0 0 25 0 5 0 517836597 1267597312 261930 33554432000 4194304 4338100 140737338431584 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 644.03
Current children cumulated vsize (KiB) 1237888


################
# More data... #
################

[pid=31590/tid=31599] ppid=31588 vsize=1852240 CPUtime=757.64 cores=0,2,4,6
/proc/31590/task/31599/stat : 31599 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 100548 0 0 0 75562 202 0 0 25 0 5 0 517836597 1896693760 427839 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=31590/tid=31600] ppid=31588 vsize=1852240 CPUtime=759.31 cores=0,2,4,6
/proc/31590/task/31600/stat : 31600 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 96623 0 0 0 75661 270 0 0 18 0 5 0 517836597 1896693760 427839 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=31590/tid=31601] ppid=31588 vsize=1852240 CPUtime=760.06 cores=0,2,4,6
/proc/31590/task/31601/stat : 31601 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 107096 0 0 0 75794 212 0 0 25 0 5 0 517836597 1896693760 427839 33554432000 4194304 4338100 140737338431584 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3036.75
Current children cumulated vsize (KiB) 1852240

[startup+822.301 s]
/proc/loadavg: 8.04 7.83 7.27 10/186 31622
/proc/meminfo: memFree=11245164/32950928 swapFree=67111528/67111528
[pid=31590] ppid=31588 vsize=1897484 CPUtime=3276.11 cores=0,2,4,6
/proc/31590/stat : 31590 (ttsth-5-4) S 31588 31590 31098 0 -1 4202496 441017 0 0 0 326588 1023 0 0 25 0 5 0 517836533 1943023616 440934 33554432000 4194304 4338100 140737338431584 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/31590/statm: 474371 440934 176 36 0 472276 0
[pid=31590/tid=31598] ppid=31588 vsize=1897484 CPUtime=818.99 cores=0,2,4,6
/proc/31590/task/31598/stat : 31598 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 122800 0 0 0 81586 313 0 0 18 0 5 0 517836597 1943023616 440934 33554432000 4194304 4338100 140737338431584 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=31590/tid=31599] ppid=31588 vsize=1897484 CPUtime=817.2 cores=0,2,4,6
/proc/31590/task/31599/stat : 31599 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 104605 0 0 0 81511 209 0 0 18 0 5 0 517836597 1943023616 440934 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=31590/tid=31600] ppid=31588 vsize=1897484 CPUtime=819.22 cores=0,2,4,6
/proc/31590/task/31600/stat : 31600 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 99988 0 0 0 81642 280 0 0 20 0 5 0 517836597 1943023616 440934 33554432000 4194304 4338100 140737338431584 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=31590/tid=31601] ppid=31588 vsize=1897484 CPUtime=820.04 cores=0,2,4,6
/proc/31590/task/31601/stat : 31601 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 111293 0 0 0 81784 220 0 0 25 0 5 0 517836597 1943023616 440934 33554432000 4194304 4338100 140737338431584 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3276.11
Current children cumulated vsize (KiB) 1897484

[startup+882.3 s]
/proc/loadavg: 8.04 7.87 7.32 10/186 31624
/proc/meminfo: memFree=11161324/32950928 swapFree=67111528/67111528
[pid=31590] ppid=31588 vsize=1926692 CPUtime=3515.48 cores=0,2,4,6
/proc/31590/stat : 31590 (ttsth-5-4) S 31588 31590 31098 0 -1 4202496 448979 0 0 0 350484 1064 0 0 25 0 5 0 517836533 1972932608 448896 33554432000 4194304 4338100 140737338431584 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/31590/statm: 481673 448896 176 36 0 479578 0
[pid=31590/tid=31598] ppid=31588 vsize=1926692 CPUtime=878.88 cores=0,2,4,6
/proc/31590/task/31598/stat : 31598 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 124852 0 0 0 87563 325 0 0 25 0 5 0 517836597 1972932608 448896 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254226 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=31590/tid=31599] ppid=31588 vsize=1926692 CPUtime=876.82 cores=0,2,4,6
/proc/31590/task/31599/stat : 31599 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 105913 0 0 0 87465 217 0 0 24 0 5 0 517836597 1972932608 448896 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=31590/tid=31600] ppid=31588 vsize=1926692 CPUtime=879.12 cores=0,2,4,6
/proc/31590/task/31600/stat : 31600 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 102653 0 0 0 87619 293 0 0 25 0 5 0 517836597 1972932608 448896 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=31590/tid=31601] ppid=31588 vsize=1926692 CPUtime=879.98 cores=0,2,4,6
/proc/31590/task/31601/stat : 31601 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 113230 0 0 0 87772 226 0 0 25 0 5 0 517836597 1972932608 448896 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3515.48
Current children cumulated vsize (KiB) 1926692

[startup+942.301 s]
/proc/loadavg: 8.16 7.93 7.37 10/186 31625
/proc/meminfo: memFree=10881704/32950928 swapFree=67111528/67111528
[pid=31590] ppid=31588 vsize=2021984 CPUtime=3754.8 cores=0,2,4,6
/proc/31590/stat : 31590 (ttsth-5-4) S 31588 31590 31098 0 -1 4202496 474092 0 0 0 374353 1127 0 0 25 0 5 0 517836533 2070511616 474009 33554432000 4194304 4338100 140737338431584 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/31590/statm: 505496 474009 176 36 0 503401 0
[pid=31590/tid=31598] ppid=31588 vsize=2021984 CPUtime=938.75 cores=0,2,4,6
/proc/31590/task/31598/stat : 31598 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 131488 0 0 0 93529 346 0 0 25 0 5 0 517836597 2070511616 474009 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=31590/tid=31599] ppid=31588 vsize=2021984 CPUtime=936.38 cores=0,2,4,6
/proc/31590/task/31599/stat : 31599 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 108251 0 0 0 93408 230 0 0 25 0 5 0 517836597 2070511616 474009 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=31590/tid=31600] ppid=31588 vsize=2021984 CPUtime=939.04 cores=0,2,4,6
/proc/31590/task/31600/stat : 31600 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 110426 0 0 0 93592 312 0 0 25 0 5 0 517836597 2070511616 474009 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=31590/tid=31601] ppid=31588 vsize=2021984 CPUtime=939.96 cores=0,2,4,6
/proc/31590/task/31601/stat : 31601 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 121596 0 0 0 93759 237 0 0 25 0 5 0 517836597 2070511616 474009 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254451 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3754.8
Current children cumulated vsize (KiB) 2021984

[startup+1002.3 s]
/proc/loadavg: 8.15 7.95 7.41 10/186 31627
/proc/meminfo: memFree=10747892/32950928 swapFree=67111528/67111528
[pid=31590] ppid=31588 vsize=2067228 CPUtime=3993.84 cores=0,2,4,6
/proc/31590/stat : 31590 (ttsth-5-4) S 31588 31590 31098 0 -1 4202496 486922 0 0 0 398203 1181 0 0 25 0 5 0 517836533 2116841472 486839 33554432000 4194304 4338100 140737338431584 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/31590/statm: 516807 486839 176 36 0 514712 0
[pid=31590/tid=31598] ppid=31588 vsize=2067228 CPUtime=998.57 cores=0,2,4,6
/proc/31590/task/31598/stat : 31598 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 137216 0 0 0 99491 366 0 0 25 0 5 0 517836597 2116841472 486839 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=31590/tid=31599] ppid=31588 vsize=2067228 CPUtime=995.96 cores=0,2,4,6
/proc/31590/task/31599/stat : 31599 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 110453 0 0 0 99353 243 0 0 25 0 5 0 517836597 2116841472 486839 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=31590/tid=31600] ppid=31588 vsize=2067228 CPUtime=998.85 cores=0,2,4,6
/proc/31590/task/31600/stat : 31600 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 111716 0 0 0 99559 326 0 0 25 0 5 0 517836597 2116841472 486839 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=31590/tid=31601] ppid=31588 vsize=2067228 CPUtime=999.79 cores=0,2,4,6
/proc/31590/task/31601/stat : 31601 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 125206 0 0 0 99735 244 0 0 25 0 5 0 517836597 2116841472 486839 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3993.84
Current children cumulated vsize (KiB) 2067228

[startup+1062.3 s]
/proc/loadavg: 8.12 7.98 7.45 10/186 31629
/proc/meminfo: memFree=10663924/32950928 swapFree=67111528/67111528
[pid=31590] ppid=31588 vsize=2096832 CPUtime=4233.16 cores=0,2,4,6
/proc/31590/stat : 31590 (ttsth-5-4) S 31588 31590 31098 0 -1 4202496 495633 0 0 0 422100 1216 0 0 25 0 5 0 517836533 2147155968 495550 33554432000 4194304 4338100 140737338431584 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/31590/statm: 524208 495550 176 36 0 522113 0
[pid=31590/tid=31598] ppid=31588 vsize=2096832 CPUtime=1058.25 cores=0,2,4,6
/proc/31590/task/31598/stat : 31598 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 139404 0 0 0 105447 378 0 0 25 0 5 0 517836597 2147155968 495550 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=31590/tid=31599] ppid=31588 vsize=2096832 CPUtime=1055.76 cores=0,2,4,6
/proc/31590/task/31599/stat : 31599 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 110811 0 0 0 105327 249 0 0 25 0 5 0 517836597 2147155968 495550 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=31590/tid=31600] ppid=31588 vsize=2096832 CPUtime=1058.76 cores=0,2,4,6
/proc/31590/task/31600/stat : 31600 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 113660 0 0 0 105538 338 0 0 25 0 5 0 517836597 2147155968 495550 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=31590/tid=31601] ppid=31588 vsize=2096832 CPUtime=1059.73 cores=0,2,4,6
/proc/31590/task/31601/stat : 31601 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 129427 0 0 0 105723 250 0 0 25 0 5 0 517836597 2147155968 495550 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254239 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4233.16
Current children cumulated vsize (KiB) 2096832

[startup+1122.3 s]
/proc/loadavg: 8.04 7.98 7.48 10/186 31630
/proc/meminfo: memFree=10513368/32950928 swapFree=67111528/67111528
[pid=31590] ppid=31588 vsize=2146260 CPUtime=4472.24 cores=0,2,4,6
/proc/31590/stat : 31590 (ttsth-5-4) S 31588 31590 31098 0 -1 4202496 509106 0 0 0 445934 1290 0 0 25 0 5 0 517836533 2197770240 509023 33554432000 4194304 4338100 140737338431584 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/31590/statm: 536565 509023 176 36 0 534470 0
[pid=31590/tid=31598] ppid=31588 vsize=2146260 CPUtime=1117.72 cores=0,2,4,6
/proc/31590/task/31598/stat : 31598 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 142805 0 0 0 111367 405 0 0 25 0 5 0 517836597 2197770240 509023 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=31590/tid=31599] ppid=31588 vsize=2146260 CPUtime=1115.68 cores=0,2,4,6
/proc/31590/task/31599/stat : 31599 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 111338 0 0 0 111302 266 0 0 25 0 5 0 517836597 2197770240 509023 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=31590/tid=31600] ppid=31588 vsize=2146260 CPUtime=1118.6 cores=0,2,4,6
/proc/31590/task/31600/stat : 31600 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 116976 0 0 0 111504 356 0 0 18 0 5 0 517836597 2197770240 509023 33554432000 4194304 4338100 140737338431584 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=31590/tid=31601] ppid=31588 vsize=2146260 CPUtime=1119.58 cores=0,2,4,6
/proc/31590/task/31601/stat : 31601 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 135656 0 0 0 111695 263 0 0 25 0 5 0 517836597 2197770240 509023 33554432000 4194304 4338100 140737338431584 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4472.24
Current children cumulated vsize (KiB) 2146260

[startup+1182.3 s]
/proc/loadavg: 8.10 8.01 7.52 10/186 31632
/proc/meminfo: memFree=10419864/32950928 swapFree=67111528/67111528
[pid=31590] ppid=31588 vsize=2197760 CPUtime=4711.5 cores=0,2,4,6
/proc/31590/stat : 31590 (ttsth-5-4) S 31588 31590 31098 0 -1 4202496 523759 0 0 0 469813 1337 0 0 25 0 5 0 517836533 2250506240 523676 33554432000 4194304 4338100 140737338431584 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/31590/statm: 549440 523676 176 36 0 547345 0
[pid=31590/tid=31598] ppid=31588 vsize=2197760 CPUtime=1177.58 cores=0,2,4,6
/proc/31590/task/31598/stat : 31598 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 147736 0 0 0 117336 422 0 0 18 0 5 0 517836597 2250506240 523676 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=31590/tid=31599] ppid=31588 vsize=2197760 CPUtime=1175.54 cores=0,2,4,6
/proc/31590/task/31599/stat : 31599 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 112082 0 0 0 117278 276 0 0 25 0 5 0 517836597 2250506240 523676 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=31590/tid=31600] ppid=31588 vsize=2197760 CPUtime=1178.18 cores=0,2,4,6
/proc/31590/task/31600/stat : 31600 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 123411 0 0 0 117450 368 0 0 20 0 5 0 517836597 2250506240 523676 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=31590/tid=31601] ppid=31588 vsize=2197760 CPUtime=1179.54 cores=0,2,4,6
/proc/31590/task/31601/stat : 31601 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 138199 0 0 0 117684 270 0 0 25 0 5 0 517836597 2250506240 523676 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254275 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4711.5
Current children cumulated vsize (KiB) 2197760



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1204.51 s]
/proc/loadavg: 8.07 8.00 7.53 10/186 31633
/proc/meminfo: memFree=10407204/32950928 swapFree=67111528/67111528
[pid=31590] ppid=31588 vsize=2207540 CPUtime=4800.17 cores=0,2,4,6
/proc/31590/stat : 31590 (ttsth-5-4) S 31588 31590 31098 0 -1 4202496 526053 0 0 0 478672 1345 0 0 25 0 5 0 517836533 2260520960 525970 33554432000 4194304 4338100 140737338431584 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/31590/statm: 551885 525970 176 36 0 549790 0
[pid=31590/tid=31598] ppid=31588 vsize=2207540 CPUtime=1199.78 cores=0,2,4,6
/proc/31590/task/31598/stat : 31598 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 147815 0 0 0 119554 424 0 0 25 0 5 0 517836597 2260520960 525970 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=31590/tid=31599] ppid=31588 vsize=2207540 CPUtime=1197.62 cores=0,2,4,6
/proc/31590/task/31599/stat : 31599 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 114095 0 0 0 119484 278 0 0 25 0 5 0 517836597 2260520960 525970 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=31590/tid=31600] ppid=31588 vsize=2207540 CPUtime=1200.36 cores=0,2,4,6
/proc/31590/task/31600/stat : 31600 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 123512 0 0 0 119666 370 0 0 25 0 5 0 517836597 2260520960 525970 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=31590/tid=31601] ppid=31588 vsize=2207540 CPUtime=1201.74 cores=0,2,4,6
/proc/31590/task/31601/stat : 31601 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 138300 0 0 0 119902 272 0 0 25 0 5 0 517836597 2260520960 525970 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.17
Current children cumulated vsize (KiB) 2207540

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1204.51 s]
/proc/loadavg: 8.07 8.00 7.53 10/186 31633
/proc/meminfo: memFree=10407204/32950928 swapFree=67111528/67111528
[pid=31590] ppid=31588 vsize=2207540 CPUtime=4800.17 cores=0,2,4,6
/proc/31590/stat : 31590 (ttsth-5-4) S 31588 31590 31098 0 -1 4202496 526053 0 0 0 478672 1345 0 0 25 0 5 0 517836533 2260520960 525970 33554432000 4194304 4338100 140737338431584 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/31590/statm: 551885 525970 176 36 0 549790 0
[pid=31590/tid=31598] ppid=31588 vsize=2207540 CPUtime=1199.78 cores=0,2,4,6
/proc/31590/task/31598/stat : 31598 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 147815 0 0 0 119554 424 0 0 25 0 5 0 517836597 2260520960 525970 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=31590/tid=31599] ppid=31588 vsize=2207540 CPUtime=1197.62 cores=0,2,4,6
/proc/31590/task/31599/stat : 31599 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 114095 0 0 0 119484 278 0 0 25 0 5 0 517836597 2260520960 525970 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=31590/tid=31600] ppid=31588 vsize=2207540 CPUtime=1200.36 cores=0,2,4,6
/proc/31590/task/31600/stat : 31600 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 123512 0 0 0 119666 370 0 0 25 0 5 0 517836597 2260520960 525970 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=31590/tid=31601] ppid=31588 vsize=2207540 CPUtime=1201.74 cores=0,2,4,6
/proc/31590/task/31601/stat : 31601 (ttsth-5-4) R 31588 31590 31098 0 -1 4202560 138300 0 0 0 119902 272 0 0 25 0 5 0 517836597 2260520960 525970 33554432000 4194304 4338100 140737338431584 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.17
Current children cumulated vsize (KiB) 2207540

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1204.61
CPU time (s): 4800.28
CPU user time (s): 4786.74
CPU system time (s): 13.5419
CPU usage (%): 398.494
Max. virtual memory (cumulated for all children) (KiB): 2207540

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4786.74
system time used= 13.5419
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 526054
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 62724
involuntary context switches= 25336

runsolver used 2.08268 second user time and 4.47432 second system time

The end

Launcher Data 

Begin job on node125 at 2011-04-29 09:32:49
IDJOB=3260110
IDBENCH=83169
IDSOLVER=1654
FILE ID=node125/3260110-1304062369
RUNJOBID= node125-1304062369-31571
PBS_JOBID= 13170441
Free space on /tmp= 72764 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/mosoi/289/289-sat-6x20.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3260110-1304062369/watcher-3260110-1304062369 -o /tmp/evaluation-result-3260110-1304062369/solver-3260110-1304062369 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3260110-1304062369.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= d7291dc9e56649ed6dcce35d6dd461ae
RANDOM SEED=1372614850

node125.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5266.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      15118464 kB
Buffers:       2219220 kB
Cached:       15018368 kB
SwapCached:          0 kB
Active:        9119728 kB
Inactive:      8181892 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      15118464 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:             528 kB
Writeback:           0 kB
AnonPages:       63420 kB
Mapped:          14880 kB
Slab:           468716 kB
PageTables:       4316 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   186700 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72764 MiB
End job on node125 at 2011-04-29 09:52:55