[prev in list] [next in list] [prev in thread] [next in thread]
List: linux-ha-dev
Subject: Re: [Linux-ha-dev] CTS test result on April 8
From: Huang Zhen <zhenhltc () cn ! ibm ! com>
Date: 2005-04-09 7:05:45
Message-ID: 42577EC9.9070507 () cn ! ibm ! com
[Download RAW message or body]
The file attached is the output of the cts.
The whole log is on bugzilla #450 now.
Guochun Shi wrote:
> can you send the whole log (100 iterations) to me?
>
> -Guochun
>
> At 03:51 PM 4/8/2005 +0800, you wrote:
>
> > 2005/04/08_12:19:51 ****************
> > 2005/04/08_12:19:51 Overall Results:{'auditfail': 14, 'failure': 3, \
> > 'success': 97, 'BadNews': 57} 2005/04/08_12:19:51 ****************
> > 2005/04/08_12:19:51 Detailed Results
> > 2005/04/08_12:19:51 Test flip: {'elapsed_time': 122.44435095787048, \
> > 'skipped': 0, 'calls': 8, 'success': 8, 'started': 4, 'down->up': 4, 'auditfail': \
> > 1, 'failure': 0, 'stopped': 4, 'max_time': 29.82415509223938, 'min_time': \
> > 2.9243118762969971, 'up->down': 4} 2005/04/08_12:19:51 Test Restart: \
> > {'elapsed_time': 550.46135592460632, 'skipped': 0, 'calls': 9, 'success': 9, \
> > 'WasStopped': 2, 'node:hadev3': 3, 'node:hadev2': 3, 'node:hadev1': 3, \
> > 'auditfail': 2, 'failure': 0, 'max_time': 212.46228098869324, 'min_time': \
> > 14.486721992492676} 2005/04/08_12:19:51 Test Stonith: {'elapsed_time': \
> > 2020.8436036109924, 'skipped': 0, 'calls': 16, 'success': 16, 'auditfail': 1, \
> > 'failure': 0, 'max_time': 163.37847399711609, 'min_time': 97.073056936264038} \
> > 2005/04/08_12:19:51 Test StartOnebyOne: {'elapsed_time': \
> > 389.56310272216797, 'skipped': 0, 'calls': 6, 'success': 6, 'stops': 11, \
> > 'auditfail': 6, 'failure': 0, 'max_time': 51.912569046020508, 'min_time': \
> > 46.898149967193604} 2005/04/08_12:19:51 Test SimulStart: \
> > {'elapsed_time': 353.45350575447083, 'skipped': 0, 'calls': 6, 'success': 6, \
> > 'stops': 11, 'auditfail': 4, 'failure': 0, 'max_time': 56.954465866088867, \
> > 'min_time': 21.812397956848145} 2005/04/08_12:19:51 Test SimulStop: \
> > {'elapsed_time': 387.32645988464355, 'skipped': 0, 'calls': 11, 'success': 11, \
> > 'started': 13, 'auditfail': 0, 'failure': 0, 'max_time': 20.110054969787598, \
> > 'min_time': 8.3391361236572266} 2005/04/08_12:19:51 Test StopOnebyOne: \
> > {'elapsed_time': 407.02541613578796, 'skipped': 0, 'calls': 8, 'success': 8, \
> > 'started': 6, 'auditfail': 0, 'failure': 0, 'max_time': 49.408849000930786, \
> > 'min_time': 19.35324501991272} 2005/04/08_12:19:51 Test RestartOnebyOne: \
> > {'elapsed_time': 998.68342161178589, 'skipped': 0, 'calls': 9, 'success': 9, \
> > 'started': 3, 'auditfail': 0, 'failure': 0, 'max_time': 149.5040431022644, \
> > 'min_time': 73.079960107803345} 2005/04/08_12:19:51 Test Bandwidth: \
> > {'elapsed_time': 180.210116147995, 'skipped': 1, 'calls': 15, 'success': 14, \
> > 'min': 12304.581583091836, 'max': 100415.58264101903, 'totalbandwidth': \
> > 684736.8338568199, 'auditfail': 0, 'failure': 0, 'max_time': 33.621577024459839, \
> > 'min_time': 7.2956085205078125e-05} 2005/04/08_12:19:51 Test ResourceRecover: \
> > {'elapsed_time': 183.56434273719788, 'skipped': 0, 'calls': 12, 'success': 9, \
> > 'auditfail': 0, 'failure': 3, 'max_time': 50.778339147567749, 'min_time': \
> > 1.1215839385986328} 2005/04/08_12:19:51 <<<<<<<<<<<<<<<< TESTS COMPLETED
> >
> > 1. one ccm dumped core error, has been reported to bugzilla, #450
> > 2005/04/08_11:43:48 Running test RestartOnebyOne (hadev3) [63 : \
> > 6787853] 2005/04/08_11:44:50 Apr 8 11:44:44 hadev1 crmd: [9643]: info: \
> > fn(do_state_transition): State transition S_IDLE -> S_ELECTION [ input=I_ELECTION \
> > cause=C_FSA_INTERNAL origin=do_election_count_vote ] 2005/04/08_11:44:50 Apr \
> > 8 11:44:50 hadev1 crmd: [9643]: info: fn(do_state_transition): State transition \
> > S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_TIMER_POPPED \
> > origin=crm_timer_popped ] 2005/04/08_11:45:12 Apr 8 11:44:22 hadev2 \
> > heartbeat: [8629]: ERROR: Exiting /usr/lib/heartbeat/ccm process 8638 killed by \
> > signal 11. 2005/04/08_11:45:12 Apr 8 11:44:22 hadev2 heartbeat: [8629]: \
> > ERROR: Exiting /usr/lib/heartbeat/ccm process 8638 dumped core \
> > 2005/04/08_11:45:12 Apr 8 11:44:22 hadev2 heartbeat: [8629]: ERROR: Client \
> > /usr/lib/heartbeat/ccm killed by signal 11. 2005/04/08_11:45:12 Apr 8 \
> > 11:44:22 hadev2 heartbeat: [8629]: ERROR: Respawning client \
> > "/usr/lib/heartbeat/ccm": 2005/04/08_11:45:12 Apr 8 11:44:22 hadev2 cib: \
> > [8639]: ERROR: fn(cib_ccm_dispatch): CCM connection appears to have failed: \
> > rc=-1. 2005/04/08_11:45:12 Apr 8 11:44:22 hadev2 crmd: [8642]: ERROR: \
> > fn(ccm_dispatch): CCM connection appears to have failed: rc=-1. \
> > 2005/04/08_11:45:12 Apr 8 11:44:22 hadev2 crmd: [8642]: info: \
> > fn(do_state_transition): State transition S_IDLE -> S_RECOVERY [ input=I_ERROR \
> > cause=C_CCM_CALLBACK origin=ccm_dispatch ] 2005/04/08_11:45:12 Apr 8 \
> > 11:44:22 hadev2 crmd: [8642]: ERROR: fn(do_recover): Action A_RECOVER \
> > (0000000001000000) not supported 2005/04/08_11:45:14 Apr 8 11:44:28 hadev2 \
> > crmd: [8642]: CRIT: fn(crmd_ha_msg_dispatch): Lost connection to heartbeat \
> > service. 2005/04/08_11:45:14 Apr 8 11:44:28 hadev2 crmd: [8642]: CRIT: \
> > fn(crmd_ha_connection_destroy): Heartbeat has left us 2005/04/08_11:45:14 Apr \
> > 8 11:44:28 hadev2 crmd: [8642]: info: fn(do_state_transition): State transition \
> > S_STOPPING -> S_TERMINATE [ input=I_ERROR cause=C_HA_DISCONNECT \
> > origin=crmd_ha_connection_destroy ] 2005/04/08_11:45:14 Apr 8 11:44:28 \
> > hadev2 heartbeat: [8629]: ERROR: Client /usr/lib/heartbeat/crmd exited with \
> > return code 2. 2005/04/08_11:45:14 Apr 8 11:44:28 hadev2 heartbeat: [8629]: \
> > ERROR: Respawning client "/usr/lib/heartbeat/crmd":
> > 2. CrmdStateAudit Failed 12 times, most in the test of start.
> > (StartOnebyOne: failed 6 times in 6 runs, SimulStart: failed 4 times in 6 runs)
> > 2005/04/08_10:55:24 Cluster is not stable: 1 (of 3).
> > 2005/04/08_10:55:24 Status of crmd@hadev1: S_TRANSITION_ENGINE (ok)
> > 2005/04/08_10:55:24 Audit CrmdStateAudit Failed.
> >
> > 3. ResourceRecover failed 3 times in 12 run. All failures like this:
> > 36104:Apr 8 11:18:50 plinuxt11 CTS: Running test ResourceRecover (hadev3) \
> > [35 : 3964090] 36105:Apr 8 11:18:50 plinuxt11 CTS: CIB already installed on node \
> > hadev1 36106:Apr 8 11:18:50 plinuxt11 CTS: Starting linux-ha-v2 on node hadev1
> > 36231:Apr 8 11:19:20 plinuxt11 CTS: CIB already installed on node hadev3
> > 36232:Apr 8 11:19:20 plinuxt11 CTS: Starting linux-ha-v2 on node hadev3
> > 36390:Apr 8 11:19:33 hadev1 tengine: [4824]: info: fn(initiate_action): \
> > Executing rsc-op (1): start DcIPaddr on hadev1 36391:Apr 8 11:19:33 hadev1 \
> > tengine: [4824]: info: fn(initiate_action): Executing rsc-op (3): start hadev1 on \
> > hadev1 36392:Apr 8 11:19:33 hadev1 tengine: [4824]: info: fn(initiate_action): \
> > Executing rsc-op (5): start hadev3 on hadev3 36393:Apr 8 11:19:33 hadev1 \
> > tengine: [4824]: info: fn(initiate_action): Executing rsc-op (7): start hadev2 on \
> > hadev3 36440:Apr 8 11:19:33 hadev1 tengine: [4824]: info: fn(initiate_action): \
> > Executing rsc-op (6): monitor hadev3 on hadev3 36449:Apr 8 11:19:33 hadev1 \
> > tengine: [4824]: info: fn(initiate_action): Executing rsc-op (8): monitor hadev2 \
> > on hadev3 36469:Apr 8 11:19:34 hadev1 tengine: [4824]: info: \
> > fn(initiate_action): Executing rsc-op (2): monitor DcIPaddr on hadev1 36475:Apr \
> > 8 11:19:34 hadev1 tengine: [4824]: info: fn(initiate_action): Executing rsc-op \
> > (4): monitor hadev1 on hadev1 36481:Apr 8 11:19:34 plinuxt11 CTS: CIB already \
> > installed on node hadev2 36482:Apr 8 11:19:34 plinuxt11 CTS: Starting \
> > linux-ha-v2 on node hadev2 36861:Apr 8 11:19:38 hadev1 tengine: [4824]: info: \
> > fn(initiate_action): Executing rsc-op (1): stop hadev2 on hadev3 36965:Apr 8 \
> > 11:19:38 hadev1 tengine: [4824]: info: fn(initiate_action): Executing rsc-op (2): \
> > start hadev2 on hadev2 36980:Apr 8 11:19:38 plinuxt11 CTS: Choice resource \
> > IPaddr::hadev2 on hadev3 36987:Apr 8 11:19:39 plinuxt11 CTS: Test \
> > ResourceRecover failed [reason:IPaddr::hadev2 is now active on more than one \
> > node: ['hadev3', 'hadev2']] 36988:Apr 8 11:19:39 plinuxt11 CTS: Test \
> > ResourceRecover (hadev3) [FAILED]
> > The cause is that the lo IP can not be removed when stop the resource.
> >
> > 4. Timeout Error log from tengine when running the Stonith test.
> > 2005/04/08_10:43:29 Running test Stonith (hadev2) [3 : 321555]
> > ** INFO: Initiating ssh-reset on host: hadev2
> > 2005/04/08_10:45:52 Apr 8 10:44:08 hadev3 tengine: [23748]: ERROR: \
> > fn(send_complete): Transition status: Timed out after 30000ms
> > 5. Restart test caused split-brain. Reported to bugzilla, #451
> > 2005/04/08_10:58:46 Running test Restart (hadev3) [17 : 1902204]
> > 2005/04/08_11:02:18 Warn: Startup pattern not found: crmd:.*State \
> > transition.*-> S_IDLE 2005/04/08_11:02:18 Warn: Apr 8 10:59:17 hadev3 crmd: \
> > [7265]: info: fn(do_state_transition): State transition S_ELECTION -> \
> > S_INTEGRATION [ input=I_ELECTION_DC cause=C_TIMER_POPPED origin=crm_timer_popped \
> > ] 2005/04/08_11:02:18 linux-ha-v2 on hadev3 joined the cluster as DC
> > 2005/04/08_11:02:20 Warn: 2 cluster partitions detected:
> > 2005/04/08_11:02:20 hadev1 hadev2
> > 2005/04/08_11:02:20 hadev3
> >
> > 6. An ERROR log in ccm, will report to bugzilla soon.
> > 2005/04/08_11:07:58 Running test Restart (hadev2) [24 : 2999592]
> > 2005/04/08_11:08:15 Apr 8 11:08:05 hadev1 ccm: [23901]: ERROR: \
> > ccm_update_table:duplicate entry hadev1
> > --
> > Best Regards,
> > Huang Zhen
> > LTC and Power Linux Testing
> > IBM China Development Lab, Beijing
> > Telno: (8610)82782244-2845
> > _______________________________________________________
> > Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org
> > http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
> > Home Page: http://linux-ha.org/
>
>
> _______________________________________________________
> Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org
> http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
> Home Page: http://linux-ha.org/
>
--
Best Regards,
Huang Zhen
LTC and Power Linux Testing
IBM China Development Lab, Beijing
Telno: (8610)82782244-2845
["cts.out" (text/plain)]
plinuxt11:~ # /usr/lib/heartbeat/cts/CTSlab.py -2 -D /tmp/cts/ -L /var/log/messages \
-r -c 100 2005/04/08_10:40:06 Random seed is: (20, 233, 7)
2005/04/08_10:40:06 Enabling DC resource
2005/04/08_10:40:06 >>>>>>>>>>>>>>>> BEGINNING 100 TESTS
2005/04/08_10:40:06 HA configuration directory: /tmp/cts/
2005/04/08_10:40:06 System log files: /var/log/messages
2005/04/08_10:40:06 Enable Stonith: 1
2005/04/08_10:40:06 Enable Standby: 1
2005/04/08_10:40:06 Cluster nodes: ['hadev1', 'hadev3', 'hadev2']
2005/04/08_10:40:07 Stopping Cluster Manager on all nodes
2005/04/08_10:40:07 Starting Cluster Manager on all nodes.
2005/04/08_10:41:30 Running test SimulStart (hadev3) [1 : 100241]
2005/04/08_10:43:21 Cluster is not stable: 1 (of 3).
2005/04/08_10:43:21 Status of crmd@hadev3: S_TRANSITION_ENGINE (ok)
2005/04/08_10:43:21 Audit CrmdStateAudit Failed.
2005/04/08_10:43:24 Running test ResourceRecover (hadev3) [2 : 317900]
2005/04/08_10:43:25 Choice resource IPaddr::DcIPaddr on hadev3
2005/04/08_10:43:29 Running test Stonith (hadev2) [3 : 321555]
** INFO: Initiating ssh-reset on host: hadev2
2005/04/08_10:45:52 Apr 8 10:44:08 hadev3 tengine: [23748]: ERROR: \
fn(send_complete): Transition status: Timed out after 30000ms 2005/04/08_10:45:55 \
Running test Stonith (hadev3) [4 : 384151]
** INFO: Initiating ssh-reset on host: hadev3
2005/04/08_10:47:42 Running test Bandwidth (hadev1) [5 : 474240]
2005/04/08_10:47:56 ...heartbeat bandwidth: 12599 bits/sec
2005/04/08_10:47:59 Running test Stonith (hadev3) [6 : 476937]
** INFO: Initiating ssh-reset on host: hadev3
2005/04/08_10:49:42 Running test Stonith (hadev3) [7 : 536029]
** INFO: Initiating ssh-reset on host: hadev3
2005/04/08_10:51:24 Running test Bandwidth (hadev1) [8 : 595246]
2005/04/08_10:51:38 ...heartbeat bandwidth: 12907 bits/sec
2005/04/08_10:51:41 Running test Restart (hadev2) [9 : 597943]
2005/04/08_10:52:01 Running test ResourceRecover (hadev2) [10 : 648671]
2005/04/08_10:52:02 Choice resource IPaddr::hadev3 on hadev3
2005/04/08_10:52:06 Running test RestartOnebyOne (hadev3) [11 : 652635]
2005/04/08_10:52:47 Apr 8 10:52:40 hadev2 crmd: [4415]: info: \
fn(do_state_transition): State transition S_NOT_DC -> S_ELECTION [ input=I_ELECTION \
cause=C_FSA_INTERNAL origin=do_election_count_vote ] 2005/04/08_10:52:47 Apr 8 \
10:52:40 hadev3 crmd: [4529]: info: fn(do_state_transition): State transition S_IDLE \
-> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL origin=do_election_count_vote ] \
2005/04/08_10:52:47 Apr 8 10:52:46 hadev3 crmd: [4529]: info: \
fn(do_state_transition): State transition S_ELECTION -> S_INTEGRATION [ \
input=I_ELECTION_DC cause=C_TIMER_POPPED origin=crm_timer_popped ] \
2005/04/08_10:53:41 Running test StartOnebyOne (hadev2) [12 : \
1075644] 2005/04/08_10:55:24 Cluster is not stable: 1 (of 3).
2005/04/08_10:55:24 Status of crmd@hadev1: S_TRANSITION_ENGINE (ok)
2005/04/08_10:55:24 Audit CrmdStateAudit Failed.
2005/04/08_10:55:27 Running test ResourceRecover (hadev1) [13 : \
1274670] 2005/04/08_10:55:28 Choice resource IPaddr::hadev1 on hadev1
2005/04/08_10:55:32 Running test ResourceRecover (hadev3) [14 : \
1278498] 2005/04/08_10:55:33 Choice resource IPaddr::hadev1 on hadev1
2005/04/08_10:55:37 Running test StopOnebyOne (hadev3) [15 : \
1282109] 2005/04/08_10:56:21 Running test RestartOnebyOne (hadev2) [16 \
: 1432232] 2005/04/08_10:58:46 Running test Restart (hadev3) [17 : \
1902204] 2005/04/08_11:02:18 Warn: Startup pattern not found: crmd:.*State \
transition.*-> S_IDLE 2005/04/08_11:02:18 Warn: Apr 8 10:59:17 hadev3 crmd: \
[7265]: info: fn(do_state_transition): State transition S_ELECTION -> S_INTEGRATION [ \
input=I_ELECTION_DC cause=C_TIMER_POPPED origin=crm_timer_popped ] \
2005/04/08_11:02:18 linux-ha-v2 on hadev3 joined the cluster as DC \
2005/04/08_11:02:20 Warn: 2 cluster partitions detected: 2005/04/08_11:02:20 \
hadev1 hadev2 2005/04/08_11:02:20 hadev3
2005/04/08_11:02:22 Running test Bandwidth (hadev1) [18 : 1940716]
2005/04/08_11:02:35 ...heartbeat bandwidth: 13033 bits/sec
2005/04/08_11:02:37 Warn: 2 cluster partitions detected:
2005/04/08_11:02:37 hadev1 hadev2
2005/04/08_11:02:37 hadev3
2005/04/08_11:02:39 Running test Bandwidth (hadev1) [19 : 1943899]
2005/04/08_11:02:51 ...heartbeat bandwidth: 13440 bits/sec
2005/04/08_11:02:54 Warn: 2 cluster partitions detected:
2005/04/08_11:02:54 hadev1 hadev2
2005/04/08_11:02:54 hadev3
2005/04/08_11:02:55 Running test StopOnebyOne (hadev3) [20 : \
1947082] 2005/04/08_11:03:20 Running test StartOnebyOne (hadev2) [21 \
: 2022610] 2005/04/08_11:04:08 Cluster is not stable: 1 (of 3).
2005/04/08_11:04:08 Status of crmd@hadev1: S_TRANSITION_ENGINE (ok)
2005/04/08_11:04:08 Audit CrmdStateAudit Failed.
2005/04/08_11:04:11 Running test RestartOnebyOne (hadev3) [22 : \
2142377] 2005/04/08_11:05:50 Apr 8 11:05:44 hadev3 crmd: [7529]: info: \
fn(do_state_transition): State transition S_IDLE -> S_ELECTION [ input=I_ELECTION \
cause=C_FSA_INTERNAL origin=do_election_count_vote ] 2005/04/08_11:05:50 Apr 8 \
11:05:44 hadev2 crmd: [7067]: info: fn(do_state_transition): State transition \
S_NOT_DC -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL \
origin=do_election_count_vote ] 2005/04/08_11:05:50 Apr 8 11:05:50 hadev3 crmd: \
[7529]: info: fn(do_state_transition): State transition S_ELECTION -> S_INTEGRATION [ \
input=I_ELECTION_DC cause=C_TIMER_POPPED origin=crm_timer_popped ] \
2005/04/08_11:06:41 Running test RestartOnebyOne (hadev2) [23 : \
2604992] 2005/04/08_11:07:58 Running test Restart (hadev2) [24 : \
2999592] 2005/04/08_11:08:15 Apr 8 11:08:05 hadev1 ccm: [23901]: ERROR: \
ccm_update_table:duplicate entry hadev1 2005/04/08_11:08:18 Running test Restart \
(hadev1) [25 : 3056171] 2005/04/08_11:10:14 Apr 8 11:09:00 hadev3 crmd: \
[8599]: info: fn(do_state_transition): State transition S_ELECTION -> S_INTEGRATION [ \
input=I_ELECTION_DC cause=C_TIMER_POPPED origin=crm_timer_popped ] \
2005/04/08_11:10:14 Apr 8 11:09:12 hadev3 crmd: [8599]: info: \
fn(do_state_transition): State transition S_IDLE -> S_ELECTION [ input=I_ELECTION \
cause=C_FSA_INTERNAL origin=do_election_count_vote ] 2005/04/08_11:10:14 Apr 8 \
11:09:18 hadev3 crmd: [8599]: info: fn(do_state_transition): State transition \
S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_TIMER_POPPED \
origin=crm_timer_popped ] 2005/04/08_11:10:14 Apr 8 11:09:30 hadev3 crmd: \
[8599]: info: fn(do_state_transition): State transition S_IDLE -> S_ELECTION [ \
input=I_ELECTION cause=C_FSA_INTERNAL origin=do_election_count_vote ] \
2005/04/08_11:10:14 Apr 8 11:09:36 hadev3 crmd: [8599]: info: \
fn(do_state_transition): State transition S_ELECTION -> S_INTEGRATION [ \
input=I_ELECTION_DC cause=C_TIMER_POPPED origin=crm_timer_popped ] \
2005/04/08_11:10:14 Apr 8 11:09:48 hadev3 crmd: [8599]: info: \
fn(do_state_transition): State transition S_IDLE -> S_ELECTION [ input=I_ELECTION \
cause=C_FSA_INTERNAL origin=do_election_count_vote ] 2005/04/08_11:10:14 Apr 8 \
11:09:54 hadev3 crmd: [8599]: info: fn(do_state_transition): State transition \
S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_TIMER_POPPED \
origin=crm_timer_popped ] 2005/04/08_11:10:14 Apr 8 11:10:06 hadev3 crmd: \
[8599]: info: fn(do_state_transition): State transition S_IDLE -> S_ELECTION [ \
input=I_ELECTION cause=C_FSA_INTERNAL origin=do_election_count_vote ] \
2005/04/08_11:10:14 Apr 8 11:10:12 hadev3 crmd: [8599]: info: \
fn(do_state_transition): State transition S_ELECTION -> S_INTEGRATION [ \
input=I_ELECTION_DC cause=C_TIMER_POPPED origin=crm_timer_popped ] \
2005/04/08_11:10:15 Cluster is not stable: 1 (of 3). 2005/04/08_11:10:15 \
Status of crmd@hadev3: S_TRANSITION_ENGINE (ok) 2005/04/08_11:10:15 Audit \
CrmdStateAudit Failed. 2005/04/08_11:10:18 Running test Stonith (hadev3) \
[26 : 3205780]
** INFO: Initiating ssh-reset on host: hadev3
2005/04/08_11:11:57 Apr 8 11:11:13 hadev2 tengine: [8734]: ERROR: \
fn(send_complete): Transition status: Timed out after 30000ms 2005/04/08_11:12:01 \
Running test Bandwidth (hadev1) [27 : 3280236] 2005/04/08_11:12:14 \
...heartbeat bandwidth: 12838 bits/sec 2005/04/08_11:12:18 Running test Stonith \
(hadev1) [28 : 3282937]
** INFO: Initiating ssh-reset on host: hadev1
2005/04/08_11:14:41 Running test StartOnebyOne (hadev2) [29 : \
3350360] 2005/04/08_11:15:46 Cluster is not stable: 1 (of 3).
2005/04/08_11:15:46 Status of crmd@hadev1: S_TRANSITION_ENGINE (ok)
2005/04/08_11:15:46 Audit CrmdStateAudit Failed.
2005/04/08_11:15:49 Running test Bandwidth (hadev1) [30 : 3468373]
2005/04/08_11:15:57 ...heartbeat bandwidth: 100029 bits/sec
2005/04/08_11:16:00 Running test Bandwidth (hadev1) [31 : 3471764]
2005/04/08_11:16:09 ...heartbeat bandwidth: 97213 bits/sec
2005/04/08_11:16:12 Running test RestartOnebyOne (hadev3) [32 : \
3475294] 2005/04/08_11:18:12 Running test flip (hadev3) [33 : \
3893324] 2005/04/08_11:18:26 Running test SimulStop (hadev3) [34 : \
3909961] 2005/04/08_11:18:50 Running test ResourceRecover (hadev3) [35 \
: 3964090] 2005/04/08_11:19:38 Choice resource IPaddr::hadev2 on hadev3
2005/04/08_11:19:39 Test ResourceRecover failed [reason:IPaddr::hadev2 is now \
active on more than one node: ['hadev3', 'hadev2']] 2005/04/08_11:19:39 Test \
ResourceRecover (hadev3) [FAILED] 2005/04/08_11:19:43 Running test \
SimulStop (hadev2) [36 : 4060949] 2005/04/08_11:20:03 Running test \
StopOnebyOne (hadev1) [37 : 4081853] 2005/04/08_11:21:34 Running \
test Stonith (hadev2) [38 : 4312946]
** INFO: Initiating ssh-reset on host: hadev2
2005/04/08_11:24:19 Running test Bandwidth (hadev3) [39 : 4330089]
2005/04/08_11:24:52 ...heartbeat bandwidth: 83077 bits/sec
2005/04/08_11:24:55 Running test SimulStart (hadev1) [40 : \
4378822] 2005/04/08_11:26:03 Cluster is not stable: 1 (of 3).
2005/04/08_11:26:03 Status of crmd@hadev3: S_TRANSITION_ENGINE (ok)
2005/04/08_11:26:03 Audit CrmdStateAudit Failed.
2005/04/08_11:26:06 Running test SimulStart (hadev3) [41 : \
4493769] 2005/04/08_11:27:09 Running test flip (hadev1) [42 : \
4653136] 2005/04/08_11:27:24 Running test SimulStop (hadev1) [43 : \
4669070] 2005/04/08_11:27:45 Apr 8 11:27:39 hadev3 crmd: [7485]: info: \
fn(do_state_transition): State transition S_IDLE -> S_ELECTION [ input=I_ELECTION \
cause=C_FSA_INTERNAL origin=do_election_count_vote ] 2005/04/08_11:27:45 Apr 8 \
11:27:39 hadev2 crmd: [4573]: info: fn(do_state_transition): State transition \
S_NOT_DC -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL \
origin=do_election_count_vote ] 2005/04/08_11:27:45 Apr 8 11:27:45 hadev3 crmd: \
[7485]: info: fn(do_state_transition): State transition S_ELECTION -> S_INTEGRATION [ \
input=I_ELECTION_DC cause=C_TIMER_POPPED origin=crm_timer_popped ] \
2005/04/08_11:28:06 Running test flip (hadev2) [44 : 4743239] \
2005/04/08_11:28:37 Running test Restart (hadev1) [45 : 4759252] \
2005/04/08_11:29:24 Running test StartOnebyOne (hadev3) [46 : \
4837693] 2005/04/08_11:30:25 Cluster is not stable: 1 (of 3).
2005/04/08_11:30:25 Status of crmd@hadev1: S_TRANSITION_ENGINE (ok)
2005/04/08_11:30:25 Audit CrmdStateAudit Failed.
2005/04/08_11:30:28 Running test ResourceRecover (hadev3) [47 : \
4942525] 2005/04/08_11:30:28 Choice resource IPaddr::hadev2 on hadev3
2005/04/08_11:30:29 Test ResourceRecover failed [reason:IPaddr::hadev2 is now \
active on more than one node: ['hadev3', 'hadev2']] 2005/04/08_11:30:29 Test \
ResourceRecover (hadev3) [FAILED] 2005/04/08_11:30:33 Running test \
RestartOnebyOne (hadev1) [48 : 4946833] 2005/04/08_11:31:32 Apr 8 \
11:31:26 hadev3 crmd: [7972]: info: fn(do_state_transition): State transition S_IDLE \
-> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL origin=do_election_count_vote ] \
2005/04/08_11:31:32 Apr 8 11:31:26 hadev2 crmd: [5605]: info: \
fn(do_state_transition): State transition S_NOT_DC -> S_ELECTION [ input=I_ELECTION \
cause=C_FSA_INTERNAL origin=do_election_count_vote ] 2005/04/08_11:31:32 Apr 8 \
11:31:32 hadev3 crmd: [7972]: info: fn(do_state_transition): State transition \
S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_TIMER_POPPED \
origin=crm_timer_popped ] 2005/04/08_11:32:25 Running test StopOnebyOne \
(hadev2) [49 : 5387313] 2005/04/08_11:33:15 Running test StartOnebyOne \
(hadev2) [50 : 5602753] 2005/04/08_11:34:06 Cluster is not stable: 1 (of \
3). 2005/04/08_11:34:06 Status of crmd@hadev1: S_TRANSITION_ENGINE (ok)
2005/04/08_11:34:06 Audit CrmdStateAudit Failed.
2005/04/08_11:34:09 Running test Bandwidth (hadev3) [51 : 5684894]
2005/04/08_11:34:16 ...heartbeat bandwidth: 100255 bits/sec
2005/04/08_11:34:20 Running test StopOnebyOne (hadev2) [52 : \
5688425] 2005/04/08_11:35:04 Running test SimulStop (hadev2) [53 : \
5836591] 2005/04/08_11:36:12 Running test Restart (hadev3) [54 : \
5936606] 2005/04/08_11:37:30 Running test SimulStop (hadev3) [55 : \
5974274] 2005/04/08_11:38:07 Running test SimulStart (hadev1) [56 \
: 6075692] 2005/04/08_11:38:32 Running test ResourceRecover (hadev1) \
[57 : 6154713] 2005/04/08_11:38:33 Choice resource IPaddr::hadev3 on hadev3
2005/04/08_11:38:37 Running test StartOnebyOne (hadev2) [58 : \
6158324] 2005/04/08_11:39:44 Cluster is not stable: 1 (of 3).
2005/04/08_11:39:44 Status of crmd@hadev1: S_TRANSITION_ENGINE (ok)
2005/04/08_11:39:44 Audit CrmdStateAudit Failed.
2005/04/08_11:39:47 Running test SimulStop (hadev1) [59 : 6318649]
2005/04/08_11:40:07 Running test ResourceRecover (hadev1) [60 : \
6338918] 2005/04/08_11:40:55 Choice resource IPaddr::DcIPaddr on hadev1
2005/04/08_11:40:59 Running test RestartOnebyOne (hadev3) [61 : \
6429662] 2005/04/08_11:41:49 Apr 8 11:41:43 hadev3 crmd: [10839]: info: \
fn(do_state_transition): State transition S_IDLE -> S_ELECTION [ input=I_ELECTION \
cause=C_FSA_INTERNAL origin=do_election_count_vote ] 2005/04/08_11:41:49 Apr 8 \
11:41:43 hadev2 crmd: [8142]: info: fn(do_state_transition): State transition \
S_NOT_DC -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL \
origin=do_election_count_vote ] 2005/04/08_11:41:49 Apr 8 11:41:49 hadev3 crmd: \
[10839]: info: fn(do_state_transition): State transition S_ELECTION -> S_INTEGRATION \
[ input=I_ELECTION_DC cause=C_TIMER_POPPED origin=crm_timer_popped ] \
2005/04/08_11:42:34 Apr 8 11:42:34 hadev3 crmd: [11279]: info: \
fn(do_state_transition): State transition S_ELECTION -> S_INTEGRATION [ \
input=I_ELECTION_DC cause=C_TIMER_POPPED origin=crm_timer_popped ] \
2005/04/08_11:43:29 Apr 8 11:41:56 hadev1 crmd: [9186]: ERROR: \
fn(do_cl_join_request): Expected a welcome from hadev3, but hadev2 replied \
2005/04/08_11:43:29 Apr 8 11:43:28 hadev1 tengine: [9318]: ERROR: \
fn(send_complete): Transition status: Timed out after 30000ms 2005/04/08_11:43:31 \
Warn: 2 cluster partitions detected: 2005/04/08_11:43:31 hadev1 hadev2
2005/04/08_11:43:31 hadev3
2005/04/08_11:43:32 Running test Bandwidth (hadev1) [62 : 6784448]
2005/04/08_11:43:45 ...heartbeat bandwidth: 13134 bits/sec
2005/04/08_11:43:47 Warn: 2 cluster partitions detected:
2005/04/08_11:43:47 hadev1 hadev2
2005/04/08_11:43:47 hadev3
2005/04/08_11:43:48 Running test RestartOnebyOne (hadev3) [63 : \
6787853] 2005/04/08_11:44:50 Apr 8 11:44:44 hadev1 crmd: [9643]: info: \
fn(do_state_transition): State transition S_IDLE -> S_ELECTION [ input=I_ELECTION \
cause=C_FSA_INTERNAL origin=do_election_count_vote ] 2005/04/08_11:44:50 Apr 8 \
11:44:50 hadev1 crmd: [9643]: info: fn(do_state_transition): State transition \
S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_TIMER_POPPED \
origin=crm_timer_popped ] 2005/04/08_11:45:12 Apr 8 11:44:22 hadev2 heartbeat: \
[8629]: ERROR: Exiting /usr/lib/heartbeat/ccm process 8638 killed by signal 11. \
2005/04/08_11:45:12 Apr 8 11:44:22 hadev2 heartbeat: [8629]: ERROR: Exiting \
/usr/lib/heartbeat/ccm process 8638 dumped core 2005/04/08_11:45:12 Apr 8 \
11:44:22 hadev2 heartbeat: [8629]: ERROR: Client /usr/lib/heartbeat/ccm killed by \
signal 11. 2005/04/08_11:45:12 Apr 8 11:44:22 hadev2 heartbeat: [8629]: ERROR: \
Respawning client "/usr/lib/heartbeat/ccm": 2005/04/08_11:45:12 Apr 8 11:44:22 \
hadev2 cib: [8639]: ERROR: fn(cib_ccm_dispatch): CCM connection appears to have \
failed: rc=-1. 2005/04/08_11:45:12 Apr 8 11:44:22 hadev2 crmd: [8642]: ERROR: \
fn(ccm_dispatch): CCM connection appears to have failed: rc=-1. 2005/04/08_11:45:12 \
Apr 8 11:44:22 hadev2 crmd: [8642]: info: fn(do_state_transition): State transition \
S_IDLE -> S_RECOVERY [ input=I_ERROR cause=C_CCM_CALLBACK origin=ccm_dispatch ] \
2005/04/08_11:45:12 Apr 8 11:44:22 hadev2 crmd: [8642]: ERROR: fn(do_recover): \
Action A_RECOVER (0000000001000000) not supported 2005/04/08_11:45:14 Apr 8 \
11:44:28 hadev2 crmd: [8642]: CRIT: fn(crmd_ha_msg_dispatch): Lost connection to \
heartbeat service. 2005/04/08_11:45:14 Apr 8 11:44:28 hadev2 crmd: [8642]: CRIT: \
fn(crmd_ha_connection_destroy): Heartbeat has left us 2005/04/08_11:45:14 Apr 8 \
11:44:28 hadev2 crmd: [8642]: info: fn(do_state_transition): State transition \
S_STOPPING -> S_TERMINATE [ input=I_ERROR cause=C_HA_DISCONNECT \
origin=crmd_ha_connection_destroy ] 2005/04/08_11:45:14 Apr 8 11:44:28 hadev2 \
heartbeat: [8629]: ERROR: Client /usr/lib/heartbeat/crmd exited with return code 2. \
2005/04/08_11:45:14 Apr 8 11:44:28 hadev2 heartbeat: [8629]: ERROR: Respawning \
client "/usr/lib/heartbeat/crmd": 2005/04/08_11:45:17 Running test flip \
(hadev3) [64 : 8415837] 2005/04/08_11:45:32 Running test flip \
(hadev3) [65 : 8432489] 2005/04/08_11:45:38 Running test ResourceRecover \
(hadev3) [66 : 8469231] 2005/04/08_11:45:39 Choice resource IPaddr::hadev2 \
on hadev1 2005/04/08_11:45:40 Test ResourceRecover failed [reason:IPaddr::hadev2 \
is now active on more than one node: ['hadev1', 'hadev2']] 2005/04/08_11:45:40 \
Test ResourceRecover (hadev3) [FAILED] 2005/04/08_11:45:44 Running test \
Stonith (hadev3) [67 : 8473203]
** INFO: Initiating ssh-reset on host: hadev3
2005/04/08_11:47:27 Running test Restart (hadev2) [68 : 8539243]
2005/04/08_11:47:46 Apr 8 11:47:34 hadev1 ccm: [9639]: ERROR: \
ccm_update_table:duplicate entry hadev1 2005/04/08_11:47:50 Running test \
Bandwidth (hadev3) [69 : 8603432] 2005/04/08_11:47:58 ...heartbeat \
bandwidth: 100367 bits/sec 2005/04/08_11:48:01 Running test StopOnebyOne \
(hadev1) [70 : 8606837] 2005/04/08_11:48:45 Running test ResourceRecover \
(hadev2) [71 : 8761296] 2005/04/08_11:49:35 Choice resource \
IPaddr::DcIPaddr on hadev1 2005/04/08_11:49:40 Running test RestartOnebyOne \
(hadev2) [72 : 8852759] 2005/04/08_11:50:21 Apr 8 11:50:14 hadev3 crmd: \
[5075]: info: fn(do_state_transition): State transition S_IDLE -> S_ELECTION [ \
input=I_ELECTION cause=C_FSA_INTERNAL origin=do_election_count_vote ] \
2005/04/08_11:50:21 Apr 8 11:50:14 hadev2 crmd: [10351]: info: \
fn(do_state_transition): State transition S_NOT_DC -> S_ELECTION [ input=I_ELECTION \
cause=C_FSA_INTERNAL origin=do_election_count_vote ] 2005/04/08_11:50:21 Apr 8 \
11:50:20 hadev3 crmd: [5075]: info: fn(do_state_transition): State transition \
S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_TIMER_POPPED \
origin=crm_timer_popped ] 2005/04/08_11:51:13 Running test Bandwidth (hadev3) \
[73 : 9253344] 2005/04/08_11:51:20 ...heartbeat bandwidth: 100416 bits/sec
2005/04/08_11:51:24 Running test SimulStop (hadev2) [74 : 9256836]
2005/04/08_11:51:44 Running test SimulStart (hadev1) [75 : \
9277758] 2005/04/08_11:52:26 Cluster is not stable: 1 (of 3).
2005/04/08_11:52:26 Status of crmd@hadev3: S_TRANSITION_ENGINE (ok)
2005/04/08_11:52:26 Audit CrmdStateAudit Failed.
2005/04/08_11:52:29 Running test StopOnebyOne (hadev2) [76 : \
9374312] 2005/04/08_11:52:49 Running test Stonith (hadev2) [77 : \
9446988]
** INFO: Initiating ssh-reset on host: hadev2
2005/04/08_11:55:34 Running test ResourceRecover (hadev1) [78 : \
9464132] 2005/04/08_11:55:57 Choice resource IPaddr::DcIPaddr on hadev2
2005/04/08_11:56:01 Running test Restart (hadev3) [79 : 9548615]
2005/04/08_11:56:19 Running test SimulStart (hadev3) [80 : \
9597972] 2005/04/08_11:57:15 Cluster is not stable: 1 (of 3).
2005/04/08_11:57:15 Status of crmd@hadev3: S_TRANSITION_ENGINE (ok)
2005/04/08_11:57:15 Audit CrmdStateAudit Failed.
2005/04/08_11:57:18 Running test Bandwidth (hadev1) [81 : 9709768]
2005/04/08_11:57:32 ...heartbeat bandwidth: 12305 bits/sec
2005/04/08_11:57:35 Running test SimulStop (hadev1) [82 : 9712469]
2005/04/08_11:57:55 Running test StopOnebyOne (hadev1) [83 : \
9737518] 2005/04/08_11:59:28 Running test Bandwidth (hadev1) [84 : \
9972984] 2005/04/08_11:59:29 Running test SimulStop (hadev1) [85 : \
9973487] 2005/04/08_12:00:38 Running test Stonith (hadev2) [86 : \
10078853]
** INFO: Initiating ssh-reset on host: hadev2
2005/04/08_12:03:23 Running test Stonith (hadev3) [87 : 10095998]
** INFO: Initiating ssh-reset on host: hadev3
2005/04/08_12:05:16 Running test SimulStop (hadev2) [88 : 10159820]
2005/04/08_12:05:29 Running test Stonith (hadev3) [89 : 10241045]
** INFO: Initiating ssh-reset on host: hadev3
2005/04/08_12:07:37 Running test SimulStop (hadev1) [90 : 10257428]
2005/04/08_12:08:17 Apr 8 12:08:11 hadev3 crmd: [4536]: info: \
fn(do_state_transition): State transition S_IDLE -> S_ELECTION [ input=I_ELECTION \
cause=C_FSA_INTERNAL origin=do_election_count_vote ] 2005/04/08_12:08:17 Apr 8 \
12:08:11 hadev1 crmd: [13844]: info: fn(do_state_transition): State transition \
S_NOT_DC -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL \
origin=do_election_count_vote ] 2005/04/08_12:08:17 Apr 8 12:08:17 hadev3 crmd: \
[4536]: info: fn(do_state_transition): State transition S_ELECTION -> S_INTEGRATION [ \
input=I_ELECTION_DC cause=C_TIMER_POPPED origin=crm_timer_popped ] \
2005/04/08_12:08:38 Running test flip (hadev1) [91 : 10386292] \
2005/04/08_12:09:09 Running test Stonith (hadev3) [92 : \
10402534]
** INFO: Initiating ssh-reset on host: hadev3
2005/04/08_12:11:02 Running test Stonith (hadev1) [93 : 10447238]
** INFO: Initiating ssh-reset on host: hadev1
2005/04/08_12:13:34 Apr 8 12:11:56 hadev3 tengine: [4782]: ERROR: \
fn(send_complete): Transition status: Timed out after 30000ms 2005/04/08_12:13:36 \
Resource DcIPaddr is running on non-DC node hadev1 2005/04/08_12:13:37 Audit \
PartitionAudit Failed. 2005/04/08_12:13:37 Running test flip (hadev2) \
[94 : 10533911] 2005/04/08_12:13:51 Resource DcIPaddr is running on non-DC node \
hadev1 2005/04/08_12:13:52 Audit PartitionAudit Failed.
2005/04/08_12:13:52 Running test Restart (hadev1) [95 : 10559019]
2005/04/08_12:14:26 Apr 8 12:14:19 hadev3 crmd: [4537]: info: \
fn(do_state_transition): State transition S_IDLE -> S_ELECTION [ input=I_ELECTION \
cause=C_FSA_INTERNAL origin=do_election_count_vote ] 2005/04/08_12:14:26 Apr 8 \
12:14:19 hadev2 crmd: [4389]: info: fn(do_state_transition): State transition \
S_NOT_DC -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL \
origin=do_election_count_vote ] 2005/04/08_12:14:26 Apr 8 12:14:25 hadev3 crmd: \
[4537]: info: fn(do_state_transition): State transition S_ELECTION -> S_INTEGRATION [ \
input=I_ELECTION_DC cause=C_TIMER_POPPED origin=crm_timer_popped ] \
2005/04/08_12:14:27 Cluster is not stable: 1 (of 3). 2005/04/08_12:14:27 \
Status of crmd@hadev3: S_TRANSITION_ENGINE (ok) 2005/04/08_12:14:27 Audit \
CrmdStateAudit Failed. 2005/04/08_12:14:30 Running test Stonith (hadev3) \
[96 : 10624509]
** INFO: Initiating ssh-reset on host: hadev3
2005/04/08_12:16:11 Running test Bandwidth (hadev3) [97 : 10713715]
2005/04/08_12:16:24 ...heartbeat bandwidth: 13127 bits/sec
2005/04/08_12:16:28 Running test ResourceRecover (hadev2) [98 : \
10716406] 2005/04/08_12:16:28 Choice resource IPaddr::hadev1 on hadev1
2005/04/08_12:16:33 Running test flip (hadev1) [99 : 10720009]
2005/04/08_12:16:47 Running test Stonith (hadev2) [100 : 10736513]
** INFO: Initiating ssh-reset on host: hadev2
2005/04/08_12:19:24 Stopping Cluster Manager on all nodes
2005/04/08_12:19:51 ****************
2005/04/08_12:19:51 Overall Results:{'auditfail': 14, 'failure': 3, 'success': \
97, 'BadNews': 57} 2005/04/08_12:19:51 ****************
2005/04/08_12:19:51 Detailed Results
2005/04/08_12:19:51 Test flip: {'elapsed_time': 122.44435095787048, \
'skipped': 0, 'calls': 8, 'success': 8, 'started': 4, 'down->up': 4, 'auditfail': 1, \
'failure': 0, 'stopped': 4, 'max_time': 29.82415509223938, 'min_time': \
2.9243118762969971, 'up->down': 4} 2005/04/08_12:19:51 Test Restart: \
{'elapsed_time': 550.46135592460632, 'skipped': 0, 'calls': 9, 'success': 9, \
'WasStopped': 2, 'node:hadev3': 3, 'node:hadev2': 3, 'node:hadev1': 3, 'auditfail': \
2, 'failure': 0, 'max_time': 212.46228098869324, 'min_time': 14.486721992492676} \
2005/04/08_12:19:51 Test Stonith: {'elapsed_time': 2020.8436036109924, \
'skipped': 0, 'calls': 16, 'success': 16, 'auditfail': 1, 'failure': 0, 'max_time': \
163.37847399711609, 'min_time': 97.073056936264038} 2005/04/08_12:19:51 Test \
StartOnebyOne: {'elapsed_time': 389.56310272216797, 'skipped': 0, 'calls': 6, \
'success': 6, 'stops': 11, 'auditfail': 6, 'failure': 0, 'max_time': \
51.912569046020508, 'min_time': 46.898149967193604} 2005/04/08_12:19:51 Test \
SimulStart: {'elapsed_time': 353.45350575447083, 'skipped': 0, 'calls': 6, \
'success': 6, 'stops': 11, 'auditfail': 4, 'failure': 0, 'max_time': \
56.954465866088867, 'min_time': 21.812397956848145} 2005/04/08_12:19:51 Test \
SimulStop: {'elapsed_time': 387.32645988464355, 'skipped': 0, 'calls': 11, \
'success': 11, 'started': 13, 'auditfail': 0, 'failure': 0, 'max_time': \
20.110054969787598, 'min_time': 8.3391361236572266} 2005/04/08_12:19:51 Test \
StopOnebyOne: {'elapsed_time': 407.02541613578796, 'skipped': 0, 'calls': 8, \
'success': 8, 'started': 6, 'auditfail': 0, 'failure': 0, 'max_time': \
49.408849000930786, 'min_time': 19.35324501991272} 2005/04/08_12:19:51 Test \
RestartOnebyOne: {'elapsed_time': 998.68342161178589, 'skipped': 0, 'calls': 9, \
'success': 9, 'started': 3, 'auditfail': 0, 'failure': 0, 'max_time': \
149.5040431022644, 'min_time': 73.079960107803345} 2005/04/08_12:19:51 Test \
Bandwidth: {'elapsed_time': 180.210116147995, 'skipped': 1, 'calls': 15, \
'success': 14, 'min': 12304.581583091836, 'max': 100415.58264101903, \
'totalbandwidth': 684736.8338568199, 'auditfail': 0, 'failure': 0, 'max_time': \
33.621577024459839, 'min_time': 7.2956085205078125e-05} 2005/04/08_12:19:51 Test \
ResourceRecover: {'elapsed_time': 183.56434273719788, 'skipped': 0, 'calls': 12, \
'success': 9, 'auditfail': 0, 'failure': 3, 'max_time': 50.778339147567749, \
'min_time': 1.1215839385986328} 2005/04/08_12:19:51 <<<<<<<<<<<<<<<< TESTS \
COMPLETED
_______________________________________________________
Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
Home Page: http://linux-ha.org/
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic