'Re: Multi-Master Replication Manager - monitor works but mmm_control show executes very long and som'

[prev in list] [next in list] [prev in thread] [next in thread] 

List:       mysql
Subject:    Re: Multi-Master Replication Manager - monitor works but mmm_control show executes very long and som
From:       Rafał_Radecki <radecki.rafal () gmail ! com>
Date:       2013-03-13 15:32:02
Message-ID: CAHd9_iTCSii2heT8ewAXSgF88tgwP-A1bJ1445r2qa5Frj-aqg () mail ! gmail ! com
[Download RAW message or body]

I have used debug on monitor and agent:
- on agent I see:
/etc/init.d/mysql-mmm-agent start
Starting MMM Agent Daemon: 2013/03/13 16:19:52 DEBUG Created pid file
'/var/run/mysql-mmm/mmm_agentd.pid' with pid 13640
2013/03/13 16:19:52 DEBUG Listener: Waiting for connection...
2013/03/13 16:19:53 DEBUG Listener: Connect!
2013/03/13 16:19:53 DEBUG Daemon: Command =
'SET_STATUS|1|db2|ONLINE|reader(192.168.0.1)|db1'
2013/03/13 16:19:53 DEBUG Received Command
SET_STATUS|1|db2|ONLINE|reader(192.168.0.1)|db1
2013/03/13 16:19:53  INFO We have some new roles added or old rules deleted!
2013/03/13 16:19:53  INFO Added:   reader(192.168.0.1)
2013/03/13 16:19:53 DEBUG Executing
/usr/libexec/mysql-mmm//agent/configure_ip vlan1503 192.168.0.1
2013/03/13 16:19:53 DEBUG Fetching uptime from /proc/uptime
2013/03/13 16:19:53 DEBUG Uptime is 4327.64
2013/03/13 16:19:53 DEBUG Daemon: Answer = 'OK: Status applied successfully!'
2013/03/13 16:19:53 DEBUG Listener: Disconnect!
2013/03/13 16:19:53 DEBUG Executing
/usr/libexec/mysql-mmm//agent/configure_ip vlan1503 192.168.0.1
2013/03/13 16:19:53 DEBUG Listener: Waiting for connection...
2013/03/13 16:19:56 DEBUG Listener: Connect!
2013/03/13 16:19:56 DEBUG Daemon: Command =
'SET_STATUS|1|db2|ONLINE|reader(192.168.0.1)|db1'
2013/03/13 16:19:56 DEBUG Received Command
SET_STATUS|1|db2|ONLINE|reader(192.168.0.1)|db1
2013/03/13 16:19:56 DEBUG Fetching uptime from /proc/uptime
2013/03/13 16:19:56 DEBUG Uptime is 4330.64
2013/03/13 16:19:56 DEBUG Daemon: Answer = 'OK: Status applied successfully!'
2013/03/13 16:19:56 DEBUG Listener: Disconnect!
2013/03/13 16:19:56 DEBUG Executing
/usr/libexec/mysql-mmm//agent/configure_ip vlan1503 192.168.0.1
2013/03/13 16:19:56 DEBUG Listener: Waiting for connection...
2013/03/13 16:19:59 DEBUG Listener: Connect!
2013/03/13 16:19:59 DEBUG Daemon: Command =
'SET_STATUS|1|db2|ONLINE|reader(192.168.0.1)|db1'
2013/03/13 16:19:59 DEBUG Received Command
SET_STATUS|1|db2|ONLINE|reader(192.168.0.1)|db1
2013/03/13 16:19:59 DEBUG Fetching uptime from /proc/uptime
2013/03/13 16:19:59 DEBUG Uptime is 4333.62
2013/03/13 16:19:59 DEBUG Daemon: Answer = 'OK: Status applied successfully!'
2013/03/13 16:19:59 DEBUG Listener: Disconnect!
2013/03/13 16:19:59 DEBUG Executing
/usr/libexec/mysql-mmm//agent/configure_ip vlan1503 192.168.0.1
2013/03/13 16:19:59 DEBUG Listener: Waiting for connection...
2013/03/13 16:20:02 DEBUG Listener: Connect!
2013/03/13 16:20:02 DEBUG Daemon: Command =
'SET_STATUS|1|db2|ONLINE|reader(192.168.0.1)|db1'
2013/03/13 16:20:02 DEBUG Received Command
SET_STATUS|1|db2|ONLINE|reader(192.168.0.1)|db1
2013/03/13 16:20:02 DEBUG Fetching uptime from /proc/uptime
2013/03/13 16:20:02 DEBUG Uptime is 4336.63
2013/03/13 16:20:02 DEBUG Daemon: Answer = 'OK: Status applied successfully!'
2013/03/13 16:20:02 DEBUG Listener: Disconnect!
2013/03/13 16:20:02 DEBUG Executing
/usr/libexec/mysql-mmm//agent/configure_ip vlan1503 192.168.0.1
2013/03/13 16:20:02 DEBUG Listener: Waiting for connection...
2013/03/13 16:20:05 DEBUG Listener: Connect!
2013/03/13 16:20:05 DEBUG Daemon: Command =
'SET_STATUS|1|db2|ONLINE|reader(192.168.0.1)|db1'
2013/03/13 16:20:05 DEBUG Received Command
SET_STATUS|1|db2|ONLINE|reader(192.168.0.1)|db1
2013/03/13 16:20:05 DEBUG Fetching uptime from /proc/uptime
2013/03/13 16:20:05 DEBUG Uptime is 4339.61
2013/03/13 16:20:05 DEBUG Daemon: Answer = 'OK: Status applied successfully!'
2013/03/13 16:20:05 DEBUG Listener: Disconnect!
2013/03/13 16:20:05 DEBUG Executing
/usr/libexec/mysql-mmm//agent/configure_ip vlan1503 192.168.0.1
2013/03/13 16:20:05 DEBUG Listener: Waiting for connection...
2013/03/13 16:20:08 DEBUG Listener: Waiting for connection...
2013/03/13 16:20:08 DEBUG Listener: Connect!
2013/03/13 16:20:08 DEBUG Daemon: Command =
'SET_STATUS|1|db2|ONLINE|reader(192.168.0.1)|db1'
2013/03/13 16:20:08 DEBUG Received Command
SET_STATUS|1|db2|ONLINE|reader(192.168.0.1)|db1
2013/03/13 16:20:08 DEBUG Fetching uptime from /proc/uptime
2013/03/13 16:20:08 DEBUG Uptime is 4342.64
2013/03/13 16:20:08 DEBUG Daemon: Answer = 'OK: Status applied successfully!'
2013/03/13 16:20:08 DEBUG Listener: Disconnect!
2013/03/13 16:20:08 DEBUG Executing
/usr/libexec/mysql-mmm//agent/configure_ip vlan1503 192.168.0.1
2013/03/13 16:20:08 DEBUG Listener: Waiting for connection...
2013/03/13 16:20:11 DEBUG Listener: Connect!
2013/03/13 16:20:11 DEBUG Daemon: Command =
'SET_STATUS|1|db2|ONLINE|reader(192.168.0.1)|db1'
2013/03/13 16:20:11 DEBUG Received Command
SET_STATUS|1|db2|ONLINE|reader(192.168.0.1)|db1
2013/03/13 16:20:11 DEBUG Fetching uptime from /proc/uptime
2013/03/13 16:20:11 DEBUG Uptime is 4345.60
2013/03/13 16:20:11 DEBUG Daemon: Answer = 'OK: Status applied successfully!'
2013/03/13 16:20:11 DEBUG Listener: Disconnect!
2013/03/13 16:20:11 DEBUG Executing
/usr/libexec/mysql-mmm//agent/configure_ip vlan1503 192.168.0.1
2013/03/13 16:20:11 DEBUG Listener: Waiting for connection...
2013/03/13 16:20:14 DEBUG Listener: Connect!
2013/03/13 16:20:14 DEBUG Daemon: Command =
'SET_STATUS|1|db2|ONLINE|reader(192.168.0.1)|db1'
2013/03/13 16:20:14 DEBUG Received Command
SET_STATUS|1|db2|ONLINE|reader(192.168.0.1)|db1
2013/03/13 16:20:14 DEBUG Fetching uptime from /proc/uptime
2013/03/13 16:20:14 DEBUG Uptime is 4348.61
2013/03/13 16:20:14 DEBUG Daemon: Answer = 'OK: Status applied successfully!'
2013/03/13 16:20:14 DEBUG Listener: Disconnect!
2013/03/13 16:20:14 DEBUG Executing
/usr/libexec/mysql-mmm//agent/configure_ip vlan1503 192.168.0.1
2013/03/13 16:20:14 DEBUG Listener: Waiting for connection...
2013/03/13 16:20:17 DEBUG Listener: Connect!
2013/03/13 16:20:17 DEBUG Daemon: Command =
'SET_STATUS|1|db2|ONLINE|reader(192.168.0.1)|db1'
2013/03/13 16:20:17 DEBUG Received Command
SET_STATUS|1|db2|ONLINE|reader(192.168.0.1)|db1
2013/03/13 16:20:17 DEBUG Fetching uptime from /proc/uptime
2013/03/13 16:20:17 DEBUG Uptime is 4351.64
2013/03/13 16:20:17 DEBUG Daemon: Answer = 'OK: Status applied successfully!'
2013/03/13 16:20:17 DEBUG Listener: Disconnect!
2013/03/13 16:20:17 DEBUG Executing
/usr/libexec/mysql-mmm//agent/configure_ip vlan1503 192.168.0.1
2013/03/13 16:20:17 DEBUG Listener: Waiting for connection...
2013/03/13 16:20:20 DEBUG Listener: Connect!
2013/03/13 16:20:20 DEBUG Daemon: Command =
'SET_STATUS|1|db2|ONLINE|reader(192.168.0.1)|db1'
2013/03/13 16:20:20 DEBUG Received Command
SET_STATUS|1|db2|ONLINE|reader(192.168.0.1)|db1
2013/03/13 16:20:20 DEBUG Fetching uptime from /proc/uptime
2013/03/13 16:20:20 DEBUG Uptime is 4354.65
2013/03/13 16:20:20 DEBUG Daemon: Answer = 'OK: Status applied successfully!'
2013/03/13 16:20:20 DEBUG Listener: Disconnect!
2013/03/13 16:20:20 DEBUG Executing
/usr/libexec/mysql-mmm//agent/configure_ip vlan1503 192.168.0.1
2013/03/13 16:20:20 DEBUG Listener: Waiting for connection...
2013/03/13 16:20:23 DEBUG Listener: Connect!
2013/03/13 16:20:23 DEBUG Daemon: Command =
'SET_STATUS|1|db2|ONLINE|reader(192.168.0.1)|db1'
2013/03/13 16:20:23 DEBUG Received Command
SET_STATUS|1|db2|ONLINE|reader(192.168.0.1)|db1
2013/03/13 16:20:23 DEBUG Fetching uptime from /proc/uptime
2013/03/13 16:20:23 DEBUG Uptime is 4357.61
2013/03/13 16:20:23 DEBUG Daemon: Answer = 'OK: Status applied successfully!'
2013/03/13 16:20:23 DEBUG Listener: Disconnect!
2013/03/13 16:20:23 DEBUG Executing
/usr/libexec/mysql-mmm//agent/configure_ip vlan1503 192.168.0.1
2013/03/13 16:20:23 DEBUG Listener: Waiting for connection...
2013/03/13 16:20:26 DEBUG Listener: Waiting for connection...
2013/03/13 16:20:26 DEBUG Listener: Connect!
2013/03/13 16:20:26 DEBUG Daemon: Command =
'SET_STATUS|1|db2|ONLINE|reader(192.168.0.1)|db1'
2013/03/13 16:20:26 DEBUG Received Command
SET_STATUS|1|db2|ONLINE|reader(192.168.0.1)|db1
2013/03/13 16:20:26 DEBUG Fetching uptime from /proc/uptime
2013/03/13 16:20:26 DEBUG Uptime is 4360.65
2013/03/13 16:20:26 DEBUG Daemon: Answer = 'OK: Status applied successfully!'
2013/03/13 16:20:26 DEBUG Listener: Disconnect!
2013/03/13 16:20:26 DEBUG Executing
/usr/libexec/mysql-mmm//agent/configure_ip vlan1503 192.168.0.1
2013/03/13 16:20:26 DEBUG Listener: Waiting for connection...
2013/03/13 16:20:29 DEBUG Listener: Connect!
2013/03/13 16:20:29 DEBUG Daemon: Command =
'SET_STATUS|1|db2|ONLINE|reader(192.168.0.1)|db1'
2013/03/13 16:20:29 DEBUG Received Command
SET_STATUS|1|db2|ONLINE|reader(192.168.0.1)|db1
2013/03/13 16:20:29 DEBUG Fetching uptime from /proc/uptime
2013/03/13 16:20:29 DEBUG Uptime is 4363.62
2013/03/13 16:20:29 DEBUG Daemon: Answer = 'OK: Status applied successfully!'
2013/03/13 16:20:29 DEBUG Listener: Disconnect!
2013/03/13 16:20:29 DEBUG Executing
/usr/libexec/mysql-mmm//agent/configure_ip vlan1503 192.168.0.1
2013/03/13 16:20:29 DEBUG Listener: Waiting for connection..

- on monitor:
2013/03/13 14:05:36 DEBUG Received Answer: OK: Status applied
successfully!|UP:1895857.62
2013/03/13 14:05:39 DEBUG Sending command 'SET_STATUS(ONLINE,
reader(readip2), db1)' to db2 (192.168.0.1:9989)
2013/03/13 14:05:39 DEBUG Received Answer: OK: Status applied
successfully!|UP:1895857.62
2013/03/13 14:05:39 DEBUG Sending command 'SET_STATUS(ONLINE,
reader(readip1),writer(writeip), db1)' to db1 (192.168.0.2:99
89)
2013/03/13 14:05:39 DEBUG Received Answer: OK: Status applied
successfully!|UP:1895860.62
2013/03/13 14:05:42 DEBUG Sending command 'SET_STATUS(ONLINE,
reader(readip2),2013/03/13 14:04:22 DEBUG Listener: Waiting for
connection.
..
2013/03/13 14:04:25 DEBUG Listener: Waiting for connection...
2013/03/13 14:04:28 DEBUG Listener: Waiting for connection...
2013/03/13 14:04:31 DEBUG Listener: Waiting for connection...
2013/03/13 14:04:34 DEBUG Listener: Waiting for connection...
2013/03/13 14:04:37 DEBUG Listener: Waiting for connection...
2013/03/13 14:04:40 DEBUG Listener: Waiting for connection...
2013/03/13 14:04:42 DEBUG Listener: Connect!
2013/03/13 14:04:42 DEBUG Listener: Disconnect!
2013/03/13 14:04:42 DEBUG Listener: Waiting for connection...
2013/03/13 14:04:42 DEBUG Listener: Connect!
2013/03/13 14:04:45 DEBUG Listener: Disconnect!
2013/03/13 14:04:45 DEBUG Listener: Waiting for connection...
2013/03/13 14:04:45 DEBUG Listener: Connect!
2013/03/13 14:04:48 DEBUG Listener: Disconnect!
2013/03/13 14:04:48 DEBUG Listener: Waiting for connection...
2013/03/13 14:04:48 DEBUG Listener: Connect!
2013/03/13 14:04:51 DEBUG Listener: Disconnect!
2013/03/13 14:04:51 DEBUG Listener: Waiting for connection...
2013/03/13 14:04:51 DEBUG Listener: Connect!
2013/03/13 14:04:54 DEBUG Listener: Disconnect!
2013/03/13 14:04:54 DEBUG Listener: Waiting for connection...
2013/03/13 14:04:54 DEBUG Listener: Connect!
2013/03/13 14:04:57 DEBUG Listener: Disconnect!
2013/03/13 14:04:57 DEBUG Listener: Waiting for connection...
2013/03/13 14:04:57 DEBUG Listener: Connect!
2013/03/13 14:05:00 DEBUG Listener: Disconnect!
2013/03/13 14:05:00 DEBUG Listener: Waiting for connection...
2013/03/13 14:05:00 DEBUG Listener: Connect!
2013/03/13 14:05:03 DEBUG Listener: Disconnect!

On host with monitor I have more and more connections to mmm agents
and mysql instances in TIME_WAIT state.
I modified net.ipv4.tcp_fin_timeout = 5 (from 60 default) but that
does not help.

netstat -an ->
tcp        0      0 192.168.0.100:44075         192.168.0.1:3306
   TIME_WAIT
tcp        0      0 192.168.0.100:57280         192.168.0.2:9989
   TIME_WAIT
tcp        0      0 192.168.0.100:57374         192.168.0.2:3306
   TIME_WAIT
tcp        0      0 192.168.0.100:44077         192.168.0.1:3306
   TIME_WAIT
tcp        0      0 192.168.0.100:57240         192.168.0.2:9989
   TIME_WAIT
tcp        0      0 192.168.0.100:43897         192.168.0.1:3306
   TIME_WAIT
tcp        0      0 192.168.0.100:57250         192.168.0.2:9989
   TIME_WAIT
tcp        0      0 192.168.0.100:39126         192.168.0.1:9989
   TIME_WAIT
...

Any tips?


Best regards,
Rafal.


2013/3/11 Manuel Arostegui <manuel@tuenti.com>:
>
>
> 2013/3/11 Rafał Radecki <radecki.rafal@gmail.com>
>>
>> Hi All.
>>
>> I use:
>>
>> cat /etc/redhat-release
>> CentOS release 6.3 (Final)
>>
>> uname -a
>> Linux prod1.local 2.6.32-279.14.1.el6.x86_64 #1 SMP Tue Nov 6 23:43:09
>> UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
>>
>> on db host:
>> rpm -qa | grep mmm
>> mysql-mmm-2.2.1-1.el6.noarch
>> bmysql-mmm-agent-2.2.1-1.el6.noarch
>>
>> on monitor host:
>> rpm -qa | grep mmm
>> mysql-mmm-monitor-2.2.1-1.el6.noarch
>> mysql-mmm-2.2.1-1.el6.noarch
>> mysql-mmm-tools-2.2.1-1.el6.noarch
>>
>> When I make mmm_control show/checks the command executes for example
>> for ~15 seconds and sometimes returns no output. In logs there is no
>> info about problems and overall the monitor performs well. But I
>> cannot use mmm_control to check its status. The servers are not over
>> loaded. I have restarted agents and monitor but that has not resolved
>> the problem. When I have rebooted the monitor host first use of
>> mmm_control show was ok but then the problem was active again.
>>
>> Any advice? Have you had any similar problems?
>
>
> Hello,
>
> Have you tried to set "debug 1" in your mmm_common.conf and start the
> monitor? You might find useful output in there.
>
> Manuel.
>
>
> --
> Manuel Aróstegui
> Systems Team
> tuenti.com

--
MySQL General Mailing List
For list archives: http://lists.mysql.com/mysql
To unsubscribe:    http://lists.mysql.com/mysql

[prev in list] [next in list] [prev in thread] [next in thread]
Configure | About | News | Add a list | Sponsored by KoreLogic