[prev in list] [next in list] [prev in thread] [next in thread] 

List:       haproxy
Subject:    Re: Debugging Backendforwarding and UP status
From:       Willy Tarreau <w () 1wt ! eu>
Date:       2013-08-31 6:45:05
Message-ID: 20130831064505.GC17316 () 1wt ! eu
[Download RAW message or body]

Hi Sebastian,

On Fri, Aug 30, 2013 at 11:30:36AM +0200, Sebastian Fohler wrote:
> Concerning the load balancing, I have experience with load balancing, 
> and yes I knew it was a backend Problem. Most of the backends have been 
> shown as down in my stats, as I already written in my last message. The 
> only thing I thought strange was, that one was shown up and still got me 
> that 503 error.

That can happen when it's impossible to establish a connection to the
server after multiple retries and there is no more server to connect
to.

> About that debugging, that was the question. How much information does 
> HAProxy provide to find the error concerning those backend health checks 
> and shuting down those systems.
> I've set the log to debug mode but everything I got were this sort of 
> log entries:
> 
> Aug 30 09:48:49 localhost haproxy[17568]: Connect from 
> 81.44.136.142:54570 to 192.168.48.12:80 (www.adworxs.net-merged/HTTP)

Got it, in fact your logs are not properly configured, you're on the
default old useless format. Please enable "option httplog" in your
frontend. You will get much more detailed reports (eg: the number of
retries, was this a connect timeout or a reject, etc...).

> So I couldn't find the reason, why all the backends have been shutdown. 
> Obviously cause the check thought they were not availabe, but the 
> problem is, that the same configuration has been working already.
> I had a network problem yesterday and had to reboot those haproxy 
> systems, since that moment none of the websites configured did work 
> anymore.

I'm suspecting that your network issue is still present. Might it be
a missing route somewhere, a firewall rule that was never saved and
lost upon the reboot, etc...

If none of your connections manage to pass through, simply run tcpdump
on the interface between haproxy and the servers. There won't me much
traffic and that should be easier to diagnose. You can do the same as
well on the servers themselves to see if they respond, and if so who
they're responding to.

> So my question was, which log interface gives me the correct information 
> about the checks and what would be the best way to analyze this problem.

The http logs give you all the details about what happened to your request.
The health checks also provide logs. You can even enable
"option log-health-checks" to have details about each succeeded/failed
check. Sometimes it's useful because you'll discover connect timeouts
or such things in your logs.

But at this point it's not easy to guess from here, all I can understand
is that you're having connectivity issues between haproxy and the servers.

Regards,
Willy


[prev in list] [next in list] [prev in thread] [next in thread]