[prev in list] [next in list] [prev in thread] [next in thread] 

List:       haproxy
Subject:    Re: Issue with tcp-request content and keep alive
From:       Willy Tarreau <w () 1wt ! eu>
Date:       2013-08-31 7:20:29
Message-ID: 20130831072029.GF17316 () 1wt ! eu
[Download RAW message or body]

Hello Ricardo,

On Fri, Aug 30, 2013 at 11:27:45AM +0200, Ricardo F wrote:
> Hello,
> 
> I have an issue when trying to track a connection based on a header, with
> tcp-request, and with keep alive enable in a listen section.
> Over the haproxy i have a cdn, which pass the ip of the client at the
> beginning of the X-Forwarded-For header. All the requests are pass through
> this cdn.
> 
> This is the configuration:
> 
> global
>         maxconn 1000
>         log 127.0.0.1 local5 info err
>         stats socket /var/run/haproxy.sock mode 0600 level admin
>         pidfile /var/run/haproxy.pid
> 
> defaults
>         mode    http
>         log global
>         retries 3
>         option redispatch
>         timeout connect          5s
>         timeout client          10s
>         timeout server          10s
>         timeout http-keep-alive 60s
>         timeout http-request    5s
> 
> listen proxy-http 192.168.1.100:80
>         mode http
>         maxconn 1000
>         balance roundrobin
>         stats enable
>         option httplog
>         option http-server-close
>         #option httpclose
>         option forwardfor
> 
>         stick-table type ip size 128m expire 30m store gpc0
>         tcp-request inspect-delay 5s
>         tcp-request content track-sc0 req.hdr_ip(X-Forwarded-For,1) if HTTP
> 
>         acl rule_marked_deny sc0_get_gpc0 gt 0
> 
>         use_backend back-deny if rule_marked_deny
> 
>         default_backend back-http
> 
> backend back-deny
>         server web-deny 192.168.1.133:80
> 
> backend back-http
>         server web-http 192.168.1.101:80
> 
> 
> With this conf, all the requests with the header X-Forwarded-For are tracked
> in the sc0 counter with the ip included in it.
> 
> If the counter of one ip is update to number one, the request will be send to
> back-deny, this is doing by writing directly in the unix socket from other
> software. Like the example:
> 
> # echo "set table proxy-http key 88.64.32.11 data.gpc0 1" | socat stdio /var/run/haproxy.sock
> 
> Since the moment that this are doing (with keep alive enable) i see that in
> the log of the web-deny backserver (the log are modified for register the
> x-forwarded-for ip instead of the real tcp connection):
> 
> 88.64.32.11 - - [30/Aug/2013:09:08:22 +0200] www.server.com "GET /some/url HTTP/1.1" 301 208
> 157.55.32.236 - - [30/Aug/2013:09:08:27 +0200] www.server.com "GET /some/url HTTP/1.1" 301 208
> 88.64.32.11 - - [30/Aug/2013:09:08:27 +0200] www.server.com "GET /some/url HTTP/1.1" 301 208
> 157.55.32.236 - - [30/Aug/2013:09:08:28 +0200] www.server.com "GET /some/url HTTP/1.1" 301 208
> 88.64.32.11 - - [30/Aug/2013:09:08:29 +0200] www.server.com "GET /some/url HTTP/1.1" 301 208
> 157.56.93.186 - - [30/Aug/2013:09:08:31 +0200] www.server.com "GET /some/url HTTP/1.1" 301 208
> 157.56.93.186 - - [30/Aug/2013:09:08:31 +0200] www.server.com "GET /some/url HTTP/1.1" 301 208
> 
> As can see, there are other ips there and only one is with the "1" in the
> table of the Haproxy. This is a small piece of log, but when i try that in a
> server with more traffic, the problem is worse, more ips are redirected to
> this backend without marked for it.
> 
> But, if i change the listen secion to "option httpclose", all works well,
> only the marked ips are redirected. Problem solved, but why?
> 
> The tcp inspect have problems tracking the request when these are passed
> through the cdn, which route more than one request of various clients in the
> same tcp connection?

I like your detailed analysis, you almost found the reason. This is because
tcp-request inspects traffic at the beginning of a *session*, not for each
request. BUT! there is a trick to help you do what you need.

A tcp-request rule put in a backend will be evaluated each time a session
is transferred to a backend. Since the keep-alive with the client is handled
in the frontend, each new request will cause the session to be connected to 
the backend, and the tcp-request rules in the backend will see all requests
(which is another reason why server-side keep-alive is a nightmare to
implement).

So I suggest that you split your "listen" into "frontend" + "backend" and
move the tcp-request rule in the backend.

I know, you'll tell me "but I can't put a use_backend rule in a backend".
Then simply use "use-server" with a server weight of zero, which will never
be used by regular traffic.

> Probably the next feature (in the roadmap) http-request track-sc will solve
> this?

Yes definitely. However for having looked at how to implement it, the
remaining hair on my head stood up straight and remained like this for
2 days :-)

The real issues with track-counters is to track events that happened
before the track rule. For example, you decide that you track a session
based on an HTTP request. But then if you want to check sc0_conn_rate,
you expect to find there the connection rate for the criterion you're
tracking. But the connection was already established before you decided
to track the element, so you have to guess that you need to count it
anyway without having an event to do so. That's extremely tricky with
keep-alive because you need to know if the connection was the same as
the last one or a new one... and this for every criterion. I don't have
a complete solution to this yet, it still requires more thinking.

Regards,
Willy


[prev in list] [next in list] [prev in thread] [next in thread]