[prev in list] [next in list] [prev in thread] [next in thread] 

List:       haproxy
Subject:    Re: Random buffer-overflow crashes
From:       Willy Tarreau <w () 1wt ! eu>
Date:       2013-02-27 23:48:20
Message-ID: 20130227234820.GB10628 () 1wt ! eu
[Download RAW message or body]

Hi Jeff,

On Wed, Feb 27, 2013 at 12:19:23PM -0500, Jeff Zellner wrote:
> Hi there,
> 
> We've been running HAProxy 1.5-dev17 (+ 1.5-dev14) on Ubuntu 12.04 for
> a few months in order to SSL terminate and load balance WebSocket
> connections to our in-browser chat client.
> 
> Starting last week, we started to get mysterious crashes now and again
> where HAProxy (1.5-dev17) would simply terminate. We ran it in debug
> mode and captured the following log:
> https://gist.github.com/bdimcheff/21ee6240ed896c94af5a
> 
> Also in that gist is the config we're running -- pretty
> straightforward as far as I can tell.
> 
> After we got a few crashes in -dev17, we switched to a server that we
> had previously used running -dev14. Strangely we seemed to get the
> same sort of problem, but instead of the haproxy process dying, it
> would hang -- stay running, but not be responsive.
> 
> We're happy to dig into this more to provide more information, please
> let me know how we can help!

This is a major issue, we absolutely need to find what is happening. The
fact that the problem did not happen in the past could be related to a
minor change in recent browsers that triggered a sleeping bug, or someone
who found a real bug and is trying to exploit it on your platform.

There are two things that could help us a lot :

  - could you please try the most recent snapshot ? I don't like to
    ask people to do this but since dev17 we mostly focused on fixing
    bugs, and very few features were added yet, so it should be at
    least as safe, if not more. It could be possible that one of the
    bugs fixed there was responsible for your problem and was not
    identified as dangerous ;

  - could you manage to get a core and along with the executable
    (not stripped please) ? For this you need to disable chroot,
    uid and gid settings, and to do ulimit -c unlimited *before*
    starting haproxy. If you feal uneasy with running this way in
    production with a daemon that has such a bug, don't do it, we
    will figure other solutions (eg: running in gdb).

I'm seeing nothing wrong in your config, so the issue could be
anywhere. Note that the "accept" on the last debug line means it
crashed *after* finishing processing the WS request, so we really
have no idea where at the moment.

Thanks,
Willy


[prev in list] [next in list] [prev in thread] [next in thread]