[prev in list] [next in list] [prev in thread] [next in thread] 

List:       haproxy
Subject:    Re: Thoughts on the master-worker model
From:       Simon Horman <horms () verge ! net ! au>
Date:       2011-07-29 5:18:33
Message-ID: 20110729051833.GA10520 () verge ! net ! au
[Download RAW message or body]

Hi Willy,

sorry for not responding earlier.

On Mon, Jul 18, 2011 at 04:40:56PM +0200, Willy Tarreau wrote:
> Hi Simon !
> 
> Last week I could find some time with a quiet place at work to dig again
> into the master-worker patch series.
> 
> I went back to what we discussed a few weeks ago concerning the pid management
> and the cmdline parsing which must be performed only once, until I realized
> that I was having trouble again with pid management due to two things that
> we cannot easily cover :
>   - a new process starting with -sf which would replace the old processes
>     running in master/worker mode.
> 
>   - binding issues within the master process when reloading a configuration
> 
> The first point causes a simple issue : if we start a new process which wants
> to replace the old ones that were working in master-worker mode, it will send
> them a SIGTTOU+SIGUSR1. The issue there is that the master should release all
> of its listening sockets, forward the signal to all children then passively
> wait for all of them to leave (including old ones). This is not undoable but
> adds a bit of complexity, which I was planning on implementing anyway.

I agree that this may be a bit tricky, but it does sound like
a reasonable approach.

> The second point is more complex. I realized that there normally is the same
> config in the master process and in all of its active children. What happens
> if the master fails to reload a configuration ? I'm thinking about the worst
> issues, the one related to binding IP:ports, which can only be detected by an
> active process. If the issue is just a conflict with another socket, then
> surely the new process can restart with -sf instead and get rid of the issue
> (back to point 1). But still, the issue of keeping a master process alive with
> a config that has nothing to do with what its children are doing is messy at
> best, and very dangerous.

I agree that keeping a master going with a new (broken) configuration
that has an unknown relationship to what the slaves are using is messy.
But I suspect that as things stand it is actually quite safe.

Still, it would be nice not to be messy.

> My long-term solution would be to have everything related to the configuration
> behind a pointer, and have that pointer referenced in sessions, health checks
> etc... That way, each session would use the config it was instanciated with
> and it would be much easier to allow an old and a new conf to coexist. It
> would not completely fix the issues with some global settings though (eg:
> nbproc, pollers, tuning options, ...) but it would be a gain. That way, if
> the new config fails, we can switch pointers back to old config and forget
> everything that was attempted.

I think such a pointer would be rather nice, even if it turns out
not to solve the problem at hand.

> Then my thinking went a bit further : before doing what is described above,
> we could have the current master fork a new master which would handle the
> new config and issue new processes. If for any reason that new process fails
> to start, it simply disappears and nothing changes. What I like with this
> method is that it also implicitly allows changing many global settings, even
> the master-worker mode may be changed. New sessions would simply use the new
> config from the new process and old sessions would remain on the old one. The
> only downside I can think of is that it will never provide any possibility to
> maintain stats or any state between the old and the new process, but that's
> secondary as the master-worker model is not meant for that either.

I think that such an approach could work quite well. Its really
just an extension of the way the (non master-worker) soft restart
mechanism works. Of course, its a bit complex, but I suspect that
it can be made to work in a clean manner.

The only slight complication that springs to mind is the case where
haproxy isn't detached from the terminal. Using a parent process of
the master (both old and new) attached to the terminal should
be able to handle that. Actually, I think the current code
uses such a parent-process (grand-master?) for such purposes anyway.

> Another advantage I was seeing to forking a new master from the old one was
> that we could probably keep the soft-restart semantics for the situations
> where the new process cannot bind : it could send a SIGTTOU to old processes
> to relinquish the ports and send a SIGTTIN in case of failure.

Yes I agree. Your suggested approach does sounds like it should
fit well with the soft-restart semantics that are currently
implemented for non-master-worker mode.

> That thinking gave me another idea that I have not developped yet. The core
> of your work is the socket cache and is what makes the system reliable. One
> possibility would that the master doesn't own any configuration at all, just
> the sockets. It would be the new processes that would connect to the socket
> cache to grab some ports, then fork the new workers as it is done today.
> I must say I'm not completely at ease with such a model because I think that
> an instanciator is needed, but I like the idea of an autonomous socket cache,
> which becomes sort of an interface between haproxy and the kernel. It will
> also help if one day we want to implement FTP support, as we'll have to be
> able to bind outgoing sockets to local port 20, and that could be performed
> by the central socket cache.

While a cache server sounds fun, I think it adds additional complexity.
And my gut-feeling is that with that complexity will come more problems
than are solved.

On the other hand, I think it is also worth considering that there may
be other caches needed. For instance, it would be nice if stick table
entries could survive a soft restart. And as that data lives in processes
that are not persistent across soft-restart (even when using master-worker)
your suggestion of a cache or caches in a separate process may be
a good solution.

> With all this in mind, I think we need to discuss a bit more before going
> back to the keyboard. Given that a number of bugs have been fixed since
> 1.5-dev6, I'll probably issue -dev7 soon and that should not stop us from
> trying to elaborate a model that suits all needs.

Understood, thanks for keeping me up to date.

[prev in list] [next in list] [prev in thread] [next in thread]