[prev in list] [next in list] [prev in thread] [next in thread] 

List:       freebsd-hackers
Subject:    Re: Best practice for accepting TCP connections on multicore?
From:       Adrian Chadd <adrian () freebsd ! org>
Date:       2014-06-09 17:54:12
Message-ID: CAJ-Vmon26CoXwj2L_MOHgTYJgLSDP_0E+64FG_6eow2ErVe_0w () mail ! gmail ! com
[Download RAW message or body]

freebsd-rss is not freebsd; it's just the evaluation code Ive'
written. 'master' is the branch there.



-a


On 9 June 2014 13:12, Hooman Fazaeli <hoomanfazaeli@gmail.com> wrote:
> On 6/7/2014 6:32 AM, Adrian Chadd wrote:
>>
>> Hi,
>>
>> In the RSS model I'm hacking on:
>>
>> * you query the kernel for the current RSS bucket -> CPU mapping.
>> Right now it's one bucket -> one CPU but at some point it may be one
>> bucket -> cpuset;
>> * you spawn a thread for each RSS bucket;
>> * you pin each thread to the RSS bucket current CPU id;
>> * you create a listen socket in that thread, marked as BINDMULTI (ie,
>> multiple things can listen on this and the kernel will load balance
>> between them) and RSS_BUCKET (ie, please place this socket in the
>> given RSS bucket, rather than the global/wildcard list);
>> * then when a connection comes in, the kernel will first do a lookup
>> for a matching wildcard socket in the per-RSS PCBGROUP, rather than
>> the global wildcard table;
>> * if it finds it, that socket gets the incoming connection.
>>
>> At some point I'll add some notification via kqueue or what not that
>> the RSS buckets need rebalancing, and userland can then re-pin the
>> per-bucket threads.
>>
>> At the moment the hacks I've done only support one listen socket per
>> entry. My hope is that BINDMULTI will do some basic hash to load
>> balance within a set of matching PCB entries - and it'll be combined,
>> so if you do BINDMULTI without RSS, it'll just load balance between
>> multiple sockets with no CPU affinity knowledge. If you do RSS, it'll
>> distribute only CPU-local requests to a thread that's sitting in the
>> right RSS bucket. if you enable both, you can use a thread pool for
>> each RSS bucket CPU and (eventually, when I write it) it'll load
>> balance among those.
>>
>> But for now I'm assuming one incoming thread per RSS bucket will be
>> enough for people to experiment with.
>>
>> anyway, I guess I should email out the details:
>>
>> * http://github.com/erikarn/freebsd - the 'local/rss' branch has the
>> RSS changes to dev/e1000/if_igb.c and netinet/
>
> would you please point to the exact URL for the branch?
>
>> * http://github.com/erikarn/freebsd-rss - has some RSS examples. Look
>> at rss-http.
>>
>> I haven't yet tested this at > 1GE because all I have at home are igb
>> and em NICs. If someone would like to donate ixgbe and T4 hardware,
>> i'll gratefully take it and do up RSS patches for those drivers.
>>
>>
>
> --
>
> Best regards.
> Hooman Fazaeli
>
_______________________________________________
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"
[prev in list] [next in list] [prev in thread] [next in thread]