'Re: Working on NUMA support'

[prev in list] [next in list] [prev in thread] [next in thread] 

List:       freebsd-hackers
Subject:    Re: Working on NUMA support
From:       "Chad J. Milios" <milios () ccsys ! com>
Date:       2014-01-08 12:50:55
Message-ID: 00E55D89-BDD1-41AD-BBF6-6752B90E8324 () ccsys ! com
[Download RAW message or body]

I'm very interested and excited to watch the progress you are making. Many thank yous \
for grappling this task on FreeBSD!

Are you designing/implementing a fully hierarchy-aware approach (nested domains) or \
is each memory domain separate (not nested, simply "mine" or "not mine" in respect to \
a particular thread context)?

> On Jan 8, 2014, at 3:23 AM, Andrew Bates <andrewbates09@gmail.com> wrote:
> 
> Hey Adrian,
> 
> We spent the last few months on research/design and plan on spending the
> next few months preparing tests, building the physical server(s) to test
> on, and finishing the prototype'd functions.
> 
> There is some white-board and pen/paper design that hasn't yet made it to
> the github repo, but the prototypes online are hopefully a solid base for
> what we intend to complete. In the meantime, we are excited to hear who
> else is interested in NUMA and if anyone has suggestions or concerns about
> our approach.
> 
> 
> > On Wed, Jan 8, 2014 at 12:03 AM, Adrian Chadd <adrian@freebsd.org> wrote:
> > 
> > Cool! Do you have any working code to implement the API, or is this
> > just in the design phase right now?
> > 
> > 
> > -a
> > 
> > 
> > > On 6 January 2014 12:11, Andrew Bates <andrewbates09@gmail.com> wrote:
> > > Hey all,
> > > 
> > > My name is Andrew Bates, and I would like to take a bit of your time to
> > > talk about NUMA support.
> > > 
> > > Supporting Non-Uniform Memory Access in FreeBSD is something that has
> > been
> > > brought up in the past <
> > http://freebsd.1045724.n5.nabble.com/NUMA-Support-is-there-in-FreeBSD-td4865200.html
> > 
> > > .
> > > This is becoming increasingly important now that multiprocessor
> > > systems
> > > are an expanding technology, thus performance is scaling in terms of cpu
> > > count, rather than just clock rate.
> > > 
> > > There is a great opportunity here to optimize performance.  After being
> > > asked to look into this by the EMC Isilon Storage Division, myself and a
> > > few colleagues advised by Andrew Pilloud and Jeff Roberson would like to
> > > propose APIs to handle basic memory allocation/management to specific
> > NUMA
> > > domains.
> > > 
> > > What we have devised so far consists of two levels.  First there are the
> > > KPIs, to expose NUMA functionality at a thread level of domain affinity.
> > > Secondly, there would be a userspace/interface to take advantage of the
> > > proposed APIs, thus giving users the capability to make their
> > applications
> > > NUMA-aware.
> > > 
> > > We took the time to look into how many other systems (Linux, Macintosh,
> > > Solaris, Windows) already approach this problem, so there are some
> > aspects
> > > of our solution that are similar to how Linux and Solaris handle NUMA.
> > > Unlike Linux libnuma, we are only proposing a few additions and a minimal
> > > library that can easily be expanded later to suit users' needs.
> > > 
> > > 
> > > KISS in mind, we came up with the following KPI prototypes
> > (freebsdnuma.h)
> > > to uncover NUMA in a usable fashion:
> > > 
> > > 
> > > -
> > > 
> > > cpuset_get_memory_affinity()
> > > -
> > > 
> > > cpuset_set_memory_affinity()
> > > -
> > > 
> > > move_pages()
> > > -
> > > 
> > > migrate_pages()
> > > -
> > > 
> > > get_numa_cpus()
> > > -
> > > 
> > > get_numa_weights()
> > > 
> > > 
> > > Then to the second part, we have the following userspace API prototypes
> > > (numanor.h) for our interface and testing purposes:
> > > 
> > > 
> > > -
> > > 
> > > is_numa_available()
> > > -
> > > 
> > > set_thread_on_domain()
> > > -
> > > 
> > > set_memory_policy()
> > > -
> > > 
> > > move_thread()
> > > 
> > > 
> > > In much much more detail, you can learn more about these prototypes, this
> > > project, view our progress, track along, and give input on our github
> > repo
> > > < https://github.com/andrewbates09/freebsd-numa > or simply via email.
> > This
> > > repo currently includes fully commented prototypes (like a mini man page)
> > > and will later include additions to the project.
> > > 
> > > If anyone has any comments, suggestions, concerns, quandaries, or just
> > > general thoughts please feel free to contact us, as we would love to hear
> > > your input!
> > > 
> > > The Leaders: Sakire Arslan Ay, Andrew Pilloud, Jeff Roberson
> > > The Team: Andrew Bates, Joshua Clark, Alex Schuldberg, Dustin Walker
> > > 
> > > --
> > > V/Respectfully,
> > > Andrew M Bates
> > > _______________________________________________
> > > freebsd-hackers@freebsd.org mailing list
> > > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> > > To unsubscribe, send any mail to "
> > freebsd-hackers-unsubscribe@freebsd.org"
> 
> 
> 
> -- 
> V/Respectfully,
> Andrew M Bates
> _______________________________________________
> freebsd-hackers@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"
_______________________________________________
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

[prev in list] [next in list] [prev in thread] [next in thread] 