[prev in list] [next in list] [prev in thread] [next in thread] 

List:       freebsd-hackers
Subject:    Re: sysctl is too slow
From:       Alan Somers <asomers () freebsd ! org>
Date:       2021-08-23 12:54:31
Message-ID: CAOtMX2jWhmQ+z3unztN9+Jd=cs5y588EqL5SxAajdjQ=mdABVg () mail ! gmail ! com
[Download RAW message or body]


Ideally, but it's not very high priority, since it's merely a performance
issue in a monitoring tool.

On Mon, Aug 23, 2021 at 6:05 AM Mateusz Guzik <mjguzik@gmail.com> wrote:

> So is this something you plan on fixing?
>
> On 8/17/21, Alan Somers <asomers@freebsd.org> wrote:
> > Actually, I did get a flamegraph, and only 0.77% of samples were in ZFS.
> >
> > On Mon, Aug 16, 2021 at 7:19 PM Mateusz Guzik <mjguzik@gmail.com> wrote:
> >
> >> On 8/16/21, Alan Somers <asomers@freebsd.org> wrote:
> >> > Yes, I see what you're talking about now.  There are a bunch of linked
> >> > lists in sysctl_find_oid etc.  Good point.
> >> > -Alan
> >> >
> >>
> >> You still want to get a flamegraph, chances are most of the problem is
> in
> >> zfs.
> >>
> >> > On Mon, Aug 16, 2021 at 1:30 PM Mateusz Guzik <mjguzik@gmail.com>
> >> > wrote:
> >> >
> >> >> Last time I checked lookup of a sysctl was very bad with linear scans
> >> all
> >> >> over.
> >> >>
> >> >> Short of complete revamp of the entire thing I would start with
> >> >> replacing the scans with a RB tree at each level. As is if you indeed
> >> >> have 5000 datasets, you are doing increasingly longer walks.
> >> >>
> >> >> On 8/16/21, Alan Somers <asomers@freebsd.org> wrote:
> >> >> > ztop feels very sluggish on a server with 5000 ZFS datasets.
> Dtrace
> >> >> shows
> >> >> > that almost all of its time is spent in sys_sysctl.  ktrace shows
> >> >> > that
> >> >> both
> >> >> > ztop and sysctl(8) call sys_sysctl a total of five times for each
> >> >> > sysctl
> >> >> > they care about:
> >> >> >
> >> >> > 1) To get the next oid
> >> >> > 2) To get the sysctl's name
> >> >> > 3) To get the oidfmt
> >> >> > 4) To get the size of the value
> >> >> > 5) To get the value itself.
> >> >> >
> >> >> > Each of these steps takes about equal time, and together all five
> >> >> > take
> >> >> > about 100us.  If the time per call is mostly syscall overhead, then
> >> the
> >> >> > process could be sped up by 80% by combining all of these things
> >> >> > into
> >> a
> >> >> > single syscall: return the next oid, its name, its format, the size
> >> >> > of
> >> >> its
> >> >> > value, and optimistically the value itself, assuming the user
> passed
> >> >> > a
> >> >> > sufficiently large buffer.
> >> >> >
> >> >> > Am I missing something?  Is there any other reason why sysctl is so
> >> >> > slow?
> >> >> > Or should I forget about it, and try to export ZFS's dataset stats
> >> >> through
> >> >> > devstat instead?
> >> >> > -Alan
> >> >> >
> >> >>
> >> >>
> >> >> --
> >> >> Mateusz Guzik <mjguzik gmail.com>
> >> >>
> >> >
> >>
> >>
> >> --
> >> Mateusz Guzik <mjguzik gmail.com>
> >>
> >
>
>
> --
> Mateusz Guzik <mjguzik gmail.com>
>


[prev in list] [next in list] [prev in thread] [next in thread]