[prev in list] [next in list] [prev in thread] [next in thread] 

List:       freebsd-hackers
Subject:    Re: Precision Hardware Clocks
From:       Josef 'Jeff' Sipek <jeffpc () josefsipek ! net>
Date:       2024-05-10 14:57:13
Message-ID: Zj42gihcrRhr2mBf () satis
[Download RAW message or body]

On Fri, May 10, 2024 at 05:37:50 +0000, Poul-Henning Kamp wrote:
> Josef 'Jeff' Sipek writes:
> > To summarize, the goals/non-goals for this work are:
> >
> >   Goals:
> >    * read-only interface to various precision hardware clocks (PHCs)
> >    * support for both absolute time and counter-only PHCs
> >    * ability to use software like chrony to stabilize system clocks
> 
> I should and will be the last to discourage anybody from having fun
> with timekeeping.

:)

> But I do feel a responsibility to point out, that you are trying
> to solve already solved problems, in a way that does not work nearly
> as well as those solutions.

I disagree.  I'm sure there are ways of improving my approach, but I don't
think this is a completely solved problem.

There are a *lot* systems out there that currently have only NTP for sync,
they would benefit from a PPS but adding an actual PPS source is not
possible.  Many of the same systems have additional oscillators with counters
that can be used as reference clocks.

(I'm using 82574 e1000e cards for my experiments since I have a number of
them and they have a 25MHz clock & counter.  There are other sources which
are better, but I don't have any on hand.)

Thorough benchmarking is necessary, of course.

> Chesterton's fence:  Before you throw it out or bypass it, you
> should find out why the current timekeeping infrastructure is built
> like it is.

100% agreed.  What I have shared is a very work-in-progress
proof-of-concept.  As I said before, I don't like the code duplication, etc.

If you have any pointers to docs with that context, please let me know.
I've read what I could find (including timecounters & nanokernel), but there
are probably more.

..
> For starters, exposing the hardware count though a char-dev is going
> to be very jittery (= time-noise).  The "userland->kernel->userland"
> context switches are very unpredictable timewise, because it is
> anyones guess how many memory operations it will take, even in the
> best case.  Worse, there is a high risk that you loose the CPU to
> another (kernel)thread which is going to /really/ introduce jitter.

I'm not sure that matters.  The PHC ioctl used by my hacked-up chrony
returns <PHC, system> time pair.  So, from userspace's perspective, it
doesn't matter how long the ioctl took to execute since the result tells the
application the CLOCK_REALTIME timestamp of the PHC reading.  Regardless of
how long it took to get the timestamp pair, the application knows exactly
how long ago (according to the system clock) the PHC reading was made and it
can apply the various filtering and statistics to derive the parameters for
ntp_adjtime(2) or however it decides to tweak the system clock.

This (unpredictable) ioctl latency is different from the
software-timestamped PPS interrupt latency, which is a problem.

> That is why the PPS-API, timecounters and kernel_pll exists:  To
> keep the "real-time" aspect of the timekeeping firmly inside the
> kernel and undisturbed by userland and lower priority kernel
> activities.

Sure, but having the code in the kernel has its limitations too.  A huge one
is that it is much harder to fuse multiple time sources into an estimate
that can be used to steer the system clock.  Exposing additional time
sources to userspace and having it figure out what to do seems more
practical than trying to implement time source combining in the kernel.

To be clear, I think a hybrid approach is the way to go - userspace does
fancy filtering and feeds the kernel pre-processed information so it can
steer the system clock more effectively.  This is already what happens, I
just want to give userspace more time sources to work with.

> Unless you can expose the hardware directly to userland, via mmap(2),
> timekeeping in userland is simply not going to perform.

Counterexample: Linux's PTP kernel API exposes time via char dev ioctls and
it performs very well.  With good PTP-capable hardware, you can get to tens
of ns.

> With that said, a lot of our timekeeping is stuff I wrote 25 years
> old, and it is absolutely due for both a rethink and a refresh,

Let me know if you have any ideas how the timekeeping code should change,
I'm more than happy to work on it.

> but
> if you decide to throw it all out and start from fresh, you will
> not get to the interesting parts for years.

I definitely don't want to throw it all out.  My current code ignores it
because that's the quick & dirty way to experiment with an idea.  A
production-ready version must be more cohesive, of course.

> So before you continue, at the very least, read this:
> 
> 	https://papers.freebsd.org/2002/phk-timecounters/

I've read it a few times over the past couple of months while I was playing
around with the PHC idea. :)

> And you should think a LOT about page 91 in this one too:
> 
> 	https://www.am1.us/wp-content/uploads/Documents/U11625_VIG-TUTORIAL.pdf

I didn't see this presentation before, but that slide is pretty much the one
figure in every timekeeping presentation.  So, I have been thinking about it
already ;)

> (The other 307 pages are also interesting :-)

Indeed!  It looks like a really information-dense set of slides.  Thanks for
the link & feedback.

Jeff.

[prev in list] [next in list] [prev in thread] [next in thread]