[prev in list] [next in list] [prev in thread] [next in thread] 

List:       fedora-directory-devel
Subject:    =?utf-8?q?=5B389-devel=5D?= Re: Should stopping the server interrupt an import job?
From:       William Brown <wbrown () suse ! de>
Date:       2019-10-16 20:00:51
Message-ID: D4FD52C2-E3F3-4DFD-B87E-8CB95B26C10B () suse ! de
[Download RAW message or body]



> On 16 Oct 2019, at 00:27, Mark Reynolds <mreynolds@redhat.com> wrote:
> 
> 
> On 10/14/19 11:13 PM, William Brown wrote:
> > 
> > > On 15 Oct 2019, at 15:51, Mark Reynolds <mreynolds@redhat.com> wrote:
> > > 
> > > 
> > > On 10/14/19 6:35 PM, William Brown wrote:
> > > > > On 15 Oct 2019, at 06:58, Mark Reynolds <mreynolds@redhat.com> wrote:
> > > > > 
> > > > > So we are finding these race conditions (leading to heap-use-after-free) \
> > > > > when you stop the server while an import task is running.  The current code \
> > > > > aborts the task which leaves the database unusable until it is fully \
> > > > > reinitialized at a later time.  Unfortunately the code that handles this is \
> > > > > complex, and very error prone. 
> > > > > I'd like to allow the import task to complete before fully shutting the \
> > > > > server down.  Code fix is trivial, but do we want the import to finish, or \
> > > > > should the import be aborted (and database left broken)?  Thoughts? \
> > > > > Opinions?
> > > > The question is "what does the admin expect"? I could envisage if you start \
> > > > an import and then cancel the task, you expect: 
> > > > * The task to be immediately stopped
> > > > * The db content rolled back.
> > > > 
> > > > Shouldn't we be in a betxn or similar during an import so we can revert?
> > > > 
> > > > Failing this, I'd assume the user would expect a ctrl-c to immediately cancel \
> > > > the task. 
> > > > What kind of use-after-frees are we seeing?
> > > See
> > > 
> > > https://pagure.io/389-ds-base/issue/50646
> > > 
> > > Pretty sure the first thing the import does is delete the db directory, but I \
> > > have not found that in the code yet, but there are definitely no transactions \
> > > used during an "import".  It's a very different process.  Now rolling back the \
> > > database would be nice, but I can imagine very large databases(100+ million \
> > > entries) where disk space could be an issue if you have to keep the old \
> > > database around until the new one is imported. 
> > > As for aborting, currently there is no abort mechanism except for stopping the \
> > > server.  So a ctrl-C is not really an option at this time.  Keep in mind I can \
> > > still easily keep the current abort behavior during a shutdown, but in the \
> > > current design if you abort an import the database is hosed.
> > What about at  least hosing it but nicely somehow? So we rm db, then we start the \
> > import into some temp location, so on ctrl-c even if we crash, db is an empty dir \
> > and still mildly hosed? 
> > I can see your point though about the db size stuff. So I guess my thinking then \
> > is: 
> > * Could we just fix the use after free *and* make the db hosing nicer somehow?
> I can fix the use-after-free, that's easy, but fixing the "hosing" is not trivial, \
> and it is out of the scope of what I am trying to do in 1.3.x.

fair

> > * What happens if we ignore the ctrl-c and just block on import? I think someone \
> > would reach for ctrl+\ pretty fast ...
> The import code launches a thread to the do the work.  So hitting control-C on a \
> CLI tool will not stop the import, and there is currently no abort process for \
> Slapi Tasks that I am aware of besides creating a new "Abort Import" task.  This is \
> something we could do in 1.4.3, or 1.5.x along with the new backend work.

Yeah, that seems like the possible solution to have each thread check periodically \
for a cancel signal in it's work loop? 

> > 
> > 
> > 
> > > > > Thanks,
> > > > > 
> > > > > Mark
> > > > > 
> > > > > -- 
> > > > > 
> > > > > 389 Directory Server Development Team
> > > > > _______________________________________________
> > > > > 389-devel mailing list -- 389-devel@lists.fedoraproject.org
> > > > > To unsubscribe send an email to 389-devel-leave@lists.fedoraproject.org
> > > > > Fedora Code of Conduct: \
> > > > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List \
> > > > > Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List \
> > > > > Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
> > > > > 
> > > > —
> > > > Sincerely,
> > > > 
> > > > William Brown
> > > > 
> > > > Senior Software Engineer, 389 Directory Server
> > > > SUSE Labs
> > > > _______________________________________________
> > > > 389-devel mailing list -- 389-devel@lists.fedoraproject.org
> > > > To unsubscribe send an email to 389-devel-leave@lists.fedoraproject.org
> > > > Fedora Code of Conduct: \
> > > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List \
> > > > Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List \
> > > > Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
> > > > 
> > > -- 
> > > 
> > > 389 Directory Server Development Team
> > —
> > Sincerely,
> > 
> > William Brown
> > 
> > Senior Software Engineer, 389 Directory Server
> > SUSE Labs
> > 
> -- 
> 
> 389 Directory Server Development Team

—
Sincerely,

William Brown

Senior Software Engineer, 389 Directory Server
SUSE Labs
_______________________________________________
389-devel mailing list -- 389-devel@lists.fedoraproject.org
To unsubscribe send an email to 389-devel-leave@lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org



[prev in list] [next in list] [prev in thread] [next in thread]