[prev in list] [next in list] [prev in thread] [next in thread] 

List:       sleuthkit-users
Subject:    Re: [sleuthkit-users] How long does it take for fls (from the Sleuth Kit) to run on a ntfs partition
From:       Brian Carrier <carrier () sleuthkit ! org>
Date:       2012-08-10 19:08:48
Message-ID: 1BE81DD2-E5EA-4645-91AC-9538206AC72E () sleuthkit ! org
[Download RAW message or body]


On Aug 10, 2012, at 1:36 PM, Michael Cohen wrote:

> Just another point of view. When integrating tsk as a library there needs to be a \
> way for the library to report progress. This is normally done by registering a \
> callback. 
> In many applications it is important to detect if tsk has hung on a particular \
> image, especially for fully automated analysis. A progress callback is a way for \
> the library to notify to the caller that its doing fine. This allows the caller to \
> distinguish between the case of tsk taking too long and hanging.

Related to this, is a backlog task to abstract out the verbose/debug messages that \
TSK generates.  Currently, they are blindly printed to stderr.  We have some use \
cases where we want them printed to a file w/out wanting to redirect all of stderr to \
that file.  So, I want to do a callback / override type of thing for verbose messages \
and let the method deal with where they go.  That could be a way to at least get a \
pulse that it is still working (even if it doesn't give specific percent complete \
progress).

Getting the details is easier said than done.  Let's take fls as an example.  It is \
slowest on FAT file systems when it is run with '-r' and needs to recursively go into \
                get Orphan Files.  The basic algorithm is:
- Traverse the directory hierarchy and keep track of which metadata structures were \
                seen by file names
- Do a metadata walk and consider the structures that were not pointed to by names to \
be orphans. 

The percent done at any point during the directory traversal depends on if we are \
going to do orphan hunting or not (if we want to provide an overall percent done).  \
The code that does the metadata walk is used both stand-alone (i.e. calling ils) or \
after the directory traversal. Therefore, its notion of percent done is complicated \
by what occurred before it.  It's a lot of work to figure out that value.  

In an automated system, we've had much better luck with the approach of quickly \
enumerating all of the files (into a database) and not doing analysis on them.  Then \
you know the total number of files. We can then go through those files and analyze \
them and then we know how many files there are left to analyze and can give some \
level of feedback.

> That's the approach we are using when adapting volatility for example. By default \
> the callback function writes to stdout (writing to stderr can cause problems with \
> interleaving with the standard output stream) only if it is a terminal a kind of \
> progress message as described here. But when we integrate volatility into \
> applications we install a different progress callback which just resets the process \
> watchdogs timer. 
> So for example tsk should report progress for each part of the initial Mft parse \
> then each file in the walk etc. A different problem is loop detection for e.g. \
> recursive directory traversal, since this will never completed but Will report \
> progress.

We should have recursion checks in enough places to detect that problem case \
(although, I just fixed a bug related to that, so it can still happen).

brian


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
sleuthkit-users mailing list
https://lists.sourceforge.net/lists/listinfo/sleuthkit-users
http://www.sleuthkit.org


[prev in list] [next in list] [prev in thread] [next in thread]