[prev in list] [next in list] [prev in thread] [next in thread]
List: haskell-cafe
Subject: Re: [Haskell-cafe] Large JSON File Processing
From: erik <eraker () gmail ! com>
Date: 2018-01-25 4:32:34
Message-ID: CALe4-LeMs6sVi2KQHNuyZScOURYhR1JzsYribJA-rDK09gzytw () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
> Parallelization in conduit can usually be achieved with the stm-conduit
library, which I believe provides the functionality you're looking for.
That's what I thought, but I couldn't figure out how one typically would
fan-out work across one of the middle conduits. I'd like to run one or two
functions across a lot of inputs in parallel, not each section of the whole
pipeline in parallel.
My attempt at using channels with stm-conduit, dumped all the filepaths
into a channel and then I tried to spawn some async conduit workers that
would read from the channels and yield their results into a shared
downstream channel.
Hmm. It still sounds like it should work, actually. I wonder what went
wrong. Maybe I should try again.
On Wed, Jan 24, 2018 at 8:27 PM, Michael Snoyman <michael@snoyman.com>
wrote:
> Parallelization in conduit can usually be achieved with the stm-conduit
> library, which I believe provides the functionality you're looking for.
>
> On Wed, Jan 24, 2018, 11:52 PM erik <eraker@gmail.com> wrote:
>
> > With Michael Snoyman's help, I rewrote my Conduit version of the
> > application (without using stm-conduit). This was a large improvement: my
> > first Conduit version was operating over all data and I didn't realize
> > this.
> >
> > I also increased the nursery size.
> >
> > My revised function ended up looking like this:
> >
> > module Search where
> > import Conduit ((.|))import qualified Conduit \
> > as Cimport Control.Monadimport Control.Monad.IO.Class \
> > (MonadIO, liftIO)import Control.Monad.Trans.Resource \
> > (MonadResource)import qualified Data.ByteString as Bimport \
> > Data.List (isPrefixOf)import Data.Maybe \
> > (fromJust, isJust)import System.Path.NameManip (guess_dotdot, \
> > absolute_path)import System.FilePath (addTrailingPathSeparator, \
> > normalise)import System.Directory (getHomeDirectory) import \
> > Filters
> >
> > sourceFilesFilter :: (MonadResource m, MonadIO m) => ProjectFilter -> FilePath -> \
> > C.ConduitM () String m () sourceFilesFilter projFilter dirname' =
> > C.sourceDirectoryDeep False dirname'
> > .| parseProject projFilter
> >
> > parseProject :: (MonadResource m, MonadIO m) => ProjectFilter -> C.ConduitM \
> > FilePath String m () parseProject (ProjectFilter filterFunc) = do
> > C.awaitForever go
> > where
> > go path' = do
> > bytes <- liftIO $ B.readFile path'
> > let isProj = validProject bytes
> > when (isJust isProj) $ do
> > let proj' = fromJust isProj
> > when (filterFunc proj') $ C.yield path'
> >
> > My main just runs the conduit and prints those that pass the filter:
> >
> > mainStreamingConduit :: IO ()
> > mainStreamingConduit = do
> > options <- getRecord "Search JSON Files"
> > let filterFunc = makeProjectFilter options
> > searchDir <- absolutize (searchPath options)
> > itExists <- doesDirectoryExist searchDir
> > case itExists of
> > False -> putStrLn "Search Directory does not exist" >> exitWith (ExitFailure 1)
> > True -> C.runConduitRes $ sourceFilesFilter filterFunc searchDir .| C.mapM_ \
> > (liftIO . putStrLn)
> > I run it like this (without the stats, typically):
> >
> > stack exec search-json -- --searchPath $FILES --name NAME +RTS -s -A32m -n4m
> >
> > Without increasing nursery size, I get a productivity around 30%. With
> > the above, however, it looks like this:
> >
> > 72,308,248,744 bytes allocated in the heap
> > 733,911,752 bytes copied during GC
> > 7,410,520 bytes maximum residency (8 sample(s))
> > 863,480 bytes maximum slop
> > 187 MB total memory in use (27 MB lost due to fragmentation)
> >
> > Tot time (elapsed) Avg pause Max pause
> > Gen 0 580 colls, 580 par 2.731s 0.772s 0.0013s 0.0105s
> > Gen 1 8 colls, 7 par 0.163s 0.044s 0.0055s 0.0109s
> >
> > Parallel GC work balance: 35.12% (serial 0%, perfect 100%)
> >
> >
> >
> > TASKS: 10 (1 bound, 9 peak workers (9 total), using -N4)
> >
> > SPARKS: 0 (0 converted, 0 overflowed, 0 dud, 0 GC'd, 0 fizzled)
> >
> >
> >
> > INIT time 0.001s ( 0.006s elapsed)
> > MUT time 26.155s ( 31.602s elapsed)
> > GC time 2.894s ( 0.816s elapsed)
> > EXIT time -0.003s ( 0.008s elapsed)
> > Total time 29.048s ( 32.432s elapsed)
> >
> > Alloc rate 2,764,643,665 bytes per MUT second
> >
> > Productivity 90.0% of total user, 97.5% of total elapsed
> >
> > gc_alloc_block_sync: 3494
> > whitehole_spin: 0
> > gen[0].sync: 15527
> > gen[1].sync: 177
> >
> > I'd still like to figure out how to parallelize the filterProj .
> > parseJson . readFile part, but for now I'm satisfied with what I have.
> >
> > (I also isolated my crashing to another process launched from the same
> > terminal window.)
> >
> > On Sun, Jan 21, 2018 at 10:12 PM, Michael Snoyman <michael@snoyman.com>
> > wrote:
> >
> > > I just wanted to comment on the conduit aspect of this in particular.
> > > Looking at your first version:
> > >
> > > conduitFilesFilter :: ProjectFilter -> Path Abs Dir -> IO [Path Abs File]
> > > conduitFilesFilter projFilter dirname' = do
> > > (_, allFiles) <- listDirRecur dirname'
> > > C.runConduit $
> > > C.yieldMany allFiles
> > > .| C.filterMC (filterMatchingFile projFilter)
> > > .| C.sinkList
> > >
> > >
> > > This isn't taking full advantage of conduit: you're reading in a list of
> > > the files in the file system, instead of streaming those values. And the
> > > output is a list of `String`, instead of streaming out those `String`s.
> > > More idiomatic would look something like:
> > >
> > > sourceFilesFilter projFilter dirname' =
> > > sourceDirectoryDeep False dirname' .| filterMC (filterMatchingFile
> > > projFilter)
> > >
> > > And then, wherever you're consuming the output, to do so in a streaming
> > > fashion, e.g.:
> > >
> > > runConduitRes $ sourceFilesFilter projFilter dirname' .| mapM_C print
> > >
> > > This should help with the increasing memory usage, though it will do
> > > nothing about the runtime overhead of parsing the JSON itself.
> > >
> > > On Mon, Jan 22, 2018 at 1:38 AM, erik <eraker@gmail.com> wrote:
> > >
> > > > Hello Haskell Cafe,
> > > >
> > > > I have written a small, pretty simple program but I am finding it hard
> > > > to reason about its behavior (and also about the best way to do what I
> > > > want), so I would like to ask you all for some suggestions.
> > > >
> > > > For reference, here's a Stack Overflow question
> > > > <https://stackoverflow.com/questions/48330690/haskell-conduit-aeson-parsing-large-jsons-and-filter-matching-key-values/48348153#48348153>
> > > > where I described what's going on, but I'll also describe it below.
> > > >
> > > > My program does the following:
> > > >
> > > > 1. Recursively list a directory,
> > > > 2. Parse the JSON files from the directory list into identifiable
> > > > objects/records,
> > > > 3. Look for matching key-value pairs, and
> > > > 4. Return filenames where matches have been found.
> > > >
> > > > A few details for more context:
> > > >
> > > > - I have to filter between 500,000 and 1 million files (I'm
> > > > typically trying to reduce down to between 1,000 and 40,000 that represent
> > > > a particular project). I usually just need the filenames.
> > > > - Each file is quite large, some of them 5mb or 10mb, and it's not
> > > > uncommon for them to have deeply nested keys (40,000 keys or so).
> > > >
> > > > My first version of this program was simple, synchronous, and as
> > > > straightforward as I could come up with. However, the memory usage
> > > > increased monotonically. Profiling, I found that most of the time was spent
> > > > in JSON-parsing into Objects before my code could turn the objects into
> > > > records (also, as you might imagine, tons of time in garbage collection).
> > > >
> > > > For my second version, I switched to conduit and it seemed to solve the
> > > > increasing memory issue. My core function now looked like this:
> > > >
> > > > conduitFilesFilter :: ProjectFilter -> Path Abs Dir -> IO [Path Abs File]
> > > > conduitFilesFilter projFilter dirname' = do
> > > > (_, allFiles) <- listDirRecur dirname'
> > > > C.runConduit $
> > > > C.yieldMany allFiles
> > > > .| C.filterMC (filterMatchingFile projFilter)
> > > > .| C.sinkList
> > > >
> > > >
> > > > This was still slow and certainly still synchronous. What I really
> > > > wanted was to run that "filterMatchingFile..." part in parallel across a
> > > > number of CPUs. As an aside, my filtering function looks like this:
> > > >
> > > > filterMatchingFile :: ProjectFilter -> Path Abs File -> IO Bool
> > > > filterMatchingFile (ProjectFilter filterFunc) fpath = do
> > > > let fp = toFilePath fpath
> > > > bs <- B.readFile fp
> > > > case validImplProject bs of -- this is pretty much just
> > > > `decodeStrict`
> > > > Nothing -> pure False
> > > > (Just proj') -> pure $ filterFunc proj'
> > > >
> > > > Here are the stats from running this:
> > > >
> > > > 115,961,554,600 bytes allocated in the heap
> > > > 35,870,639,768 bytes copied during GC
> > > > 56,467,720 bytes maximum residency (681 sample(s))
> > > > 1,283,008 bytes maximum slop
> > > > 145 MB total memory in use (0 MB lost due to fragmentation)
> > > >
> > > > Tot time (elapsed) Avg pause Max pause
> > > > Gen 0 108716 colls, 108716 par 76.915s 20.571s 0.0002s 0.0266s
> > > > Gen 1 681 colls, 680 par 0.530s 0.147s 0.0002s 0.0009s
> > > >
> > > > Parallel GC work balance: 14.99% (serial 0%, perfect 100%)
> > > >
> > > > TASKS: 10 (1 bound, 9 peak workers (9 total), using -N4)
> > > >
> > > > SPARKS: 0 (0 converted, 0 overflowed, 0 dud, 0 GC'd, 0 fizzled)
> > > >
> > > > INIT time 0.001s ( 0.007s elapsed)
> > > > MUT time 34.813s ( 42.938s elapsed)
> > > > GC time 77.445s ( 20.718s elapsed)
> > > > EXIT time 0.000s ( 0.010s elapsed)
> > > > Total time 112.260s ( 63.672s elapsed)
> > > >
> > > > Alloc rate 3,330,960,996 bytes per MUT second
> > > >
> > > > Productivity 31.0% of total user, 67.5% of total elapsed
> > > >
> > > > gc_alloc_block_sync: 188614
> > > > whitehole_spin: 0
> > > > gen[0].sync: 33
> > > > gen[1].sync: 811204
> > > >
> > > >
> > > > I thought about writing a plainer (non-conduit) parallel version but I
> > > > was afraid of the memory issue. I tried to write a Conduit-plus-channels
> > > > version but it didn't work.
> > > >
> > > > Finally, I wrote a version using stm-conduit, which I thought might be
> > > > a bit more efficient. It seems to be slightly better, but it's not really
> > > > the kind of parallelization I was imagining:
> > > >
> > > > conduitAsyncFilterFiles :: ProjectFilter -> Path Abs Dir -> IO [String]
> > > > conduitAsyncFilterFiles projFilter dirname' = do
> > > > (_, allFiles) <- listDirRecur dirname'
> > > > buffer 10
> > > > (C.yieldMany allFiles
> > > > .| (C.mapMC (readFileWithPath . toFilePath)))
> > > > (C.mapC (filterProjForFilename projFilter)
> > > > .| C.filterC isJust
> > > > .| C.mapC fromJust
> > > > .| C.sinkList)
> > > >
> > > > The first conduit passed to `buffer` does something like the following: \
> > > > parseStrict
> > > > . B.readFile.
> > > >
> > > > This still wasn't too great, but after reading about handing garbage
> > > > collection in smarter ways, I found that I could run my application like
> > > > this:
> > > >
> > > > stack exec search-json -- --searchPath $FILES --name hello +RTS -s -A32m -n4m
> > > >
> > > > And the "productivity" would shoot up quite a lot presumably because
> > > > I'm doing less frequent garbage collection. My program also got a bit
> > > > faster:
> > > >
> > > > 36,379,265,096 bytes allocated in the heap
> > > > 1,238,438,160 bytes copied during GC
> > > > 22,996,264 bytes maximum residency (85 sample(s))
> > > > 3,834,152 bytes maximum slop
> > > > 207 MB total memory in use (14 MB lost due to fragmentation)
> > > >
> > > > Tot time (elapsed) Avg pause Max pause
> > > > Gen 0 211 colls, 211 par 1.433s 0.393s 0.0019s 0.0077s
> > > > Gen 1 85 colls, 84 par 0.927s 0.256s 0.0030s 0.0067s
> > > >
> > > > Parallel GC work balance: 67.93% (serial 0%, perfect 100%)
> > > >
> > > > TASKS: 10 (1 bound, 9 peak workers (9 total), using -N4)
> > > >
> > > > SPARKS: 0 (0 converted, 0 overflowed, 0 dud, 0 GC'd, 0 fizzled)
> > > >
> > > > INIT time 0.001s ( 0.004s elapsed)
> > > > MUT time 12.636s ( 12.697s elapsed)
> > > > GC time 2.359s ( 0.650s elapsed)
> > > > EXIT time -0.015s ( 0.003s elapsed)
> > > > Total time 14.982s ( 13.354s elapsed)
> > > >
> > > > Alloc rate 2,878,972,840 bytes per MUT second
> > > >
> > > > Productivity 84.2% of total user, 95.1% of total elapsed
> > > >
> > > > gc_alloc_block_sync: 9612
> > > > whitehole_spin: 0
> > > > gen[0].sync: 2044
> > > > gen[1].sync: 47704
> > > >
> > > >
> > > > Thanks for reading thus far. I now have three questions.
> > > >
> > > > 1. I understand that my program necessarily creates tons of garbage
> > > > because it parses and then throws away 5mb of JSON 500,000 times. However,
> > > > I don't really understand why this helps "+RTS -A32m -n4m" and I'm
> > > > always reluctant to sprinkle in magic I don't fully understand. Can anyone
> > > > help me understand what this means?
> > > >
> > > > 2. It seems that the allocation limit is really something I should be
> > > > using, but I can't figure out how to successfully add it to my package.yml
> > > > with the other options. From the documentation for GHC 8.2, I thought it
> > > > needed to look like this but it never works, usually telling me that -A32m
> > > > and -n4m are not recognizable flags (how do I add them in to my package.yml
> > > > so I don't have to pass them when running the program?):
> > > >
> > > > ghc-options:
> > > > - -threaded
> > > > - -rtsopts
> > > > - "-with-rtsopts=-N4 -A32m -n4m"
> > > >
> > > > 3. Finally, the most important question I have is this. When I run this
> > > > program on OSX, it runs successfully through to completion. However, *a
> > > > few minutes after terminating*, my terminal becomes unresponsive. I
> > > > use emacs for my editor, typically launched from a terminal window and that
> > > > too becomes unresponsive. This is not a typical outcome for any programs I
> > > > write and it happens *every time* I run this particular application,
> > > > so I know that this application is to blame.
> > > >
> > > > The crazy thing is that force quitting the terminal or logging out
> > > > doesn't help: I have to actually restart my computer to use the terminal
> > > > application again. Other details that may help:
> > > >
> > > > - This crash happens after the process id for my program has
> > > > terminated.
> > > > - Watching its progress in HTOP, it never comes close to running
> > > > out of memory: the value hovers in the same place.
> > > >
> > > > I can't really deploy an application that has this potential-crashing
> > > > problem, but I don't know to debug this issue. My total stab-in-the-dark
> > > > idea is that heap allocations somehow are unrecoverable even after the
> > > > process has terminated? Can anyone offer suggestions on things to look for
> > > > or ways to debug and/or fix this issue?
> > > >
> > > > Finally, if anyone has suggestions on better ways to structure my
> > > > application or parallelize the slow parts, I'll happily take those.
> > > >
> > > > Thanks again for reading. I appreciate any suggestions you may have.
> > > >
> > > > Best,
> > > >
> > > > --
> > > > Erik Aker
> > > >
> > > > _______________________________________________
> > > > Haskell-Cafe mailing list
> > > > To (un)subscribe, modify options or view archives go to:
> > > > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
> > > > Only members subscribed via the mailman list are allowed to post.
> > > >
> > >
> > >
> >
> >
> > --
> > Erik Aker
> >
>
--
Erik Aker
[Attachment #5 (text/html)]
<div dir="ltr"><div>> Parallelization in conduit can usually be achieved with the \
stm-conduit library, which I believe provides the functionality you're looking \
for.</div><div><br></div><div>That's what I thought, but I couldn't figure \
out how one typically would fan-out work across one of the middle conduits. I'd \
like to run one or two functions across a lot of inputs in parallel, not each section \
of the whole pipeline in parallel.<br></div><div><br></div><div>My attempt at using \
channels with stm-conduit, dumped all the filepaths into a channel and then I tried \
to spawn some async conduit workers that would read from the channels and yield their \
results into a shared downstream channel. <br></div><div><br></div><div>Hmm. It still \
sounds like it should work, actually. I wonder what went wrong. Maybe I should try \
again.<br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, \
Jan 24, 2018 at 8:27 PM, Michael Snoyman <span dir="ltr"><<a \
href="mailto:michael@snoyman.com" target="_blank">michael@snoyman.com</a>></span> \
wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px \
#ccc solid;padding-left:1ex"><p dir="ltr">Parallelization in conduit can usually be \
achieved with the stm-conduit library, which I believe provides the functionality \
you're looking for.</p><div class="HOEnZb"><div class="h5"> <br><div \
class="gmail_quote"><div dir="ltr">On Wed, Jan 24, 2018, 11:52 PM erik <<a \
href="mailto:eraker@gmail.com" target="_blank">eraker@gmail.com</a>> \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 \
.8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">With Michael \
Snoyman's help, I rewrote my Conduit version of the application (without using \
stm-conduit). This was a large improvement: my first Conduit version was operating \
over all data and I didn't realize this.
<p>I also increased the nursery size.</p>
<p>My revised function ended up looking like this:</p>
<pre class="m_8845810541154921453m_8418980463297255581gmail-lang-hs \
m_8845810541154921453m_8418980463297255581gmail-prettyprint \
m_8845810541154921453m_8418980463297255581gmail-prettyprinted"><code><span \
class="m_8845810541154921453m_8418980463297255581gmail-kwd">module</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> Search </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-kwd">where</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">
</span><span class="m_8845810541154921453m_8418980463297255581gmail-kwd">import</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> Conduit \
</span><span class="m_8845810541154921453m_8418980463297255581gmail-pun">((.|))</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-kwd">import</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> qualified Conduit \
as C </span><span class="m_8845810541154921453m_8418980463297255581gmail-kwd">import</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> Control.Monad \
</span><span class="m_8845810541154921453m_8418980463297255581gmail-kwd">import</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> \
Control.Monad.IO.Class </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">MonadIO</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> liftIO</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-kwd">import</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> \
Control.Monad.Trans.Resource </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">MonadResource</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-kwd">import</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> qualified \
Data.ByteString as B </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-kwd">import</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> Data.List \
</span><span class="m_8845810541154921453m_8418980463297255581gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">isPrefixOf</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-kwd">import</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> Data.Maybe \
</span><span class="m_8845810541154921453m_8418980463297255581gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">fromJust</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> isJust</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-kwd">import</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> \
System.Path.NameManip </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">guess_dotdot</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> \
absolute_path</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-kwd">import</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> \
System.FilePath </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">addTrailingPathSeparator</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> normalise</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-kwd">import</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> \
System.Directory </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">getHomeDirectory</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">
</span><span class="m_8845810541154921453m_8418980463297255581gmail-kwd">import</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> Filters
sourceFilesFilter </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">::</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">MonadResource \
m</span><span class="m_8845810541154921453m_8418980463297255581gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> MonadIO m</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">=></span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> ProjectFilter \
</span><span class="m_8845810541154921453m_8418980463297255581gmail-pun">-></span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> FilePath </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">-></span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> C.ConduitM </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">()</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> String m </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">()</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> sourceFilesFilter \
projFilter dirname' </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">=</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> C.sourceDirectoryDeep \
False dirname' </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">.|</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> parseProject projFilter
parseProject </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">::</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">MonadResource \
m</span><span class="m_8845810541154921453m_8418980463297255581gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> MonadIO m</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">=></span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> ProjectFilter \
</span><span class="m_8845810541154921453m_8418980463297255581gmail-pun">-></span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> C.ConduitM FilePath \
String m </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">()</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> parseProject \
</span><span class="m_8845810541154921453m_8418980463297255581gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">ProjectFilter \
filterFunc</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">=</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-kwd">do</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> C.awaitForever go
</span><span class="m_8845810541154921453m_8418980463297255581gmail-kwd">where</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> go path' \
</span><span class="m_8845810541154921453m_8418980463297255581gmail-pun">=</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-kwd">do</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> bytes </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun"><-</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> liftIO </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">$</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> B.readFile path' \
</span><span class="m_8845810541154921453m_8418980463297255581gmail-kwd">let</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> isProj </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">=</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> validProject bytes when \
</span><span class="m_8845810541154921453m_8418980463297255581gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">isJust isProj</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">$</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-kwd">do</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-kwd">let</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> proj' </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">=</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> fromJust isProj when \
</span><span class="m_8845810541154921453m_8418980463297255581gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">filterFunc \
proj'</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">$</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> C.yield \
path'</span></code></pre>
<p>My main just runs the conduit and prints those that pass the filter:</p>
<pre class="m_8845810541154921453m_8418980463297255581gmail-lang-hs \
m_8845810541154921453m_8418980463297255581gmail-prettyprint \
m_8845810541154921453m_8418980463297255581gmail-prettyprinted"><code><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">mainStreamingConduit \
</span><span class="m_8845810541154921453m_8418980463297255581gmail-pun">::</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> IO </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">()</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> mainStreamingConduit \
</span><span class="m_8845810541154921453m_8418980463297255581gmail-pun">=</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-kwd">do</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> options </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun"><-</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> getRecord </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-str">"Search JSON \
Files"</span><span class="m_8845810541154921453m_8418980463297255581gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581gmail-kwd">let</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> filterFunc </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">=</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> makeProjectFilter \
options searchDir </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun"><-</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> absolutize </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">searchPath \
options</span><span class="m_8845810541154921453m_8418980463297255581gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> itExists </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun"><-</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> doesDirectoryExist \
searchDir </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-kwd">case</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> itExists </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-kwd">of</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> False </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">-></span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> putStrLn </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-str">"Search Directory \
does not exist"</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">>></span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> exitWith </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">ExitFailure </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">1</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> True </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">-></span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> C.runConduitRes \
</span><span class="m_8845810541154921453m_8418980463297255581gmail-pun">$</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> sourceFilesFilter \
filterFunc searchDir </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">.|</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> C.mapM_ </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">liftIO </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">.</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> putStrLn</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">)</span></code></pre>
<p>I run it like this (without the stats, typically):</p>
<pre class="m_8845810541154921453m_8418980463297255581gmail-lang-hs \
m_8845810541154921453m_8418980463297255581gmail-prettyprint \
m_8845810541154921453m_8418980463297255581gmail-prettyprinted"><code><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">stack exec \
search</span><span class="m_8845810541154921453m_8418980463297255581gmail-pun">-</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">json </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-com">-- --searchPath $FILES \
--name NAME +RTS -s -A32m -n4m</span></code></pre>
<p>Without increasing nursery size, I get a productivity around 30%. With the above, \
however, it looks like this:</p>
<pre class="m_8845810541154921453m_8418980463297255581gmail-lang-hs \
m_8845810541154921453m_8418980463297255581gmail-prettyprint \
m_8845810541154921453m_8418980463297255581gmail-prettyprinted"><code><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">72</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">308</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">248</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">744</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> bytes allocated \
</span><span class="m_8845810541154921453m_8418980463297255581gmail-kwd">in</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> the heap </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">733</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">911</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">752</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> bytes copied during GC \
</span><span class="m_8845810541154921453m_8418980463297255581gmail-lit">7</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">410</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">520</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> bytes maximum residency \
</span><span class="m_8845810541154921453m_8418980463297255581gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">8</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> sample</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">s</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">))</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">863</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">480</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> bytes maximum slop \
</span><span class="m_8845810541154921453m_8418980463297255581gmail-lit">187</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> MB total memory \
</span><span class="m_8845810541154921453m_8418980463297255581gmail-kwd">in</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> use </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">27</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> MB lost due to \
fragmentation</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">
Tot time </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">elapsed</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> Avg pause Max pause \
Gen </span><span class="m_8845810541154921453m_8418980463297255581gmail-lit">0</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">580</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> colls</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">580</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> par </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">2.731</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">s </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">0.772</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">s </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">0.0013</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">s </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">0.0105</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">s Gen </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">1</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">8</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> colls</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">7</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> par </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">0.163</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">s </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">0.044</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">s </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">0.0055</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">s </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">0.0109</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">s
Parallel GC work balance</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">:</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">35.12</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">%</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">serial </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">0</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">%,</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> perfect </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">100</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">%)</span></code></pre></div><div \
dir="ltr"><pre class="m_8845810541154921453m_8418980463297255581gmail-lang-hs \
m_8845810541154921453m_8418980463297255581gmail-prettyprint \
m_8845810541154921453m_8418980463297255581gmail-prettyprinted"><code><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">
TASKS</span><span class="m_8845810541154921453m_8418980463297255581gmail-pun">:</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">10</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">1</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> bound</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">9</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> peak workers \
</span><span class="m_8845810541154921453m_8418980463297255581gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">9</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> total</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">),</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> using </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">-</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">N4</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">
SPARKS</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">:</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">0</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">0</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> converted</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">0</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> overflowed</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">0</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> dud</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">0</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> GC'd</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">0</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> fizzled</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">)</span></code></pre></div><div \
dir="ltr"><pre class="m_8845810541154921453m_8418980463297255581gmail-lang-hs \
m_8845810541154921453m_8418980463297255581gmail-prettyprint \
m_8845810541154921453m_8418980463297255581gmail-prettyprinted"><code><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">
INIT time </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">0.001</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">s </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">0.006</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">s elapsed</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> MUT time \
</span><span class="m_8845810541154921453m_8418980463297255581gmail-lit">26.155</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">s </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">31.602</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">s elapsed</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> GC time \
</span><span class="m_8845810541154921453m_8418980463297255581gmail-lit">2.894</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">s </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">0.816</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">s elapsed</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> EXIT time \
</span><span class="m_8845810541154921453m_8418980463297255581gmail-pun">-</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">0.003</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">s </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">0.008</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">s elapsed</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> Total time \
</span><span class="m_8845810541154921453m_8418980463297255581gmail-lit">29.048</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">s </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">32.432</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">s elapsed</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">
Alloc rate </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">2</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">764</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">643</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">665</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> bytes per MUT second
Productivity </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">90.0</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">%</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-kwd">of</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> total user</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">97.5</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">%</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-kwd">of</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> total elapsed
gc_alloc_block_sync</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">:</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">3494</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> \
whitehole_spin</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">:</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">0</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> gen</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">[</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">0</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">].</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">sync</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">:</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">15527</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> gen</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">[</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">1</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">].</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln">sync</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pun">:</span><span \
class="m_8845810541154921453m_8418980463297255581gmail-pln"> </span><span \
class="m_8845810541154921453m_8418980463297255581gmail-lit">177</span></code></pre>
<p>I'd still like to figure out how to parallelize the <code>filterProj . \
parseJson . readFile</code> part, but for now I'm satisfied with what I have. \
<br></p><p>(I also isolated my crashing to another process launched from the same \
terminal window.)<br></p></div><div class="gmail_extra"><br><div \
class="gmail_quote"></div></div><div class="gmail_extra"><div class="gmail_quote">On \
Sun, Jan 21, 2018 at 10:12 PM, Michael Snoyman <span dir="ltr"><<a \
href="mailto:michael@snoyman.com" target="_blank">michael@snoyman.com</a>></span> \
wrote:<br></div></div><div class="gmail_extra"><div class="gmail_quote"><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr"><div>I just wanted to comment on the conduit \
aspect of this in particular. Looking at your first \
version:</div><span><div><br><div><pre \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176gmail-m_746947386685124369gmail-lang-hs \
m_8845810541154921453m_8418980463297255581m_6090739341647934176gmail-m_746947386685124369gmail-prettyprint \
m_8845810541154921453m_8418980463297255581m_6090739341647934176gmail-m_746947386685124369gmail-prettyprinted"><code><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176gmail-m_746947386685124369gmail-pln">conduitFilesFilter \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176gmail-m_746947386685124369gmail-pun">::</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176gmail-m_746947386685124369gmail-pln"> \
ProjectFilter </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176gmail-m_746947386685124369gmail-pun">-></span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176gmail-m_746947386685124369gmail-pln"> \
Path Abs Dir </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176gmail-m_746947386685124369gmail-pun">-></span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176gmail-m_746947386685124369gmail-pln"> \
IO </span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176gmail-m_746947386685124369gmail-pun">[</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176gmail-m_746947386685124369gmail-pln">Path \
Abs File</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176gmail-m_746947386685124369gmail-pun">]</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176gmail-m_746947386685124369gmail-pln">
conduitFilesFilter projFilter dirname' </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176gmail-m_746947386685124369gmail-pun">=</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176gmail-m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176gmail-m_746947386685124369gmail-kwd">do</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176gmail-m_746947386685124369gmail-pln">
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176gmail-m_746947386685124369gmail-pun">(_,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176gmail-m_746947386685124369gmail-pln"> \
allFiles</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176gmail-m_746947386685124369gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176gmail-m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176gmail-m_746947386685124369gmail-pun"><-</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176gmail-m_746947386685124369gmail-pln"> \
listDirRecur dirname' C.runConduit </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176gmail-m_746947386685124369gmail-pun">$</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176gmail-m_746947386685124369gmail-pln">
C.yieldMany allFiles
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176gmail-m_746947386685124369gmail-pun">.|</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176gmail-m_746947386685124369gmail-pln"> \
C.filterMC </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176gmail-m_746947386685124369gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176gmail-m_746947386685124369gmail-pln">filterMatchingFile \
projFilter</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176gmail-m_746947386685124369gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176gmail-m_746947386685124369gmail-pln">
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176gmail-m_746947386685124369gmail-pun">.|</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176gmail-m_746947386685124369gmail-pln"> \
C.sinkList</span></code></pre></div><span \
style="font-family:monospace,monospace"></span></div><div><br></div></span><div>This \
isn't taking full advantage of conduit: you're reading in a list of the files \
in the file system, instead of streaming those values. And the output is a list of \
`String`, instead of streaming out those `String`s. More idiomatic would look \
something like:</div><div><br></div><div>sourceFilesFilter projFilter dirname' \
=</div><div> sourceDirectoryDeep False dirname' .| filterMC (filterMatchingFile \
projFilter)</div><div><br></div><div>And then, wherever you're consuming the \
output, to do so in a streaming fashion, e.g.:</div><div><br></div><div>runConduitRes \
$ sourceFilesFilter projFilter dirname' .| mapM_C \
print</div><div><br></div><div>This should help with the increasing memory usage, \
though it will do nothing about the runtime overhead of parsing the JSON \
itself.<br></div><div><span \
style="font-family:monospace,monospace"></span></div></div><div \
class="gmail_extra"><br><div class="gmail_quote"><div><div \
class="m_8845810541154921453m_8418980463297255581h5">On Mon, Jan 22, 2018 at 1:38 AM, \
erik <span dir="ltr"><<a href="mailto:eraker@gmail.com" \
target="_blank">eraker@gmail.com</a>></span> wrote:<br></div></div><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div><div \
class="m_8845810541154921453m_8418980463297255581h5"><div dir="ltr"><div>Hello \
Haskell Cafe,</div><div><br></div><div>I have written a small, pretty simple program \
but I am finding it hard to reason about its behavior (and also about the best way to \
do what I want), so I would like to ask you all for some \
suggestions.</div><div><br></div><div>For reference, here's a <a \
href="https://stackoverflow.com/questions/48330690/haskell-conduit-aeson-parsing-large-jsons-and-filter-matching-key-values/48348153#48348153" \
target="_blank">Stack Overflow question</a> where I described what's going on, \
but I'll also describe it below.</div><div><br></div><div>My program does the \
following:<br></div><div><ol><li>Recursively list a directory,</li><li>Parse the JSON \
files from the directory list into identifiable objects/records,<br></li><li>Look for \
matching key-value pairs, and</li><li>Return filenames where matches have been \
found.</li></ol><div>A few details for more context:</div><ul><li>I have to filter \
between 500,000 and 1 million files (I'm typically trying to reduce down to \
between 1,000 and 40,000 that represent a particular project). I usually just need \
the filenames.<br></li><li>Each file is quite large, some of them 5mb or 10mb, and \
it's not uncommon for them to have deeply nested keys (40,000 keys or \
so).</li></ul><div>My first version of this program was simple, synchronous, and as \
straightforward as I could come up with. However, the memory usage increased \
monotonically. Profiling, I found that most of the time was spent in JSON-parsing \
into Objects before my code could turn the objects into records (also, as you might \
imagine, tons of time in garbage collection).<br></div><div><br></div><div>For my \
second version, I switched to conduit and it seemed to solve the increasing memory \
issue. My core function now looked like this:</div><div><pre \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lang-hs \
m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-prettyprint \
m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-prettyprinted"><code><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">conduitFilesFilter \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">::</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
ProjectFilter </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">-></span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
Path Abs Dir </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">-></span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
IO </span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">[</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">Path \
Abs File</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">]</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">
conduitFilesFilter projFilter dirname' </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">=</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-kwd">do</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">(_,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
allFiles</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun"><-</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
listDirRecur dirname' C.runConduit </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">$</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">
C.yieldMany allFiles
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">.|</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
C.filterMC </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">filterMatchingFile \
projFilter</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">.|</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
C.sinkList</span></code></pre></div><div><br></div><div>This was still slow and \
certainly still synchronous. What I really wanted was to run that \
"filterMatchingFile..." part in parallel across a number of CPUs. As an \
aside, my filtering function looks like this:</div><div><br></div><div><span \
style="font-family:monospace,monospace">filterMatchingFile :: ProjectFilter -> \
Path Abs File -> IO Bool<br>filterMatchingFile (ProjectFilter filterFunc) fpath = \
do<br> let fp = toFilePath fpath<br> bs <- B.readFile fp<br> case \
validImplProject bs of -- this is pretty much just `decodeStrict`<br> Nothing \
-> pure False<br> (Just proj') -> pure $ filterFunc \
proj'</span><br></div><div><br></div><div>Here are the stats from running \
this:</div><div><br></div><div><pre \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lang-hs \
m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-prettyprint \
m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-prettyprinted"><code><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">115</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">961</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">554</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">600</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
bytes allocated </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-kwd">in</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
the heap </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">35</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">870</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">639</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">768</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
bytes copied during GC </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">56</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">467</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">720</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
bytes maximum residency </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">681</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
sample</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">s</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">))</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">1</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">283</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">008</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
bytes maximum slop </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">145</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
MB total memory </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-kwd">in</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
use </span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
MB lost due to fragmentation</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">
Tot time </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">elapsed</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
Avg pause Max pause Gen </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">108716</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
colls</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">108716</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
par </span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">76.915</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">s \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">20.571</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">s \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0.0002</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">s \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0.0266</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">s
Gen </span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">1</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">681</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
colls</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">680</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
par </span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0.530</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">s \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0.147</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">s \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0.0002</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">s \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0.0009</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">s
Parallel GC work balance</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">:</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">14.99</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">%</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">serial \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">%,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
perfect </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">100</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">%)</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">
TASKS</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">:</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">10</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">1</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
bound</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">9</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
peak workers </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">9</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
total</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">),</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
using </span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">-</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">N4</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">
SPARKS</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">:</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
converted</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
overflowed</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
dud</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
GC'd</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
fizzled</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">
INIT time </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0.001</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">s \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0.007</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">s \
elapsed</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">
MUT time </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">34.813</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">s \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">42.938</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">s \
elapsed</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">
GC time </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">77.445</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">s \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">20.718</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">s \
elapsed</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">
EXIT time </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0.000</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">s \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0.010</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">s \
elapsed</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">
Total time </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">112.260</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">s \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">63.672</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">s \
elapsed</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">
Alloc rate </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">3</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">330</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">960</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">996</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
bytes per MUT second
Productivity </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">31.0</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">%</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-kwd">of</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
total user</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">67.5</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">%</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-kwd">of</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
total elapsed
gc_alloc_block_sync</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">:</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">188614</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">
whitehole_spin</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">:</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">
gen</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">[</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">].</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">sync</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">:</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">33</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">
gen</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">[</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">1</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">].</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">sync</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">:</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_7 \
46947386685124369gmail-lit">811204</span></code></pre></div><div><br></div><div>I \
thought about writing a plainer (non-conduit) parallel version but I was afraid of \
the memory issue. I tried to write a Conduit-plus-channels version but it didn't \
work. <br></div><div><br></div><div>Finally, I wrote a version using stm-conduit, \
which I thought might be a bit more efficient. It seems to be slightly better, but \
it's not really the kind of parallelization I was \
imagining:</div><div><br></div><div><span \
style="font-family:monospace,monospace">conduitAsyncFilterFiles :: ProjectFilter \
-> Path Abs Dir -> IO [String]<br>conduitAsyncFilterFiles projFilter \
dirname' = do<br> (_, allFiles) <- listDirRecur dirname'<br> buffer \
10<br> (C.yieldMany allFiles<br> .| (C.mapMC (readFileWithPath . \
toFilePath)))<br> (C.mapC (filterProjForFilename projFilter)<br> \
.| C.filterC isJust<br> .| C.mapC fromJust<br> .| \
C.sinkList)</span><br></div><div><br></div><div>The first conduit passed to `buffer` \
does something like the following: <span \
style="font-family:monospace,monospace">parseStrict . \
B.readFile</span>.</div><div><br></div><div>This still wasn't too great, but \
after reading about handing garbage collection in smarter ways, I found that I could \
run my application like this:</div><div><pre \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lang-hs \
m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-prettyprint \
m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-prettyprinted"><code><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">stack \
exec search</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">-</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">json \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-com">-- \
--searchPath $FILES --name hello +RTS -s -A32m -n4m</span></code></pre></div><div>And \
the "productivity" would shoot up quite a lot presumably because I'm \
doing less frequent garbage collection. My program also got a bit \
faster:</div><div><br></div><div><pre \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lang-hs \
m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-prettyprint \
m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-prettyprinted"><code><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">36</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">379</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">265</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">096</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
bytes allocated </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-kwd">in</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
the heap </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">1</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">238</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">438</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">160</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
bytes copied during GC </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">22</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">996</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">264</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
bytes maximum residency </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">85</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
sample</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">s</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">))</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">3</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">834</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">152</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
bytes maximum slop </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">207</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
MB total memory </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-kwd">in</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
use </span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">14</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
MB lost due to fragmentation</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">
Tot time </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">elapsed</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
Avg pause Max pause Gen </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">211</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
colls</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">211</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
par </span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">1.433</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">s \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0.393</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">s \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0.0019</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">s \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0.0077</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">s
Gen </span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">1</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">85</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
colls</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">84</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
par </span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0.927</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">s \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0.256</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">s \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0.0030</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">s \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0.0067</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">s
Parallel GC work balance</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">:</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">67.93</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">%</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">serial \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">%,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
perfect </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">100</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">%)</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">
TASKS</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">:</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">10</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">1</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
bound</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">9</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
peak workers </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">9</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
total</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">),</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
using </span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">-</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">N4</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">
SPARKS</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">:</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
converted</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
overflowed</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
dud</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
GC'd</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
fizzled</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">
INIT time </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0.001</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">s \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0.004</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">s \
elapsed</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">
MUT time </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">12.636</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">s \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">12.697</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">s \
elapsed</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">
GC time </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">2.359</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">s \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0.650</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">s \
elapsed</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">
EXIT time </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">-</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0.015</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">s \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0.003</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">s \
elapsed</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">
Total time </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">14.982</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">s \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">(</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">13.354</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">s \
elapsed</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">)</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">
Alloc rate </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">2</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">878</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">972</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">840</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
bytes per MUT second
Productivity </span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">84.2</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">%</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-kwd">of</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
total user</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">,</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">95.1</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">%</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-kwd">of</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
total elapsed
gc_alloc_block_sync</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">:</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">9612</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">
whitehole_spin</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">:</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">
gen</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">[</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">0</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">].</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">sync</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">:</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">2044</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">
gen</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">[</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-lit">1</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">].</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln">sync</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pun">:</span><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-pln"> \
</span><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_7 \
46947386685124369gmail-lit">47704</span></code></pre></div><div><br></div><div>Thanks \
for reading thus far. I now have three questions.</div><div><br></div>1. I understand \
that my program necessarily creates tons of garbage because it parses and then throws \
away 5mb of JSON 500,000 times. However, I don't really understand why this helps \
"<code><span class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-com">+RTS \
-A32m -n4m"</span></code> and I'm always reluctant to sprinkle in magic I \
don't fully understand. Can anyone help me understand what this \
means?<br><div><br></div><div>2. It seems that the allocation limit is really \
something I should be using, but I can't figure out how to successfully add it to \
my package.yml with the other options. From the documentation for GHC 8.2, I thought \
it needed to look like this but it never works, usually telling me that -A32m and \
-n4m are not recognizable flags (how do I add them in to my package.yml so I \
don't have to pass them when running the \
program?):<br></div><div><br></div><div><span \
style="font-family:monospace,monospace">ghc-options:<br> - -threaded<br> \
- -rtsopts<br> - "-with-rtsopts=-N4 <code><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-com">-A32m \
-n4m"<br></span></code></span></div><div><br></div><div>3. Finally, the most \
important question I have is this. When I run this program on OSX, it runs \
successfully through to completion. However, <i>a few minutes after terminating</i>, \
my terminal becomes unresponsive. I use emacs for my editor, typically launched from \
a terminal window and that too becomes unresponsive. This is not a typical outcome \
for any programs I write and it happens <i>every time</i> I run this particular \
application, so I know that this application is to blame. \
<br></div><div><br></div><div>The crazy thing is that force quitting the terminal or \
logging out doesn't help: I have to actually restart my computer to use the \
terminal application again. Other details that may help: <br></div><ul><li>This \
crash happens after the process id for my program has terminated. </li><li>Watching \
its progress in HTOP, it never comes close to running out of memory: the value hovers \
in the same place.</li></ul><div>I can't really deploy an application that has \
this potential-crashing problem, but I don't know to debug this issue. My total \
stab-in-the-dark idea is that heap allocations somehow are unrecoverable even after \
the process has terminated? Can anyone offer suggestions on things to look for or \
ways to debug and/or fix this issue? </div><div><code><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail-com"></span></code></div><div><br></div></div><div>Finally, \
if anyone has suggestions on better ways to structure my application or parallelize \
the slow parts, I'll happily take those.</div><div><br></div><div>Thanks again \
for reading. I appreciate any suggestions you may \
have.</div><div><br></div><div>Best,<span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176HOEnZb"><font \
color="#888888"><br></font></span></div><span \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176HOEnZb"><font \
color="#888888"><div><br>-- <br><div \
class="m_8845810541154921453m_8418980463297255581m_6090739341647934176m_746947386685124369gmail_signature"><div \
dir="ltr">Erik Aker</div></div> </div></font></span></div>
<br></div></div>______________________________<wbr>_________________<br>
Haskell-Cafe mailing list<br>
To (un)subscribe, modify options or view archives go to:<br>
<a href="http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe" \
rel="noreferrer" target="_blank">http://mail.haskell.org/cgi-<wbr>bin/mailman/listinfo/haskell-<wbr>cafe</a><br>
Only members subscribed via the mailman list are allowed to \
post.<br></blockquote></div><br></div> </blockquote></div></div><div \
class="gmail_extra"><br><br clear="all"><br>-- <br><div \
class="m_8845810541154921453m_8418980463297255581gmail_signature" \
data-smartmail="gmail_signature"><div dir="ltr">Erik Aker</div></div> \
</div></blockquote></div> </div></div></blockquote></div><br><br clear="all"><br>-- \
<br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Erik \
Aker</div></div> </div>
[Attachment #6 (text/plain)]
_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic