[prev in list] [next in list] [prev in thread] [next in thread]
List: markus-dev
Subject: Re: pdfs and MarkUs
From: Byron Weber Becker <bwbecker () uwaterloo ! ca>
Date: 2012-02-10 14:25:00
Message-ID: CCA7CC69-FA1F-4953-BAB9-061AEC8F3828 () uwaterloo ! ca
[Download RAW message or body]
Nicolas -- I didn't run the test personally, but that's my understanding
of the results. That allocating more memory did not improve the time
to collect and convert the files.
Carrie or Paul, anything else to add?
Byron
On 2012-02-10, at 5:46 AM, nicolas.bouillon@eleves.ec-nantes.fr wrote:
> Hi Byron,
>
> I work, with Camille, on pdf conversion here in Nantes, and I have a question \
> concerning the tests you had.
> Just to be sure, what you say is that using 100 M or 1G memory limit gives the same \
> result ? In both cases, 7 minutes to collect the files ?
>
> So, increasing the memory limit is not a solution, am I right ?
>
> Best regards
>
>
> ----- Original Message -----
> From: "Byron Weber Becker" <bwbecker@uwaterloo.ca>
> To: "Severin Gehwolf" <jerboaa@gmail.com>
> Cc: "Benjamin Vialle" <benjaminvialle@gmail.com>, "markus-dev" \
> <markus-dev@cs.toronto.edu>, "Paul Kates" <pkates@math.uwaterloo.ca>, "carrie \
> howells" <carrie.howells@uwaterloo.ca>
> Sent: Tuesday, January 31, 2012 11:37:49 PM
> Subject: Re: pdfs and MarkUs
>
> Some of our folks working on this project rans some benchmarks. They say:
>
> > I've run the same test twice now, once with the ImageMagick memory limit set to \
> > the default of 100M and once since you upped the limit to 1000M.
> > I had 10 different people submit the same 11 page pdf, which is 74KB. And in both \
> > tests, it took MarkUs 7 minutes to collect/convert all of the submissions. \
> > (Extrapolate to 1500 MATH 135/137 students and this conversion would take ~18 \
> > hours.) :(
>
> Byron
>
>
> On 2012-01-30, at 12:51 PM, Severin Gehwolf wrote:
>
> > Hi,
> >
> > IIRC MarkUs forks in order to do conversions. However, it does this
> > only to keep the UI responsive and have the conversion happening in
> > the background (e.g. over night). I think it was designed so that only
> > one conversion runs at a time (it uses a queue and some attributes in
> > the database in order to check if a conversion finished, if so kicks
> > off the next). That said, I *highly* recommend to enforce a limit on
> > ImageMagick. It'll happily eat up all available memory otherwise,
> > which will result in the system crashing. This happening isn't *that*
> > rare with no enforced memory limit. Feel free to play with values
> > other than 100Mb, although I suggest to do that on a test system
> > rather than a production instance.
> >
> > We'd love to hear back which config worked reasonably well for your
> > setup. Note that the (rather conservative) value of 100Mb has been
> > picked as default so as to not trash setups which have PDFs enabled
> > and sys admins didn't bother looking at the memory value. Better play
> > safe was the idea :)
> >
> > My $0.02
> >
> > --Severin
> >
> > On Mon, Jan 30, 2012 at 11:43 AM, Benjamin Vialle
> > <benjaminvialle@gmail.com> wrote:
> > > Hi Byron,
> > >
> > > We have two students here in Centrale Nantes (France) working on PDF side of
> > > MarkUs.
> > >
> > > They are working on the memory consumption of ImageMagick.
> > >
> > > All the best,
> > >
> > >
> > > 2012/1/30 Byron Weber Becker <bwbecker@uwaterloo.ca>
> > > >
> > > > Hi, all --
> > > >
> > > > We've had some folks at UW investigating the PDF side of MarkUs. They
> > > > think
> > > > increasing the memory ImageMagick is allowed to have would increase the
> > > > performance.
> > > > See notes below. I'm wondering if the MarkUs team has done any tuning on
> > > > these
> > > > parameters; any reason 100M was chosen?
> > > >
> > > > One of my concerns is whether the conversion happens one at a time or
> > > > several
> > > > in parallel. If in parallel, we probably don't want to give as much
> > > > memory
> > > > as Paul proposed below (1/2 of physical RAM).
> > > >
> > > > Thanks,
> > > > Byron
> > > >
> > > > On 2012-01-30, at 9:27 AM, Carrie Howells wrote:
> > > >
> > > > > Hi Byron.
> > > > >
> > > > > Paul Kates has been running some experiments with ImageMagick and has
> > > > > found some settings that may improve the performance of MarkUs converting
> > > > > pdfs to images. Is it possible to change a couple MarkUs parameters on your
> > > > > test server to see if performance improves? (Paul's notes about which two
> > > > > parameters to tweak are below.)
> > > > >
> > > > >
> > > > > *******
> > > > > Carrie Howells, Instructional Support Coordinator
> > > > > Mathematics Faculty Computing Facility
> > > > > University of Waterloo
> > > > > MC 3054, 519-888-4567 x36272
> > > > >
> > > > >
> > > > > --------------------------------------------------------------
> > > > > Configuation of Markus source:
> > > > >
> > > > > The 100 MB memory limit for ImageMagick is mentioned in Markus source
> > > > > file
> > > > > config/environments/production.rb
> > > > >
> > > > > See line: PDF_CONV_MEMORY_ALLOWANCE = 100
> > > > >
> > > > > The ImageMagick command conversion call line is in Markus source file
> > > > > app/models/submission_file.rb
> > > > >
> > > > > See line:
> > > > > `convert -limit memory
> > > > > #{MarkusConfigurator.markus_config_pdf_conv_memory_allowance} -limit map 0
> > > > > -density 150 -resize 66% #{File.join(storage_path, self.filename)} -append
> > > > > #{file_path} >> #{File.join(RAILS_ROOT, "log", "export-pdf.log")}`
> > > > >
> > > > > For a trial, either change the memory limit from
> > > > > 100 to 4000 (e.g. 100 MB to 4 GB; i.e. pick a value that is half
> > > > > the available RAM memory in the machine), or change the "convert" call
> > > > > by
> > > > > dropping the -limit memory option value
> > > > > i.e.
> > > > > `convert -limit map 0 -density 150 -resize 66%
> > > > > #{File.join(storage_path, self.filename)} -append #{file_path} >>
> > > > > #{File.join(RAILS_ROOT, "log", "export-pdf.log")}`
> > > > >
> > > > > Paul
> > > > >
> > > >
> > > > ---------------------------------------------------------
> > > > Byron Weber Becker Voice: 519-888-4567 x34661
> > > > School of Computer Science Fax: 519-885-1208
> > > > University of Waterloo Office: DC3105
> > > > Waterloo, ON N2L 3G1
> > > >
> > > > Advising FAQ: http://www.cs.uwaterloo.ca/current/faq/index.shtml
> > > >
> > >
> > >
> > >
> > > --
> > > Benjamin
> >
>
> ---------------------------------------------------------
> Byron Weber Becker Voice: 519-888-4567 x34661
> School of Computer Science Fax: 519-885-1208
> University of Waterloo Office: DC3105
> Waterloo, ON N2L 3G1
>
> Advising FAQ: http://www.cs.uwaterloo.ca/current/faq/index.shtml
>
>
---------------------------------------------------------
Byron Weber Becker Voice: 519-888-4567 x34661
School of Computer Science Fax: 519-885-1208
University of Waterloo Office: DC3105
Waterloo, ON N2L 3G1
Advising FAQ: http://www.cs.uwaterloo.ca/current/faq/index.shtml
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic