[prev in list] [next in list] [prev in thread] [next in thread]
List: qemu-discuss
Subject: Re: [Qemu-discuss] apt-get fsync in VM (Was: Disk Performance)
From: Quentin Hartman <qhartman () gmail ! com>
Date: 2014-06-09 17:32:34
Message-ID: CAA1jYCPwsqwOmykH05C08AdPWe1YM-RwLTtQE4wXwN23W+4j0Q () mail ! gmail ! com
[Download RAW message or body]
Just to follow up on this for others. I ended up needing to change the
ceph file system backend to xfs (from btrfs) and enable the
"network=writeback" filesystem cache setting in my qemu config to get
apt-get to behave well in VMs.
QH
On Fri, Jun 6, 2014 at 2:07 PM, Quentin Hartman <qhartman@gmail.com> wrote:
> So! I've corrected the configuration of my Ceph cluster and am now getting
> about 70MB/s writes and 85MB/s reads inside VM's backed by rbd volumes.
> Generally performance of these machines "feels" quite good, which supports
> the consistent results I'm getting from iozone, dbench, and dd [1].
> However, specifically when installing packages using apt-get it's still
> taking forever, often causing chef-client timeouts and all sorts of other
> pain. This does not seem to happen with instances that are booting from
> locally-stored images. Presumably this is due to apt-get's somewhat
> notorious fsync behavior [2].
>
> Have any of you seen this behavior in Ubuntu or Debian based VMs? If so,
> have you discovered a way to overcome it?
>
> Thanks!
>
> QH
>
> [1] - I got criticized previously for using dd for benchmarking, and I
> admit it is not the most in-depth test, but the results I got from it have
> been consistent with what I got from "better" tools, so I think it is
> probably fine if you just need something quick to see which way the wind is
> blowing.
> [2] - google "apt-get fsync" for lots of talk on the topic if you're not
> familiar. especially
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=588254
>
> On Fri, May 30, 2014 at 11:13 AM, Stephan von Krawczynski <
> skraw@ithnet.com> wrote:
>
>> On Fri, 30 May 2014 16:53:50 +0000
>> dw+qemu-discuss@hmmz.org wrote:
>>
>> > On Fri, May 30, 2014 at 06:47:15PM +0200, Stephan von Krawczynski wrote:
>> >
>> > > > I'm no expert on this, but it sounds to me like the file copy is
>> using
>> > > > memory to cache disk blocks, swapping out the memory that is the
>> > > > emulated memory of the guests. It seems to me that the solution to
>> > > > that is to force the Qemu process's memory space to remain in RAM.
>> > > >
>> > > > Is there a way to do that in Linux?
>> > >
>> > > No.
>> >
>> > Unrelated to solving the original problem, just wanted to point out this
>> > is in fact trivially possible, and there have been patches in the past
>> > to implement it.
>> >
>> > See http://lists.gnu.org/archive/html/qemu-devel/2012-09/msg05081.html
>> > and http://linux.die.net/man/2/mlockall
>> >
>> >
>> > David
>>
>> Well, I guess you mean "-realtime mlock=on" option. It did not help me.
>>
>> Regards,
>> Stephan
>>
>>
>>
>>
>
[Attachment #3 (text/html)]
<div dir="ltr">Just to follow up on this for others. I ended up needing to change \
the ceph file system backend to xfs (from btrfs) and enable the \
"network=writeback" filesystem cache setting in my qemu config to get \
apt-get to behave well in VMs.<div>
<br></div><div>QH</div></div><div class="gmail_extra"><br><br><div \
class="gmail_quote">On Fri, Jun 6, 2014 at 2:07 PM, Quentin Hartman <span \
dir="ltr"><<a href="mailto:qhartman@gmail.com" \
target="_blank">qhartman@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr">So! I've corrected the configuration of my \
Ceph cluster and am now getting about 70MB/s writes and 85MB/s reads inside VM's \
backed by rbd volumes. Generally performance of these machines "feels" \
quite good, which supports the consistent results I'm getting from iozone, \
dbench, and dd [1]. However, specifically when installing packages using apt-get \
it's still taking forever, often causing chef-client timeouts and all sorts of \
other pain. This does not seem to happen with instances that are booting from \
locally-stored images. Presumably this is due to apt-get's somewhat notorious \
fsync behavior [2].<div>
<br></div><div>Have any of you seen this behavior in Ubuntu or Debian based VMs? If \
so, have you discovered a way to overcome it?</div><div \
class="gmail_extra"><br></div><div class="gmail_extra">Thanks!</div><div \
class="gmail_extra">
<br></div><div class="gmail_extra">QH</div><div class="gmail_extra"><br></div><div \
class="gmail_extra">[1] - I got criticized previously for using dd for benchmarking, \
and I admit it is not the most in-depth test, but the results I got from it have been \
consistent with what I got from "better" tools, so I think it is probably \
fine if you just need something quick to see which way the wind is blowing.</div>
<div class="gmail_extra">[2] - google "apt-get fsync" for lots of talk on \
the topic if you're not familiar. especially <a \
href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=588254" \
target="_blank">https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=588254</a><br>
<br><div class="gmail_quote">On Fri, May 30, 2014 at 11:13 AM, Stephan von \
Krawczynski <span dir="ltr"><<a href="mailto:skraw@ithnet.com" \
target="_blank">skraw@ithnet.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div><div>On Fri, 30 May 2014 16:53:50 +0000<br>
<a href="mailto:dw%2Bqemu-discuss@hmmz.org" \
target="_blank">dw+qemu-discuss@hmmz.org</a> wrote:<br> <br>
> On Fri, May 30, 2014 at 06:47:15PM +0200, Stephan von Krawczynski wrote:<br>
><br>
> > > I'm no expert on this, but it sounds to me like the file copy is \
using<br> > > > memory to cache disk blocks, swapping out the memory that is \
the<br> > > > emulated memory of the guests. It seems to me that the \
solution to<br> > > > that is to force the Qemu process's memory space \
to remain in RAM.<br> > > ><br>
> > > Is there a way to do that in Linux?<br>
> ><br>
> > No.<br>
><br>
> Unrelated to solving the original problem, just wanted to point out this<br>
> is in fact trivially possible, and there have been patches in the past<br>
> to implement it.<br>
><br>
> See <a href="http://lists.gnu.org/archive/html/qemu-devel/2012-09/msg05081.html" \
target="_blank">http://lists.gnu.org/archive/html/qemu-devel/2012-09/msg05081.html</a><br>
> and <a href="http://linux.die.net/man/2/mlockall" \
target="_blank">http://linux.die.net/man/2/mlockall</a><br> ><br>
><br>
> David<br>
<br>
</div></div>Well, I guess you mean "-realtime mlock=on" option. It did not \
help me.<br> <br>
Regards,<br>
Stephan<br>
<br>
<br>
<br>
</blockquote></div><br></div></div>
</blockquote></div><br></div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic