[prev in list] [next in list] [prev in thread] [next in thread]
List: lxc-users
Subject: Re: [lxc-users] How properly to find what consumes memory inside the container.
From: Ivan Kurnosov <zerkms () zerkms ! ru>
Date: 2017-10-11 21:51:19
Message-ID: CABbRoj6Wtc0a-u=r96kFNivGfEQUiSt9cQ8o33xxfWB6wXP+nw () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Fajar,
yep, that is helpful. Thanks.
On 12 October 2017 at 10:42, Fajar A. Nugraha <list@fajar.net> wrote:
> On Tue, Sep 19, 2017 at 11:20 AM, Ivan Kurnosov <zerkms@zerkms.ru> wrote:
>
>>
>> But if I clear the system caches on the host
>>
>> echo 3 > /proc/sys/vm/drop_caches
>>
>>
>> the container memory consumption drops to the expected <100mb.
>>
>> So the question, how to monitor the memory consumption from the container
>> reliably?
>>
>
> As in 'know what the kernel thinks the container is using? You're doing it
> already.
>
> If you use lxcfs (which should be the case on ubuntu), /proc entries
> inside containers are fake, created using data from the relevant cgroups.
>
>
>
>> And why does `free` count caches as used memory inside container?
>>
>>
>>
>
> From https://www.kernel.org/doc/Documentation/cgroup-v2.txt:
> "
> Memory
> ------
>
> The "memory" controller regulates distribution of memory. Memory is
> stateful and implements both limit and protection models. Due to the
> intertwining between memory usage and reclaim pressure and the
> stateful nature of memory, the distribution model is relatively
> complex.
>
> While not completely water-tight, all major memory usages by a given
> cgroup are tracked so that the total memory consumption can be
> accounted and controlled to a reasonable extent. Currently, the
> following types of memory usages are tracked.
>
> - Userland memory - page cache and anonymous memory.
>
> - Kernel data structures such as dentries and inodes.
>
> - TCP socket buffers.
>
> The above list may expand in the future for better coverage.
>
> ...
> Memory Ownership
> ~~~~~~~~~~~~~~~~
>
> A memory area is charged to the cgroup which instantiated it and stays
> charged to the cgroup until the area is released. Migrating a process
> to a different cgroup doesn't move the memory usages that it
> instantiated while in the previous cgroup to the new cgroup.
>
> A memory area may be used by processes belonging to different cgroups.
> To which cgroup the area will be charged is in-deterministic; however,
> over time, the memory area is likely to end up in a cgroup which has
> enough memory allowance to avoid high reclaim pressure.
>
> If a cgroup sweeps a considerable amount of memory which is expected
> to be accessed repeatedly by other cgroups, it may make sense to use
> POSIX_FADV_DONTNEED to relinquish the ownership of memory areas
> belonging to the affected files to ensure correct memory ownership.
> "
>
>
>
> An interesting question would be "can you exclude page cache from cgroup
> memory accounting". I don't think that's possible though.
>
> --
> Fajar
>
> _______________________________________________
> lxc-users mailing list
> lxc-users@lists.linuxcontainers.org
> http://lists.linuxcontainers.org/listinfo/lxc-users
>
--
With best regards, Ivan Kurnosov
[Attachment #5 (text/html)]
<div dir="ltr">Fajar,<div><br></div><div>yep, that is helpful. \
Thanks.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On 12 \
October 2017 at 10:42, Fajar A. Nugraha <span dir="ltr"><<a \
href="mailto:list@fajar.net" target="_blank">list@fajar.net</a>></span> \
wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px \
#ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div \
class="gmail_quote"><span class="">On Tue, Sep 19, 2017 at 11:20 AM, Ivan Kurnosov \
<span dir="ltr"><<a href="mailto:zerkms@zerkms.ru" \
target="_blank">zerkms@zerkms.ru</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><br></div><div>But if I \
clear the system caches on the host</div><div><br></div></div><blockquote \
style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div>echo 3 > \
/proc/sys/vm/drop_caches </div></div></blockquote><div><div><br></div><div>the \
container memory consumption drops to the expected \
<100mb.</div><div><br></div><div>So the question, how to monitor the memory \
consumption from the container \
reliably?</div></div></div></blockquote><div><br></div></span><div>As in 'know \
what the kernel thinks the container is using? You're doing it \
already.</div><div><br></div><div>If you use lxcfs (which should be the case on \
ubuntu), /proc entries inside containers are fake, created using data from the \
relevant cgroups.</div><span class=""><div><br></div><div> </div><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div> And why does `free` \
count caches as used memory inside container?</div><span \
class="m_169603496233069706gmail-HOEnZb"><font \
color="#888888"><div><br></div><div><br></div></font></span></div></div></blockquote><div><br></div><div><br></div></span><div>From \
<a href="https://www.kernel.org/doc/Documentation/cgroup-v2.txt" \
target="_blank">https://www.kernel.org/<wbr>doc/Documentation/cgroup-v2.<wbr>txt</a>:</div><div>"</div><div>Memory</div><div>------</div><div><br></div><div>The \
"memory" controller regulates distribution of memory. Memory \
is</div><div>stateful and implements both limit and protection models. Due to \
the</div><div>intertwining between memory usage and reclaim pressure and \
the</div><div>stateful nature of memory, the distribution model is \
relatively</div><div>complex.</div><div><br></div><div>While not completely \
water-tight, all major memory usages by a given</div><div>cgroup are tracked so that \
the total memory consumption can be</div><div>accounted and controlled to a \
reasonable extent. Currently, the</div><div>following types of memory usages are \
tracked.</div><div><br></div><div>- Userland memory - page cache and anonymous \
memory.</div><div><br></div><div>- Kernel data structures such as dentries and \
inodes.</div><div><br></div><div>- TCP socket buffers.</div><div><br></div><div>The \
above list may expand in the future for better \
coverage.</div><div><br></div><div>...</div><div><div>Memory \
Ownership</div><div>~~~~~~~~~~~~~~~~</div><div><br></div><div>A memory area is \
charged to the cgroup which instantiated it and stays</div><div>charged to the cgroup \
until the area is released. Migrating a process</div><div>to a different cgroup \
doesn't move the memory usages that it</div><div>instantiated while in the \
previous cgroup to the new cgroup.</div><div><br></div><div>A memory area may be used \
by processes belonging to different cgroups.</div><div>To which cgroup the area will \
be charged is in-deterministic; however,</div><div>over time, the memory area is \
likely to end up in a cgroup which has</div><div>enough memory allowance to avoid \
high reclaim pressure.</div><div><br></div><div>If a cgroup sweeps a considerable \
amount of memory which is expected</div><div>to be accessed repeatedly by other \
cgroups, it may make sense to use</div><div>POSIX_FADV_DONTNEED to relinquish the \
ownership of memory areas</div><div>belonging to the affected files to ensure correct \
memory ownership.</div></div><div>"</div><div><br></div><div><br></div><div><br></div><div>An \
interesting question would be "can you exclude page cache from cgroup memory \
accounting". I don't think that's possible though.</div><span \
class="HOEnZb"><font color="#888888"><div><br></div><div>-- \
</div><div>Fajar</div></font></span></div></div></div> \
<br>______________________________<wbr>_________________<br> lxc-users mailing \
list<br> <a href="mailto:lxc-users@lists.linuxcontainers.org">lxc-users@lists.<wbr>linuxcontainers.org</a><br>
<a href="http://lists.linuxcontainers.org/listinfo/lxc-users" rel="noreferrer" \
target="_blank">http://lists.linuxcontainers.<wbr>org/listinfo/lxc-users</a><br></blockquote></div><br><br \
clear="all"><div><br></div>-- <br><div class="gmail_signature" \
data-smartmail="gmail_signature">With best regards, Ivan Kurnosov</div> </div>
[Attachment #6 (text/plain)]
_______________________________________________
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic