[prev in list] [next in list] [prev in thread] [next in thread]
List: lxc-users
Subject: Re: [lxc-users] LXD 2.14 - Ubuntu 16.04 - kernel 4.4.0-57-generic - SWAP continuing to grow
From: "Fajar A. Nugraha" <list () fajar ! net>
Date: 2017-07-16 4:30:29
Message-ID: CAG1y0scoA048QSERSt0kGpgP0pOePEheq-VLFUU3jji+yRoVSw () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
On Sat, Jul 15, 2017 at 10:48 PM, Ron Kelley <rkelleyrtp@gmail.com> wrote:
> Thanks for the great replies.
>
> Marat/Fajar: How many servers do you guys have running in production, and
> what are their characteristics (RAM, CPU, workloads, etc)?
My biggest production one was AWS r4.16xlarge (almost 500GB memory), though
nowadays I mostly use r3.8xlarge (around half the memory, but more
cost-efficient). It's running many things from small web servers to large
hadoop instances.
My smallest production system had 16GB RAM. After some struggles (including
testing if swap would help, which did not), in the end I increase its RAM
to 128GB. MUCH better.
> I am trying to see if our systems generally align to what you are
> running. Running without swap seems rather drastic and removes the "safety
> net" in the case of a bad program.
Which is why I mentioned setting the limits beforehand.
For a memory-limited system, you should be able to follow
http://digitaloceanvps.blogspot.co.id/2014/04/best-configuration-for-512mb-1gb-ram.html
or similar. Though since you said you use nginx, instead of setting appache
you should be able to just set php-fpm to use on-demand process manager
with a small (2-4) max process.
> In the end, we must have all containers/processes running 24/7.
>
Which is EXACTLY why I disable swap. I do NOT want a misconfigured
container dragging others down. And when you configure the applications
correctly, each container should stay within its limited memory. Things
like 'sudden spike in user access' would slow it down (e.g. due to waiting
for php-fpm process becomes available) but it would not create a spike in
memory usage.
> tldr;
> ----
> After digging into this a bit, it seems "top", "top", and "free" report
> similar swap usage, however, other tools report much less swap usage. I
> found the following threads on the ‘net which include simple scripts to
> look in /proc and examine swap usage per process:
>
> https://stackoverflow.com/questions/479953/how-to-find-
> out-which-processes-are-swapping-in-linux
> https://www.cyberciti.biz/faq/linux-which-process-is-using-swap
>
> As some people pointed out, top/htop don't accurately report the swap
> usage as they combine a number of memory fields together. And, indeed,
> running the script in each container (looking at /proc) show markedly
> different results when all the numbers are added up. For example, the
> "free" command on one of our servers reports 3G of swap in use, but the
> script that scans the /proc directory only shows 1.3G of real swap in use.
> Very odd.
>
> All that said, the real issue is to find out if one of our
> containers/processes has a memory leak (per Marat's suggestion below).
> Unfortunately, LXD does not provide an easy way to track per-container
> stats, thus we must "roll our own" tools.
>
>
/proc/meminfo (and some other files) in the container is fake. It's created
by lxcfs, using numbers from cgroup.
cgroup would generally provide accurate info (e.g. 'how much memory is used
by processes under this cgroup'). If you're rolling your own tools, read
cgroup files directly (e.g. /sys/fs/cgroup/memory/lxc/ ...).
In any case, if your tools show memory usage in a container higher than its
configured limit, then its perfectly normal that it starts to swap. Even
when the host itself still have lots of memory.
--
Fajar
[Attachment #5 (text/html)]
<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Sat, Jul 15, 2017 \
at 10:48 PM, Ron Kelley <span dir="ltr"><<a href="mailto:rkelleyrtp@gmail.com" \
target="_blank">rkelleyrtp@gmail.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex">Thanks for the great replies.<br> <br>
Marat/Fajar: How many servers do you guys have running in production, and what are \
their characteristics (RAM, CPU, workloads, etc)? \
</blockquote><div><br></div><div>My biggest production one was AWS r4.16xlarge \
(almost 500GB memory), though nowadays I mostly use r3.8xlarge (around half the \
memory, but more cost-efficient). It's running many things from small web servers \
to large hadoop instances.</div><div><br></div><div>My smallest production system had \
16GB RAM. After some struggles (including testing if swap would help, which did not), \
in the end I increase its RAM to 128GB. MUCH better.</div><div> </div><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex">I am trying to see if our systems generally align \
to what you are running. Running without swap seems rather drastic and removes the \
"safety net" in the case of a bad program. </blockquote><div><br></div><div>Which \
is why I mentioned setting the limits beforehand.</div><div><br></div><div>For a \
memory-limited system, you should be able to follow <a \
href="http://digitaloceanvps.blogspot.co.id/2014/04/best-configuration-for-512mb-1gb-r \
am.html">http://digitaloceanvps.blogspot.co.id/2014/04/best-configuration-for-512mb-1gb-ram.html</a> \
or similar. Though since you said you use nginx, instead of setting appache you \
should be able to just set php-fpm to use on-demand process manager with a small \
(2-4) max process.</div><div> </div><blockquote class="gmail_quote" \
style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex">In the end, we must have all containers/processes \
running 24/7.<br></blockquote><div><br></div><div>Which is EXACTLY why I disable \
swap. I do NOT want a misconfigured container dragging others down. And when you \
configure the applications correctly, each container should stay within its limited \
memory. Things like 'sudden spike in user access' would slow it down (e.g. \
due to waiting for php-fpm process becomes available) but it would not create a spike \
in memory usage.</div><div><br></div><blockquote class="gmail_quote" \
style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"><br> tldr;<br>
----<br>
After digging into this a bit, it seems "top", "top", and "free" report similar swap \
usage, however, other tools report much less swap usage. I found the following \
threads on the ‘net which include simple scripts to look in /proc and examine swap \
usage per process:<br> <br>
<a href="https://stackoverflow.com/questions/479953/how-to-find-out-which-processes-are-swapping-in-linux" \
rel="noreferrer" target="_blank">https://stackoverflow.com/<wbr>questions/479953/how-to-find-<wbr>out-which-processes-are-<wbr>swapping-in-linux</a><br>
<a href="https://www.cyberciti.biz/faq/linux-which-process-is-using-swap" \
rel="noreferrer" target="_blank">https://www.cyberciti.biz/faq/<wbr>linux-which-process-is-using-<wbr>swap</a><br>
<br>
As some people pointed out, top/htop don't accurately report the swap usage as they \
combine a number of memory fields together. And, indeed, running the script in each \
container (looking at /proc) show markedly different results when all the numbers are \
added up. For example, the "free" command on one of our servers reports 3G of swap \
in use, but the script that scans the /proc directory only shows 1.3G of real swap in \
use. Very odd.<br> <br>
All that said, the real issue is to find out if one of our containers/processes has a \
memory leak (per Marat's suggestion below). Unfortunately, LXD does not provide an \
easy way to track per-container stats, thus we must "roll our own" tools.<br> <span \
class="gmail-HOEnZb"><font \
color="#888888"><br></font></span></blockquote><div><br></div><div>/proc/meminfo (and \
some other files) in the container is fake. It's created by lxcfs, using numbers \
from cgroup.</div><div>cgroup would generally provide accurate info (e.g. 'how \
much memory is used by processes under this cgroup'). If you're rolling your \
own tools, read cgroup files directly (e.g. /sys/fs/cgroup/memory/lxc/ \
...).</div><div><br></div><div>In any case, if your tools show memory usage in a \
container higher than its configured limit, then its perfectly normal that it starts \
to swap. Even when the host itself still have lots of \
memory.</div><div><br></div><div>-- </div><div>Fajar</div></div></div></div>
[Attachment #6 (text/plain)]
_______________________________________________
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic