[prev in list] [next in list] [prev in thread] [next in thread] 

List:       systemd-devel
Subject:    Re: [systemd-devel] protecting sshd against forkbombs, excessive memory usage by other processes
From:       Mantas_Mikulėnas <grawity () gmail ! com>
Date:       2020-08-12 13:07:18
Message-ID: CAPWNY8WrKnAW2OMhxa1zGdyOG9L1hOaonGtnGt=m29ferQKTXg () mail ! gmail ! com
[Download RAW message or body]

[Attachment #2 (multipart/alternative)]


On Wed, Aug 12, 2020 at 7:03 AM Tomasz Chmielewski <mangoo@wpkg.org> wrote:

> I've made a mistake and have executed a forkbomb-like task. Almost
> immediately, the system became unresponsive, ssh session froze or were
> very slow to output even single characters; some ssh sessions timed out
> and were disconnected.
>
> It was not possible to connect a new ssh session to interrupt the
> runaway task - new connection attempt were simply timing out.
>
> SSH is the only way to access the server. Eventually, after some 30
> mins, the system "unfroze" - but - I wonder - can systemd help sysadmins
> getting out of such situations?
>
> I realize it's a bit tricky, as there are two cases here:
>
> 1) misbehaving program is a child process of sshd (i.e. user logged in
> and executed a forkbomb)
>

I don't think "child process of sshd" is the useful part, as logged-in user
processes are actually moved to a separate cgroup for the session – so yes,
they're sshd children, but they actually have resource limits fully
separate from the main sshd daemon process.

Which means that with systemd, each user already has their own limit on the
number of processes/tasks (the default in user-.slice.d is TasksMax=33%
of...something, but it could be lowered to e.g. 10% or to 4096) without
affecting the service itself.

So I'm sure that sshd.service and user-0.slice could be tweaked somehow to
give root a higher priority at cgroup level, but that depends on what your
system actually ran out of...

-- 
Mantas Mikulėnas

[Attachment #5 (text/html)]

<div dir="ltr"><div dir="ltr">On Wed, Aug 12, 2020 at 7:03 AM Tomasz Chmielewski \
&lt;<a href="mailto:mangoo@wpkg.org">mangoo@wpkg.org</a>&gt; wrote:<br></div><div \
class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I&#39;ve made a \
mistake and have executed a forkbomb-like task. Almost <br> immediately, the system \
became unresponsive, ssh session froze or were <br> very slow to output even single \
characters; some ssh sessions timed out <br> and were disconnected.<br>
<br>
It was not possible to connect a new ssh session to interrupt the <br>
runaway task - new connection attempt were simply timing out.<br>
<br>
SSH is the only way to access the server. Eventually, after some 30 <br>
mins, the system &quot;unfroze&quot; - but - I wonder - can systemd help sysadmins \
<br> getting out of such situations?<br>
<br>
I realize it&#39;s a bit tricky, as there are two cases here:<br>
<br>
1) misbehaving program is a child process of sshd (i.e. user logged in <br>
and executed a forkbomb)<br></blockquote><div><br></div><div>I don&#39;t think \
&quot;child process of sshd&quot; is the useful part, as logged-in user processes are \
actually moved to a separate cgroup for the session – so yes, they&#39;re sshd \
children, but they actually have resource limits fully separate from the main sshd \
daemon process.</div><div><br></div><div>Which means that with systemd, each user \
already has their own limit on the number of processes/tasks (the default in \
user-.slice.d is TasksMax=33% of...something, but it could be lowered to e.g. 10% or \
to 4096) without affecting the service itself.</div><div><br></div><div>So I&#39;m \
sure that sshd.service and user-0.slice could be tweaked somehow to give root a \
higher priority at cgroup level, but that depends on what your system actually ran \
out of...</div></div><div><br></div>-- <br><div dir="ltr" \
class="gmail_signature"><div dir="ltr">Mantas Mikulėnas</div></div></div>



_______________________________________________
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel


[prev in list] [next in list] [prev in thread] [next in thread]