[prev in list] [next in list] [prev in thread] [next in thread] 

List:       mesos-dev
Subject:    Re: how is the agent available memory computed/updated?
From:       Charles-François_Natali <cf.natali () gmail ! com>
Date:       2020-05-03 10:42:27
Message-ID: CAH_1eM0dki770+GsCOQdT6cNVNxQ8ZEubRnnThkTGHrVKGBtCQ () mail ! gmail ! com
[Download RAW message or body]

Thanks!

Le ven. 1 mai 2020 =C3=A0 00:11, Vinod Kone <vinodkone@apache.org> a =C3=A9=
crit :
>
> I commented on the JIRA.
>
> On Thu, Apr 30, 2020 at 3:02 PM Charles-Fran=C3=A7ois Natali <cf.natali@g=
mail.com>
> wrote:
>
> > Thanks Vinod.
> >
> > Yes, I understand that Mesos assumes it's the only process managing
> > resources, makes sense.
> > Looking at the code and testing shows the agent reports as available
> > memory the total memory of the host, minus 1GB (or half the total
> > memory if the total memory is below 2GB)
> > (
> > https://github.com/apache/mesos/blob/master/src/slave/containerizer/con=
tainerizer.cpp#L152
> > ).
> > So basically it means that if assumes that the OS doesn't use more
> > than 1GB. I guess if it's not the case one can just specify the memory
> > manually to the agent, so that's fine.
> >
> > Actually the reason I was wondering about this is because we recently
> > had a problem where containers couldn't be destroyed because of tasks
> > stuck in uninterruptible (D) state, which caused the memory to be
> > basically leaked, i.e. the agent was advertising the memory free while
> > it was still being used by the stuck processes. We ran into a similar
> > issue with GPUs - it's a known issue
> > https://issues.apache.org/jira/browse/MESOS-8038 - I posted an
> > analysis and potential fix, it'd be great if someone could have a look
> > :).
> >
> > Cheers,
> >
> > Charles
> >
> > Le jeu. 30 avr. 2020 =C3=A0 15:36, Vinod Kone <vinodkone@apache.org> a =
=C3=A9crit :
> > >
> > > Mesos assumes that it is the only process managing resources of a box
> > (cpu,
> > > mem, disk). So if you have out of band processes using up resources i=
t
> > > won't be reflected in the resource offers and the box can be
> > overcommitted.
> > > There is no runtime periodic check of available resources, it's only
> > > calculated once at startup.
> > >
> > > Resource detection logic is here:
> > >
> > https://github.com/apache/mesos/blob/master/src/slave/containerizer/con=
tainerizer.cpp#L65
> > >
> > > On Thu, Apr 30, 2020 at 8:17 AM Charles-Fran=C3=A7ois Natali <
> > cf.natali@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > Could someone point me to some code/documentation explaining how th=
e
> > > > agent available memory is computed, and when it is refreshed?
> > > >
> > > > For example, if I have an agent started, with some outstanding offe=
rs,
> > > > and I then start a process - not as a task managed by Mesos, but as=
 an
> > > > external process which just allocates a lot of memory - and touches
> > > > it, not just committed - I can see the machine available memory go
> > > > down (as reported by free, and MemAvailable in /proc/meminfo), but =
the
> > > > agent doesn't rescind any offer, and never seems to actually refres=
h
> > > > it - event after starting/stopping tasks.
> > > >
> > > > Cheers,
> > > >
> > > > Charles
> > > >
> >
[prev in list] [next in list] [prev in thread] [next in thread]