[prev in list] [next in list] [prev in thread] [next in thread]
List: gentoo-portage-dev
Subject: Re: [gentoo-portage-dev] Performance tuning and parallelisation
From: Ed W <lists () wildgooses ! com>
Date: 2021-08-31 8:00:14
Message-ID: a6c07377-3748-da61-cd82-7e27193934d1 () wildgooses ! com
[Download RAW message or body]
On 26/08/2021 17:38, Marco Sirabella wrote:
> -
>
> Hi Ed,
>
> I've taken a dabble at trying to track down portage's bottlenecks (and have stopped for the time
> being at solving them :/ )
>
> Can anyone give me a leg up on how I could benchmark this further and look for the hotspot?
> Perhaps someone understand the architecture of this point more intimately and could point at
> whether there are opportunities to do some of the processing on mass, rather than per file?
>
> From my notes at the timem, it looks like yappi <https://pypi.org/project/yappi/> worked a bit
> better than python's built in cProfile for me because it properly dove into async calls. I used
> snakeviz <https://jiffyclub.github.io/snakeviz/> for visualizing the profile results.
>
> I was taking a look at depclean, but I found similarly that a lot of duplicate process was being
> done due to encapsulated abstractions not being able to communicate that the same thing was being
> done multiple times eg removing each package processes a massive json structure for each package
> removed, although I opted to work on the more-understandable unicode conversions.
>
> My stalled progress can be found here: #700 <https://github.com/gentoo/portage/pull/700>. Lost the
> drive to continue for now unfortunately :<
>
> Good luck! Looking forward to your optimizations
>
> – Marco Sirabella
>
Hi All, thanks for the replies. Wow, Marco, that patch touches a lot of stuff...!
OK, I will start by trying to get the profilers going and work from there...
(Alec, to avoid replying separately: Thanks for your notes. Yes, I am not clear which of the
install/merge phases specifically is the culprit, but it feels like something in that area is
"slow", especially when run under qemu user mode. I think using qmerge won't work for my build
script, but great idea to use it for benchmarking to narrow things down - thanks)
Thanks all
Ed W
[Attachment #3 (text/html)]
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<div class="moz-cite-prefix">On 26/08/2021 17:38, Marco Sirabella
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:20210826163815.vr7jtxkwuqz37wyz@Ridl3y">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<meta charset="utf-8">
<meta name="generator" content="pandoc">
<meta name="viewport" content="width=device-width,
initial-scale=1.0, user-scalable=yes">
<title>-</title>
<style>code{white-space: pre-wrap;}span.smallcaps{font-variant: \
small-caps;}span.underline{text-decoration: underline;}div.column{display: \
inline-block; vertical-align: top; width: 50%;}div.hanging-indent{margin-left: 1.5em; \
text-indent: -1.5em;}ul.task-list{list-style: none;}</style> <!--[if lt IE 9]>
<script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.3/html5shiv-printshiv.min.js"></script>
<![endif]-->
<p>Hi Ed,</p>
<p>I've taken a dabble at trying to track down portage's
bottlenecks (and have stopped for the time being at solving them
:/ )</p>
<blockquote>
<p>Can anyone give me a leg up on how I could benchmark this
further and look for the hotspot? Perhaps someone understand
the architecture of this point more intimately and could point
at whether there are opportunities to do some of the
processing on mass, rather than per file?</p>
</blockquote>
<p>From my notes at the timem, it looks like <a
href="https://pypi.org/project/yappi/" moz-do-not-send="true">yappi</a>
worked a bit better than python's built in cProfile for me
because it properly dove into async calls. I used <a
href="https://jiffyclub.github.io/snakeviz/"
moz-do-not-send="true">snakeviz</a> for visualizing the
profile results.</p>
<p>I was taking a look at depclean, but I found similarly that a
lot of duplicate process was being done due to encapsulated
abstractions not being able to communicate that the same thing
was being done multiple times eg removing each package processes
a massive json structure for each package removed, although I
opted to work on the more-understandable unicode conversions.</p>
<p>My stalled progress can be found here: <a
href="https://github.com/gentoo/portage/pull/700"
moz-do-not-send="true">#700</a>. Lost the drive to continue
for now unfortunately :<</p>
<p>Good luck! Looking forward to your optimizations</p>
<p>– Marco Sirabella</p>
</blockquote>
<p><br>
</p>
<p>Hi All, thanks for the replies. Wow, Marco, that patch touches a
lot of stuff...! <br>
</p>
<p>OK, I will start by trying to get the profilers going and work
from there...</p>
<p>(Alec, to avoid replying separately: Thanks for your notes. Yes,
I am not clear which of the install/merge phases specifically is
the culprit, but it feels like something in that area is "slow",
especially when run under qemu user mode. I think using qmerge
won't work for my build script, but great idea to use it for
benchmarking to narrow things down - thanks)</p>
<p>Thanks all</p>
<p>Ed W<br>
</p>
</body>
</html>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic