[prev in list] [next in list] [prev in thread] [next in thread]
List: kde-devel
Subject: Re: Gitlab Evaluation & Migration
From: Gleb Popov <6yearold () gmail ! com>
Date: 2019-02-28 18:21:04
Message-ID: CALH631kOhPZqb18rzjXriWb5PnZQX4ayQ0vnAtnk2n5a2OVq7A () mail ! gmail ! com
[Download RAW message or body]
On Thu, Feb 28, 2019 at 9:43 PM Ben Cooksley <bcooksley@kde.org> wrote:
> On Fri, Mar 1, 2019 at 3:13 AM Nate Graham <nate@kde.org> wrote:
> >
> > ---- On Wed, 27 Feb 2019 23:02:03 -0700 Ben Cooksley <bcooksley@kde.org>
> wrote ----
> > > In terms of server load, it would be nice if the setup of forks was
> > > still something the developer had to initiate rather than being done
> > > automatically for every repository touched by kdesrc-build (I say this
> > > mainly as if we had 50 people fork just half of the mainline
> > > repositories we have, that's ~450GB of space used up - a massive
> > > scalability issue)
> >
> > This seems like a challenge that needs to be addressed regardless of
> whether or not kdesrc-build does it automatically, because people creating
> tons and tons of forks is guaranteed to happen anyway if we move to Gitlab.
> It seems non-optimal if having more people able to submit merge requests
> results in the potential to blow up our servers.
>
> We have a little over 1,000 mainline repositories, so in the above
> example we'd be talking about 25,000 forks being created - and i'd be
> expecting quite a bit more than 50 people to use kdesrc-build. To use
> another scenario, if the metric of half the repositories being
> involved (or even a quarter) held true with say 300 users, you're now
> looking at 75,000 - 150,000 forked repositories (and probably around
> 1.4TB - 2.7TB of space used) courtesy of an automated tool.
>
> It would take quite a while for us to reach 150,000 forked
> repositories on Gitlab if humans were to be creating these manually,
> however if an automated tool is going to be creating them as part of
> it's workflow, then we will hit it much more quickly (and is a
> phenomenal waste of resources given virtually all of those forks will
> never be utilised)
>
I wonder if advanced filesystem features like ZFS deduplication may help in
this situation.
> I certainly do expect a number of forks to be created yes, but i'd
> rather they be useful forks where someone at least intends on working
> on something, rather than ones created automatically by software "just
> in case" someone decides to work on a project.
>
> >
> > Nate
> >
>
> Cheers,
> Ben
>
[Attachment #3 (text/html)]
<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" \
class="gmail_attr">On Thu, Feb 28, 2019 at 9:43 PM Ben Cooksley <<a \
href="mailto:bcooksley@kde.org">bcooksley@kde.org</a>> wrote:<br></div><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex">On Fri, Mar 1, 2019 at 3:13 AM Nate Graham <<a \
href="mailto:nate@kde.org" target="_blank">nate@kde.org</a>> wrote:<br> ><br>
> ---- On Wed, 27 Feb 2019 23:02:03 -0700 Ben Cooksley <<a \
href="mailto:bcooksley@kde.org" target="_blank">bcooksley@kde.org</a>> wrote \
----<br> > > In terms of server load, it would be nice if the setup of forks \
was<br> > > still something the developer had to initiate rather than being \
done<br> > > automatically for every repository touched by kdesrc-build (I \
say this<br> > > mainly as if we had 50 people fork just half of the \
mainline<br> > > repositories we have, that's ~450GB of space used up - a \
massive<br> > > scalability issue)<br>
><br>
> This seems like a challenge that needs to be addressed regardless of whether or \
not kdesrc-build does it automatically, because people creating tons and tons of \
forks is guaranteed to happen anyway if we move to Gitlab. It seems non-optimal if \
having more people able to submit merge requests results in the potential to blow up \
our servers.<br> <br>
We have a little over 1,000 mainline repositories, so in the above<br>
example we'd be talking about 25,000 forks being created - and i'd be<br>
expecting quite a bit more than 50 people to use kdesrc-build. To use<br>
another scenario, if the metric of half the repositories being<br>
involved (or even a quarter) held true with say 300 users, you're now<br>
looking at 75,000 - 150,000 forked repositories (and probably around<br>
1.4TB - 2.7TB of space used) courtesy of an automated tool.<br>
<br>
It would take quite a while for us to reach 150,000 forked<br>
repositories on Gitlab if humans were to be creating these manually,<br>
however if an automated tool is going to be creating them as part of<br>
it's workflow, then we will hit it much more quickly (and is a<br>
phenomenal waste of resources given virtually all of those forks will<br>
never be utilised)<br></blockquote><div><br></div><div>I wonder if advanced \
filesystem features like ZFS deduplication may help in this \
situation.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px \
0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <br>
I certainly do expect a number of forks to be created yes, but i'd<br>
rather they be useful forks where someone at least intends on working<br>
on something, rather than ones created automatically by software "just<br>
in case" someone decides to work on a project.<br>
<br>
><br>
> Nate<br>
><br>
<br>
Cheers,<br>
Ben<br>
</blockquote></div></div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic