[prev in list] [next in list] [prev in thread] [next in thread]
List: kde-devel
Subject: Re: Gitlab Evaluation & Migration
From: Ben Cooksley <bcooksley () kde ! org>
Date: 2019-02-28 18:36:42
Message-ID: CA+XidOEgxGLN6xs7q_4aJ3x82siJY5v+z+aWccVkxwEW0BD9wg () mail ! gmail ! com
[Download RAW message or body]
On Fri, 1 Mar 2019, 07:21 Gleb Popov <6yearold@gmail.com wrote:
>
>
> On Thu, Feb 28, 2019 at 9:43 PM Ben Cooksley <bcooksley@kde.org> wrote:
>
>> On Fri, Mar 1, 2019 at 3:13 AM Nate Graham <nate@kde.org> wrote:
>> >
>> > ---- On Wed, 27 Feb 2019 23:02:03 -0700 Ben Cooksley <
>> bcooksley@kde.org> wrote ----
>> > > In terms of server load, it would be nice if the setup of forks was
>> > > still something the developer had to initiate rather than being done
>> > > automatically for every repository touched by kdesrc-build (I say
>> this
>> > > mainly as if we had 50 people fork just half of the mainline
>> > > repositories we have, that's ~450GB of space used up - a massive
>> > > scalability issue)
>> >
>> > This seems like a challenge that needs to be addressed regardless of
>> whether or not kdesrc-build does it automatically, because people creating
>> tons and tons of forks is guaranteed to happen anyway if we move to Gitlab.
>> It seems non-optimal if having more people able to submit merge requests
>> results in the potential to blow up our servers.
>>
>> We have a little over 1,000 mainline repositories, so in the above
>> example we'd be talking about 25,000 forks being created - and i'd be
>> expecting quite a bit more than 50 people to use kdesrc-build. To use
>> another scenario, if the metric of half the repositories being
>> involved (or even a quarter) held true with say 300 users, you're now
>> looking at 75,000 - 150,000 forked repositories (and probably around
>> 1.4TB - 2.7TB of space used) courtesy of an automated tool.
>>
>> It would take quite a while for us to reach 150,000 forked
>> repositories on Gitlab if humans were to be creating these manually,
>> however if an automated tool is going to be creating them as part of
>> it's workflow, then we will hit it much more quickly (and is a
>> phenomenal waste of resources given virtually all of those forks will
>> never be utilised)
>>
>
> I wonder if advanced filesystem features like ZFS deduplication may help
> in this situation.
>
Deduplication can only do so much.
Even if you were to solve the capacity/resource issues, you would hit
another issue: Your personal namespace would be flooded with these
automatically created forks.
This would make your actual personal projects (which are future playground
projects in most cases) nearly impossible to find.
>
>> I certainly do expect a number of forks to be created yes, but i'd
>> rather they be useful forks where someone at least intends on working
>> on something, rather than ones created automatically by software "just
>> in case" someone decides to work on a project.
>>
>> >
>> > Nate
>> >
>>
>> Cheers,
>> Ben
>>
>
Regards,
Ben
>
[Attachment #3 (text/html)]
<div dir="auto"><div><div class="gmail_quote"><div dir="ltr">On Fri, 1 Mar 2019, \
07:21 Gleb Popov <<a href="mailto:6yearold@gmail.com">6yearold@gmail.com</a> \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 \
.8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div \
dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On \
Thu, Feb 28, 2019 at 9:43 PM Ben Cooksley <<a href="mailto:bcooksley@kde.org" \
target="_blank" rel="noreferrer">bcooksley@kde.org</a>> \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Fri, Mar 1, 2019 at \
3:13 AM Nate Graham <<a href="mailto:nate@kde.org" target="_blank" \
rel="noreferrer">nate@kde.org</a>> wrote:<br> ><br>
> ---- On Wed, 27 Feb 2019 23:02:03 -0700 Ben Cooksley <<a \
href="mailto:bcooksley@kde.org" target="_blank" \
rel="noreferrer">bcooksley@kde.org</a>> wrote ----<br> > > In terms of \
server load, it would be nice if the setup of forks was<br> > > still \
something the developer had to initiate rather than being done<br> > > \
automatically for every repository touched by kdesrc-build (I say this<br> > \
> mainly as if we had 50 people fork just half of the mainline<br> > > \
repositories we have, that's ~450GB of space used up - a massive<br> > > \
scalability issue)<br> ><br>
> This seems like a challenge that needs to be addressed regardless of whether or \
not kdesrc-build does it automatically, because people creating tons and tons of \
forks is guaranteed to happen anyway if we move to Gitlab. It seems non-optimal if \
having more people able to submit merge requests results in the potential to blow up \
our servers.<br> <br>
We have a little over 1,000 mainline repositories, so in the above<br>
example we'd be talking about 25,000 forks being created - and i'd be<br>
expecting quite a bit more than 50 people to use kdesrc-build. To use<br>
another scenario, if the metric of half the repositories being<br>
involved (or even a quarter) held true with say 300 users, you're now<br>
looking at 75,000 - 150,000 forked repositories (and probably around<br>
1.4TB - 2.7TB of space used) courtesy of an automated tool.<br>
<br>
It would take quite a while for us to reach 150,000 forked<br>
repositories on Gitlab if humans were to be creating these manually,<br>
however if an automated tool is going to be creating them as part of<br>
it's workflow, then we will hit it much more quickly (and is a<br>
phenomenal waste of resources given virtually all of those forks will<br>
never be utilised)<br></blockquote><div><br></div><div>I wonder if advanced \
filesystem features like ZFS deduplication may help in this \
situation.</div></div></div></blockquote></div></div><div dir="auto"><br></div><div \
dir="auto">Deduplication can only do so much.</div><div dir="auto"><br></div><div \
dir="auto">Even if you were to solve the capacity/resource issues, you would hit \
another issue: Your personal namespace would be flooded with these automatically \
created forks. </div><div dir="auto"><br></div><div dir="auto">This would make your \
actual personal projects (which are future playground projects in most cases) nearly \
impossible to find.</div><div dir="auto"><br></div><div dir="auto"><div \
class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 \
.8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div \
class="gmail_quote"><div><br></div><blockquote class="gmail_quote" style="margin:0px \
0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <br>
I certainly do expect a number of forks to be created yes, but i'd<br>
rather they be useful forks where someone at least intends on working<br>
on something, rather than ones created automatically by software "just<br>
in case" someone decides to work on a project.<br>
<br>
><br>
> Nate<br>
><br>
<br>
Cheers,<br>
Ben<br></blockquote></div></div></blockquote></div></div><div \
dir="auto"><br></div><div dir="auto">Regards,</div><div dir="auto">Ben </div><div \
dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 \
0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div \
class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> \
</blockquote></div></div> </blockquote></div></div></div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic