[prev in list] [next in list] [prev in thread] [next in thread]
List: vdsm-devel
Subject: =?utf-8?q?=5Bovirt-devel=5D?= Re: Container-based CI backend is now available for use
From: Ales Musil <amusil () redhat ! com>
Date: 2020-03-26 12:41:58
Message-ID: CALVEqe7nonP1eRHy=4e6VHGx6uDQAUL4WhXsDRNnuQ_soKw=kg () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
On Tue, Mar 24, 2020 at 1:50 PM Ales Musil <amusil@redhat.com> wrote:
>
>
> On Tue, Dec 31, 2019 at 9:01 AM Barak Korren <bkorren@redhat.com> wrote:
>
>> *Update #2: *We have now merged all the patches that deal with artifact
>> and log collection. And have updated the documentation
>> <https://ovirt-infra-docs.readthedocs.io/en/latest/CI/STDCI-Containers/index.html>
>> accordingly.
>> The container-based backend should now be usable for the vast majority of
>> the CI use cases.
>>
>> We do have some more features coming down the line geared towards more
>> sophisticated use cases such as running OST suits and integrating with
>> gating and change-queue flows. those Include:
>>
>> 1. Supporting the use of privileged containers
>>
>> What can I do to actually try this? I am failing because our container is
> not on the whitelist. Can you please point me in the right direction?
> Btw. The patch that I was trying this on is
> https://gerrit.ovirt.org/c/107831/
>
Any update on this?
>
>
>>
>> 1. Invoking the container-based backed from the gating jobs
>> 2. Generating and providing the `extra_sources` file
>> 3. Runtime injection of YUM mirror URLs
>> 4. Support for storing and using secret data such as password and
>> auth tokens.
>>
>> I invite everyone to start moving workloads to the new system and enjoy
>> the enhanced speed and reliability.
>>
>> On Sun, 15 Dec 2019 at 14:23, Barak Korren <bkorren@redhat.com> wrote:
>>
>>> *Update: *We have now merged the automated cloning support feature, the
>>> currently merged code should already be applicable for a wide range of uses
>>> including running 'check-patch' workloads.
>>>
>>> On Thu, 12 Dec 2019 at 09:00, Barak Korren <bkorren@redhat.com> wrote:
>>>
>>>> A little less then a month ago I sent an email to this list telling you
>>>> all about ongoing work to create a new container-based backend for the
>>>> oVirt CI system.
>>>>
>>>> I'm pleased to announce that we have managed to finally merged an
>>>> initial set of patches implementing that backend yesterday, and it is now
>>>> available for general use.
>>>>
>>>> *What? Were? How do I use it?*
>>>>
>>>> Documentation about how to use the new backend is now available in read
>>>> the docs
>>>> <https://ovirt-infra-docs.readthedocs.io/en/latest/CI/STDCI-Containers/index.html>
>>>> .
>>>>
>>>> *Wait! I needed it to do X which it doesn't!*
>>>>
>>>> For the time being the new backend lacks some features that some may
>>>> consider to be essential, such as automated cloning of patch source code
>>>> and build artifact collection. We already have implemented patches
>>>> providing a substantial amount of additional functionality, and hopefully
>>>> we will be able to merge them soon. Following is a list of those patches
>>>> and the features they implement:
>>>>
>>>> 1. Automated source cloning support:
>>>> - 104213 <https://gerrit.ovirt.org/104213>: Implement STDCI DSL
>>>> support for initContainers
>>>> - 104590 <https://gerrit.ovirt.org/104590>: STDCI DSL: Add the
>>>> `decorate` option
>>>> - 104668 <https://gerrit.ovirt.org/104668>: Document source
>>>> cloning extension for containers
>>>> 2. Artifact collection support
>>>> - 104690 <https://gerrit.ovirt.org/104690>: Added NFS server
>>>> container image
>>>> - 104273 <https://gerrit.ovirt.org/104273>: STDCI PODS: Unique
>>>> UID for each job build's POD
>>>> - 104756 <https://gerrit.ovirt.org/104756>: pipeline-loader:
>>>> refactor: separate podspec func
>>>> - 104757 <https://gerrit.ovirt.org/104757>: pipeline-loader:
>>>> refactor: Use podspec struct def
>>>> - 104766 <https://gerrit.ovirt.org/104766>: STDCI PODs: Add
>>>> artifact collection logic
>>>> - 105522 <https://gerrit.ovirt.org/105522>: Documented artifact
>>>> collection in containers
>>>> 3. Extended log collection
>>>> - 104842 <https://gerrit.ovirt.org/104842>: STDCI PODs: Add POD
>>>> log collection
>>>> - 105523 <https://gerrit.ovirt.org/105523>: Documented log
>>>> collection in containers
>>>> 4. Privileged container support
>>>> - 104786 <https://gerrit.ovirt.org/104786>: STDCI DSL: Enable
>>>> privileged containers
>>>> 5. Support for using containers in gating jobs:
>>>> - 104804 <https://gerrit.ovirt.org/104804>: standard-stage:
>>>> refactor: move DSL to a library
>>>> - 104811 <https://gerrit.ovirt.org/104811>: gate: Support
>>>> getting suits from STDCI DSL
>>>> 6. Providing the `extra_sources` file to OST suit containers:
>>>> - 104843 <https://gerrit.ovirt.org/104843>: stdci_runner: Create
>>>> extra_sources for PODs
>>>> 7. Support for mirror injection and upstream source cloning
>>>> - 104917 <https://gerrit.ovirt.org/104917>: Added a container
>>>> with STDCI tools
>>>> - 104918 <https://gerrit.ovirt.org/104918>: decorate.py: Add
>>>> script
>>>> - 104989 <https://gerrit.ovirt.org/104989>: STDCI DSL: Use
>>>> `tools` container for `decorate`
>>>> - 104994 <https://gerrit.ovirt.org/104994>: stdci_runner: Inject
>>>> mirrors in PODs
>>>>
>>>>
>>>> As you can see, we have quite a big pile of reviews to do, as always,
>>>> help is very welcome...
>>>>
>>>> Regards,
>>>> Barak.
>>>>
>>>> --
>>>> Barak Korren
>>>> RHV DevOps team , RHCE, RHCi
>>>> Red Hat EMEA
>>>> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
>>>>
>>>
>>>
>>> --
>>> Barak Korren
>>> RHV DevOps team , RHCE, RHCi
>>> Red Hat EMEA
>>> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
>>>
>>
>>
>> --
>> Barak Korren
>> RHV DevOps team , RHCE, RHCi
>> Red Hat EMEA
>> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
>> _______________________________________________
>> Devel mailing list -- devel@ovirt.org
>> To unsubscribe send an email to devel-leave@ovirt.org
>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>> oVirt Code of Conduct:
>> https://www.ovirt.org/community/about/community-guidelines/
>> List Archives:
>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/HYZVFWHYDNUY7332WH3T3UPIAZ4HZLKY/
>>
>
>
> --
>
> Ales Musil
>
> Software Engineer - RHV Network
>
> Red Hat EMEA <https://www.redhat.com>
>
> amusil@redhat.com IM: amusil
> <https://red.ht/sig>
>
--
Ales Musil
Software Engineer - RHV Network
Red Hat EMEA <https://www.redhat.com>
amusil@redhat.com IM: amusil
<https://red.ht/sig>
[Attachment #5 (text/html)]
<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" \
class="gmail_attr">On Tue, Mar 24, 2020 at 1:50 PM Ales Musil <<a \
href="mailto:amusil@redhat.com" target="_blank">amusil@redhat.com</a>> \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div \
dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On \
Tue, Dec 31, 2019 at 9:01 AM Barak Korren <<a href="mailto:bkorren@redhat.com" \
target="_blank">bkorren@redhat.com</a>> wrote:<br></div><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><b>Update #2: </b>We have now \
merged all the patches that deal with artifact and log collection. And have updated \
<a href="https://ovirt-infra-docs.readthedocs.io/en/latest/CI/STDCI-Containers/index.html" \
target="_blank">the documentation</a> accordingly.</div><div>The container-based \
backend should now be usable for the vast majority of the CI use \
cases.</div><div><br></div><div>We do have some more features coming down the line \
geared towards more sophisticated use cases such as running OST suits and integrating \
with gating and change-queue flows. those Include:</div><div><ol><li>Supporting the \
use of privileged containers</li></ol></div></div></blockquote><div>What can I do to \
actually try this? I am failing because our container is not on the whitelist. Can \
you please point me in the right direction?</div><div>Btw. The patch that I was \
trying this on is <a href="https://gerrit.ovirt.org/c/107831/" \
target="_blank">https://gerrit.ovirt.org/c/107831/</a></div></div></div></blockquote><div><br></div><div>Any \
update on this?<br></div><div> </div><blockquote class="gmail_quote" \
style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div> \
</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px \
solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><ol><li>Invoking the \
container-based backed from the gating jobs</li><li>Generating and providing the \
`extra_sources` file</li><li>Runtime injection of YUM mirror URLs</li><li>Support for \
storing and using secret data such as password and auth tokens.</li></ol></div><div>I \
invite everyone to start moving workloads to the new system and enjoy the enhanced \
speed and reliability.</div><br><div class="gmail_quote"><div dir="ltr" \
class="gmail_attr">On Sun, 15 Dec 2019 at 14:23, Barak Korren <<a \
href="mailto:bkorren@redhat.com" target="_blank">bkorren@redhat.com</a>> \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div \
dir="ltr"><div><b>Update: </b>We have now merged the automated cloning support \
feature, the currently merged code should already be applicable for a wide range of \
uses including running 'check-patch' workloads. </div><br><div \
class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, 12 Dec 2019 at 09:00, \
Barak Korren <<a href="mailto:bkorren@redhat.com" \
target="_blank">bkorren@redhat.com</a>> wrote:<br></div><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div \
dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div \
dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div \
dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div \
dir="ltr"><div dir="ltr">A little less then a month ago I sent an email to this list \
telling you all about ongoing work to create a new container-based backend for the \
oVirt CI system.<div><br></div><div>I'm pleased to announce that we have managed \
to finally merged an initial set of patches implementing that backend yesterday, and \
it is now available for general use.</div><div><br></div><div><b>What? Were? How do \
I use it?</b></div><div><b><br></b></div><div>Documentation about how to use the new \
backend is now available in <a \
href="https://ovirt-infra-docs.readthedocs.io/en/latest/CI/STDCI-Containers/index.html" \
target="_blank">read the docs</a>.<br clear="all"><div><br></div><div><b>Wait! I \
needed it to do X which it doesn't!</b></div><div><br></div><div>For the time \
being the new backend lacks some features that some may consider to be essential, \
such as automated cloning of patch source code and build artifact collection. We \
already have implemented patches providing a substantial amount of additional \
functionality, and hopefully we will be able to merge them soon. Following is a list \
of those patches and the features they implement:</div><div><ol><li>Automated source \
cloning support:<br></li><ul><li><a href="https://gerrit.ovirt.org/104213" \
target="_blank">104213</a>: Implement STDCI DSL support for initContainers \
</li><li><a href="https://gerrit.ovirt.org/104590" target="_blank">104590</a>: STDCI \
DSL: Add the `decorate` option </li><li><a href="https://gerrit.ovirt.org/104668" \
target="_blank">104668</a>: Document source cloning extension for \
containers<br></li></ul><li>Artifact collection support</li><ul><li><a \
href="https://gerrit.ovirt.org/104690" target="_blank">104690</a>: Added NFS server \
container image <br></li><li><span><a \
href="https://gerrit.ovirt.org/104273" target="_blank">104273</a>: STDCI PODS: Unique \
UID for each job build's POD </span> <br></li><li><a \
href="https://gerrit.ovirt.org/104756" target="_blank">104756</a>: pipeline-loader: \
refactor: separate podspec func<br></li><li><a href="https://gerrit.ovirt.org/104757" \
target="_blank">104757</a>: pipeline-loader: refactor: Use podspec struct \
def<br></li><li><a href="https://gerrit.ovirt.org/104766" target="_blank">104766</a>: \
STDCI PODs: Add artifact collection logic<br></li><li><a \
href="https://gerrit.ovirt.org/105522" target="_blank">105522</a>: Documented \
artifact collection in containers<br></li></ul><li>Extended log \
collection</li><ul><li><a href="https://gerrit.ovirt.org/104842" \
target="_blank">104842</a>: STDCI PODs: Add POD log collection<br></li><li><a \
href="https://gerrit.ovirt.org/105523" target="_blank">105523</a>: Documented log \
collection in containers</li></ul><li>Privileged container support</li><ul><li><a \
href="https://gerrit.ovirt.org/104786" target="_blank">104786</a>: STDCI DSL: Enable \
privileged containers<br></li></ul><li>Support for using containers in gating \
jobs:</li><ul><li><a href="https://gerrit.ovirt.org/104804" \
target="_blank">104804</a>: standard-stage: refactor: move DSL to a library \
<br></li><li><a href="https://gerrit.ovirt.org/104811" target="_blank">104811</a>: \
gate: Support getting suits from STDCI DSL<br></li></ul><li>Providing the \
`extra_sources` file to OST suit containers:</li><ul><li><a \
href="https://gerrit.ovirt.org/104843" target="_blank">104843</a>: stdci_runner: \
Create extra_sources for PODs<br></li></ul><li>Support for mirror injection and \
upstream source cloning</li><ul><li><a href="https://gerrit.ovirt.org/104917" \
target="_blank">104917</a>: Added a container with STDCI tools</li><li><a \
href="https://gerrit.ovirt.org/104918" target="_blank">104918</a>: decorate.py: Add \
script<br></li><li><a href="https://gerrit.ovirt.org/104989" \
target="_blank">104989</a>: STDCI DSL: Use `tools` container for \
`decorate`</li><li><a href="https://gerrit.ovirt.org/104994" \
target="_blank">104994</a>: stdci_runner: Inject mirrors in \
PODs</li></ul></ol></div><div><br></div><div>As you can see, we have quite a big pile \
of reviews to do, as always, help is very \
welcome...</div><div><br></div><div>Regards,</div><div>Barak.</div><div><br></div>-- \
<br><div dir="ltr">Barak Korren<br>RHV DevOps team , RHCE, RHCi<br>Red Hat EMEA<br><a \
href="http://redhat.com" target="_blank">redhat.com</a> | TRIED. TESTED. TRUSTED. | \
<a href="http://redhat.com/trusted" \
target="_blank">redhat.com/trusted</a></div></div></div></div></div></div></div></div> \
</div></div></div></div></div></div></div></div></div></div></div></div></div></div> \
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr">Barak \
Korren<br>RHV DevOps team , RHCE, RHCi<br>Red Hat EMEA<br><a href="http://redhat.com" \
target="_blank">redhat.com</a> | TRIED. TESTED. TRUSTED. | <a \
href="http://redhat.com/trusted" target="_blank">redhat.com/trusted</a></div></div> \
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr">Barak \
Korren<br>RHV DevOps team , RHCE, RHCi<br>Red Hat EMEA<br><a href="http://redhat.com" \
target="_blank">redhat.com</a> | TRIED. TESTED. TRUSTED. | <a \
href="http://redhat.com/trusted" target="_blank">redhat.com/trusted</a></div></div> \
_______________________________________________<br> Devel mailing list -- <a \
href="mailto:devel@ovirt.org" target="_blank">devel@ovirt.org</a><br> To unsubscribe \
send an email to <a href="mailto:devel-leave@ovirt.org" \
target="_blank">devel-leave@ovirt.org</a><br> Privacy Statement: <a \
href="https://www.ovirt.org/site/privacy-policy/" rel="noreferrer" \
target="_blank">https://www.ovirt.org/site/privacy-policy/</a><br> oVirt Code of \
Conduct: <a href="https://www.ovirt.org/community/about/community-guidelines/" \
rel="noreferrer" target="_blank">https://www.ovirt.org/community/about/community-guidelines/</a><br>
List Archives: <a href="https://lists.ovirt.org/archives/list/devel@ovirt.org/message/HYZVFWHYDNUY7332WH3T3UPIAZ4HZLKY/" \
rel="noreferrer" target="_blank">https://lists.ovirt.org/archives/list/devel@ovirt.org/message/HYZVFWHYDNUY7332WH3T3UPIAZ4HZLKY/</a><br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr"><div \
dir="ltr"><div><div dir="ltr"><div>
<p style="font-weight:bold;margin:0px;padding:0px;font-size:14px;text-transform:capitalize;font-family:"RedHatText",sans-serif">
<span>Ales</span> <span>Musil</span><span \
style="color:rgb(170,170,170);margin:0px"></span> </p>
<p style="font-weight:normal;font-size:12px;margin:0px;text-transform:capitalize;font-family:"RedHatText",sans-serif"><span>Software \
Engineer - RHV Network</span> </p>
<p style="font-weight:normal;margin:0px 0px \
4px;font-size:12px;font-family:"RedHatText",sans-serif"> <a \
style="color:rgb(0,136,206);font-size:12px;margin:0px;text-decoration:none;font-family:"RedHatText",sans-serif" \
href="https://www.redhat.com" target="_blank">Red Hat <span>EMEA</span></a> </p>
<div style="margin-bottom:4px">
</div>
<p style="font-weight:normal;margin:0px;font-size:12px;font-family:"RedHatText",sans-serif">
<span style="margin:0px;padding:0px"><a \
style="color:rgb(0,0,0);font-size:12px;margin:0px;text-decoration:none;font-family:"RedHatText",sans-serif" \
href="mailto:amusil@redhat.com" target="_blank">amusil@redhat.com</a> </span>
<span>IM: <span>amusil</span></span>
</p>
<div style="margin-top:12px">
<table border="0">
<tbody><tr>
<td width="100px"><a href="https://red.ht/sig" target="_blank"> <img \
src="https://static.redhat.com/libs/redhat/brand-assets/latest/corp/logo.png" \
width="90" height="auto"></a> </td>
</tr>
</tbody></table>
</div>
</div></div></div></div></div></div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr"><div dir="ltr"><div><div \
dir="ltr"><div>
<p style="font-weight:bold;margin:0px;padding:0px;font-size:14px;text-transform:capitalize;font-family:"RedHatText",sans-serif">
<span>Ales</span> <span>Musil</span><span \
style="color:rgb(170,170,170);margin:0px"></span> </p>
<p style="font-weight:normal;font-size:12px;margin:0px;text-transform:capitalize;font-family:"RedHatText",sans-serif"><span>Software \
Engineer - RHV Network</span> </p>
<p style="font-weight:normal;margin:0px 0px \
4px;font-size:12px;font-family:"RedHatText",sans-serif"> <a \
style="color:rgb(0,136,206);font-size:12px;margin:0px;text-decoration:none;font-family:"RedHatText",sans-serif" \
href="https://www.redhat.com" target="_blank">Red Hat <span>EMEA</span></a> </p>
<div style="margin-bottom:4px">
</div>
<p style="font-weight:normal;margin:0px;font-size:12px;font-family:"RedHatText",sans-serif">
<span style="margin:0px;padding:0px"><a \
style="color:rgb(0,0,0);font-size:12px;margin:0px;text-decoration:none;font-family:"RedHatText",sans-serif" \
href="mailto:amusil@redhat.com" target="_blank">amusil@redhat.com</a> </span>
<span>IM: <span>amusil</span></span>
</p>
<div style="margin-top:12px">
<table border="0">
<tbody><tr>
<td width="100px"><a href="https://red.ht/sig" target="_blank"> <img \
src="https://static.redhat.com/libs/redhat/brand-assets/latest/corp/logo.png" \
width="90" height="auto"></a> </td>
</tr>
</tbody></table>
</div>
</div></div></div></div></div></div>
[Attachment #6 (text/plain)]
_______________________________________________
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/UFESJBSZDGPVCUIT6KIVK4LGS74DKLK2/
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic