Re: [atomic-devel] Meeting item for 20170802: Atomic Host + Modularity - Enabling CI/CD

[Josh Boyer <jwboyer redhat com>]

On Thu, Aug 3, 2017 at 10:16 AM, Ari LiVigni <ari redhat com> wrote:
>
>
> On Thu, Aug 3, 2017 at 9:23 AM, Josh Boyer <jwboyer redhat com> wrote:
>>
>> On Thu, Aug 3, 2017 at 8:59 AM, Dusty Mabe <dusty dustymabe com> wrote:
>> >
>> >
>> > On 08/03/2017 08:32 AM, Josh Boyer wrote:
>> >> On Wed, Aug 2, 2017 at 11:15 AM, Dusty Mabe <dusty dustymabe com>
>> >> wrote:
>> >>> Recently we talked about adding more CI/CD to the Atomic Host
>> >>> processes. Since we release every two weeks we'd like to automatically
>> >>> run tests on every change that goes in to Atomic Host and prevent
>> >>> it from going into Atomic Host if it causes test failures.
>> >>>
>> >>> This was suggested to the Atomic Working Group from the new CI Working
>> >>> Group [1] in the 2017-07-12 meeting [2]. Stef, from the CI working
>> >>> group also sent a mail to the cloud list about it [3].
>> >>>
>> >>> Here are some of the things we want to be able to do:
>> >>>
>> >>> 1  Have a rigorous definition (including specific versions, buildroot)
>> >>> of
>> >>>    what goes into an Atomic Host, including dependencies.
>> >>>
>> >>> 2  Triggering the CI/CD pipeline based on a change to definition
>> >>>    of what goes into Atomic Host.
>> >>>
>> >>> 3  A way to revert a package in the Atomic Host compose to an earlier
>> >>> version.
>> >>>
>> >>> 4  A place to store higher level tests along with rigorous definition
>> >>>    of Atomic Host, including them being versioned.
>> >>>
>> >>> 5  Landing multiple changes that need to land together to pass
>> >>> testing.
>> >>>
>> >>>
>> >>> Out of that list we think we require that:
>> >>>
>> >>> 1R The definition of what is composed into Atomic Host artifacts
>> >>> should
>> >>>    include specific versions of packages, and all dependencies
>> >>> included.
>> >>
>> >> Is this an output of or input to the compose?  I ask because
>> >> dependencies will change, so assuming they are static would be bad.
>> >
>> > I believe we want this to be an input, but an input that can be updated
>> > by
>> > bots. i.e. a new version of rpmxyz is available and a bot opens a PR to
>> > add it to the definition of atomic host and tests run on that PR. This
>> > is
>> > not set in stone, however. Definitely looking from input here,
>> > especially
>> > from people who know more about modularity.
>>
>> Hm.  OK.  I can see that working if it's automated as you suggest.  I
>> guess I would say that the goal of a "rigorous" definition is still
>> met, it's just that the definition changes a lot.
>>
>> >> They can also vary by architecture.
>> >
>> > This is something that was previously glossed over in our discussions.
>> > Thanks
>> > for bringing it up. Does modularity handle this at all today?
>>
>> Good question, and I don't know the answer.  I suspect it would have
>> to, particularly for things like low-level modules where the packages
>> are close to the architecture sets.  Atomic Host itself is certainly
>> in that category.  The higher level modules (e.g. httpd) are likely
>> identical in content and RPM NVRs though.
>>
>> >>> 2R The definition of what is composed in an Atomic Host should be
>> >>> stored
>> >>>    in a git repository so that changes can be detected easily. The
>> >>>    CI/CD pipeline can be triggered off of changes to this reposiroty.
>> >>
>> >> If the answer to the above question is "output of", then this item is
>> >> essentially a package manifest for everything in the compose, correct?
>> >
>> > Answer was 'input of'. We can debate on whether that is a good or a bad
>> > thing :)
>>
>> I was going for the easier way out with "output of", but input isn't
>> inherently bad either.  I guess for CI/CD, it doesn't matter much
>> either way.  The important thing there is that the changes are made
>> available to the pipeline and it can act on those at the appropriate
>> time.
>>
>> josh
>>
> I assume the CI/CD pipeline you refer to is the one upstream CI (my team) is
> working on?
> Otherwise this seems to be a duplication since we are already doing this
> work.

Dusty would have to answer that.  I was just talking about CI/CD in
general, but I think the specific pipeline Dusty is looking at is
indeed the one your team is working on.

josh

>> >>> 3R A mechanism to make a future composed Atomic Host artifact, contain
>> >>>    an earlier (in RPM NVR parlance) version of a package.
>> >>>
>> >>> 4R The high level functional Atomic Host tests should live in the same
>> >>>    git repository with the rigorous definition of what goes into an
>> >>> Atomic
>> >>>    Host.
>> >>>
>> >>> 5R A mechanism to tell the CI Pipeline that multiple dist-git
>> >>> repository
>> >>>    changes (i.e. multiple changing RPMs) should be built and tested
>> >>> together.
>> >>>
>> >>> We believe that moving to an Atomic Host defined by a module [4] can
>> >>> satisfy our needs and enable the improved control of artifacts within
>> >>> Atomic Host as well as the improved triggering and coupling with a
>> >>> testing pipeline that will help us move forward. Since modularity is
>> >>> new
>> >>> this will take some experimentation to prove out. Let's chat about
>> >>> this
>> >>> in the Atomic Working Group meeting today at 1700UTC in
>> >>> #fedora-meeting-1.
>> >>>
>> >>> Dusty
>> >>>
>> >>>
>> >>> [1] https://fedoraproject.org/wiki/CI
>> >>> [2]
>> >>> https://meetbot.fedoraproject.org/fedora-meeting-1/2017-07-12/fedora_atomic_wg.2017-07-12-17.00.log.html
>> >>> [3]
>> >>> https://lists.fedoraproject.org/archives/list/cloud lists fedoraproject org/message/ZS7XY7NHXGGOPOB2YKNQUWSDUGCMYIL5/
>> >>> [4] https://docs.pagure.org/modularity/
>> >>>
>> >> _______________________________________________
>> >> Atomic Working Group mailing list -- atomic lists fedoraproject org
>> >> To unsubscribe send an email to atomic-leave lists fedoraproject org
>> >>
>>
>
>
>
> --
> -== @ri ==-
> My PGP fingerprint is F87F1EE7CD8BEE13