[atomic-devel] Why Atomic Host should be built using Modularity

[Colin Walters <walters verbum org>]

There was a discussion today in the Atomic WG about using Modules.
Meeting log: https://lists.projectatomic.io/projectatomic-archives/atomic-devel/2017-August/msg00004.html
Agenda discussion: https://lists.projectatomic.io/projectatomic-archives/atomic-devel/2017-August/msg00002.html
(Side note; this doc was originally stored at <https://gist.github.com/cgwalters/c69bf2091eab7c1af316d0d7dd41f530>)

This post is the "why" that I'd written earlier.  Some of the rationale
here crosscuts with other active discussions; for example, keeping
Python 2 in Atomic Host, as well as Bodhi and update cadences.

There’s an effort in Fedora called
[“Modularity”](https://docs.pagure.org/modularity/) that effectively gives
multiple lifecycles for different components.

Today (and since the start), Fedora Atomic Host *is* Fedora - we have just added
another build process at the end of the "Everything" build process (and the same
is true of the `registry.fedoraproject.org/fedora` container). There's a lot of
benefits to that; any security fixes to e.g. `glibc` or `openssl` automatically
propagate.

Content streams
---------------

But on the other hand, Fedora Atomic Host is locked into the current Fedora
cycles; the freezes, then the "open the floodgates" model of Bodhi.

The "at most once a day updates" that Bodhi implements is quite suboptimal
for the server case.  We go to quite a bit of effort to implement a "two
week" cadence for the ostree updates, distinct from whatever happens in Bodhi
day to day.

Further, we really have no idea what might land in Bodhi day to day.  There
are a lot of updates that are basically non-critical, and should ideally
be batched together.  Things like `rsync`.

Another issue is that we currently don't have any ability to do *overrides* when we need
to. If a package gets karma or the maintainer decides to ship it anyways, that's
basically it. Yes, ideally we don't have overrides, but the reality is we need
the ability, we just want to avoid using it in any long term capacity.


Moving towards a single stream
------------------------------

One of the major benefits of the rpm-ostree technology is that there's
no special distinction between minor and major upgrades, as we've ended
up with between `yum upgrade` and `dnf system-upgrade`.  The underlying
ostree side is "image like" - we could easily do changes on the scale of
[UsrMove](https://fedoraproject.org/wiki/Features/UsrMove) without requiring
any special work or downtime, in the same way we can do small incremental updates
atomically.  For example, imagine that we decided to switch to
the [Debian Multiarch](https://wiki.debian.org/Multiarch) layout.  That'd
just be another atomic upgrade (with rollback, etc.)

See [this issue](https://pagure.io/atomic-wg/issue/228) for more background
discussion on this topic.

We'd like "Fedora Atomic Host" to simply be a single stream for most users.
This implies more than just having a unified update mechanism; we also
need to ensure that any Changes that land in "Fedora" are minimally
disruptive for the server case.

Good examples here are:

 - https://lists.fedoraproject.org/archives/list/devel lists fedoraproject org/thread/ZNQW72UP36UAFMX53HPFFQTWTQDZVJ3M/
   We'd probably take this change, but delay it quite a while.  It makes
   sense for desktops, and as an opt-in for server admins who want it.
 - Dropping /usr/bin/python: https://lists.fedoraproject.org/archives/list/devel lists fedoraproject org/thread/JO4WGEQVGRJMLNDLMZURHIQEGPXUZCOH/
   This badly affects Ansible (which is quite important for Atomic Host)

Now, a lot of how this works comes down to how quickly Modularity
takes over Fedora and the changes in general.  We're not going to get
out of being able to make changes like the above; but I think
we want more control over when they land - for example, allow
Workstation to actually do something earlier.

Basically, we want to create a minimal server OS focused
on container hosting which you can install to e.g. a
bare metal machine, and upgrade seamlessly in place
without any special work for years.

How many years?  This proposal doesn't address that,
but it certainly seems to me like we could easily handle
2-3 years without major breaking changes.

Obviously again, a lot of this relies on containerization to be
truly successful.  (And containerization in turn really requires
Modularity too!)

One important aspect of this the "system containers" effort to
move even things like Kubernetes into containers that are decoupled from
the host lifecycle.  We'd likely have Kubernetes containers as modules,
with likely at least 2 major versions available.

Development benefits for AH
---------------------------

We want to minimize the number of versions of our critical path
components that we maintain; think `docker`, `ostree` etc.  Now,
perhaps in a Modularity world we would end up offering multiple
`docker` RPMs (and containers), but I don't see a reason to maintain
multiple `ostree/rpm-ostree` for example.  Another good example is
`cloud-init`.  Further, the more we can move away from the generic
"rawhide" model to one where we have only transient branches for
e.g. `anaconda`, the better.

Or to phrase this another way - it's better if our "current stable" is the development stream target. 
The at-most-once-a-day updates model is too slow for development  (we want to test smaller incremental changes as they come in).  The once-a-day model is also too fast for most end users (see above).  It's also easier to link automated testing policy to modules rather than karma.   And if something goes wrong in testing (say we encounter a kernel crasher), we want the ability to roll back versions - which `rpm-ostree` supports very well; `rpm-ostree upgrade` behaves like `yum distro-sync` always.

Development benefits for Fedora in general
------------------------------------------

By staying in sync with the base runtime etc., we unify development
and help push Modularity forwards.

Benefits to users
-----------------

Not much *directly*.  However, for any users who want to e.g. file bugs, we can
make it more clear that e.g. if they have a problem with `coreutils`, that comes
from the base runtime, and ideally Bugzilla allows them to file bugs there.
If things like the Changes end up being written in terms of modules, we'll be able
to automatically pick those up.

What next
-------------

We're going to experiment with this!