Re: [atomic-devel] RFC Upstream Docker Layered Image Build tooling

[Tomas Tomecek <ttomecek redhat com>]

In-line comments everywhere.

Quoting Clayton Coleman (2015-07-15 17:54:41)
> When you say "run / maintain the entire PaaS", can you describe the
> portions of the PaaS that you don't want to run?  You need worker nodes
> (with Docker installed) to build docker images, an agent on each node (to
> manage the docker image builds), and a central server to have an API for
> starting / scheduling / running builds.  For a small deployment, that's
> what OpenShift is providing.  Is it concerns with SDN (which you don't need
> for builds), or the registry (which you don't need), or the router (which
> you don't need)?  I would hope that you would be able to run just the bits
> you need for image builds in OpenShift on their own.

Maintenance issues from my PoV are mainly related to everything changes too
often (but it's not just OpenShift, it's also docker, docker-py...). Basically
with every release we need to fix/workaround/rewrite our code. This is the
reason we are not running 1.0 still, we just didn't have enough resources to get
there.

> I can certainly understand there are gaps in build functionality.  Part of
> the original goal here was to build something that would be useful to
> people who needed to build images in a farm, with a long term goal of being
> able to manage other types of workloads.  I don't think it's true that the
> build system exists only to serve the PaaS - we plan on using it for other
> things, like hosting builds securely for others.
> 
> Are you saying that long term, you don't believe that a generic compute
> service that happens to have very close ties to Docker, has integrated
> metering, usage, security, and quota, and is supported on a wide range of
> platforms, is *not* a good tool to use to build images?
> 
> 
> 
> On Wed, Jul 15, 2015 at 10:15 AM, Adam Miller <maxamillion fedoraproject org
> > wrote:
> 
> > Hello all,
> >     Something that is in the works in Fedora right now is the Docker
> > Layered Image Build Service[0], and we've been looking at using
> > OSBS[1][2] for this but the more we dig into it there is uncertainty
> > about the OpenShift[3] component being the best solution for a build
> > system. OpenShift is a great Platform as a Service(PaaS) cloud
> > environment that happens to feature the logic for scheduling container
> > builds, but that function exists to enable other functionality in the
> > PaaS. Meanwhile from an administrative perspective, we have to
> > run/maintain the entire PaaS to use an extremely tiny subset of it's
> > functionality.

I share your concerns wrt maintenance. On the other hand, I also agree with
Clayton: you don't need to maintain stuff which is not used.

There's also learning curve and you need to understand essentials, plus stuff
related to scheduling, building etc. (luckily OpenShift's documentation is
really great).

So in the end the maintenance is not just taking care of the deployment, you
also need to know how it works and what's wrong when something blows (e.g.
yesterday we had an issue when node stopped scheduling builds, it took us some
time to figure out that capacity of the node was exhausted and garbage collector
didn't do its thing by purging completed pods).

> >     I spoke with KB from the CentOS team to try and get some
> > notes/pointers on OSBS because as I understood it, the CentOS team
> > already gone through a setup of the system before. As we spoke I
> > mentioned my concerns about the overhead and complexity of using a
> > full blown PaaS just for build scheduling and it turns out the CentOS
> > team shares those concerns. Also, the CentOS team largely just uses
> > atomic-reactor[4] on it's own outside of OSBS currently because of the
> > PaaS overhead (at this time, they are still using a version under the
> > previous name of "dock" but that's more or less just a side detail).
> > Using atomic-reactor as close to at face value as possible is
> > something that we on the Fedora Team are also interested in pursuing
> > because the atomic-reactor functionality is great for layered builds,
> > it's simple/concise, has a very clean code base, and it has a very
> > good python API.

That's what atomic-reactor is for: python library with cli and pre/post build
hooks to build docker images where you are in charge of defining the workflow
for the build process.

> >     Largely as a side note, I attempted to dig into things a little
> > bit and found that there was originally a smaller in scope
> > builder/scheduler service being worked on called dbs[5][6][7] which is
> > based around Celery[8] but this project was left behind in favor of
> > using OpenShift for OSBS.

You are totally right. Originally, the build system was split into 4 pieces:
api/web, worker, cli and build tool. We've been very close to a functional
prototype.

> >     All that being said, I wanted to start a discussion around the
> > topic to see if there is something that we could all work on
> > collectively upstream to target the very specific need of performing
> > these kinds of builds using atomic-reactor in a way that Fedora,
> > CentOS, and $others could benefit from using and collaborating on
> > contributions. Whether that be to investigate dbs and possibly
> > "resurrect" that work or fork (if necessary) to continue development,
> > or pursue something entirely new. The end goal would be that we could
> > collaborate on as much as possible and not re-invent build tooling for
> > the different distros. Also, it might be nice if this tooling could
> > build nulecule[9] apps.

Since koji is used heavily, we could possibly create a "reactor-build" type of
task which would use atomic-reactor directly.

Wrt resurrecting dbs, the positive is that we would have a build system which
would be capable of building images and nothing else. We would also be in a full
control of the codebase (atm with OpenShift/k8s it's not that easy to get some
changes to code [10]).

But again, you would end up with the same architecture: whole new build system
for just one type of artifacts. I can also imagine there would be a lot of
"reinventing the wheel".

> > Please let me know what your thoughts are on the topic. If there's
> > something I'm not thinking of or if the idea of removing OpenShift
> > from the build tooling seems completely crazy, please let me know why
> > because it's certainly possible I'm missing something and I'm always
> > open to new ideas.
> >
> > Thank you,
> > -AdamM
> >
> > [0] -
> > https://fedoraproject.org/wiki/Changes/Layered_Docker_Image_Build_Service
> > [1] - https://github.com/projectatomic/osbs-client
> > [2] - https://github.com/projectatomic/ansible-osbs
> > [3] - https://github.com/openshift/origin
> > [4] - https://github.com/projectatomic/atomic-reactor
> > [5] - https://github.com/DBuildService/dbs-server
> > [6] - https://github.com/DBuildService/dbs-worker
> > [7] - https://github.com/DBuildService/dbs-client
> > [8] - http://www.celeryproject.org/
> > [9] - https://github.com/projectatomic/nulecule
> >
> >
> 
> 
> -- 
> Clayton Coleman | Lead Engineer, OpenShift

[10] https://github.com/GoogleCloudPlatform/kubernetes/issues/9013

~~
Tomáš Tomeček
Software Engineer
Developer Experience
UTC+2 (CEST)