Re: [atomic-devel] 3 kinds of "docker build"

[Sam Thursfield <sam thursfield codethink co uk>]

On 17/10/14 14:33, Colin Walters wrote:
> On Thu, Oct 16, 2014, at 12:42 PM, Sam Thursfield wrote:
> > Hi guys. First post here I think although I've been watching with
> > interest for a bit.
>
> Welcome!

Thanks :)

> > This is (one of) the ideas behind Baserock[1]. Baserock considers
> > everything to be throwaway except the input source code, which is kept
> > in Git, and the build tool and build environment... which are generated
> > from source code kept in Git.
>
> Yes, though when I was talking to Rob last weekend, I think lorry needs
> to fix the submodule handling.  What gnome-continuous does here has
> worked out quite well; it does recursive mirroring, and crucially
> rewrites the submodule references to refer to the local mirror.  See:
> https://git.gnome.org/browse/gnome-continuous/tree/src/js/vcs.js#n187

OK. In Baserock we end up rewriting the .gitmodule file manually to have 
a keyed URL that points to a Git mirror server (e.g. [1]). Then we also 
set 'git config submodule.foo.url=xxx' on local checkouts.

It'd be nice to do this a better way :) the fact that we have a Git 
mirror *and* a local Git cache makes it a bit tricky, perhaps.

[1]: 
http://git.baserock.org/cgi-bin/cgit.cgi/delta/gstreamer.git/tree/.gitmodules?h=baserock/genivi/baseline

> > That's an interesting use case. Morph, the build tool Baserock uses,
> > could be adapted into being an OK tool for this situation. It has pretty
> > good caching of builds so if you build two systems that share a lot of
> > stuff, the shared things only build once (or not at all, if there were
> > no new commits).
>
> Are caches here like packages, or more like bitbake's sstate cache?

What's a package? :)

If you have all the source code of a system, and construct a build graph 
from it, you can then use the cached artifacts to save yourself building 
things that were already built. Given a single cached artifact, you 
can't go back and construct a build graph from that.

Given a built system (i.e. the end result of a `morph build`) and all 
the input Git repos you can reconstruct the exact build graph that 
produced that system, so the cached build artifacts *are* a real cache 
... but it may cost you a lot of time in rebuilding stuff if you empty it!

> > The main problem with Morph as a developer tool is that it's very
> > paranoid about build-dependencies. So to follow your Glance example, if
> > the developer hacks on python-jsonpatch, Morph will rebuild that from
> > source, but then also rebuild everything above it in the build graph.
>
> Ther'es no one right answer here - it needs to be tunable.
> gnome-continuous only rebuilds modules which changed, which has worked
> out quite well for its intended use case, but we've certainly seen
> breakage when e.g. vala changes and then only later is geary built or
> something.  On the other hand you do at least need to periodically check
> everything builds from source.

Yeah; I guess for continuous delivery you need both.

> That's just source -> artifact policy, a whole other level is the
> artifact -> Docker image policy.  Should the distribution's Docker base
> images be updated for every RPM change, or only security?  And what
> about layered images?  There's almost certainly going to need to be some
> way to communicate that a new image build is security-sensitive, and
> layers really should rebuild.

I guess if the images in the Docker Registry don't get updated often 
enough, people start to add 'yum update' to their Dockerfiles and 
Fedora's hosting bill increases. ...

I don't really think the Docker Registry or the whole "golden master" 
approach is a step forward, but we shall see!

> Anyways so...circling back to my desire for some tool that lets me build
> multiple containers with common artifacts, I was basically thinking of
> something like "mockchain in Docker".  mockchain being:
> http://skvidal.wordpress.com/2012/04/20/mockchain-use-cases-and-examples/
>
> This is actually one of the nicest things about COPR - you can upload
> *multiple* sources and it will (eventually through brute force) build
> them in a working order.

mockchain is a bit over my head, as an relative outsider to working on 
RPM-based distros. COPR does look incredibly handy if I ever need to 
publish my own Fedora package repository!

Sam

--
Sam Thursfield, Codethink Ltd.
Office telephone: +44 161 236 5575