Docker builds vs package builds

[Colin Walters <walters verbum org>]

I posted something related to this on atomic@, but it's more of a 
development thing, so moving here.

After playing with Dockerfiles for some of my code, one thing that's 
becoming very clear to me is that using Dockerfiles certainly make it 
easy to get started, but the simplicity of the model/format seriously 
hampers efficient RPM (or many other buildsystems) integration, and has 
some subtle traps.

For example, a simple thing you see in a lot of Dockerfiles is:

RUN yum -y update

Except...this gets cached by Docker which is **not** what you want in 
general.  This is https://github.com/dotcloud/docker/issues/1996

I ended up doing the recommended workaround in that bug of adding a 
comment, so it's 
RUN yum -y update #nocache20140523.0
And whenever I want to avoid the cache I change the comment.

The right thing depends on circumstance, but it would be lot more 
efficient to do a check of the repository timestamps, and reuse the 
cached layer if they haven't changed.  A further optimization would be 
to reuse the cached layer if none of the packages have changed (this 
matters a lot for Fedora, where the repo changes a lot but not always 
for packages you actually use).

Doing this sort of intelligence requires nontrivial code; it's not 
clear to me whether it should live inside or outside of the container.

It's hard for it to live inside the container as the container itself 
(AFAIK) can't drive the Docker layer caching.  We'd have to do 
something like include within each layer cache metadata (such as the 
repository timestamps), and then enhance yum to leave the system 
untouched if the timestamps haven't changed.

This gets really quite ugly as we'd need to do pervasive O_NOATIME 
inside the container to avoid any changes from simply running code, 
clean up every temporary file, etc.

Then the other model is to try to entirely drive the process from 
outside of Docker.  But if you do that, it's again hard to reuse the 
Docker image cache; you'd have to keep cached state on the host system.

Is anyone aware of any advanced work on package (or other advanced 
buildsystem/deployment) integration with Dockerfiles?  This topic could 
probably use a wiki page or something, it seems like a fairly open area.