Re: [atomic-devel] [fedora-atomic f23] remove Python source files

[Fabian Deutsch <fdeutsch redhat com>]

On Tue, Dec 1, 2015 at 1:36 PM, Giuseppe Scrivano <gscrivan redhat com> wrote:
> Hi,
>
> I was experimenting with reducing the size of the Atomic Host image and
> it seems that a lot of space is used by Python source files.

For many (6?) years we have been blacklisting .py files in oVirt Node
(a slimmed down fedora/CentOS based appliance).

The removal caused some trouble:
- removing informations from drawbacks
- Making debugging - testing changes - very cumbersome
- Breaks "plugin" mechanisms of a surprising large number of tools

In our next release we will finally re-introduce .py files again:
Bug 1233106 - [RFE] Remove all kernel, firmware, and .py file blacklisting

I'd suggest to consider and not underestimate these factors.

- fabian

> I had to deal differently with the two versions as Python 3 handles
> source-less distributions in a different way than Python 2.
>
> Python 2 simply loads the *.pyc file when the *.py file is missing.
>
> Python 3 requires an additional step, as it puts the precompiled version
> under the __pycache__ directory, but it expects the file to be one level
> upper when the source file is missing:
>
> /foo/__pycache__/test.cpython-34.pyo -> /foo/test.pyc
>
>
> This patch reduces the used disk space by around 55 MB.
>
> Any comments?
>
> Thanks,
> Giuseppe
>
>
>
> >From 277e5fcdd6fbc2f3d51589a4065f2026f8becccd Mon Sep 17 00:00:00 2001
> From: Giuseppe Scrivano <gscrivan redhat com>
> Date: Mon, 30 Nov 2015 13:42:10 +0100
> Subject: [PATCH] treecompose-post.sh: delete any .py file
>
> and leave only the precompiled version.
>
> It frees around 55MB of disk space.
>
> Signed-off-by: Giuseppe Scrivano <gscrivan redhat com>
> ---
>  treecompose-post.sh | 37 +++++++++++++++++++++++++++++++++++++
>  1 file changed, 37 insertions(+)
>
> diff --git a/treecompose-post.sh b/treecompose-post.sh
> index 73b6573..39f1ba0 100755
> --- a/treecompose-post.sh
> +++ b/treecompose-post.sh
> @@ -8,3 +8,40 @@ find /usr/share/locale -mindepth  1 -maxdepth 1 -type d -not -name "${KEEPLANG}"
>  localedef --list-archive | grep -a -v ^"${KEEPLANG}" | xargs localedef --delete-from-archive
>  mv -f /usr/lib/locale/locale-archive /usr/lib/locale/locale-archive.tmpl
>  build-locale-archive
> +
> +# Compile all the files.
> +find /usr/lib*/python2.* -type d -exec python2 -OO -m compileall -l {} +
> +find /usr/lib*/python3.* -type d -exec python3 -OO -m compileall -l {} +
> +
> +# Here we treat Python 2 and Python 3 differently:
> +
> +# Python 2
> +# *.pyo files are basically *.pyc files, except that when the source
> +# file is missing Python will load only the .pyc file:
> +find /usr/lib*/python2.* -type f -name "*.pyo" | while read i
> +do
> +    destination_pyc=$(echo $i | sed -e's|pyo$|pyc|')
> +    rm -f $destination_pyc
> +    mv $i $destination_pyc
> +done
> +find /usr/lib*/python2.* -type f -name "*.py" | while read i
> +do
> +    test -f ${i}c && rm ${i}
> +done
> +
> +# Python 3 instead keeps the cache file under __pycache__, these files
> +# won't be loaded if the source file is missing, but: "In order to
> +# continue to support source-less distributions though, if the source
> +# file is missing, Python will import a lone pyc file if it lives
> +# where the source file would have been." (PEP-3147)
> +find /usr/lib*/python3.* -type f -name "*.py[co]" | while read i
> +do
> +    destination_pyc=$(echo $i | sed -e "s|/__pycache__/|/|" -e "s|\.cpython-3[0-9]*\.pyo|.pyc|")
> +    rm -f $destination_pyc
> +    mv $i $destination_pyc
> +done
> +find /usr/lib*/python3.* -type d -name "__pycache__" -exec rm -rf {} +
> +find /usr/lib*/python3.* -type f -name "*.py" | while read i
> +do
> +    test -f ${i}c && rm ${i}
> +done
> --
> 2.5.0
>



-- 
Fabian Deutsch <fdeutsch redhat com>
RHEV Hypervisor
Red Hat