Re: [atomic-devel] [fedora-atomic f23] remove Python source files

[Nick Coghlan <ncoghlan gmail com>]

On 2 December 2015 at 22:54, Giuseppe Scrivano <gscrivan redhat com> wrote:
> Fabian Deutsch <fdeutsch redhat com> writes:
>
>> On Tue, Dec 1, 2015 at 1:36 PM, Giuseppe Scrivano <gscrivan redhat com> wrote:
>>> Hi,
>>>
>>> I was experimenting with reducing the size of the Atomic Host image and
>>> it seems that a lot of space is used by Python source files.
>>
>> For many (6?) years we have been blacklisting .py files in oVirt Node
>> (a slimmed down fedora/CentOS based appliance).
>>
>> The removal caused some trouble:
>> - removing informations from drawbacks
>> - Making debugging - testing changes - very cumbersome
>> - Breaks "plugin" mechanisms of a surprising large number of tools
>>
>> In our next release we will finally re-introduce .py files again:
>> Bug 1233106 - [RFE] Remove all kernel, firmware, and .py file blacklisting
>>
>> I'd suggest to consider and not underestimate these factors.
>
> thanks for the feedbacks.  Were .pyc files used instead of the .py
> version in oVirt?  IIUIC, pyc files maintain the same information as the
> original source file, while .pyo files are a optimized version that has
> not all the original content.  Using the optimized version, I could
> strip 55 MB of space from the Fedora Atomic image, if I use .pyc then
> the reduction is only 27 MB.

Even pyc files still drop a lot of info (e.g. comments, the original
source text), just not as much as -OO does (since that drops
docstrings as well, while both -O and -OO drop assert statements and
"if __debug__:" blocks)

> Nick, would .pyc be a safe alternative to .py files or are there still
> cases where it doesn't work in the same way?

There are still cases where it doesn't work the same way. For the
first two points Fabian mentions, tracebacks read the original source
file to figure out the context of an error, so leaving the source
files out means less informative messages in error logs and error
aggregation tools like Sentry, and you also lose the ability to do
in-place editing while debugging a developmental tree.

For the plugin breakage, any project that does its own "read, compile,
exec" processing could encounter problems if the source files are
missing - the import system and runtime provided APIs like execfile()
and runpy.run_path() are designed to handle source-or-bytecode files,
but most other systems only handle the source file case (since
bytecode-only distribution is a relatively rare scenario).

Something that might be interesting is a "strip-and-squash" tool that
post-processes any existing image to remove "unneeded" files, and then
squashes all the layers together to minimise the total image
footprint. However, using such a tool effectively could be difficult,
since you'd *also* lose any benefits of layer sharing on execution
nodes.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan gmail com   |   Brisbane, Australia