The Python Podcast.__init__
The Python Podcast.__init__
#technology

The podcast about Python and the people who make it great

Kay Hayen on Nuitka

October 23, 2015 01:34:35 109.38 MB Downloads: 0

Visit our site to listen to past episodes, support the show, and sign up for our mailing list.

Summary

Kay Hayen is a systems engineer from Germany who has dedicated his spare time to the creation of Nuitka, a library that will compile your Python project to C++. In this episode we talked to Kay about what inspired him to create the project, how it operates, and some of the challenges he has faced. It is a very interesting project and it has the potential to let you run your Python code in a whole new way!

Brief Introduction

  • Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.
  • Subscribe on iTunes, Stitcher, TuneIn or RSS
  • Follow us on Twitter or Google+
  • Give us feedback! Leave a review on iTunes, Tweet to us, send us an email, leave us a message on Google+, or leave a comment on our show notes
  • I would like to thank everyone who has donated to the show. Your contributions help us make the show sustainable. For details on how to support the show you can visit our site at pythonpodcast.com
  • I would also like to thank Hired, a job marketplace for developers, for sponsoring this episode of Podcast.__init__. Use the link hired.com/podcastinit to double your signing bonus. Linode has also sponsored this episode and you can get a $10 credit using the link linode.com/podcastinit to try out their fast and reliable linux virtual servers.
  • We are recording today on October 6th, 2015 and your hosts as usual are Tobias Macey and Chris Patti
  • Today we are interviewing Kay Hayen about the Nuitka project
Hired LogoOn Hired software engineers & designers can get 5+ interview requests in a week and each offer has salary and equity upfront. With full time and contract opportunities available, users can view the offers and accept or reject them before talking to any company. Work with over 2,500 companies from startups to large public companies hailing from 12 major tech hubs in North America and Europe. Hired is totally free for users and If you get a job you’ll get a $2,000 “thank you” bonus. If you use our special link to signup, then that bonus will double to $4,000 when you accept a job. If you’re not looking for a job but know someone who is, you can refer them to Hired and get a $1,337 bonus when they accept a job. Linode Sponsor BannerUse the promo code podcastinit10 to get a $10 credit when you sign up!

Interview with Kay Hayen

  • Introductions
    • German, family with 2 kids, one cat
    • Working in ATM (Air Traffic Management), tracker product
    • Systems Engineer
    • Nuitka as a hobbyist
  • How did you get introduced to Python?
    • Once was Perl “Guru”.
    • Python was getting a lot of positive press
    • Team decision to want to use readable stuff
    • CPAN was still more complete, but Python was making inroads
  • Can you describe how to pronounce the name of your project?
    • Wife Anna, Russian, Annuitka -> Nuitka
  • Can you briefly describe what Nuitka is and what your motivation was for creating it?
    • I was thinking a fully integrated and compatible compiler should be possible.
    • Why is nobody doing it?
    • I can do it.
    • I am doing it.
    • Take Python beyond current use cases.
      • Everbody currently using Python needs no compiler, or wouldn’t use it
      • Less need for time consuming C++/Python hybrid coding
      • Simple code should compile to fast code by default
      • Complex code should still work
  • On the project web site it says that Nuitka does a lot of clever things after being fed a Python project. Can you provide some details as to what some of that cleverness is?
    • Re-formulations of Python into simpler Python
      • No “class”
      • No “assert”
      • No complex assignments
    • SSA tracing
      • Attaching uses to assignments properly
        • Despite try/finally
        • Loops
      • Avoids checks for known defined/undefined values
    • Function inlining (coming)
    • Constant propagation
    • Closure variable removal
  • What is libpython and how is it used in both Nuitka and CPython?
    • Core of the Python interpreter
    • With Python VM and C interface
    • Nuitka can fall back to it
    • Avoiding it as often as we can, key to performance
  • Is there any way to provide hints to Nuitka to generate more optimized output?
    • Nuitka is yet to make a difference based on type information
    • Not yet there, but coming soonish. SSA was pre-requisite
    • PEP 484 will be unreliable type information, mostly useless
    • I want type hints that are checked at Python run time
  • What are some of the biggest challenges in generating statically compiled code from a language as dynamic as Python?
    • Python is compiled to .pyc files
    • Compatible Frame stack, cached
    • Exception handling of Python is terrible
    • CPython type system designed to be extensible
      • Extension types for functions, bound/unbound methods, generators, etc.
    • Many details to get right
  • Are there any particular Python constructs that Nuitka is unable to translate and as a corollary to that is the compilation step lossy at all or do you have some way of ensuring that the functionality of the program remains unaltered?
    • Big point, no price attached
    • Except for not having bytecode, there is nothing missing
    • No pdb support
    • Edit / run cycle is not accelerated
    • That said: PyQt (integrated), PySide (available, unmerged), wxPython (available, maybe merged) needed patches to take compiled function/method objects for function objects too
  • Are there any particular types of programs that benefit the most from Nuitka’s compilation?
    • Bindings with ctypes of cffi compile into zero overhead C calls (planned)
    • Scientific programs are the most obvious goal (float type inference)
    • CPU bound or low latency programs
  • Is it possible to feed an entire project with multiple modules into Nuitka all at once or is the standard use to perform compilation one source file or submodule at a time?
    • You give it the main program and it recurses imports according to “PYTHONPATH”
    • nuitka –recurse-all “/usr/bin/hg” supposed to work
    • Might have to give directories with program plug-ins
  • I’m curious about what led you to choose compilation to C++ for Nuitka rather than making Nuitka an LLVM back end like Numba?
    • When I started Nuitka, I was using C++0x and variadic templates
    • Wanted to make a proof of concept that compatibility and integration is feasible
    • From there, code generation got less high level to goto ridden C
  • How does Nuitka compare to projects like Numba or Cython?
    • Graceful degradation goal
    • Complete compatibility with Python whole stack
  • How does Nuitka compare to PyPy? – Kay
    • PyPy is the coolest project ever
    • Pure Python goals shared
  • How can users evaluate the performance of Nuitka – Kay
    • They currently cannot
    • Developing a tool to compare CPython and Nuitka runs
      • Based on vmprof from PyPy people
      • Identify parts of program where Nuitka is slower
      • Links to source code
    • To be done, help needed.
    • Nuitka is only starting to get to serious performance
      • Compatibility is such a high bar to take
      • C++ to C took a year (avoiding C++ exceptions)
      • SSA literally took forever

Picks

Keep In Touch

The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA