OSDC 2010 presentation on cool stuff that can be done with Python Abstract Syntax Trees and the compile() function.
Demo Python compiler (incomplete and hacky!) can be found here: http://github.com/thomaslee/viking-poc
2. About Me <3
● Consultant for Shine Technologies
● Software consultancy with offices in Melbourne and Brisbane
● http://www.shinetech.com
● Python contributor
● try/except/finally
● compile() ASTs
● … and other bits and bobs.
● Would-be compiler geek.
3. Overview <3
● What are we discussing?
● Python Compiler Architecture
● Abstract Syntax Trees (ASTs)
● Abstract Syntax Trees in CPython
● Writing Compilers for the Python VM
● Benefits
● Demo?
4. What are we discussing? <3
● A bird's-eye view of bytecode compilers.
● Features introduced in Python 2.5 & 2.6.
● Abstract Syntax Trees and their use in Python.
● What they are.
● Why they're cool!
● Implement your own language in Python for the Python VM.
6. Abstract Syntax Trees <3
● Data structure.
● Constructed by a parser.
● Consumed by a code generator.
● In-memory, logical representation of a program.
● Syntax and bytecode independent.
● Contrast to parse trees.
7. Abstract Syntax Trees in CPython <3
● Python 2.5: Used internally by the Python compiler.
● API access from Python, but limited utility.
● Python 2.6: compile() ASTs to bytecode.
● Parse Python source to an AST.
● Analyze & manipulate ASTs in-memory.
– Optimisation
– Static analysis
● Generate and execute bytecode from an AST.
● What if you construct your own ASTs?
8. Writing Compilers for the Python VM <3
● Write a scanner (in Python!), generate tokens.
● Write a parser (in Python!), construct an AST.
● Use Python's ast module.
● Use compile() to construct a code object from the AST.
● Use exec/eval() to run the code object on the Python VM.
9. Benefits <3
● Write compilers in Python!
● ASTs more easily manipulated than generated source.
● Optimise code at the AST level.
● Use Python modules from within your language.
● OR Compile your language to .pyc files and call 'em from
Python.
11. About Viking <3
● Think of it as Python with braces...
● A front end for the Python bytecode compiler.
● Compiles source to an equivalent using Python's AST.
● Written over the course of a few hours.
● <400 LOPC (including spaces and comments).
● Quick, dirty, horrible, incomplete...
● … but surprisingly powerful as a demonstration.
