Python+gradle
•
5 j'aime•7,155 vues
A presentation I gave at GradleSummit 2015 that discusses building Python code bases with Gradle.
Recommandé
Contenu connexe
Tendances
Tendances (20)
En vedette
Similaire à Python+gradle
Similaire à Python+gradle (20)
Dernier
Dernier (20)
Python+gradle
- 2. Python
- 3. What is setuptools? #!/usr/bin/env python from setuptools import setup setup( description=README, author='Stephen Holsapple', author_email='sholsapp@linkedin.com', packages=['foo'], install_requires=[], )
- 4. What is setuptools? Everything. • To Python, setuptools is everything. – Is the core API that most tools nowadays use. – Is the brains behind: • Build • Packaging • Metadata discovery • Dependency management
- 5. Building w/ setuptools • Tools like virtualenv and pip make life preHy good.
- 6. More about virtualenv $ virtualenv my-env $ source my-env/bin/activate (my-env)$ python Python 2.7.5 >>> import sys >>> sys.path ['./my-env/lib/python2.7/site- packages'] (my-env) $ deactivate $
- 7. More about virtualenv • A command line tool useful for: – CreaJng isolated Python environments. • Really, a plaLorm to solve dependency and version problems.
- 8. More about pip • A command line tool useful for: – Finding dependencies – Installing dependencies – ArJfact management – Caching • Really, an implementaJon of the setuptools programming interface.
- 9. More about pip (my-env)$ pip install requests Downloading/unpacking requests Downloading requests-2.7.0-py2.py3-none- any.whl (470kB): 470kB downloaded Installing collected packages: requests Successfully installed requests Cleaning up...
- 10. Building w/ setuptools (my-env)$ pip install requests (my-env)$ python >>> import requests >>> ...
- 11. Packaging w/ setuptools • Tools like setuptools, pip, pex make life preHy good. – Source distribuJons – Binary distribuJons – Deployable distribuJon
- 12. Metadata w/ setuptools • Doesn't provide great ways to: – Reason about builds before/aOer – Reason about arJfacts programaJcally • Does expose programming interfaces to integrate with exisJng metadata systems. • Requires us to evaluate Python to query about metadata.
- 13. Metadata w/ setuptools (my-env)$ pip show requests --- Name: requests Version: 2.7.0 Location: my-env/lib/python2.7/site-packages Requires: (my-env)$ pip freeze requests==2.7.0 wsgiref==0.1.2
- 14. Dependencies w/ setuptools #!/usr/bin/env python import setuptools setuptools.setup(, # ... install_requires=[ "requests>=2.6", "pyOpenSSL==0.15", "urllib3", ], # ... )
- 15. Dependencies w/ pip $ cat requirements.txt requests>=2.6 pyOpenSSL==0.15 urllib3
- 16. [pip/issues/988] “Requirements files are used to force pip to properly resolve dependencies. As it is now, pip doesn't have true dependency resolu=on, but instead simply uses the first specifica=on it finds for a project.”
- 17. pkg_resources is greedy • Simply use the first specificaJon found for a dependency. • Ask developers to pin all of their dependencies. – pip freeze > requirements.txt
- 19. [DistributionNotFound] pip.exceptions.DistributionNotFound: No distributions at all found for requests
- 23. Future Work • Sincere thanks to Donald StuS et al. on PyPA working to improve state of Python packaging. • Python is rapidly changing for the beHer.
- 24. Python • Report card – Ecosystem (great) – Build System (good) – Packaging (good) – Metadata (okay) – Dependency Management (bad)
- 25. Scaling Products • Builds • ArJfacts – Downloading – Caching – Metadata • Dependencies – Dependency ResoluJon – Conflict ResoluJon • TesJng
- 26. Building BeHer Building Systems • We have a core set of great tools to work with: – virtualenv – setuptools – pip – pex
- 27. A polite tool • The pip tool is polite because: – Allows for customizaJon or disabling of setuptools and pip features: • --index-url • --extra-index-url • --no-deps – Allows us to use local repositories. – Allows us to solve our problems our way.
- 28. #!/bin/build.sh vinit && source activate python setup.py install py.test python setup.py sdist
- 29. #!/bin/build.sh vinit && source activate python setup.py install py.test python setup.py sdist pip wheel .
- 30. #!/bin/build.sh vinit && source activate for dep in $( cache/ ); do pip install --no-deps dep done python setup.py install py.test python setup.py sdist
- 31. #!/bin/build.sh vinit && source activate for dep in $( ???/ ); do pip install --no-deps dep done python setup.py install py.test python setup.py sdist
- 32. Gradle
- 33. #!/bin/build.sh vinit && source activate for dep in $( gradle/ ); do pip install --no-deps dep done python setup.py install py.test python setup.py sdist
- 34. Philosophy Let’s let Python do what Python is good at and let Gradle do what Gradle is good at.
- 36. Gradle as a Build Orchestrator • First, resolve my dependencies • Second, run my build – pip install --no-deps dep • Third, run my package – python setup.py sdist • Last, run my publish
- 37. build.gradle apply plugin: 'python-venv' apply plugin: 'python-sdist' sdist { sourceDirectory 'src' }
- 38. build.gradle apply plugin: 'python-venv' apply plugin: 'python-pex' sdist { sourceDirectory 'src' }
- 39. Enhancing Python • Represent arJfacts and dependencies in a way Gradle can understand. • Integrate exisJng metadata with setuptools using custom distribuJon class.
- 40. RepresenJng Dependencies <ivy-module version="1.0"> <info module="cryptography" revision="0.8.2" /> <publications> <artifact name="cryptography" conf="default"/> </publications> <dependencies> <dependency name="pyasn1" rev="0.1.7" /> <dependency name="six" rev="1.9.0" /> <dependency name="setuptools" rev="3.4.4" /> <dependency name="enum34" rev="1.0.4" /> <dependency name="cffi" rev="0.9.2" /> </dependencies> </ivy-module>
- 41. The Cheese Shop • Python Package Index (PyPI) – Community hosts packages here – Cannot modify packages here • Internal Python Package Index – ArJfactory – Add Ivy metadata for packages
- 42. IntegraJng Metadata • How to marry setuptools with custom metadata systems. – Product names – Product versions – SpecificaJon files – Dependencies
- 43. IntegraJng Metadata #!/usr/bin/env python2.6 import setuptools from distgradle import GradleDistribution setuptools.setup( distclass=GradleDistribution, package_dir={'': 'src'}, packages=setuptools.find_packages('src'), include_package_data=True, namespace_packages=['linkedin'], )
- 44. Gradle Infrastructure • python-product – Your enterprise build logic, covering everything from your resoluJon strategies, your repositories, your metadata, and more. – You could write a simple plugin that simply defers to setuptools, if you want.
- 45. Gradle Plugins • python-venv – Builds a local environment • python-sdist – Build a source distribuJon using setuptools. • python-wheel – Build a wheel distribuJon using setuptools. • python-pex – Build a pex distribuJon using pex.
- 46. python-venv • Under the hood: – CreaJng a virtual environment. – Installing build requirements. – Installing your resolved dependencies. – Installing your project. – Running tests – Packaging arJfacts
- 47. python-sdist • Under the hood: – Applying python-venv – Invoking ./setup.py sdist
- 48. python-wheel • Under the hood: – Applying python-venv – Invoking pip wheel • Making Python's wheel file compaJble with Ivy is hard.
- 49. python-pex • Under the hood: – Applying python-venv – Invoking pex • Prepare to baHle with pex's dependency resolver.
- 50. python-??? • ExisJng plugins easy to add now. • AddiJonal plugins easy to add later. • Mix and match any way we like. – We might want a source and a wheel distribuJon.
- 51. Python+Gradle • Report card – Ecosystem (great) – Build System (good) – Packaging (good) – Metadata (good) – Dependency Management (great)
- 52. What is Gradle doing? • Reading specificaJon files • Dependency and conflict resoluJon • Downloading and caching • OrchestraJng creaJon of Python arJfacts • CreaJng build metadata • Uploading arJfacts • PyPI
- 53. What is Gradle not doing? • Changing how you develop with Python • Changing how Python packages arJfacts • Changing Python arJfacts themselves • Pushing Gradle-‐isms into Python
- 54. Major Wins • Dependency Management • Plugin Architecture • Reusable Logic • MulJ-‐language Builds
- 55. MulJ-‐language Builds • One build system to rule them all. – Java – Scala – C/C++ – Javascript – Python • Products can contain different languages
- 56. Why Python? • Why was Python so easy to build with Gradle? – Can decouple dep. mgmt. from setuptools – Can decouple dep. mgmt. from PyPI – Can represent dependencies with Ivy • Other languages like Python can also easily use Gradle
- 57. QuesJons?