1. Over the past 1.5 years LinkedIn has been using {py}gradle on over 800 different projects successfully. Given
the success of these projects we are going to offer the plugin to the open source community. We plan to open
source {py}gradle in it’s current state, after LinkedIn-isms are removed with a 0 major version. When it’s open
sourced, we will also present the plan forward for 1.0 and beyond.
OPEN SOURCE
BUILDING A POLYGLOT SERVICE?
A Polyglot Service has a backend API server written in Python that uses gunicorn, Flask, and other internal
and external Python dependencies, it has a frontend server written in Ember.js that uses NPM under the hood,
it uses Python clients generated from Rest.li interface definition language (IDL) using a Java tool, and must
emit the proper metadata that our deployment system uses to configure, canary, and deploy our service.
How can we build a Polyglot Service?
generateJava
generatePython
buildJava
buildEmber
buildRestli
buildPython
build
uploadMetadata
publish
uploadArtifacts
packDeployable
runPytest
pipInstallDeps
makeDists
createVenv
Gradle is an open source modular build system. Gradle started its life building JVM-based code, and
gained additional traction when they moved to Android and C/C++ over the past few years. Gradle
gives developers a rich API to implement plugins: the primitive units that compose a Gradle build
system. A plugin developer is able to implement a DSL to model any process or language that they
wish, ultimately enabling us to build polyglot build systems. A plugin can be reused, mixed and
matched with other plugins, extended, configured, and more. Gradle’s plugin architecture lets us
build better build systems.
ABOUT GRADLE
BUILD.GRADLE
apply plugin: 'python'
apply plugin: 'python-api-server'
def insertFromFile = { filePath ->
// Read a Python requirements.txt file...
}
dependencies {
// Installation requirements from file
installRequires insertFromFile('requirements.txt')
// Installation requirements
installRequires 'pypi:Flask:0.0.0'
installRequires 'linkedin:utils:0.0.0'
// Test requirements
testsRequire 'pypi:mock:0.0.0'
// Setup requirements
setupRequires 'pypi:setuptools-git:0.0.0'
// Front end static resources requirements
resource project(path: ':ember-frontend', configuration:
'static')
}
- Dependency management for FREE!
- No need to emit metadata. The build system does that for me.
- Different languages work together.
Some cases where Gradle works better for version conflict resolution than the traditional Python build
workflow.
- Python picks up a new version on each rebuild. Gradle can reproduce the builds.
- Python gets version conflict for major version updates. Gradle picks up the highest.
- Python gets version conflict when one branch of the graph blacklists version as bad, but another
branch does not. Gradle reliably blacklists the bad version.
In complex dependency systems, we may
want to do what LinkedIn calls “semantic
version” testing. Upon a change to D, we
want to test its changes against C, F, and G.
If the application does not pin all its dependencies,
it can get into cascading version conflicts during
dependency updates (see below). In this example
we can see that C, F, and G all depend on D at dif-
ferent major versions. What should we do?
Dependency management happens on graphs, where nodes represent packages and edges represent ver-
sions or ranges of versions on other nodes. We care about dependency management because it gives us
the ability to reproduce builds and safely evolve software as it changes time over time.
Dependency management is challenging to get right, and solutions exist on a spectrum. On one side,
we can practice version pinning in which a dependency pins all direct and transitive dependencies. This
approach achieves reproducible builds, but makes safely evolving difficult. On the other side, we can
employ floating versions and only specify direct dependencies. This approach achieves fluidity in evolv-
ing software but makes reproducible builds impossible and doesn’t make evolution necessarily safe.
DEPENDENCY MANAGEMENT
A
B
E
D
C
G
F
A
B
E
D
C
G
F
A
B
E
D
C
G
F
ABOUT SEMANTIC VERSIONING
[
V 1 2 5Major Minor Patch
[
[
A {py}gradle project uses setuptools, vir-
tualenv, and pip to maintain a development
and build environment that makes use of ev-
erything that a Python developer is already fa-
miliar with: setup.py, setup.cfg, require-
ments.txt, tox.ini, and more. A {py}gradle
project looks and feels like a Python project
and produces backwards-compatible artifacts
that work with non-{py}gradle-based Python
projects.
HOW TO USE
$ tree
├── build.gradle
├── gradlew
├── backend-api
│ ├── build.gradle
│ ├── setup.cfg
│ ├── setup.py
│ ├── src
│ │ └── backend
│ │ └── __init__.py
│ ├── requirements.txt
│ └── test
│ └── test_backend.py
├── <subproject>
│ └── ...
└── settings.gradle
$ ./gradlew build
{py}gradle uses the power of Ivy for dependency
management. Then, it goes out of your way and
lets you work as a pure Python developer. After
the first build, the standard virtualenv develope-
ment workflow can be used.
BUILD SYSTEM COMPARISON
ABSTRACT
As our Python development team grew over the years and the number of packages and interdepen-
dencies grew from tens to hundreds to thousands, we found that we were struggling to maintain
our code base using the standard tooling that was available to us: setuptools, virtualenv, and pip.
At the same time, our Java development team converged on a new build system – Gradle – that al-
lowed them to write extendable build logic. We found that Gradle wasn’t just for Java, and we wrote
a powerful and idiomatic Python build system that reused much of the build logic as our Java
peers, without changing the way our Python developers work.
{py}gradle
Stephen Holsapple (sholsapp@linkedin.com, , sholsapp)
Zvezdan Petković (zpetkovic@linkedin.com, zvezdan)
Ethan Hall (ethall@linkedin.com, ethankhall)
Loren Carvalho (lcarvalho@linkedin.com, sixninetynine)
Gradle Maven Ant Setuptools Buildout Pants CMake
Powerful
Dependency Resolution
API
Decoupled Dependency
Metadata
Ivy-based
Metadata
Pluggable
Scriptable
Human-readable
Build File
Natively Polyglot
Generic Artifact
Hosting Support
(*) to a certain extent, or not easily
*
*
*
Gradle allows us to have multiple projects, each in a separate directory, for each language used. Each project
uses the gradle plugin(s) specific to that language. The artifacts produced in a build of each project can be
used in another project. The ordering of build tasks gives us the full control of the overall service build. For
advanced use, one can exclude tasks from running, or use multiple languages in the same project.
