In an environment where software is constantly changing and new versions of a library should be distributed across thousands of projects, how can you know that your projects are using the right version of a given dependency? What if a OSS library introduces a security vulnerability and you need to make sure that no one is using it in your company? What if an internal library introduces a bad change and you need everyone to upgrade/downgrade? Automating these for hundreds or thousands of engineers is crucial. At Netflix, engineers are not immune to the cost of dependency updates. Library owners publish new versions of their code without a comprehensive understanding of the organizational impact. Application owners ingest new library versions that can fail in obvious or subtle ways, leading to decreased confidence and slower organizational velocity. But these are problems we understand, and tooling can help. After years of evolving our build, we've developed a few conceptual models of dependency management. Dependency management is hard, and in all cases there are compromises, and you the build owner should be conscious of which choices you're making and what else is available. On this session, I'll introduce some tools we've developed at Netflix which attack dependency issues on a large scale to make it easier for every JVM engineer at Netflix and how we react to the scenarios mentioned above.

1. Dependency
Management in
a Complex World
September 28, 2022

2. Roberto Perez Alcolea
Senior Software Engineer @ Net
fl
ix
JVM Ecosystem Team
rperezalcolea@net
fl
ix.com
@rpalcolea
Dependency Management in a Complex World

3. Life of a
developer
Dependency Management in a Complex World

4. Dependency Management in a Complex World
Tuesday Afternoon

5. Dependency Management in a Complex World
Wednesday
Morning

6. Dependency Management in a Complex World

7. Dependency Management in a Complex World
Are we in
dependency hell?

8. Dependency Management in a Complex World
Most likely, yes!
🔥😱 🔥😱 🔥😱

9. Dependency Management in a Complex World
This is just one
project… what if…

10. Dependency Management in a Complex World
Your dependency graph looks like this

11. Dependency Management in a Complex World
— Multi/Many repositories setup
(~3k)
— Binary integration (JARs)
— Microservices with fat clients
— Hundreds/Thousands of
Engineers
— Thousands of builds per day
— Thousands of artifacts generated
per day

12. Dependency Management in a Complex World

13. Dependency Management in a Complex World
How to keep up with all these changes?
Build often, release often!

14. Dependency Management in a Complex World
Yes, but this comes with
some costs 💸 💸

15. Common
Problems
in dependency
Management
(That we have seen)
Dependency Management in a Complex World

16. Compatibility issues
— 2 components depending on the
same module but with different,
incompatible, APIs
— Multiple libraries providing the same
feature (logger APIs)
— Dealing with incompatible versions
of a runtime (e.g: Scala 2.11 vs
Scala 2.12)
— Misaligned dependencies of a
component (e.g: Jackson Databind
2.9.0 with Jackson Core 2.9.4)
— Intentional and unintentional
Breaking API changes in libraries
— Java version
Dependency Management in a Complex World

17. Build / Dependency graph Hygiene
— Build does not declares the
dependencies it needs
— Build declares dependencies that
doesn’t need
Dependency Management in a Complex World

18. Publishers lack visibility
— Who uses my library?
— Who will I break if I release a
change?
— Who is exercising/using a
particular method I want to
deprecate and/or replace?
Dependency Management in a Complex World

19. Semver is not enough
— Which version should I use?
— Pinning versions is technical debt
— In a dynamic environment where
projects get changes constantly,
moving them between majors can
be challenging
Dependency Management in a Complex World

20. Constant change
— How can we migrate a project
from dependency A to B?
— Can it be done only by
dependency management
machinery rules?
— Do we need to ship code changes
also?
— Can we do this on behalf of the
users?
Dependency Management in a Complex World

21. Security Vulnerabilities
— Prevent projects from using a
given version of a dependency
— Reject a particular module from
the dependency graph
— Let users know that they might be
in danger
Dependency Management in a Complex World

22. Ownership
When my project is broken…
— Who owns the module/library
breaking my project?
— How can I contact them?
— Is there a support channel to
reach out?
— Can I open an incident directly
with them?
Dependency Management in a Complex World

23. Are there any
solutions?
Dependency Management in a Complex World

24. Dependency Management in a Complex World
Sure, there are 😃😃!

25. Dependency Management in a Complex World
Dependency Resolution
— Resolution Rules Plugin
— Resolution Rules
— Gradle Plugins with
Resolution Strategies
— Gradle Module Metadata

26. Dependency Management in a Complex World
Build Hygiene
— Nebula Lint plugin
— Dependency Analysis
Gradle Plugin

27. Dependency Management in a Complex World
Who uses my library?
Collection of services and UIs
that enable artifact
observability and the ability to
effect change in the Netflix
ecosystem.
Astrid to the rescue ⛑!

28. Dependency Management in a Complex World
Who uses my library?

29. Dependency Management in a Complex World
Who uses my library?

30. Dependency Management in a Complex World
Who uses my library?

31. Dependency Management in a Complex World
But what about
security
vulnerabilities?

32. Dependency Management in a Complex World
CVE + Astrid + Nebula
Practical Approach to Automate the Discovery &
Eradication of Open-Source Software Vulnerabilities
Aladdin Almubayed | 2019 ▶

33. Dependency Management in a Complex World
Astrid Campaign

34. Dependency Management in a Complex World
And you said
constant change…

35. Dependency Management in a Complex World
Automated dependencies and
lint updates

36. Dependency Management in a Complex World
Automated dependencies and
lint updates

37. Dependency Management in a Complex World
Automated dependencies and
lint updates

38. Dependency Management in a Complex World
Managed Delivery

39. Dependency Management in a Complex World
Paved Path vs non-Paved Path (log4j edition)
~minutes-hour vs ~hours-days

40. Dependency Management in a Complex World
Paved Path vs non-Paved Path (log4j edition)

41. Dependency Management in a Complex World
Who owns this
library?

42. Dependency Management in a Complex World
Software Ownership

43. Dependency Management in a Complex World
Unfortunately, some of these
things are not OSS Today.
However there are OSS and
Enterprise options…

44. Dependency Management in a Complex World
Update dependencies
Security
Constant change
Code usage
Refactoring
Comby
Ownership

45. What’s next for
us?
Dependency Management in a Complex World

46. Publisher feedback
— When a new version of a library is
published, build all the consumers
and verify if the change breaks
them.
• This requires proper verification
steps on each project
— Use Sourcegraph Code Intelligence
capabilities to understand when a
modified/removed code is being
used by consumers or not
Dependency Management in a Complex World

47. Dependency Management in a Complex World
Distributed Refactoring

48. Dependency Management in a Complex World
Library changesets with
publications

49. Last thoughts…
— Dependency management is hard
— Dependency hell is inevitable, let’s
try to reduce the pain
— Build often, release often. Avoid
conflict resolution by limiting
version skew
— Producers and consumers play an
important role
— Moving projects from A to B is
hard but there are options out
there
Dependency Management in a Complex World

50. Questions?
Dependency Management in a Complex World

51. Thank
You.
Roberto Perez Alcolea
rperezalcolea@net
fl
ix.com