Software architecture is often overlooked: pushed by deadlines and hurry, we tend to make our applications a giant ball of mud. Here is where the unix philosophy comes to help.
In this talk I will give an overview of the unix philosophy and I will explain how I have applied it to django development for the OpenWISP and NetJSON projects (a set of software modules and standard formats that can be used to deploy wireless networks and implement a public wifi service) with encouraging results. In this talk I will cover the benefits and the downside of such approach, showing example implementations that are being tested in the real world.
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Applying the Unix Philosophy to Django projects: a report from the real world
1. Unix Philosophy & Django projects
Applying the Unix Philosophy to
Django projects
a report from the real world
2. Who am I?
Federico Capoano (A.K.A. Nemesis)
OpenWISP core developer
Working on public wif at Cineca since 2012
3.
4. What is OpenWISP?
OpenWISP is a set of software modules that can be
used to deploy and manage wireless networks
(public wif, backbone, mesh networks)
5. OpenWISP 1
OpenWISP 1: started in 2008-2016
Focused on public wif for Italian cities
Fast growth of features to accommodate requirements
12. 1. "Giant ball of mud" problems
Many features in a single codebase
Hard to add new features in a clean way
Hard to maintain over time (eg: upgrade
dependencies, fx bugs)
13. 2. Inflexibility
Hard to extend without changing the core
Hard to adapt it and reuse it in different contexts
Specialized features available in the
codebase are not easily reusable
14. 3. Little or no contributions
Complexity scared away
potential external contributors
We only got low quality patches
which were never merged
15. 4. Costs and risk of death
Increased costs to keep the project alive
Many years of work risked to fade into oblivion
16. 5. Duplication of efforts
Little reusability means:
reinventing the wheel
no ecosystem of interoperable tools
26. 3. Change tolerant
Born during the digital revolution 1970-2010
It kept pace with the dramatic
evolution of technology
27. The Unix Philosophy in short
What are the main concepts
of the Unix philosophy?
28. "Do one thing and do it well"
● Break down big problems in smaller problems
29. "Do one thing and do it well"
● Break down big problems in smaller problems
● Write simple programs that solve one problem well
30. "Do one thing and do it well"
● Break down big problems in smaller problems
● Write simple programs that solve one problem well
● Write programs to work together
31. "Do one thing and do it well"
● Break down big problems in smaller problems
● Write simple programs that solve one problem well
● Write programs to work together
● Value simplicity and maintainability over
performance and complexity
32. What advantages does it bring?
What are the main advantages
of applying the Unix Philosophy?
37. 5. Agility
A simple program will be more easily
readapted to new requirements
38. 6. Reuse
Being easier to use, understand, read and maintain
it will be readapted to work in different contexts
and will attract more contributors
39. 7. Ecosystem
Many such programs form an ecosystem of
interoperable tools
(like basic lego building blocks)
and
Allow to solve complex problems faster
41. The Unix Philosophy in detail
The 17 rules of the Unix Philosophy
From "The art of unix programming"
by Eric Raymond
42. 1. Rule of Modularity
Write simple parts connected
by clean interfaces
source
43. 2. Rule of Clarity
Clarity is better than cleverness
source
44. 3. Rule of Composition
Design programs to be connected
to other programs
source
45. 4. Rule of Separation
Separate policy from mechanism;
separate interfaces from engines
source
46. 5. Rule of Simplicity
Design for simplicity;
add complexity only where you must
source
47. 6. Rule of Parsimony
Write a big program only when it is clear by
demonstration that nothing else will do
source
48. 7. Rule of Transparency
Design for visibility to make inspection
and debugging easier
source
49. 8. Rule of Robustness
Software is said to be robust when it performs well
under unexpected conditions which stress the
designer's assumptions, as well as under normal
conditions
source
50. 9. Rule of Representation
Fold knowledge into data so program logic
can be stupid and robust
source
51. 10. Rule of Least Surprise
In interface design, always do
the least surprising thing
source
52. 11. Rule of Silence
When a program has nothing surprising
to say, it should say nothing
source
53. 12. Rule of Repair
When you must fail, fail noisily
and as soon as possible
source
54. 13. Rule of Economy
Programmer time is expensive; conserve it
in preference to machine time
source
55. 14. Rule of Generation
Avoid hand-hacking; write programs to
write programs when you can
source
56. 15. Rule of Optimization
Prototype before polishing;
get it working before you optimize it
source
57. 16. Rule of Diversity
Distrust all claims for “one true way”
source
58. 17. Rule of Extensibility
Design for the future, because it will
be here sooner than you think
source
59.
60. How to apply it to Django?
How to apply the Unix philosophy
to django projects?
Let's see some real world
examples from OpenWISP 2
61. 1. Rule of Modularity
● Develop main features as reusable django apps
62. 1. Rule of Modularity
● Develop main features as reusable django apps
● One django app for each group of related features
63. 1. Rule of Modularity
● Develop main features as reusable django apps
● One django app for each group of related features
● Document public API
64. 1. Rule of Modularity
● Develop main features as reusable django apps
● One django app for each group of related features
● Document its public API
● Include a license and a changelog
65. 1. Rule of Modularity
● Develop main features as reusable django apps
● One django app for each group of related features
● Document its public API
● Include a license and a changelog
● Publish your app on pypi and djangopackages.org
66. 1. Real world examples
● Confgurations of routers and VPNs:
django-netjsonconfg
67. 1. Real world examples
● Confgurations of routers and VPNs:
django-netjsonconfg
● PKI management (x509 certifcates): django-x509
68. 1. Real world examples
● Confgurations of routers and VPNs:
django-netjsonconfg
● PKI management (x509 certifcates): django-x509
● Multi-tenancy: contributed to django-organizations
reuse existing projects when possible!
69. 2. Rule of Clarity
Explicit is better than implicit
This is already a widely accepted concept in
the python world
70. 3. Rule of Composition
Combine, extend and customize django
reusable apps in your fnal django project
71. 3. Real world examples
● openwisp-users: extends django-organizations
72. 3. Real world examples
● openwisp-users: extends django-organizations
● openwisp-controller: depends on openwisp-users
and extends django-netjsonconfg and django-x509
73. 3. Real world examples
● openwisp-users: extends django-organizations
● openwisp-controller: depends on openwisp-users
and extends django-netjsonconfg and django-x509
● Final result handled by ansible-openwisp2 in an
(almost) transparent manner
74.
75. 4. Rule of Separation
● implement mechanisms as libraries
76. 4. Rule of Separation
● implement mechanisms as libraries
● implement policy as a confguration
77. 4. Rule of Separation
● implement mechanisms as libraries
● implement policy as a confguration
● make these libraries highly confgurable
78. 4. Rule of Separation
● implement mechanisms as libraries
● implement policy as a confguration
● make these libraries highly confgurable
● make these libraries work with data
79. 4. Rule of Separation
● implement mechanisms as libraries
● implement policy as a confguration
● make these libraries highly confgurable
● make these libraries work with data
● clearly defne input and output
80. 4. Real world examples
● netjsonconfg: python library for generating
router & VPN confgurations from NetJSON
objects
81. 4. Real world examples
● netjsonconfg: python library for generating
router & VPN confgurations from NetJSON
objects
● django-netjsonconfg: web interface that uses
netjsonconfg under the hood
84. 5. Rule of Simplicity
● start with very basic features
85. 5. Rule of Simplicity
● start with very basic features
● release your project early
(even if you feel it's incomplete)
86. 5. Rule of Simplicity
● start with very basic features
● release your project early
(even if you feel it's incomplete)
● add one feature at time as your understanding grows
87. 5. Rule of Simplicity
● start with very basic features
● release your project early
(even if you feel it's incomplete)
● add one feature at time as your understanding grows
● add complexity only when necessary
89. 5. Real world examples
as of April 2017:
● 24 releases of netjsonconfg
90. 5. Real world examples
as of April 2017:
● 24 releases of netjsonconfg
● 25 releases of django-netjsonconfg
91. 5. Real world examples
as of April 2017:
● 24 releases of netjsonconfg
● 25 releases of django-netjsonconfg
● 5 releases of django-x509
92. 5. Real world examples
as of April 2017:
● 24 releases of netjsonconfg
● 25 releases of django-netjsonconfg
● 5 releases of django-x509
OpenWISP 2 still lacks some features of OpenWISP 1
93. 6. Rule of Parsimony
Prefer creating new reusable apps
when adding big features
unless doing this complicates things a lot
with no real advantages
94. 6. Real world examples
PKI management in new app: django-x509
VPN confgurations added to existing app:
django-netjsonconfg
95. 7. Rule of Transparency
Log unexpected events using
the python logging facility
96. Logging of bad requests in django-netjsonconfg
import logging
logger = logging.getLogger(__name__)
def invalid_response(request, error, status):
logger.warning(error, extra={'request':
request, 'stack': True})
return ControllerResponse(error,
status=status)
97. 7. Real world examples
Provide good default logging
Provide support for sentry
Take a look at a real example
98. 8. Rule of Robustness
Add constraints to your reusable django apps only
when necessary:
● avoid very strict validation rules
● provide confgurable settings
99. 8. Real world examples
● settings in django-netjsonconfg
● settings in django-x509
100. 9. Rule of Representation
Fold complex information in data structures
Process these data structures with algorithms
Try to make algorithms framework-agnostic
(if possible)
101. 9. Real world examples
in OpenWISP 2, confguration of routers is
implemented as a single text feld formatted as
NetJSON
(in OpenWISP 1 each available confguration had its
own database table and model,
very hard to maintain and evolve)
105. 10. Rule of Least Surprise
This has become a widely accepted
concept in the IT industry
106. 11. Rule of Silence
Just don't make your apps annoying
It's not that hard :-P
107. 12. Rule of Repair
Errors should never pass silently
Unless explicitly silenced
(from the zen of python)
Let's add: fail fast, noisily and early
108. from django.core.exceptions import ImproperlyConfigured
from .settings import REGISTRATION_ENABLED, SHARED_SECRET
if REGISTRATION_ENABLED and not SHARED_SECRET:
msg = 'NETJSONCONFIG_SHARED_SECRET not set!'
raise ImproperlyConfigured(msg)
Full real world example available in
django-netjsonconfg
109. 13. Rule of Economy
When Unix was born this was a radical idea:
assembler was the norm, C was considered a higher
level language
Python is a consequence of the success of that
radical idea
Embrace this concept
110. 14. Rule of Generation
When you fnd yourself writing lots of boilerplate
code, try to use
meta-programming or code generators
117. 17. Extensibility: abstract models
● Provide abstract models in your
most important modules
● Store these in a python fle which does not import
concrete models
(otherwise other django apps won't be able to import them)
120. 17. Extensibility: base admin
● Provide base admin classes in your main modules
(avoid duplication)
121. 17. Extensibility: base admin
● Provide base admin classes in your main modules
(avoid duplication)
● Store them in a python fle which does not import
concrete models
(otherwise other django-apps won't be able to import them)
123. 17. Extensibility: base views
● If your reusable django app has views, provide
generic views that can be extended
● Store them in a python fle which does not import
concrete models
(otherwise other django-apps won't be able to import them)
125. 17. Extensibility: reusable urls
If your app provides views that can be extended,
third party apps will have to redefne their URLs
You may want to avoid this required duplication by
providing a mechanism to import urls
126. 17. Extensibility: reusable urls
reusable urls in django-netjsonconfg
example usage:
from django_netjsonconfig.utils import get_controller_urls
from . import views # customized views
# creates new url patterns hooked to customized views
urlpatterns = get_controller_urls(views)
127. 17. Extensibility: AppConfig
If your reusable django app relies on signal
connection for some features,
provide a base AppConfig class
128. 17. Extensibility: AppConfig
base AppConfig class in django-netjsonconfg
from django_netjsonconfig.apps import OpenWispAppConfig
class MyOwnApp(OpenWispAppConfig):
name = 'yourapp.config'
label = 'config'
def __setmodels__(self):
# these are your custom models
from .models import Config, VpnClient
self.config_model = Config
self.vpnclient_model = VpnClient
130. 1. Faster release cycle & evolution
We release new features more often
The project is evolving rapidly
131. 2. Easier maintainance
Once a bug in a specifc module is replicated
fxing it is easier compared to the work needed to
fx bugs in OpenWISP 1
132. 3. More derivative work
OpenWISP 2 was released officially
less than 1 year ago (as of April 2017)
notwithstanding that, there are already
a couple of derivative works
133. 4. Growth
The user base has been growing rapidly
Users send feedback and patches
139. 1. Integration issues
Combining more django apps may
result in integration issues
Changes to project confguration can result
in bugs that are not caught by unit tests
140. 1. Solution
Integration tests are important in this case
Shortcut: I was able to import tests of a base app
and repeat them in the extension app
This easy fx was good enough for my case
141. 2. Big features
When I needed to introduce a new major feature I
needed to change at least a couple of modules
142. 2. Solution
many times this will be necessary
If you have to change many modules every time,
refactor your code to be more loosely coupled
144. 3. Repository management overhead
● working with many repositories can be overwhelming
● more git tags, pypi releases, versioning, changelogs
145. 3. Repository management overhead
● working with many repositories can be overwhelming
● more git tags, pypi releases, versioning, changelogs
● more announcements for new releases
147. 3. Solutions
● import several projects in a single window of your editor
● automate versioning and changelogs
(I haven't done this yet)
148. 3. Solutions
● import several projects in a single window of your editor
● automate versioning and changelogs
(I haven't done this yet)
● do not send announcements for minor releases which
contain no real advantage for end users
149. 4. Documentation fragmentation
We have many repositories with
their own READMEs and docs
We don't have a single documentation website yet
150. 4. Solutions
I added links to specifc module docs from the website
I often have to send those links on the mailing list
But a central comprehensive documentation website
would be better (we don't have this yet)
151. 5. Bug reports
Users don't know where to send bug reports
and they write to the mailing list
152. 5. Solution
● For the moment I reply by pointing to them where
to open issues
153. 5. Solution
● For the moment I reply by pointing to them where
to open issues
● But a single issue tracker for all the repositories
would be better
(even though it may not really solve the "which module?" problem)
154. 6. Dispersion of popularity
We no longer have a single popular
repository with many github stars
155. 6. Solution
Stop caring about this
Github stars are not an accurate indicator
of the success of a project
164. Thank you
Ideas, critical feedback, suggestions?
Talk to me!
Find me also on:
● twitter (@nemesisdesign)
● github (@nemesisdesign)
● linkedin (Federico Capoano)
You can fnd the slides on slideshare