1.
Keep your repo clean
Héctor Canto
@hectorcanto_dev

2.
● Software Developer ++
● Barcelona & Vigo
● @ Blueliv
○ CyberSecurity
○ Threat Intelligence
● Interested in
○ Best practices
○ Great documentation
○ Teach and Learn
About me

3.
● Maintenance
● Best practices
● Standards
● Auxiliary tools
○ Security
● Documentation
● Collaboration
Topics
Keep your repo clean

4.
Main ideas
1. Code, config, deploy and documentation matched together
○ joined through CI/CD and versioning
2. Code not only for yourself, but for others
3. Holistic view of documentation
“Code is read much more often than written.”
- paraphrasing Clean Code, Robert C.Martin (2008)
Keep your repo clean

5.
Why maintaining a project
● “Future you”
● Collaboration
○ team-mates
○ non-tech colleagues (product, sales)
○ external workers
○ partners
● Holiday, medical leaves, bus factor & rotation
● Do to others what you hope they do to you
“Always code as if the guy who ends up
maintaining your code will be a violent
psychopath who knows where you live.
Code for readability.”
- John F. Woods (1991)
Keep your repo clean

6.
I- Code, config and deploy
Keep your repo clean

7.
They have some common folder and files
> Let's build a template
References
- https://medium.com/worldsensing-techblog/project-templates-an
d-cookiecutter-6d8f99a06374
- https://realpython.com/python-application-layouts/
- http://cookiecutter-templates.sebastianruml.name/
- PyConEs 2018 https://www.youtube.com/watch?v=Ttco5BTJI7I
- PyBCN 2019 https://www.youtube.com/watch?v=TkrTM2Byj34
How a repo should look like
Cookiecutter, Yeoman
PEP-8, keepachangelog.com
├── $source/
├── config/
├── deploy/
├── docs/
├── requirements/
├── scripts/
├── README.md
├── CHANGELOG.md
├── setup.cfg
├── Makefile
├── docker-compose.yml
└── sonar-project.properties
Keep your repo clean

8.
Versioning
● Versioning is very useful
○ rollbacks, debugging, regressions
● It should span to:
○ Tags, Releases, Documentation
○ Artifact repositories (pkg manager, Docker registry)
● Major.minor.patch-prerelease.x+build#commit
○ 1.0.0-alpha.2+201303131abde44700
○ 0.7.2-rc.5
bumpversion, Nexus (artifacts), CI/CD tools
SemVer.org 1.0+, PEP-404
# setup.cfg
[bumpversion]
current_version = 1.2.3
commit = False
tag = True
tag_name = {new_version}
[bumpversion:file:source/__nit__.py]
[bumpversion:file:sonar-project.prop
erties]
# bash
bumpversion patch --new-version
2.1.3
Keep your repo clean

9.
● Should work out-of-the-box
● Templates
○ With comments and loops!
○ same template, multiple environments
● Infrastructure as Code
● Passwords out of repo
# db.env
POSTGRES_USER=dbuser
POSTGRES_DB=dbname
POSTGRES_HOST=localhost
POSTGRES_PORT=5432
POSTGRES_PASSWORD=notadmin
# db.env.j2
name: {{ db.schema_name|default('dbname') }}
user: {{ db.user | default('dbuse') }}
{# do not use same dbname and username #}
host: {{ db.host | default('postgres') }}
port: {{ db.port | default(5432) }}
{%- if 'password' in db %}
password: '{{ db.password }}'
{%- endif %}
Maintain a workable config
Jinja2, direnv, docker, ConfigLoad
IaC, YAML, .env
Keep your repo clean

10.
Utils configuration: setup.cfg
● Keep your utils configurations together
● And the root of your folder lighter
setup.cfg
# setup.cfg
[flake8]
...
[pylint]
...
[pep8]
...
[coverage:run]
...
[coverage:report]
...
[coverage:xml]
...
[coverage:html]
...
[tool:pytest]
...
Keep your repo clean

11.
Connect it all: Code, config and deploy
● A repository should have the means to be deployed
○ instructions in documentation
○ simple commands in Makefile, manage.py, scripts ...
● Code, config and deploy should match together in a point-in-time
● Strategy 1: all in one repo Strategy II: multi-repo
● CI/CD and config management tools join all together
Python-Fabric, Chef, Puppet, Ansible, Salt, Terraform …
IaC, DevOps, SRE
Keep your repo clean

12.
Connect it all: branching and ticketing
● Ticket numbering
○ type-of-branch/functional-name-#PROJ-123
■ feature, bugfix, release, docs …
■ what are you implementing here, don't repeat the ticket title
■ what ticket are you solving
● Versioning reference:
○ git - changelog - docker - docs - ticketing …
○ links on the wiki to the "latest", prevents obsolescence
Any CI-CD tool, Git GUI Clients
git-flow
Keep your repo clean

13.
II - Code for others
Keep your repo clean

14.
To be a good programmer you need to be a bit of a writer
● Be coherent and consistent
○ try: …. except SpecificException as exc:
● Answer some of the 6 'W' of journalism:
○ Who, What, Where, Why, When and hoW
● An image is worth more than 1000 words
○ Diagrams!
Code style I
class WhoService
def what_i_do(p1, p2, p3)
"""
why I do this, and a bit
how
p1: where to
p2: who
p3: when or how often
"""
how_i_do_this()
and_that()
return what_you_need
Keep your repo clean
PlantUML, Draw.io, InkScape
UML, SVG

15.
Code Style II
1. Avoid generic vocabulary, use specifics:
a. Frontend app, API REST, auth service, TwitterAPIClient
2. Use collective nouns and adjectives:
a. for choice in choice_options
b. for element in selected_items
c. for item in raw_params
3. Avoid custom abbreviations,
use conventions.
4. Add a glossary to your project docs
or to your wiki.
https://hackernoon.com/software-complexity-naming-6e02e7e6c8cb
“Vocabulary is not just a way of communication,
it is a toolset.
It evokes ideas, it enables abstraction and specification.
But it also introduces implicit meaning, irony,
sarcasm, hidden assumptions.”
Keep your repo clean
Sphinx-docs

16.
● Main on top, detail out or below
● Don't put all code together, modules!
○ my rule of thumb 500 lines (200 even better)
○ 30 lines per method or function
● Follow a style guide
○ extend it with your own rules
Code style III
Writing Idiomatic Python (Jeff Knup), Google's pyguide
black, flake8, pylint, Sonarqube, IDE plugins
PEP-8, PEP-20
Keep your repo clean
- Put the main function first, leave
fine details to other methods
- Use submodules and classes to
order things and encapsulate logic
together (in classes f.i.)
- Use _ and __ to tip scope
- import namespaces, not classes and
funcitons, to give context

17.
Code style IV
● Know your Design Patterns
○ client-server, singleton, adapter, pub/sub, factory ...
● Follow well-known principles and practices:
○ Object Calisthenics
■ if-else inversion
○ SOLID, KISS, DRY, YAGNI…
○ Learn about DDD
https://sourcemaking.com/
https://williamdurand.fr/2013/06/03/object-calisthenics/
# SOLID
S: validate_filter_transform_and_upload
O: BaseClass, CoolMixin, HigherOrderComponent
LD: func1(data, param1, param2)
func2(data, param2, param1)
I: def func3(injection1, injection2):
that = injection1.search(this)
injection2.save(that)
Keep your repo clean

18.
Comments and docstrings
● Do not comment everything
● Use triple-quote comments
● Save comments and docstrings
with good naming
● Type hinting! Python 3.5+
● Follow a docstring standard:
○ reST, numpy, Google, Epytext
"""
This is a multiline
comment
"""
"""
This is a Google docstring
Args:
param1 (int): The 1st parameter.
param2 (str): The 2nd parameter.
Returns:
bool: The return value. True for
success, False otherwise.
Raise:
TypeError: parameters are wrong
"""
http://daouzli.com/blog/docstring.html
PEP-257
Keep your repo clean

19.
Security Tools
● Safety (pyup.io)
● Docker Image scans: Clair, Trivy
● Docker secrets (Swarm, K8S)
● Ansible Vault
● SonarQube
● git-crypt
● KeePassXC
● Training, training and training
Keep your repo clean

20.
III - Documentation purposes
Keep your repo clean

21.
Documentation goals
▪ Explain implementation
▪ extension of the code
▪ Explain purpose
▪ Give context
▪ Evolution traceback
▪ Explain how to work together
▪ the framework
▪ Explain decisions
Keep your repo clean

22.
Extended documentation
In order of scope
- Code itself
- Comments
- Docstrings
- Tests
- Commit log
- Dev docs
- +ADRs (next slide...)
- API specification
- GUI helpers (tooltips, inline help)
- Tickets or Stories
- Project wiki
- User docs
- FAQ
Who reads the documentation:
- You
- Future you
- Your team
- Other devs
- Other colleagues
- Marketing, customer support ...
- Managers
- Partners
- Customers
- Users
Keep your repo clean

23.
Architecture Decision Records (ADR)
● Higher scope: architecture and software design
● Explain expectations
● State pros, cons, expected benefits and risk
● Write down agreements and compromises
● Re-evaluation and self-evaluation:
○ why we use such language, framework or design
ADR Template
# Intro
## Author, Approvers, links
# Context and problem statement
# Decision
## Drivers, Considered options
## Outcome
# Pros and expected benefits
# Cons and potential risks
# Possible improvements
# Links to other ADRs
Keep your repo clean
thoughtworks.com/radar/techniques/lightweight-architecture
-decision-records
ADR templates: https://adr.github.io

24.
Extended documentation caveats
Caveats:
- Maintenance
- obsolescence and inconsistency
- Redundancy
- Different scopes
- departments, docs audience
- Out of one own responsibility
- Beware of Agile principles:
- “Working SW first”
- Not good for small projects
Solutions and mitigations:
- Documentation owner and/or
docs committee
- Automation
- Documentation as Code
- One source of truth
- Bug-bashing for docs
- Spread Domain language across
teams and projects (DDD)
Keep your repo clean

25.
● Keep code, config and deploy synced with Versioning
● Think on the "future you"
● Write code for others
● Use tools, standards and automatization to your benefit
● Code explains implementation, documentation explain purpose
○ and gives context
● Consider the "extended" documentation
Summary
Keep your repo clean

26.
Thanks for your attention!
Questions?
Héctor Canto
@hectorcanto_dev
hectorcanto
WE'RE HIRING

27.
Specifics
● Repo structure and maintenance
● Versioning
● Config
● Deploy artifacts
● Test
● Security
● Code style
● Extended Documentation

28.
Each of the following topics may be
extended to a full talk.
We will keep things high level.
All icons, images and graphs are made by
me, opensource or free-to-use:
https://icons8.com
https://unsplash.com/
InkScape & GIMP
Disclaimers

29.
Deploy
A repository must have the means to run the software it represents
● Commands to simplify the process
○ Makefile, manage.py, Ansible
● Means to deploy: Docker, python-fabric, Ansible, Terraform…
● Consider several environments
make, Ansible, fabric, Terraform, Docker ...
IaC

30.
● Spend time testing
○ a lot initially, it is worth it
● Understand the different aspects of test
○ Cost
○ Unicity
○ Intention
○ Box-style
● A test can have several aspect at once
○ Unitary tests usually are cheap, white-box, non-integrative ...
○ Sometimes they may have some “integration” side
● Understand the test pyramid analogy
unittest, pytest, mock, hypothesis, Postman ...
TDD, BDD
Tests

31.
More ‘doc’ tools
- Swagger (OAI)
- Django Rest Framework browsable API
- Any Wiki
- Ticketing System
- Bug Tracking System
- Support forum and FAQs
Reference one to each other

32.
To be a good programmer you need to be a
bit of a writer
● Be coherent and consistent
○ try: … except SpecificException as exc:
● Answer some of the 6 'W' of journalism:
○ what, how, why, when, where, who
● An image is worth more than 1000 words
○ Diagrams!
● Don't put all code together, modules!
○ my rule of thumb 500 lines (200 even better)
Code style
Keep your repo clean