Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à How to Design a Great API (using flask) [ploneconf2017]
Similaire à How to Design a Great API (using flask) [ploneconf2017] (20)
Dernier
Dernier (20)
How to Design a Great API (using flask) [ploneconf2017]
- 4. Intuitive
- 7. ○
- 8. Are Hard Coded Variables Bad? Yes, here’s why: 1. When runtime variables aren’t centralized, they’re harder to find, making your code less readable 2. If the variable is related to environment, you’ll likely need to juggle git 3. If the variable is supposed to be private or secret (e.g. passwords) database = 'production_url' File.py (production) ~~ database = ’localhost_url' ++ def ban_user(email): ++ ... File.py (local machine)
- 10. Configurations / Environment database: postgresql://localhost:5432/my_app test_database: postgresql://localhost:5432/testing flask_login_secret: import yaml yaml_config = yaml.load( open(('../my-app.yaml')) print yaml_config[’database’] usage my_app.yaml.example
- 11. Configurations / Environment (2) class Config(object): DEBUG = False CSRF_ENABLED = True SECRET = yaml_config['flask_login_secret'] SQLALCHEMY_DATABASE_URI = yaml_config['database'] class DevelopmentConfig(Config): DEBUG = True class TestingConfig(Config): TESTING = True SQLALCHEMY_DATABASE_URI = yaml_config['test_database’] DEBUG = True class ProductionConfig(Config): DEBUG = False TESTING = False app_config = { 'development': DevelopmentConfig, 'testing': TestingConfig, 'production': ProductionConfig } app = create_app('development') app.run() run.py conftest.py @pytest.fixture(scope='session') def mock_app(): mock = create_app(’testing') cxt = mock.app_context() cxt.push() yield mock
- 13. Object Relational Mappers (2) class User(Base): __tablename__ = 'users' id = Column(Integer, primary_key=True) name = Column(Text, nullable=False) email = Column(Text, index=True, unique=True, nullable=False) password = Column(Text, nullable=False)
- 14. Object Relational Mappers (3) With ORM Without ORM def get_user_email(id): return conn.query(User.email) .get(id) def get_user_email(id): query = text(””” SELECT email FROM users WHERE id = %d LIMIT 1;”””%(id)) rows = conn.execute(query) if rows is not None: return rows[0][’email’] return None ^^ Much simpler ^^ Might have to worry about SQL Injection
- 15. How to Standardize Database Schemas? GIT History Database Migrations (Alembic) + Database Schema =
- 16. Database Migrations def upgrade(): op.add_column('reports', sa.Column(author', sa.Text()) ) def downgrade(): op.drop_column('reports', author’) models.py alembic/version/6a9eb6b43d7d_added_report_author.py class Report(Base): __tablename__ = 'reports’ ++ author = Column(Text) ^^ automatically generated for you by Alembic
- 17. Default Alembic Behavior $ alembic revision –-autogenerate -m “myrev” # file_template = %%(rev)s_%%(slug)s Terminal Alembic.ini
- 18. Default Alembic Behavior (2) $ alembic history 69c35d0ef5bb -> 000fd728c260 (head) b4b0fbdebf42 -> 69c35d0ef5bb 9dcf0d65e983 -> b4b0fbdebf42 ac39eb5890f4 -> 9dcf0d65e983 7054e7fae54a -> ac39eb5890f4 6a9eb6b43d7d -> 7054e7fae54a 56a1b15216da -> 6a9eb6b43d7d dbe9911f4c0b -> 56a1b15216da 2d5f5c762653 -> dbe9911f4c0b 24f34e82e6c1 -> 2d5f5c762653 ... Terminal 8 12 3 14 5 6 9 4 7 11 13 15 10 1 2
- 19. Ideal Alembic Behavior # file_template = %%(rev)s_%%(slug)s file_template = %%(year)d_%%(month).2d_%%(day).2d_%%( slug)s Alembic.ini 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
- 20. How to Standardize Database Data? Revision tools typically only manage schemas, not data, you’ll need to create a database seed Why • Go from 0 to standard instantly • Developers are free to mess with their local database • It’s easy to test your back end with a known database setup How • Dump database data to a sql file • Create a script (.py, .sh, etc) to reset your database schema and insert the seed file pg_dump --dbname=“my_app” --file=”/PATH/TO/seed.sql” --data-only --inserts
- 21. Where Are My Packages? $ python run.py ImportError: No module named flask_sqlalchemy $ pip install flask_sqlalchemy $ python run.py ImportError: No module named smtplib $ pip install smtplib $ python run.py ImportError: No module named passlib $ ... #FacePalm
- 22. Requirements.txt $ python run.py ImportError: No module named flask_sqlalchemy $ pip install –r requirements.txt Collecting flask_sqlalchemy Collecting smtplib Collecting passlib ... $ python run.py * Running on http://127.0.0.1:5000/ Flask == 0.12 Flask-Cors==3.0.3 Flask-SQLAlchemy == 2.1Flask-Login==0.3.2 psycopg2 == 2.6.1 PyYAML == 3.11 requests == 2.10.0 SQLAlchemy == 1.0.14 passlib==1.6.5 bcrypt==3.1.1 WTForms==2.1 ...
- 23. Requirements.txt (2) git+ssh://git@github.com/MY_USER/MY_REPO.git#egg=MY_REPO Install packages from github git+file:///PATH/TO/MY/PROJECT Install packages from local file system Why use these methods? • Ability to quickly test whether changes to another python project worked without having to push to github • This flow is fully local and keeps your git history cleaner • WARNING: When using the local file system method, your changes must be locally committed otherwise pip will not notice the changes.
- 24. Why aren’t my files updating? $ find . -name '*.pyc' -delete Your project may be running pre-compiled .pyc files and not overriding the old files Common .pyc conflict scenarios: • Moved or renamed python directories • Moved or renamed python files • Added or removed __init__.py files • Created a variable with the same name as a module To clear out all .pyc files and start fresh, run the following:
- 25. Why Isn’t the Setup README Working? $ cd MyApp $ cp config/app.yaml.example config/app.yaml cp: No such file or directory $ #Confused #WhatDo
- 26. Setup Scripts def install(dirs, args): env = make_virtualenv(dirs['venv_dir']) path = install_node_and_npm(dirs, env['PATH']) env['PATH'] = path install_bower(dirs, env) check_call(['bower', 'install', '--force-latest'], cwd=dirs['root_dir'], env=env) check_call(['pip', 'install', '-q', '--upgrade', 'pip'], cwd=dirs['root_dir'], env=env) check_call(['pip', 'install', '-q', '-r', 'requirements.txt'], cwd=dirs['root_dir'], env=env) return env setup.py
- 27. Durable
- 28. Should We UnitTest? Common Concerns • Writing tests takes time away from building features • Slows down development • Writing tests is boring Short Answer • Absolutely!!! • Don’t just test the golden-path, also test failure scenarios like 3xx, 4xx, or 5xx errors
- 29. Why UnitTests are Awesome Resolution of Time Concern • Requirement: You must verify your code works before shipping • Would you prefer to do that manually every time or automate that process? • Test coverage is an upfront investment saving time later on extra debugging Time Spent Testing Application Complexity Automated Manual
- 30. Why UnitTests are Awesome (2) Resolution of Boring Concern • Writing real code (tests) vs being a click farm Writing Features Writing Tests Clicking Thru App How Boring
- 31. Global PIP Cache Using the Wrong Dependencies? Project X (Flask==0.12) Project Y (Flask==0.8) Common Package Management Problems • Need to use different versions of the same package on another project or branch • Project broke because it was using existing package versions of other projects • Requirements.txt is missing an entry because previous developer already had the package installed Project Z Project W
- 33. Virtual Environments (2) Setup Issues? Just delete your venv and create a fresh one $ virtualenv ~/.venvs/X $ source ~/.venvs/X/bin/activate (X)$ pip install -r requirements.txt $ rm –rf ~/.venvs/X Clean & easy dependency management; especially when working on multiple projects Why? • Package version conflicts • v5.0.1 != v5.3.0 • Dependency contamination across multiple projects • Testing dependency upgrades
- 34. Flush vs Commit Flush Reserves a placeholder spot in database Commit Saves records into database Tips • Use flush when you need to get an auto- increment id back • Use commit ONLY when you are 100% sure you want to save rows • Every time you use `flush` or `commit`, you are adding another round-trip request Problem with always committing def save_test(student, answers): for answer in answers: conn.add(Answer(answer)) conn.commit() # SLOW! grade = calcGrade(answers) << ERROR >> conn.add(Test(student, grade)) conn.commit() Test will not get saved, but every answer will still be in the database!
- 35. Flush vs Commit (2) Only commit once def save_test(student, answers): for answer in answers: conn.add(Answer(answer)) grade = calcGrade(answers) << ? ERROR ? >> conn.add(Test(student, grade)) conn.commit() If error, nothing gets saved If no error, everything gets saved Flush (if auto-increment needed) def save_test(student, answers): for answer in answers: conn.add(Answer(answer)) conn.flush() ### use answers[0].id grade = calcGrade(answers) << ? ERROR ? >> conn.add(Test(student, grade)) conn.commit()
- 36. Errors & Rollbacks On app error, rollback session @app.errorhandler(Exception) def handle_error_response(error): conn.rollback() return jsonify(error) After requests, close sessions @app.teardown_appcontext def shutdown_session(exception=None): conn.remove()
- 37. Flexible
- 38. Project Directory Structure /app /api /models /dbms /utils /config /setup /tests /api /dbms /utils requirements.txt conftest.py run.py /app - Contains all business logic for run-time usage /config - Contains yaml and other configuration files /setup - Contain scripts and seed files for resetting database and project dependencies /tests - Contains all unit-test, ideally similar in directory structure to /app
- 39. App Factories app/__init__.py db = SQLAlchemy() def create_app(config_name): app = Flask(__name__, instance_relative_config=True) app.config.from_object(app_config[config_name]) app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False app.secret_key = app.config['SECRET'] cors = CORS(app, supports_credentials=True) db.init_app(app) login_manager.init_app(app) app.register_blueprint(api) return app
- 40. App Factories (2) app/__init__.py from flask import Flask app = Flask(__name__) import api import db app/api/__init__.py from app import app, db import endpoint_group1 import endpoint_group2 app/db/__init__.py from app.utils.config import config from sqlalchemy.orm import scoped_session conn = scoped_session(config...) import query_group1 import query_group2 Issues: • Circular imports (prevents testing) • Lack of separation of concerns between app module and app object
- 41. Blueprints admins = Blueprint( ’admins', __name__) app.register_blueprint(admins) @admins.before_request def before_admin_req(): print ”ADMIN ROUTE REQUESTED" @admins.route('/admin/secret', methods=['GET']) def admin_secret(): return jsonify({}) public = Blueprint( ’public', __name__) app.register_blueprint(public) @public.before_request def before_public_req(): print ”PUBLIC ROUTE REQUESTED" @public.route('/public/not_secret', methods=['GET']) def public_not_secret(): return jsonify({}) Blueprints can be used to: • Logically group routes • Selectively apply/add request lifecycle events to a subset of routes
- 42. Extend and Expire Sessions Every time a user makes a request, extend their session for 30 minutes @app.before_request def extend_session(): session.permanent = True app.config['PERMANENT_SESSION_LIFETIME'] = timedelta(minutes=30) session.modified = True
- 43. Flexible Testing (pdb) Typical Python Debugging def my_func(objects): print "MY_FUNC", objects if len(objects) > 0: print "NOT EMPTY" new_objects = some_func(objects) << ERROR >> print new_objects else: print "EMTPY" new_objects = [] return new_objects • Lots of print statements • Only renders prints run before the app crashes • Can’t update values Python Debugger import pdb def my_func(objects): pdb.set_trace() if len(objects) > 0: new_objects = some_func(objects) << ERROR >> else: new_objects = [] return new_objects • Sets a break-point at `.set_trace()` • Functions like JS browser debugging break-points • Can step line-by-line or continue until next break-point • At each line, there is an interactive terminal • Not only can print, but update values
- 45. Reliable
- 46. How to maximize up-time? Preventative (Before break) • Don’t ship bugs • Unit Tests • Staging servers with user testing • A/B rollouts • Stable deployment server Quick Detection • System logging (Splunk) • Slack hooks • Scheduled health check pings • E.g. cron jobs Quick Diagnosis • Failure/Exception tracking + logging • API playground • Postman
- 47. Versioning Why • New rollouts won’t break people’s existing builds • Backwards compatibility • Ability to plan obsolescence How Host a deployment for each version • Track version # in config files • Prepend version to routes Host one big deployment for all versions • Duplicate files • Create route version mapper
- 48. Analytics on API Usage Why • Shows what endpoints are most frequently used (most critical) • Optimize popular endpoints • Shows what users are still using deprecated endpoints • Notify users to upgrade How • Could be as simple as an incrementing counter in a database table • Could use deployment tools to wrap your API that provide analytics (DevOps level)
- 49. User Friendly
- 50. 101 in Endpoint Design Things to be aware of • #1 rule is being consistent • No matter what you pick, some group is going to be annoyed • Their complaints are likely valid, but only for a small component, not how it affects the whole picture • Common patterns: REST, CRUD • Use more then just GET and POST • PUT, DELETE, OPTIONS, PATCH • Use relevant status codes • 1xx info, 2xx success, 3xx redirection, 4xx client errors, 5xx server errors • Figure out if users are often instantly re-forming the data you send them
- 53. Fast
- 55. Bulk SQL Inserts Three useful methods • bulk_save_objects • bulk_insert_mappings • Mogrify ORMs were not primarily intended for bulk usage Using bulk commands gives a 5-10x speed boost # Without bulk for i in range(100000): conn.add(Employee(name="EMP #" + str(i))) conn.commit() # With bulk conn.bulk_save_objects([ Employee(name="EMP #" + str(i)) for i in range(100000) ])
- 56. Caching Store common responses in-memory for quick retrieval from flask import Flask from flask.ext.cache import Cache app = Flask(__name__) cache.init_app(app, config={'CACHE_TYPE': 'simple'}) @cache.cached(timeout=50) @app.route('/', methods=['GET']) def index(): return render_template('index.html')