2. WHAT IS SPECQL?
› The combination of clojure.spec, PostgreSQL and the power of
macros
clojure.spec PostgreSQL
λ
3. SOME BACKGROUND
› Yesql/Jeesql/Hugsql are all fine and specql does not try to ”solve”
SQL so that you don’t have to use it anymore
› When you have multiple slightly different queries, there’s
duplication or fetching too much
› Adding dynamic WHERE clauses leads to ugly NULL checking of
bound parameters
4. EXAMPLE
› SELECT it.foo, it.bar, it.baz
› FROM interesting_table it
› WHERE it.archived = FALSE
› AND it.organization = :user_org_id
› AND it.category IN (:categories)
5. EXAMPLE
› SELECT it.foo, it.bar, it.baz
› FROM interesting_table it
› WHERE it.archived = FALSE
› AND it.organization = :user_org_id
› AND it.category IN (:categories)
How to make
variants without
duplication or
selecting too much?
6. EXAMPLE
› SELECT it.foo, it.bar, it.baz
› FROM interesting_table it
› WHERE it.archived = FALSE
› AND it.organization = :user_org_id
› AND it.category IN (:categories)
How to
add/remove
clauses
dynamically?
7. EXAMPLE
› SELECT it.foo, it.bar, it.baz,
› o.name AS organization_name
› FROM interesting_table it
› JOIN organization o ON it.organization_id = o.id
› WHERE it.archived = FALSE
› AND it.organization = :user_org_id
› AND it.category IN (:categories)
8. EXAMPLE
› SELECT it.foo, it.bar, it.baz,
› o.name AS organization_name
› FROM interesting_table it
› JOIN organization o ON it.organization_id = o.id
› WHERE it.archived = FALSE
› AND it.organization = :user_org_id
› AND it.category IN (:categories)
How to cleanly add
these without code
duplication?
9. THE SPECQL WAY: DEFINE
› (define-tables db
› [”interesting_table” ::it/interesting
› {::it/organization (rel/has-one
› ::it/organization_id
› ::org/organization
› ::org/id)}])
10. THE SPECQL WAY: DEFINE
› (define-tables db
› [”interesting_table” ::it/interesting
› {::it/organization (rel/has-one
› ::it/organization_id
› ::org/organization
› ::org/id)}])
The main macro
for defining things
11. THE SPECQL WAY: DEFINE
› (define-tables db
› [”interesting_table” ::it/interesting
› {::it/organization (rel/has-one
› ::it/organization_id
› ::org/organization
› ::org/id)}])
The name of the
table in the
database
12. THE SPECQL WAY: DEFINE
› (define-tables db
› [”interesting_table” ::it/interesting
› {::it/organization (rel/has-one
› ::it/organization_id
› ::org/organization
› ::org/id)}])
The namespaced
keyword for the
table
13. THE SPECQL WAY: DEFINE
› (define-tables db
› [”interesting_table” ::it/interesting
› {::it/organization (rel/has-one
› ::it/organization_id
› ::org/organization
› ::org/id)}])
Additional
definitions for
columns
22. THE SPECQL WAY: JOIN
› ;; => ({::it/foo 1
› ::it/bar ”example”
› ::it/baz true
› ::it/organization {::org/id 1
› ::org/name ”Acme Inc”}
› …)
Joined entity is
available in a nested
map
23. SPECQL BENEFITS
› Clojure data!
› The column set is just data and can be easily manipulated
• Even given as parameters from the frontend
› Where clauses can be easily added and are (mostly) just data as well
› Every table and column has a single ns keyword definition
• No more ”order” vs ”order-id” vs ”ord” differences in returned query
results
• Namespaced keys are the way of the future
24. SPECQL BENEFITS
› Clojure data!
› The column set is just data and can be easily manipulated
• Even given as parameters from the frontend
› Where clauses can be easily added and are (mostly) just data as well
› Every table and column has a single ns keyword definition
• No more ”order” vs ”order-id” vs ”ord” differences in returned query
results
• Namespaced keys are the way of the future HERE TO STAY
25. SPECQL BENEFITS
› Works with ClojureScript
• The spec generation part, put your specs in .cljc files and enjoy the same
specs and have your db definitions drive your frontend as well
› Works well with ”typed document storage” pattern
• A column can be a user defined type or array
› Provides generic upsert!
• No need to branch code, let the db handle it
› Works with database VIEWs
26. STATUS
› The github project page (https://github.com/tatut/specql) still says
EXPERIMENTAL
• Current 0.7 alpha stage, should be ready this year
• We are already using it in production in Harja
› No concrete promises before the experimental flag is gone, but the
API should only grow
• The test suite is comprehensive and should not break
27. STATUS
› 0.7 is coming and adds support for stored procedures
• Macro for defining a stored procedure as a function
• Better JOIN handling
• Currently fetching multiple ”has many” collections at once doesn’t work properly
› How to help
• Please try it out if you are using PostgreSQL
• Report rough edges, I try to provide good error messages
Specql is a new library for using PostgreSQL from a Clojure application. It introspects your database at compile time (with the help of some macro magic) and provides clojure.spec specs for your tables and their columns.
In addition to creating specs, Specql also retains runtime information so that it can provide generic query and update operations. More on those later.
We have found that slightly different queries are painful to maintain.
Hand written SQL also suffers from the problem of having different keys for different table columns. One query might use :order and another might use :orderid. There is no single truth of what a keyword means.
Let's compare by an example. Consider some table called interesting_table that we want to query from.
How can we easily vary the set of columns we fetch without having multiple almost identical queries or fetching everything and selecting the wanted keys on the caller.
These two problems, in our experience, lead to making similar queries or make the queries you write awkwardly complicated.
And we haven’t even talked about JOINs yet. What if I have two similar queries, but the other one needs to join a table?
Adding columns from a JOINed table further changes how our SQL query looks. Making it difficult to even see that they are conceptually the same.
First we define our tables so specql can generate the specs and record information about it.
First we define our tables so specql can generate the specs and record information about it.
First we define our tables so specql can generate the specs and record information about it.
The spec for the table (a keys spec) is generated for the keyword specified here. All column specs will be generated in the same namespace.
The column options can be used to do transformations for the columns (like converting an enum to keywords on the fly).
In this example we are defining a JOIN specification. The organization virtual column is a one to one JOIN to the organization table.
Specql provides a function called fetch that can be used to make queries against any previously defined table.
The table is simply the keyword we registered the table under in a previous call to define-tables
The columns is just a Clojure set of the namespaced keywords of the table columns
The where map is interesting. The keys are columns of the table being queried the values can either be values (which are simple equality checks) or operators. The basic SQL operators are provided (equality, less than, greater than, in, like) and there is a protocol for you to define more.
Multiple maps of where clauses can be combined with op/and and op/or operators.
Finally the result is a sequence of maps which contain the namespaced keys.
How do we add the JOIN?
We simply add a vector to the column set. The first element is the join definition we defined in define-tables and the second element is a set of columns to retrieve from the join table.
We simply add a vector to the column set. The first element is the join definition we defined in define-tables and the second element is a set of columns to retrieve from the join table.
We simply add a vector to the column set. The first element is the join definition we defined in define-tables and the second element is a set of columns to retrieve from the join table.
ClojureScript: speql understands some of the constraints on fields: NOT NULL and string lengths.
PostgreSQL columns can be arbitrarily complex objects. You can have an array of a user defined type as a column and specql will work great with that.
Upsert:
Upsert, which came in pg 9.5, is fully supported.
Upsert takes the db, the table keyword, a record map to insert and a where clause for the update
Optionally the unique column set to upsert on can also be specified, and defaults to the primary key
Views: you can write your complicated queries as views that and spec those like any table.
Although it is a very young library, we are already using it in production in Harja, which is a large public sector application for road infastructure maintenance.
We started adding specql to our new features and it has been helpful.