Artjoms Iskovs and Miles Richardson show how Splitgraph can share open datasets to ClickHouse using PostgreSQL protocol.

1. Splitgraph
1
Open data and beyond

2. 2
What is Splitgraph?
● Splitgraph Core
○ Build, extend, query and share data images
○ Powered by PostgreSQL
○ github.com/splitgraph/splitgraph
● Splitgraph Cloud
○ Splitgraph peer
○ Data catalog
○ Database proxy (Data Delivery Network, data mesh, data virtualization…)
○ www.splitgraph.com

3. 3
postgresql://user@data.splitgraph.com

4. 4
Plan
● Query data from public Splitgraph DDN via ODBC
○ odbc() function (SELECT * FROM odbc(...))
○ ODBC engine (CREATE TABLE … ENGINE ODBC)
○ Limits queries to 30s / 10k rows for QoS
● Run Splitgraph locally
○ Ingest data directly into ClickHouse for faster analytics
● All code on GitHub
○ github.com/splitgraph/splitgraph/tree/master/examples/clickhouse

5. 5

6. 6
odbc.ini
[splitgraph_ddn]
Description = PostgreSQL connection
Driver = PostgreSQL Unicode
Database = ddn
Servername = data.splitgraph.com
UserName = API_KEY
Password = API_SECRET
ConnSettings = SET application_name = 'odbc-clickhouse';
Port = 5432
Protocol = 12.3
SSLmode = require
ReadOnly = Yes
RowVersioning = No
ShowSystemTables = No
UseServerSidePrepare=0

7. 7
Demo 1: odbc() function
●
●
●
●
●

8. 8
Demo 1: odbc() function
SELECT
engine,
address
FROM odbc('DSN=splitgraph_ddn',
'cityofchicago/fire-stations-28km-gtjn', 'fire_stations')
LIMIT 10

9. 9
Demo 2: ODBC engine and cross-portal joins
●
●
●
●
○
○

10. 10
Demo 2: ODBC engine and cross-portal joins
CREATE TABLE chicago_cases (
lab_report_date String,
cases_total Int32
)
ENGINE = ODBC(
'DSN=splitgraph_ddn',
'cityofchicago/covid19-daily-cases-deaths-and-hospitalizations-naz8-j4nc',
'covid19_daily_cases_deaths_and_hospitalizations'
);
CREATE TABLE cambridge_cases (
date String,
new_positive_cases Int32
)
ENGINE = ODBC(
'DSN=splitgraph_ddn',
'cambridgema-gov/covid19-cumulative-cases-by-date-tdt9-vq5y',
'covid19_cumulative_cases_by_date'
);

11. 11
Demo 3: Local Splitgraph engine
●
○
○
●

12. 12
Demo 3: Local Splitgraph engine
SELECT
candidate_normalized,
SUM(votes) AS votes
FROM odbc('DSN=splitgraph',
'splitgraph/2016_election',
'precinct_results')
WHERE state_postal = 'TX'
GROUP BY candidate_normalized
ORDER BY votes DESC

13. 13
Demo 3: Local Splitgraph engine
CREATE TABLE sg_2016_election (
state_postal String,
state_fips String,
votes Int32...
) ENGINE = ODBC('DSN=splitgraph', 'splitgraph/2016_election',
'precinct_results');
CREATE TABLE ch_2016_election AS sg_2016_election
ENGINE MergeTree ORDER BY county_fips;
INSERT INTO ch_2016_election SELECT * FROM sg_2016_election;

14. 14
Thanks!
www.splitgraph.com
{artjoms,miles}@splitgraph.com
github.com/splitgraph/splitgraph/tree/master/examples/clickhouse