ClickHouse is open source. You can build it yourself. What’s more, you can make it better! In this webinar, we’ll demonstrate how to pull the ClickHouse code from Github and build it. We’ll then walk through how to contribute a new feature to ClickHouse by developing, testing, and pushing a pull request through the community merge process. There will be demos and ample time for questions. Join us to get started as a ClickHouse developer!
2. About me
Server team lead @ Altinity
working on CH for ~3 years now
~70 merged PRs in CH
IPv4 and IPv6 datatype
Gorilla and DoubleDelta codecs
DateTime64 datatype + extended range for date/time*
system.session_log
+ many others
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
2
3. About Altinity
The #1 enterprise ClickHouse provider
Now offering Altinity.Cloud
Major committer and community sponsor for ClickHouse in US/EU
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
3
4. Overview
What CH is
How to
Clone
Make changes
Build
Run tests
Submit a PR
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
4
5. What ClickHouse is
an OLAP DBMS [0][1]
Open source software [2]
fastest growing open source project now
highly optimized modern C++
~0.5M LOC (excluding third-party)
80-ish third-party libraries
Lots of tests
Docs are in the repo [3]
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
5
6. Why contribute
Fastest way to have a bug fixed or feature implemented
Learn a lot in the process
Fame
system.contributors
ClickHouse merged contributors badge
Arctic Code Vault Contributor [4]
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
6
11. Clone
Make a fork (if you are going to contribute changes to the community)
Clone repo and fetch submodules
$ cd ~
$ git clone https://github.com/ClickHouse/ClickHouse.git ClickHouse
$ cd ClickHouse
$ git submodule update --init --recursive
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
11
13. Make changes - complexity hierarchy
[7]
Functions: regular, aggregate,
table-function
Data format
Table Engine
Codec ( None , LZ4 , DoubleDelta )
SQL feature: clause ( UNION ALL ),
section ( LIMIT BY ), request type (
SYSTEM )
DB engine
Dictionaries: layout, source
Data type ( UUID )
Client/server protocol
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
13
14. Make changes
We are going to add a helloWorld() function
SELECT helloWorld();
"Hello World!"
Follow guides [8]
$ vim ~/ClickHouse/src/Functions/helloWorld.cpp
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
14
15. Make changes
#include <DataTypes/DataTypeString.h>
#include <Functions/IFunction.h>
#include <Functions/FunctionFactory.h>
#include <Core/Field.h>
namespace DB
{
class FunctionHelloWorld : public IFunction
{
public:
static constexpr auto name = "helloWorld";
static FunctionPtr create(ContextPtr) { return std::make_shared<FunctionHelloWorld>(); }
String getName() const override { return name; }
bool useDefaultImplementationForConstants() const override { return true; }
bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo &) const override { return false; }
size_t getNumberOfArguments() const override { return 0; }
DataTypePtr getReturnTypeImpl(const DataTypes &) const override { return std::make_shared<DataTypeString>(); }
ColumnPtr executeImpl(const ColumnsWithTypeAndName &, const DataTypePtr & result_type, size_t input_rows_count) const override
{
return result_type->createColumnConst(input_rows_count, "Hello World!");
}
};
void registerFunctionHelloWrold(FunctionFactory & factory)
{
factory.registerFunction<FunctionHelloWorld>();
}
}
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
15
16. Make changes
diff --git a/src/Functions/registerFunctions.cpp b/src/Functions/registerFunctions.cpp
index 9b1a7faebb..0451ff0ead 100644
--- a/src/Functions/registerFunctions.cpp
+++ b/src/Functions/registerFunctions.cpp
@@ -68,2 +68,5 @@ void registerFunctionAESDecryptMysql(FunctionFactory & factory);
+void registerFunctionHelloWorld(FunctionFactory & factory);
+
void registerFunctions()
@@ -132,2 +135,3 @@ void registerFunctions()
registerFunctionLogTrace(factory);
+ registerFunctionHelloWorld(factory);
}
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
16
18. Build
Build in Docker
Can produce .deb , .rpm and .tar.gz
Easy to set up - no prerequisites, only docker
any OS
Build on Host
More suitable for development process (i.e. developing CH core)
Ubuntu, Prerequisites
faster (~2x)
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
18
19. Build - Hardware Requirements
16GB of RAM (32 is better)
multiple cores (4+)
20-50GB on disk
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
19
20. Build in Docker
$ docker run
--network=host
--rm
--volume=$(realpath ./PACKAGES):/output
--volume=$(realpath .):/build
-e DEB_CC=clang-11
-e DEB_CXX=clang++-11
-e ALIEN_PKGS='--rpm --tgz'
-e CMAKE_FLAGS="$CMAKE_FLAGS -DADD_GDB_INDEX_FOR_GOLD=1"
clickhouse/clickhouse-deb-builder
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
20
21. Build in Docker
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
21
22. Build in Docker
$ cd ~/ClickHouse/PACKAGE
$ ls *.deb
clickhouse-client_21.11.1_all.deb clickhouse-common-static-dbg_21.11.1_amd64.deb
clickhouse-common-static_21.11.1_amd64.deb clickhouse-server_21.11.1_all.deb
clickhouse-test_21.11.1_all.deb
$ ls *.rpm
clickhouse-client-21.11.1-2.noarch.rpm clickhouse-common-static-21.11.1-2.x86_64.rpm
clickhouse-common-static-dbg-21.11.1-2.x86_64.rpm clickhouse-server-21.11.1-2.noarch.rpm
clickhouse-test-21.11.1-2.noarch.rpm
$ ls *.tgz
clickhouse-client-21.11.1.tgz clickhouse-common-static-21.11.1.tgz
clickhouse-common-static-dbg-21.11.1.tgz clickhouse-server-21.11.1.tgz
clickhouse-test-21.11.1.tgz
$ ls *.buildinfo *.changes
clickhouse_21.11.1_amd64.buildinfo clickhouse_21.11.1_amd64.changes
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
22
23. Build on Host - Prerequisites
for Ubuntu 20.04:
$ apt-get install
clang-11
clang-tidy-11
cmake
lld-11
llvm-11
llvm-11-dev
ninja-build
tzdata
--yes --no-install-recommends
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
23
24. Build on Host - Building
$ mkdir ~/build
$ cd ~/build
$ CC=clang-11 CXX=clang++-11 cmake ~/ClickHouse -GNinja
$ ninja clickhouse
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
24
25. Build on Host - Building
~80 mins
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
25
26. Build on Host - Building
$ ls ~/build/programs/
CMakeFiles benchmark cmake_install.cmake
extract-from-config install library-bridge
odbc-bridge CTestTestfile.cmake **clickhouse**
compressor format keeper
local server bash-completion
client copier git-import
keeper-converter obfuscator static-files-disk-uploader
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
26
28. Test - Smoke-test
$ cd ~/ClickHouse/programs/server
$ ~/build/programs/clickhouse server
$ ~/build/programs/clickhouse client
ClickHouse client version 21.11.1.1.
Connecting to localhost:9000 as user default.
Connected to ClickHouse server version 21.11.1 revision 54449.
Warnings:
* Server was built in debug mode. It will work slowly.
:)
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
28
29. Test - Smoke-test
:) SELECT version()
SELECT version()
Query id: 6f8b1cd8-f5d9-4f49-87ce-125ea27e08f8
┌─version()─┐
│ 21.11.1.1 │
└───────────┘
1 rows in set. Elapsed: 0.008 sec.
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
29
30. Tests
Executed on CI, but can also be run locally
There are multiple ways to validate a build [6]
Unit-tests - src/.../tests/gtest_*.cpp ~ 81 files ~ 10K test (4 min)
Stateless* - tests/queries/0_stateless ~ 3424 tests (45min)
Integration* - tests/integration/test_x/test.py ~ 276 tests
Performance* - tests/performance/*.xml ~ 220 tests
Lots of others
* either of those must be present in PR
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
30
31. Test - add test
$ cat ~/ClickHouse/tests/queries/0_stateless/02030_hello_world.sql
SELECT helloWorld() as r, toTypeName(r);
$ cat ~/ClickHouse/tests/queries/0_stateless/02030_hello_world.reference
Hello World! String
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
31
32. Test - run test
$ cd ~/ClickHouse/tests
$ # Requires server accessible via port 9000
$ ./clickhouse-test -b ~/build/programs/server/clickhouse 02030_hello_world
WARNING: jinja2 not installed! Template tests will be skipped.
Using queries from 'queries' directory
Using ~/ClickHouse/programs/server/clickhouse as client program (expecting split build)
Connecting to ClickHouse server... OK
Running 1 stateless tests (MainProcess).
02030_hello_world: [ OK ]
1 tests passed. 0 tests skipped. 1.85 s elapsed (MainProcess).
Won't run stateful tests because test data wasn't loaded.
All tests have finished.
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
32
33. Commit & Push
Review Release
Development
clone
change
build
test
commit & push
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
33
34. Commit & Push
$ git checkout -b hello_world
$ git add src/Functions/helloWorld.cpp
src/Functions/registerFunctions.cpp
tests/queries/0_stateless/02030_hello_world.sql
tests/queries/0_stateless/02030_hello_world.reference
$ git commit
$ git push enmk HEAD #*
* enmk - name of the remote, you can add a remote after cloning a repo
with git remote add
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
34
37. Review - Description
Review Release
Development
description
CLA
fix checks
fix requests
merge to master
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
37
38. Review - Description
Changelog category (leave one):
- New Feature
- Improvement
- Bug Fix (user-visible misbehaviour in official stable or prestable release)
- Performance Improvement
- Backward Incompatible Change
- Build/Testing/Packaging Improvement
- Documentation (changelog entry is not required)
- Not for changelog (changelog entry is not required)
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
...
Detailed description / Documentation draft:
...
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
38
39. Review - Description
Changelog category (leave one):
- New Feature
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Added `helloWorld` function that outputs a "Hello World!"
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
39
40. Review - CLA
Review Release
Development
description
CLA
fix checks
fix requests
merge to master
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
40
41. Review - CLA
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
41
42. Review - checks
Review Release
Development
description
CLA
fix checks
fix requests
merge to master
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
42
43. Review - checks
There are ~50 checks [9], roughly grouped as:
style checks
fasttest
builds
stateless
Unit tests
statefull
integration
sanitizers
performance
AST Fuzzer
Stress tests
TestFlows tests
PVS Checks
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
43
44. Review - checks
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
44
45. Review - check page - Fasttest
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
45
46. Review - check page - Stateless
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
46
47. Review - check page - Integration
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
47
48. Review - fix change requests
Review Release
Development
description
CLA
fix checks
fix requests
merge to master
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
48
49. Review - Merge
Review Release
Development
description
CLA
fix checks
fix requests
merge to master
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
49
50. Review - Merge
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
50
52. References
[0]: clickhouse.com
[1]: altinity.com
[2]: github.com/ClickHouse/ClickHouse
[3]: clickhouse.com/docs
[4]: archiveprogram.github.com
[5]: clickhouse.com/docs/en/development/developer-instruction
[6]: clickhouse.com/docs/en/development/tests
[7]: youtu.be/g5CDi8Nbut4?t=3669
[8]: clickhouse.com/docs/en/development/style/
[9]: clickhouse.com/docs/en/development/continuous-integration/
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
52
53. BTW, we are hiring!
altinity.com/careers
altinity.com/job/clickhouse-server-engineer
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
53
55. !!! BONUS PAGES !!!
Examples of how to the execute most important tests on your machine
Unit-tests
Stateless
Integration
Performance
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
55
56. Tests - Unit-tests
src/.../tests/gtest_*.cpp ~ 81 files
cover specific isolated thing
fast to execute
GTest-based
easy to debug
not so easy to write
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
56
57. Tests - Unit-tests - run on Host
Need to build a test binary first
$ cd ~/build
$ ninja unit_tests_dbms
Run the tests
$ ~/build/src/unit_tests_dbms --gtest_filter="*GTEST-FILTER-CLAUSE*"
# $ ~/build/src/unit_tests_dbms --gtest_filter="*TokenExtractorTest*"
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
57
58. Tests - Stateless
tests/queries/0_stateless ~ 3424 tests
functional test
require running ClickHouse server
executed by tests/clickhouse-test
easy to write, can be written in either SQL , bash , or Python
require .reference file
simple to run
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
58
59. Tests - Stateless - run in Docker
$ docker run
--rm
-v $(realpath ~/ClickHouse/PACKAGES):/package_folder
-v $(realpath ./test_output):/test_output
-v $(realpath ./ch_server_logs):/var/log/clickhouse-server/
clickhouse/stateless-test
# image can be built locally
$ cd ~/ClickHouse/docker/test/stateless
$ docker build . -t clickhouse/stateless-test
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
59
60. Tests - Stateless - run on Host
Needs a running server
$ cd ~/ClickHouse/programs/server
$ ~/build/programs/clickhouse server
Execute test driver
$ cd ~/ClickHouse/tests
$ ./clickhouse-test -b ~/build/programs/clickhouse
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
60
61. Tests - Integration
tests/integration/test_x/test.py ~ 276 tests
check how CH cooperates with various external entities
Distributed (replicas, distributed queries, Zookeeper)
External storages (MySQL, PostgreSQL, Kafka, S3, etc)
Python-based + Docker-compose
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
61
62. Tests - Integration - run in Docker
Requires docker in docker, hence sudo
$ cd ~/ClickHouse/tests/integration
$ sudo ./runner
--binary ~/build/programs/clickhouse
--odbc-bridge-binary ~/build/programs/clickhouse-odbc-bridge
--base-configs-dir ../../programs/server/
'test_odbc_interaction -ss'
# image can be built locally
$ cd ~/ClickHouse/docker/test/integration/runner
$ docker build -t clickhouse/integration-tests-runner .
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
62
63. Tests - Performance
tests/performance/*.xml ~ 220 tests
XML-based
preconditions: <preconditions>
parameters: <substitution>
setup: <settings> , <create_query> , <fill_query>
run: <query>
tear down: <drop_query>
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
63
64. Tests - Performance - run in Docker
Usefull for measuring performance of the
Requires a running server, reachable on port 9000
$ docker run
-it
--network=host
-v $(realpath ~/ClickHouse/tests/performance/encrypt_decrypt.xml):/perf.xml
clickhouse/clickhouse-performance-comparison
bash -c "./perf.py /perf.xml"
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
64