This document discusses using distributed locks to synchronize access to shared resources in distributed applications. It provides examples of when distributed locks are needed to prevent data corruption or loss from concurrent requests. The document recommends using proven distributed lock libraries like redis-semaphore and redlock-rb to implement distributed locks instead of recreating the logic. Tips are given on testing for concurrency issues and choosing correctness over efficiency with locks.

1. Distributed locks
KnapsackPro.com API case study
Artur Trzop

2. Start with a problem

3. But know the first

4. CircleCI container
Execute Rspec tests
Run all tests in single container

5. First container
Execute Rspec tests
Second container
Execute Rspec tests
Split tests between container
CI build for git commit

6. First container
Execute Rspec
tests fetched from work queue
Second container
Execute Rspec
tests fetched from work queue
Dynamic tests split between container
with knapsack_pro gem
Tests in work queue on
KnapsackPro.com API server

7. First requests from any container
should create a new work queue
for the git commit
def test_files
create_queue unless queue_exists?
test_files_from_top_of_the_queue
end

8. When this code can happen
at the same time?
● development - webrick is single threaded (bug won’t happen)
● Production (bug happens):
○ unicorn (multiple processes)
○ puma (multiple threads and/or multiple processes)

9. What is distributed lock?
synchronize access to shared resources
It means different processes must operate with shared resources
in a mutually exclusive way

10. Why you want a lock in a
distributed application?
● Efficiency
○ Avoid expensive computation
○ Saves time & money

11. Why you want a lock in a
distributed application?
● Correctness
○ Prevent corrupted data
○ Prevent data loss
○ Prevent data inconsistency

12. What tool I could use?
https://github.com/dv/redis-semaphore

13. How I solve the problem?
● Write tests to do concurrent requests against staging
○ Requests in separate threads
● Verify if the problem exists after implementing fix

14. Implementation fix
def test_files
semaphore = Redis::Semaphore.new(:semaphore_for_ci_build_id, host: "localhost")
semaphore.lock(5) do
create_queue unless queue_exists?
end
test_files_from_top_of_the_queue
end

15. What should I remember
● Locks are hard
● Distributed locks are even harder
● Don't reinvent the wheel, use proven solutions
● Most web apps are not thread-safe due to missing locks.
○ Expect edge cases while you grow.

16. Another example
def save
build = find_build || new_build
do_something_complex_with_build
build.save
end

17. With distributed lock
def save
semaphore.lock(5) do
build = find_build || new_build
do_something_complex_with_build
build.save
end
end

18. What distributed lock
allows us?
● When single redis DB
○ You can have multiple unicorn/puma processes
○ You can have multiple machines if each use the same redis DB

19. What distributed lock
allows us?
● When multiple redis DBs
○ Use library that supports Redlock algorithm
■ redlock-rb gem
■ https://redis.io/topics/distlock
● Is Redlock perfect? Kind of, because it has timing assumptions.
https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html
Response from Redis author http://antirez.com/news/101

20. Tips
● Be aware of trade-off. Do you care about efficiency or correctness?
● Test your endpoints with concurrent requests to reproduce problem
● Use transactions when changing many records
● Use unique index to ensure data consistency in DB
● Use tested distribution lock solutions

21. Nice to read
https://makandracards.com/makandra/31937-differences-between-transactions-an
d-locking

22. Thanks