https://tw.pycon.org/2021/en-us/conference/talk/194

1. 2021/10/02 PyCon Taiwan
Queick: A Simple Job Queue
System for Python
Ryota SUENAGA
1

2. Me
• Ryota SUENAGA a.k.a asmsuechan
• A new grad software developer living in Tokyo, Japan
• Working for M3, Inc.
• Like: markdown
• Former commiter of a markdown editor
(Boostnote)
• Creator of a simple markdown parser, minute
• 实际上，我会说一点儿中文
2

3. 0. Background story
3

4. 0.1 Background story
My prof: “Could you please make entry/exit management system for our lab?”
Me: “Yes, sir”
Someday when I was a master’s student (1 year ago)
~2 days later~
Me: “OK, it’s done. The system status is all-green”
4

5. 0.2 Background story
This system consists of Raspberry Pi, PaSori RC-S380 NFC Card Scanner, a speaker, Slack
The entrance and exit time is posted on our Slack channel
5

6. 0.3 Background story
The system was working fine for a few days…
But some troubles were discovered
• Network connection was not good every few hours
• I should use a job-queue system to retry
• But using Redis is too much for other students…
So, I created an easy one!
6

7. 1. Overview
7

8. 1.1 What I did
I developed a job-queue system in Python
Its concept is “Deadly Simple.”
8

9. 1.2 Job-queue system?
• Main task: Deligating jobs to another process
• e.g: image processing, sending e-mails, uploading files
Worker process
Queue
Application
enqueue a job
take out the job
enqueue success
Executing…
time↓
9

10. 1.3 Queick
• Light and simple job-queue system
• Only Python standard libraries
• Thread-based programming
https://github.com/asmsuechan/queick
10

11. 1.4 Queick
2 jobs that only print a text with sleep are enqueued
11

12. • Asynchronous execution
• Periodic execution
• Be able to specify exact time
• Scheduling execution
• Retry
1.5 Queick’s features and characteristics
• No Redis (stored in-memory) and no persistence
• Built by only Python standard libraries
• Failed job retry when network connection returns
• Thread based job execution
Features
Characteristics
12

13. 1.6 Queick usage
enqueue(), enqueue_at()
cron()
These interfaces are implemented based on RQ
13

14. • time
• argparse
• importlib
• traceback
• logging
• random
• concurrent.futures
• types
1.7 Standard libraries used in Queick
• typing
• socket
• pickle
• multiprocessing
• os
• sched
• sys
• unittest
14

15. 2. General job-queue system
15

16. 2.1 General purpose to use job-queue system
1. Asynchronous job execution for a heavy task
• Background processing
2. Scheduling
• job management
• e.g. running a function every 1 minute
16

17. 2.2 How to realize them easily in Python
↑Job side (application)
←Job manager (ran as another process)
17

18. 2.3 GIL (Global Interpretor Lock)
• GIL prevents multiple threads working simultaneously
• multi-threads programming in Python don’t work intuitively
• CPU cannot utilize multi cores
• Processing CPU-bounded tasks are not suitable
• Only CPython
A. Because Queick’s concept is “Deadly Simple”.
Q. “Why did you choose multi-thread?”
18

19. 2.4 Why Job-queue is needed
• Abstraction thread and queue
• We can be freed from multi-thread programming and queue management
• Job management
• Jobs can be easily managed if UI is implemented
asynchrounos execution and scheduling is possible
without any job-queue systems technically. So why?
Main points
19

20. 2.5 Base implementation of a Job-queue system
Very simple example
• Multi-Thread
• Queue
simple-job-queue-example.py 20

21. 2.6 Examples
rq/rq ⭐7.9K
celery/celery ⭐17.9K
coleifer/huey ⭐3.6k
21

22. 2.7 RQ quick introduction
• RQ is Redis Queue
• ⭐8K
• Queick is affected by RQ because I like RQ’s interface and its simpleness
RQ (Redis Queue) is a simple Python library for queueing jobs and
processing them in the background with workers.
22

23. 2.8 RQ API
• Very simple
• enqueue
• enqueue_at
• enqueue_in
23

24. 2.9 RQ architecture
RQ Architecture
Application
• Job is executed on a child process
• Scheduler runs as another process
• Job retry is ran on the scheduler
Scheduler process
Worker process
Child process
fork()
fork()
Child process
RQ
Retry when failed
Check Queue every 1 sec
subscribe
enqueue
JobRegistry
redis
24

25. 2.10 For more information
• RQ: Simple job queues for Python (https://python-rq.org/)
• rq/rq (https://github.com/rq/rq)
• Full Stack Python (https://www.fullstackpython.com/task-queues.html)
• Introducing RQ (https://nvie.com/posts/introducing-rq/)
25

26. 3. Queick architecture deep dive
26

27. 3.1 Job data structure
• Dictionary
• function name, retry flag, and other options
27

28. 3.2 Finding a job
• importlib
• This method imports a module dinamically from its name.
28

29. 3.3 Architecture and its internal structure
Application
29

30. 3.3 Architecture and its internal structure
Job
30

31. 3.3 Architecture and its internal structure
Job
31

32. 3.4 Network connection check
Retry when returning to online
• Network connection check by another process
• Check connection every 1 sec
• Enqueue all when the state changes disconnected
to connected
Failed JobQueue
Worker process
Network Checker
Run
Thread
Failed retry_on_network_availabe == True
TCP Server
JobQueue
Queick
32

33. 3.4 Network connection check
Retry when returning to online
Failed JobQueue
Worker process
Network Checker
Run
Thread
Failed retry_on_network_availabe == True
TCP Server
JobQueue
Queick
Application
• Network connection check by another process
• Check connection every 1 sec
• Enqueue all when the state changes disconnected
to connected
33

34. 3.4 Network connection check
Retry when returning to online
Failed JobQueue
Worker process
Network Checker
Run
Thread
Failed retry_on_network_availabe == True
TCP Server
JobQueue
Queick
Execution
• Network connection check by another process
• Check connection every 1 sec
• Enqueue all when the state changes disconnected
to connected
34

35. 3.4 Network connection check
Retry when returning to online
Failed JobQueue
Worker process
Network Checker
Run
Thread
Failed retry_on_network_availabe == True
TCP Server
JobQueue
Queick
Execution
• Network connection check by another process
• Check connection every 1 sec
• Enqueue all when the state changes disconnected
to connected
35

36. 3.4 Network connection check
Retry when returning to online
Failed JobQueue
Worker process
Network Checker
Run
Thread
Failed retry_on_network_availabe == True
TCP Server
JobQueue
Queick
• Network connection check by another process
• Check connection every 1 sec
• Enqueue all when the state changes disconnected
to connected
36

37. 3.5 Testing
Ofcourse Queick has unit tests by unittest
But testing for multi-thread programming is difficult…
Integration tests are used to assure if Queick works as expected
37

38. 3.5 Testing
How I implement integration tests?
Integration tests inside a Docker image of Queick
Executing a procedure of “writing a line to a file” on Queick
Confirm if the file has a correct content
Job
Write “a”
Write “b”
a
b
38

39. 4. Demonstration
39
If time is left enough

40. 5. Summary and conclusion
40

41. 5.1 OSS development
• In OSS development, “Spped” is justice
• “Done is better than perfect”, especially in indivisual development
• human being is lazy animal
• Intensive development until the first release
• Publishing a blog entry and posting to Reddit are effective to reach first users
• OSS activiteis include information trasmission
41

42. 5.2 Summary / Conclusion
• Queick is a “Deadly Simple” job-queue system
• Thread based job-queue system
• built by only standard libraries
• More application examples are needed
• Building a software by only standard libraries is a lot of fun
• “Queick” comes from Queue + Quick
• The core of Queick was made in 2 days, yes, my developing speed was
quick, therefore, queick.
42

43. Thank you for listening!
Any questions?
43