1. using Queue Service
for batch processing

2. Job Queue
data source
convert the data into job and sent to queue
1

3. Job Queue
data source
1
convert the data into job and sent to queue
workerworkerworkerworkerworker
2
pull job and consume it

4. Job Queue
data source
1
convert the data into job and sent to queue
workerworkerworkerworkerworker
2
pull job and consume it
Result Queue
3
send result to another queue

5. Job Queue
data source
1
convert the data into job and sent to queue
workerworkerworkerworkerworker
2
pull job and consume it
Result Queue
3
send result to another queue
4
feed back to data
aggregator
pull result

6. Job Queue
data source
1
convert the data into job and sent to queue
workerworkerworkerworkerworker
2
pull job and consume it
Result Queue
3
send result to another queue
4
aggregator
pull result
make report

7. Example
for video resolution

8. Job Queue
video table
{id, url, how-to-parse}
1

9. Job Queue
1
workerworkerworkerworkerworker
2
pull job and consume it
video table
{id, url, how-to-parse}

10. Job Queue
1
workerworkerworkerworkerworker
2
pull job and consume it
Result Queue
3
{id, width, height, codec}
video table
{id, url, how-to-parse}

11. Job Queue
1
workerworkerworkerworkerworker
2
pull job and consume it
Result Queue
3
update video where id=?
set width=?, height=?, codec=?
video table
{id, url, how-to-parse}
{id, width, height, codec}
4
aggregator
pull result

12. Example
for streaming alive

13. Job Queue
streaming table
{id, url}
1

14. Job Queue
1
workerworkerworkerworkerworker
2
pull job and consume it
streaming table
{id, url}

15. Job Queue
1
workerworkerworkerworkerworker
2
pull job and consume it
Result Queue
3
{id, is_alive}
streaming table
{id, url}

16. Job Queue
1
workerworkerworkerworkerworker
2
pull job and consume it
Result Queue
3
update streaming_status
where id=?
set status=?
4
aggregator
pull result
streaming table
{id, url}
{id, is_alive}

17. Deployment

18. Job Queue
workerworkerworkerworkerworker
Result Queue
aggregator
Everything can deploy locally,
But

19. Job Queue
workerworkerworkerworkerworker
Result Queue
aggregator
Deploy Workers on the Cloud
let it more scalable