SlideShare une entreprise Scribd logo
1  sur  10
Télécharger pour lire hors ligne
Redis
As a Message Queue
1Wednesday, October 9, 13
The Plug
Essentially, a screenshot site
MMORPG (Massively-Multiplayer, Online
Role-Playing Game) niche market
Site visitors browse games, games have
albums
Players can upload screenshots
2Wednesday, October 9, 13
The Architecture
(2) 6-core CPU, 16GB RAM, 6-drive 3TB RAID10
(1) 8-core CPU, 16GB RAM, 2-drive 64GB RAID1 SSD, 2-drive 320GB RAID1
(2) layer-4 ipvsadm load balancers in master/backup failover via
keepalived
(3) PostgreSQL master/slave database
(3) nginx+php-fpm (5.5) app servers
Replicated file system via GlusterFS for images
Memcached instance for php sessions
Redis for caching and message queue
Amazon Cloudfront for CDN (content delivery network)
PhalconPHP framework, Gmagick, Twitter Bootstrap v3.0, REST API
3Wednesday, October 9, 13
Message Queue?
Push messages into a stack
Pop (pull) messages off a stack
You put (push) data (messages) into the
queue (stack)
Messages are things you want to process
later by pulling them from the queue
4Wednesday, October 9, 13
Messages?
A “message” is just some data
Could be an array
Could be a string
5Wednesday, October 9, 13
The Use-Case
When a user uploads images, it is bad to
block the user while your server processes
the photos
Image resizing
Creating multiple versions
Creating database entries
6Wednesday, October 9, 13
The User Flow
User selects images
Images are uploaded to app server
App server generates a UUID filename and
copies the images to the replicated file system
App server increments a Redis counter
App server creates an array with uuid, user id,
game id and pushes this “message” into the Redis
list (message queue)
7Wednesday, October 9, 13
Batch Processing
A separate PHP/other process running on any of
the app servers (crontab, forking daemon) pops
the Redis list to get a message
By using GlusterFS as a replicated file system all
app servers have all the uploads/images
When a message is obtained from Redis queue, it
processes the file upload, creates various image
sizes
Creates a database entry in PostgreSQL
8Wednesday, October 9, 13
Lets push!
// Create a new redis connection
$r = new Redis();
$r->connect($host, $port);
// Create the message array
$message = [‘uuid’ => $uuid, ‘userId’ => $userId, ‘gameId’ => $gameId];
// Get a new message id by implementing the builtin increment feature of Redis
$id = $r->incr('mmoscreens:screens-queue');
// Push the id into the queue (Redis list)
$r->lpush('mmoscreens:screens-queue', $id);
// Set the $message into Redis
$r->hmset("mmoscreens:screens-queue:$id", $message);
9Wednesday, October 9, 13
Lets pull!
// Create a new redis connection
$r = new Redis()->connect($host, $port);
// pop the message ID from the queue, or block until one is available
$id = $r->brPop('mmoscreens:screens-queue', BLOCK_WAIT_TIME);
// brPop returns [listName, element]
isset($id[1]) && $id = $id[1];
// Get the message from Redis
$message = $r->hgetall("mmoscreens:screens-queue:$id");
$message:
[
‘uuid’ => ‘fef4430c-e4a8-4874-b380-63edd2fa8472’,
‘userId’ => 10611,
‘gameId’ => 8
]
10Wednesday, October 9, 13

Contenu connexe

Tendances

Building a DSL with GraalVM (VoxxedDays Luxembourg)
Building a DSL with GraalVM (VoxxedDays Luxembourg)Building a DSL with GraalVM (VoxxedDays Luxembourg)
Building a DSL with GraalVM (VoxxedDays Luxembourg)Maarten Mulders
 
PL/Perl - New Features in PostgreSQL 9.0
PL/Perl - New Features in PostgreSQL 9.0PL/Perl - New Features in PostgreSQL 9.0
PL/Perl - New Features in PostgreSQL 9.0Tim Bunce
 
Node.js streaming csv downloads proxy
Node.js streaming csv downloads proxyNode.js streaming csv downloads proxy
Node.js streaming csv downloads proxyIsmael Celis
 
Object Storage with Gluster
Object Storage with GlusterObject Storage with Gluster
Object Storage with GlusterGluster.org
 
Application Logging in the 21st century - 2014.key
Application Logging in the 21st century - 2014.keyApplication Logging in the 21st century - 2014.key
Application Logging in the 21st century - 2014.keyTim Bunce
 
PL/Perl - New Features in PostgreSQL 9.0 201012
PL/Perl - New Features in PostgreSQL 9.0 201012PL/Perl - New Features in PostgreSQL 9.0 201012
PL/Perl - New Features in PostgreSQL 9.0 201012Tim Bunce
 
Using Node.js to Build Great Streaming Services - HTML5 Dev Conf
Using Node.js to  Build Great  Streaming Services - HTML5 Dev ConfUsing Node.js to  Build Great  Streaming Services - HTML5 Dev Conf
Using Node.js to Build Great Streaming Services - HTML5 Dev ConfTom Croucher
 
Perl at SkyCon'12
Perl at SkyCon'12Perl at SkyCon'12
Perl at SkyCon'12Tim Bunce
 
Static Typing in Vault
Static Typing in VaultStatic Typing in Vault
Static Typing in VaultGlynnForrest
 
On UnQLite
On UnQLiteOn UnQLite
On UnQLitecharsbar
 
agri inventory - nouka data collector / yaoya data convertor
agri inventory - nouka data collector / yaoya data convertoragri inventory - nouka data collector / yaoya data convertor
agri inventory - nouka data collector / yaoya data convertorToshiaki Baba
 
Building a DSL with GraalVM (CodeOne)
Building a DSL with GraalVM (CodeOne)Building a DSL with GraalVM (CodeOne)
Building a DSL with GraalVM (CodeOne)Maarten Mulders
 
Perl Memory Use 201209
Perl Memory Use 201209Perl Memory Use 201209
Perl Memory Use 201209Tim Bunce
 
SCALE 15x Minimizing PostgreSQL Major Version Upgrade Downtime
SCALE 15x Minimizing PostgreSQL Major Version Upgrade DowntimeSCALE 15x Minimizing PostgreSQL Major Version Upgrade Downtime
SCALE 15x Minimizing PostgreSQL Major Version Upgrade DowntimeJeff Frost
 
Perl Memory Use - LPW2013
Perl Memory Use - LPW2013Perl Memory Use - LPW2013
Perl Memory Use - LPW2013Tim Bunce
 
Working with databases in Perl
Working with databases in PerlWorking with databases in Perl
Working with databases in PerlLaurent Dami
 
Using ngx_lua in UPYUN 2
Using ngx_lua in UPYUN 2Using ngx_lua in UPYUN 2
Using ngx_lua in UPYUN 2Cong Zhang
 
Streams are Awesome - (Node.js) TimesOpen Sep 2012
Streams are Awesome - (Node.js) TimesOpen Sep 2012 Streams are Awesome - (Node.js) TimesOpen Sep 2012
Streams are Awesome - (Node.js) TimesOpen Sep 2012 Tom Croucher
 

Tendances (20)

Building a DSL with GraalVM (VoxxedDays Luxembourg)
Building a DSL with GraalVM (VoxxedDays Luxembourg)Building a DSL with GraalVM (VoxxedDays Luxembourg)
Building a DSL with GraalVM (VoxxedDays Luxembourg)
 
PL/Perl - New Features in PostgreSQL 9.0
PL/Perl - New Features in PostgreSQL 9.0PL/Perl - New Features in PostgreSQL 9.0
PL/Perl - New Features in PostgreSQL 9.0
 
Node.js streaming csv downloads proxy
Node.js streaming csv downloads proxyNode.js streaming csv downloads proxy
Node.js streaming csv downloads proxy
 
Object Storage with Gluster
Object Storage with GlusterObject Storage with Gluster
Object Storage with Gluster
 
Application Logging in the 21st century - 2014.key
Application Logging in the 21st century - 2014.keyApplication Logging in the 21st century - 2014.key
Application Logging in the 21st century - 2014.key
 
PL/Perl - New Features in PostgreSQL 9.0 201012
PL/Perl - New Features in PostgreSQL 9.0 201012PL/Perl - New Features in PostgreSQL 9.0 201012
PL/Perl - New Features in PostgreSQL 9.0 201012
 
Using Node.js to Build Great Streaming Services - HTML5 Dev Conf
Using Node.js to  Build Great  Streaming Services - HTML5 Dev ConfUsing Node.js to  Build Great  Streaming Services - HTML5 Dev Conf
Using Node.js to Build Great Streaming Services - HTML5 Dev Conf
 
Perl at SkyCon'12
Perl at SkyCon'12Perl at SkyCon'12
Perl at SkyCon'12
 
Static Typing in Vault
Static Typing in VaultStatic Typing in Vault
Static Typing in Vault
 
Everything as Code with Terraform
Everything as Code with TerraformEverything as Code with Terraform
Everything as Code with Terraform
 
On UnQLite
On UnQLiteOn UnQLite
On UnQLite
 
agri inventory - nouka data collector / yaoya data convertor
agri inventory - nouka data collector / yaoya data convertoragri inventory - nouka data collector / yaoya data convertor
agri inventory - nouka data collector / yaoya data convertor
 
Building a DSL with GraalVM (CodeOne)
Building a DSL with GraalVM (CodeOne)Building a DSL with GraalVM (CodeOne)
Building a DSL with GraalVM (CodeOne)
 
Perl Memory Use 201209
Perl Memory Use 201209Perl Memory Use 201209
Perl Memory Use 201209
 
SCALE 15x Minimizing PostgreSQL Major Version Upgrade Downtime
SCALE 15x Minimizing PostgreSQL Major Version Upgrade DowntimeSCALE 15x Minimizing PostgreSQL Major Version Upgrade Downtime
SCALE 15x Minimizing PostgreSQL Major Version Upgrade Downtime
 
tdc2012
tdc2012tdc2012
tdc2012
 
Perl Memory Use - LPW2013
Perl Memory Use - LPW2013Perl Memory Use - LPW2013
Perl Memory Use - LPW2013
 
Working with databases in Perl
Working with databases in PerlWorking with databases in Perl
Working with databases in Perl
 
Using ngx_lua in UPYUN 2
Using ngx_lua in UPYUN 2Using ngx_lua in UPYUN 2
Using ngx_lua in UPYUN 2
 
Streams are Awesome - (Node.js) TimesOpen Sep 2012
Streams are Awesome - (Node.js) TimesOpen Sep 2012 Streams are Awesome - (Node.js) TimesOpen Sep 2012
Streams are Awesome - (Node.js) TimesOpen Sep 2012
 

Similaire à Redis as a message queue

containerit at useR!2017 conference, Brussels
containerit at useR!2017 conference, Brusselscontainerit at useR!2017 conference, Brussels
containerit at useR!2017 conference, BrusselsDaniel Nüst
 
Hopping in clouds: a tale of migration from one cloud provider to another
Hopping in clouds: a tale of migration from one cloud provider to anotherHopping in clouds: a tale of migration from one cloud provider to another
Hopping in clouds: a tale of migration from one cloud provider to anotherMichele Orselli
 
Got database access? Own the network!
Got database access? Own the network!Got database access? Own the network!
Got database access? Own the network!Bernardo Damele A. G.
 
iRODS Globus Connector
iRODS Globus ConnectoriRODS Globus Connector
iRODS Globus ConnectorGlobus
 
Sql injection with sqlmap
Sql injection with sqlmapSql injection with sqlmap
Sql injection with sqlmapHerman Duarte
 
MacOS forensics and anti-forensics (DC Lviv 2019) presentation
MacOS forensics and anti-forensics (DC Lviv 2019) presentationMacOS forensics and anti-forensics (DC Lviv 2019) presentation
MacOS forensics and anti-forensics (DC Lviv 2019) presentationOlehLevytskyi1
 
Big Data Summer training presentation
Big Data Summer training presentationBig Data Summer training presentation
Big Data Summer training presentationHarshitaKamboj
 
IPC mechanisms in windows
IPC mechanisms in windowsIPC mechanisms in windows
IPC mechanisms in windowsVinoth Raj
 
import rdma: zero-copy networking with RDMA and Python
import rdma: zero-copy networking with RDMA and Pythonimport rdma: zero-copy networking with RDMA and Python
import rdma: zero-copy networking with RDMA and Pythongroveronline
 
maxbox starter72 multilanguage coding
maxbox starter72 multilanguage codingmaxbox starter72 multilanguage coding
maxbox starter72 multilanguage codingMax Kleiner
 
Building and Scaling Node.js Applications
Building and Scaling Node.js ApplicationsBuilding and Scaling Node.js Applications
Building and Scaling Node.js ApplicationsOhad Kravchick
 
Migrating the elastic stack to the cloud, or application logging @ travix
 Migrating the elastic stack to the cloud, or application logging @ travix Migrating the elastic stack to the cloud, or application logging @ travix
Migrating the elastic stack to the cloud, or application logging @ travixRuslan Lutsenko
 
Hopping in clouds - phpuk 17
Hopping in clouds - phpuk 17Hopping in clouds - phpuk 17
Hopping in clouds - phpuk 17Michele Orselli
 
IPC in Microkernel Systems, Capabilities
IPC in Microkernel Systems, CapabilitiesIPC in Microkernel Systems, Capabilities
IPC in Microkernel Systems, CapabilitiesMartin Děcký
 
Docker Multi-arch All The Things
Docker Multi-arch All The ThingsDocker Multi-arch All The Things
Docker Multi-arch All The ThingsDocker, Inc.
 
Towards a Serverless Platform for Edge AI
Towards a Serverless Platform for Edge AITowards a Serverless Platform for Edge AI
Towards a Serverless Platform for Edge AIThomas Rausch
 
Prosit google-cloud
Prosit google-cloudProsit google-cloud
Prosit google-cloudUC Davis
 
Охота на уязвимости Hadoop
Охота на уязвимости HadoopОхота на уязвимости Hadoop
Охота на уязвимости HadoopPositive Hack Days
 
Node.js - A practical introduction (v2)
Node.js  - A practical introduction (v2)Node.js  - A practical introduction (v2)
Node.js - A practical introduction (v2)Felix Geisendörfer
 
Building Hopsworks, a cloud-native managed feature store for machine learning
Building Hopsworks, a cloud-native managed feature store for machine learning Building Hopsworks, a cloud-native managed feature store for machine learning
Building Hopsworks, a cloud-native managed feature store for machine learning Jim Dowling
 

Similaire à Redis as a message queue (20)

containerit at useR!2017 conference, Brussels
containerit at useR!2017 conference, Brusselscontainerit at useR!2017 conference, Brussels
containerit at useR!2017 conference, Brussels
 
Hopping in clouds: a tale of migration from one cloud provider to another
Hopping in clouds: a tale of migration from one cloud provider to anotherHopping in clouds: a tale of migration from one cloud provider to another
Hopping in clouds: a tale of migration from one cloud provider to another
 
Got database access? Own the network!
Got database access? Own the network!Got database access? Own the network!
Got database access? Own the network!
 
iRODS Globus Connector
iRODS Globus ConnectoriRODS Globus Connector
iRODS Globus Connector
 
Sql injection with sqlmap
Sql injection with sqlmapSql injection with sqlmap
Sql injection with sqlmap
 
MacOS forensics and anti-forensics (DC Lviv 2019) presentation
MacOS forensics and anti-forensics (DC Lviv 2019) presentationMacOS forensics and anti-forensics (DC Lviv 2019) presentation
MacOS forensics and anti-forensics (DC Lviv 2019) presentation
 
Big Data Summer training presentation
Big Data Summer training presentationBig Data Summer training presentation
Big Data Summer training presentation
 
IPC mechanisms in windows
IPC mechanisms in windowsIPC mechanisms in windows
IPC mechanisms in windows
 
import rdma: zero-copy networking with RDMA and Python
import rdma: zero-copy networking with RDMA and Pythonimport rdma: zero-copy networking with RDMA and Python
import rdma: zero-copy networking with RDMA and Python
 
maxbox starter72 multilanguage coding
maxbox starter72 multilanguage codingmaxbox starter72 multilanguage coding
maxbox starter72 multilanguage coding
 
Building and Scaling Node.js Applications
Building and Scaling Node.js ApplicationsBuilding and Scaling Node.js Applications
Building and Scaling Node.js Applications
 
Migrating the elastic stack to the cloud, or application logging @ travix
 Migrating the elastic stack to the cloud, or application logging @ travix Migrating the elastic stack to the cloud, or application logging @ travix
Migrating the elastic stack to the cloud, or application logging @ travix
 
Hopping in clouds - phpuk 17
Hopping in clouds - phpuk 17Hopping in clouds - phpuk 17
Hopping in clouds - phpuk 17
 
IPC in Microkernel Systems, Capabilities
IPC in Microkernel Systems, CapabilitiesIPC in Microkernel Systems, Capabilities
IPC in Microkernel Systems, Capabilities
 
Docker Multi-arch All The Things
Docker Multi-arch All The ThingsDocker Multi-arch All The Things
Docker Multi-arch All The Things
 
Towards a Serverless Platform for Edge AI
Towards a Serverless Platform for Edge AITowards a Serverless Platform for Edge AI
Towards a Serverless Platform for Edge AI
 
Prosit google-cloud
Prosit google-cloudProsit google-cloud
Prosit google-cloud
 
Охота на уязвимости Hadoop
Охота на уязвимости HadoopОхота на уязвимости Hadoop
Охота на уязвимости Hadoop
 
Node.js - A practical introduction (v2)
Node.js  - A practical introduction (v2)Node.js  - A practical introduction (v2)
Node.js - A practical introduction (v2)
 
Building Hopsworks, a cloud-native managed feature store for machine learning
Building Hopsworks, a cloud-native managed feature store for machine learning Building Hopsworks, a cloud-native managed feature store for machine learning
Building Hopsworks, a cloud-native managed feature store for machine learning
 

Dernier

Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 

Dernier (20)

Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 

Redis as a message queue

  • 1. Redis As a Message Queue 1Wednesday, October 9, 13
  • 2. The Plug Essentially, a screenshot site MMORPG (Massively-Multiplayer, Online Role-Playing Game) niche market Site visitors browse games, games have albums Players can upload screenshots 2Wednesday, October 9, 13
  • 3. The Architecture (2) 6-core CPU, 16GB RAM, 6-drive 3TB RAID10 (1) 8-core CPU, 16GB RAM, 2-drive 64GB RAID1 SSD, 2-drive 320GB RAID1 (2) layer-4 ipvsadm load balancers in master/backup failover via keepalived (3) PostgreSQL master/slave database (3) nginx+php-fpm (5.5) app servers Replicated file system via GlusterFS for images Memcached instance for php sessions Redis for caching and message queue Amazon Cloudfront for CDN (content delivery network) PhalconPHP framework, Gmagick, Twitter Bootstrap v3.0, REST API 3Wednesday, October 9, 13
  • 4. Message Queue? Push messages into a stack Pop (pull) messages off a stack You put (push) data (messages) into the queue (stack) Messages are things you want to process later by pulling them from the queue 4Wednesday, October 9, 13
  • 5. Messages? A “message” is just some data Could be an array Could be a string 5Wednesday, October 9, 13
  • 6. The Use-Case When a user uploads images, it is bad to block the user while your server processes the photos Image resizing Creating multiple versions Creating database entries 6Wednesday, October 9, 13
  • 7. The User Flow User selects images Images are uploaded to app server App server generates a UUID filename and copies the images to the replicated file system App server increments a Redis counter App server creates an array with uuid, user id, game id and pushes this “message” into the Redis list (message queue) 7Wednesday, October 9, 13
  • 8. Batch Processing A separate PHP/other process running on any of the app servers (crontab, forking daemon) pops the Redis list to get a message By using GlusterFS as a replicated file system all app servers have all the uploads/images When a message is obtained from Redis queue, it processes the file upload, creates various image sizes Creates a database entry in PostgreSQL 8Wednesday, October 9, 13
  • 9. Lets push! // Create a new redis connection $r = new Redis(); $r->connect($host, $port); // Create the message array $message = [‘uuid’ => $uuid, ‘userId’ => $userId, ‘gameId’ => $gameId]; // Get a new message id by implementing the builtin increment feature of Redis $id = $r->incr('mmoscreens:screens-queue'); // Push the id into the queue (Redis list) $r->lpush('mmoscreens:screens-queue', $id); // Set the $message into Redis $r->hmset("mmoscreens:screens-queue:$id", $message); 9Wednesday, October 9, 13
  • 10. Lets pull! // Create a new redis connection $r = new Redis()->connect($host, $port); // pop the message ID from the queue, or block until one is available $id = $r->brPop('mmoscreens:screens-queue', BLOCK_WAIT_TIME); // brPop returns [listName, element] isset($id[1]) && $id = $id[1]; // Get the message from Redis $message = $r->hgetall("mmoscreens:screens-queue:$id"); $message: [ ‘uuid’ => ‘fef4430c-e4a8-4874-b380-63edd2fa8472’, ‘userId’ => 10611, ‘gameId’ => 8 ] 10Wednesday, October 9, 13