3. Gobblin on AWS Architecture
Workers Auto Scaling Group
Worker
Master ASG
Master
Network File System / EFS
W W W
Zookeeper
Helix
Task assignmentJob Submission
Master / Driver
Services:
Helix Controller
Job Conf Manager
Job Scheduler
Worker
Services:
Task Executor
Worker Worker
Launcher
Services:
AWS Security Manager
AWS Coordinator
Provisions all resourcse
By coordinating with AWS
Request ASGs
for Master and
Workers
Launch ASG and Instances
ASG = Auto Scaling Group
5. Next Steps
1 Multi master provisioning and HA
2 Replace NFS with EFS
3 Support for Spot Instances
4 Continuous Execution
5 Advanced hooks for ASG and exploit Helix
56 AMI coming up
6. Gobblin as a Service Preview Architecture
Workers Auto Scaling Group
Worker
Master ASG
Master
Network File System / EFS
W W W
Zookeeper
Helix
Task assignmentJob Submission
Master / Driver
Services:
Helix Controller
Job Conf Manager
Job Scheduler
Worker
Services:
Task Executor
Worker Worker
Request ASGs
for Master and
Workers
Launch ASG and Instances
Service / Launcher
Services:
AWS Security Manager
AWS Coordinator
UI Server
Monitoring Service
Job
Factory
Upload Job
Configs To S3
Launch
Zookeeper
ASG = Auto Scaling Group
8. Next Steps
1 Open up more Sources and Sinks
2 Add Graphs (Graphite / Graphana) and comprehensive metrics
3 More cluster types (Azure, Google cloud)
4 Continuous Execution
5 Open to public. Register for preview at: gobblin.io
8