An Intro To Azure Batch Service

1. Azure Batch Service
George Chrysovalantis Grammatikos
Cloud Platform MCSA | Microsoft Azure MVP
E-mail: george@cloudopszone.com
Blog : https://cloudopszone.com
TechNet Wiki Profile: George Chrysovalantis Grammatikos

2. What is Azure Batch Service?
“ Use Azure Batch to run large-scale
parallel and high-performance computing
(HPC) batch jobs efficiently in Azure. Azure
Batch creates and manages a pool of
compute nodes (virtual machines), installs
the applications you want to run, and
schedules jobs to run on the nodes. ”

3. Cloud Basic Principals
Resources
Self Service
Scalability
Billing by consumption

4. Microsoft Azure Virtual Machines Types
General Purpose
Compute Optimized
Memory Optimized
Storage Optimized
GPU
High Performance Compute

5. Azure Batch Features
Resource Management
Process Management
Resource & Process Monitoring

6. • Node Management
• Auto-Scaling
• Low Priority Nodes
• Applications
Resource Management

7. Node Management
Cloud Service Worker Roles
Virtual Machines
Third Party Applications

8. Auto-Scaling
Configure the number of nodes in a pool
Auto Scale Formula, based on service variables (CPU, Memory, Number of tasks)
startingNumberOfVMs = 1;
maxNumberofVMs = 25;
pendingTaskSamplePercent = $PendingTasks.GetSamplePercent(180 * TimeInterval_Second);
pendingTaskSamples = pendingTaskSamplePercent < 70 ? startingNumberOfVMs :
avg($PendingTasks.GetSample(180 * TimeInterval_Second));
$TargetDedicatedNodes=min(maxNumberofVMs, pendingTaskSamples);

9. Low Priority Nodes

10. Applications
• Zip Package containing executables
• Deployed to nodes when required
• Pools – Deployed to all nodes in pool
• Task – Deployed to nodes running the task

11. Process Management
Job and Task Management
• Scheduling
• Re-try tasks on failure
• Max task duration
• Tasks may depends from other tasks
• Explicitly defined on task creation
Task Dependencies
Multi-Instance Tasks
• Run tasks on multiple compute nodes
• Inter-node communication required on pool

12. Resource & Process Monitoring
Azure Portal
Batch Explorer (Batch Labs)
App Insights
Metrics using API
• Rest
• .NET
• Azure CLI
• Azure PowerShell

13. Parallel Job Processing
Job
Tasks
Result

14. 1. Upload
Application,
Input Files
2. Create
Pool, Jobs,
Tasks
3. Download
App, Input
File & Run
Tasks
4. Monitor
Tasks
5. Upload
Output Files
6.
Download
Output
Files
Azure Batch Architecture

15. Azure Batch Architecture (Based On The Demo)

16. Azure Batch Processing Architecture (Based On The Demo)
Application Jobs
Input Files Output Files

17. Resource Quotas
Resource Default Limit Maximum Limit
Batch accounts per region per
subscription
1-3 50
Dedicated cores per Batch account 10-100 N/A
Low-priority cores per Batch account 10-100 N/A
Active jobs and job schedules per
Batch account
100-300 1000
Pools per Batch account 20-100 500

18. Pool Size Limits
Resource Maximum Limit
Compute nodes in inter-node communication enabled pool
Batch service pool allocation mode 100
Batch subscription pool allocation mode 80
Compute nodes in
Dedicated nodes 2000
Low-priority nodes 1000

19. NOT Supported VM Sizes
Family Unsupported
Sizes
Basic A-series Basic_A0(A0)
A-series Standard_A0
B-series All
DC-series All
Extreme memory
optimized
All
Hb-series* All
Hc-series* All
Lsv2-series* All
NDv2-series* All
NVv2-series* All
SAP-HANA All
* Not currently supported, but will be supported in the future

20. Batch Account Credentials
Storage Account Credentials
Pool And Job Constants
Batch .NET Project Basic Constants

21. Simulation Examples
• Education: Monte Carlo Simulation (solve complex math problems)
• Computer Software Industry: gaming
• Dev / DevOps : (Common Test execution framework used to manage and
• schedule tasks)
• Motor Racing: F1(Complex simulation and predictions)
• Economy: banks,
• Animation Rendering (Single Frame is rendered as its own task)

22. Azure Batch Pricing
Azure Batch is a Free service. Charges accrue for the underlying resources
that run your Batch workloads. Configure Cloud Services and Virtual Machines
to estimate the cost of your Batch pools.
TIER: Low priority
INSTANCE: A1: 1 cores(s), 1.75 GB RAM, 225 GB Temporary storage, €0.027/hour
No Of Instances: 5
No Of Days: 1
Total Cost: € 3.64
Storage:
Bandwitdth: € 0.02

23. Microsoft Azure Compute

24. Demo
Azure Batch .NET File Processing With ffmpeg

25. Useful Tools & Links
Azure Batch Tools
 Azure Portal
 Azure Batch GitHub
 Azure CLI
 Azure Storage Explorer
 AzCopy Tool
 Blobxfer Tool
 Azure Batch Explorer
Docker Support
 Batch Shipyard
Feedback & Forums
 Feedback
Pricing
 Cloud Services
 Windows Virtual Machines
 Linux Virtual Machines

26. Resources
 Azure Batch Documentation
 Azure Batch Code Samples Repo
 Batch service quotas and limits Jump
 VM sizes for compute nodes in an Azure Batch pool Jump
 Tutorial: Run a parallel workload with Azure Batch using the .NET API Jump
 Azure Batch Videos - Channel 9 Jump
 https://azure.microsoft.com/en-us/pricing/details/batch/
 Create an automatic scaling formula for scaling compute nodes in a Batch pool
Jump
 Batch computing at a fraction of the price
 Real-world Applications of Simulations
 Simulation Examples

27. Help us improve and organize even better meetups