Contenu connexe Similaire à Containerising the Mule Runtime with Kubernetes & From Zero to Batch : MuleSoft Virtual Muleys Meetups (20) Plus de Angel Alberici (6) Containerising the Mule Runtime with Kubernetes & From Zero to Batch : MuleSoft Virtual Muleys Meetups1. All contents © MuleSoft, LLC
From zero to Batch
Anu Vijayamohan
Containerizing the Mule
Runtime with
Kubernetes
Arno Brugman
18th November
17:00 London (GMT) | 18:00 Europe | 22:30 India (Delhi)
9:00 San Francisco | 12:00 New York
2. All contents © MuleSoft, LLC
MuleSoft Meetups: Virtual Muleys
(Online Group - English )
November, 2020Recording
4. All contents © MuleSoft, LLC
Keep in mind!
1. For Trainings and Certifications - https://training.mulesoft.com/
○ Still NEW: Mule 4 MCD FREE Certification voucher when completing the
Mule 4 Self-Serve course!
2. For Training queries:
https://help.learn.mulesoft.com/hc/en-us/requests/new
3. Doubts?
○ We have a very good community to help you! https://help.mulesoft.com/
○ Or leverage StackOverflow (https://stackoverflow.com/) ( specially for
those working on Dataweave and Runtime)
4
5. All contents © MuleSoft, LLC
Guidelines for today
1. Pop questions in the chat and anyone can try to answer!
2. Remaining questions will be asked in your behalf at the end of the
session
3. The Recording will be shared in the Meetups page and in the
VirtualMuleys Youtube Channel
4. Your feedback drives the content of these meetups, fill the
surveys at the end of the event, let us know what you think!
5
7. All contents © MuleSoft, LLC
Start playing with DW 2.0 from your browser: http://dwlang.fun/
Pre-Alpha: DataWeave Playground & Tutorial
7
Disclaimer: It is not an officially supported MuleSoft Product. It is not supported by
MuleSoft Support, you won't be able to open MuleSoft Support cases for this Suite and
there are no SLAs for responses nor bug/features progress.
8. All contents © MuleSoft, LLC
Start playing at: http://dwlang.fun/
Pre-Alpha: DataWeave Playground & Tutorial
8
Disclaimer: It is not an officially supported MuleSoft Product. It is not supported by
MuleSoft Support, you won't be able to open MuleSoft Support cases for this Suite and
there are no SLAs for responses nor bug/features progress.
9. All contents © MuleSoft, LLC
● Repo and Pull Requests: https://github.com/mulesoft-labs/data-weave-tutorial
“Pre-Alpha” - DataWeave Playground & Tutorial
9
10. All contents © MuleSoft, LLC
Compete in the first virtual MuleSoft Hackathon!
● You have until the Nov. 29, 2020 to submit!
● Amazing prices with lot of fun
● For the event:
https://www.mulesoft.com/hackathon2020
● Recording with all the details:
https://meetups.mulesoft.com/mulesoft-hackatho
n-2020/
● Disclaimer: Country and additional eligibility
restrictions will apply.
12. All contents © MuleSoft, LLC
Marina Rivero
Anypoint Studio 7.7 deep dive with
MuleSoft engineering!
Eric Mastrangelo
25th November
17:00 London (GMT) | 18:00 Europe | 22:30 India (Delhi)
9:00 San Francisco | 12:00 New York
https://meetups.mulesoft.com/e/mphrn6/
13. All contents © MuleSoft, LLC
● 5-15 minutes videos with a lot of good
content.
● Introduction: Continuous Integration
and Continuous Deployment (CICD)
● Overview (Why, what is it) and how you
can articulate the value of CICD to
different Stakeholders
● Find it here:
https://sfdc.co/cokrDc
Friends of Max Videos
13
Build PassTest
BuildTest Pass
Automation
Server
Automated
Automated
14. All contents © MuleSoft, LLC
Simple and effective ways to protect your data
Secure your user accounts with multi-factor authentication
Something you know
Login Credentials
Something you have
MFA
Find more information in: https://docs.mulesoft.com/access-management/multi-factor-authentication
15. All contents © MuleSoft, LLC
Secure and strong verification methods
Verification methods supported by MuleSoft
SMS (Text) verification
Phone call verification
Email verification
Not Supported (
x x
Salesforce Authenticator
Mobile App
Third-Party Authenticator Apps
Security Keys Built-In Authenticators
Such as:
Google Authenticator
Microsoft Authenticator
Authy
Such as:
Yubico’s YubiKey
Google’s Titan Security Key
Desktop + mobile device biometrics, such as:
Windows Hello
Touch ID
Face ID
Fast, free authentication
sfdc.co/IntrotoAuthenticator
16. All contents © MuleSoft, LLC
● Multi-tabbing has been a highly
requested feature from our Ideas
Portal
● We are excited to announce that it has
been added to API Designer to
improve your API specification
development experience.
● Whether you have one RAML file or
12, simply CMD + click (Mac OS) or
CTRL + click (Windows) on a file in the
left-hand panel to open a new tab in
API Designer
16
Available
Now
https://blogs.mulesoft.com/dev/design-dev/how-to-use-multi-tabbing
-and-rename-symbol/
Take advantage of the Multi-tabbing Feature
17. All contents © MuleSoft, LLC
From zero to Batch
Anu Vijayamohan
Containerizing the Mule
Runtime with
Kubernetes
Arno Brugman
18th November
17:00 London (GMT) | 18:00 Europe | 22:30 India (Delhi)
9:00 San Francisco | 12:00 New York
18. All contents © MuleSoft, LLC
Arno A. Brugman, Strategic Advisor
Containerizing the Mule Runtime with K8s
MuleSoft Meetup Wed, Nov 18, 5:00 PM (GMT)
19. All contents © MuleSoft, LLC
Safe harbor statement
The information in this presentation is confidential and proprietary to MuleSoft and may not be disclosed without the
permission of MuleSoft. This presentation is not subject to your license agreement or any other service or
subscription agreement with MuleSoft. MuleSoft has no obligation to pursue any course of business outlined in this
document or any related presentation, or to develop or release any functionality mentioned therein. This document,
or any related presentation and MuleSoft's strategy and possible future developments, products and or platforms
directions and functionality are all subject to change and may be changed by MuleSoft at any time for any reason
without notice. The information on this document is not a commitment, promise or legal obligation to deliver any
material, code or functionality. This document is provided without a warranty of any kind, either express or implied,
including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or
non-infringement. This document is for informational purposes and may not be incorporated into a contract.
MuleSoft assumes no responsibility for errors or omissions in this document, except if such damages were caused by
MuleSoft intentionally or grossly negligent.
All forward-looking statements are subject to various risks and uncertainties that could cause actual results to differ
materially from expectations. Readers are cautioned not to place undue reliance on these forward-looking
statements, which speak only as of their dates, and they should not be relied upon in making purchasing decisions.
20. All contents © MuleSoft, LLC
About the Speaker
20
Arno Brugman
● 30+ years in IT
● 4 years with MuleSoft
● Strategic Advisor
○ EMEA’s most ambitious customers
○ Trusted Advisor
○ Product & Technology
○ Customer Advocate
21. All contents © MuleSoft, LLC
Agenda
1. Introduction to Containerization
2. Containerization Environments
3. Containerizing the Mule Runtime with Kubernetes
4. Building the Image
5. Deploying the application
6. CI/CD
7. Runtime Fabric
21
23. All contents © MuleSoft, LLC
Introduction to Containerization
A brief history
● MULTICS (1963)
● UNICS (1969)
● MVS/zOS (1974)
● VAX/VMS (1977)
● Chroot Jail (1979)
● Windows (1983)
● Linux (1991)
● VMWare Workstation (1999)
● Linux Containers (2008)
○ LXC
● docker (2013)
○ Based on LXC
23
24. All contents © MuleSoft, LLC
docker definition of Containers
Introduction to Containerization
A container is a standard unit of software that packages up code and all its
dependencies so the application runs quickly and reliably from one
computing environment to another
24
25. All contents © MuleSoft, LLC
docker definition of Containers (2)
Introduction to Containerization
A container is an abstraction at the application layer that packages code and
dependencies together. Multiple containers can run on the same machine
and share the OS kernel with other containers, each running as isolated
processes in userspace. Containers take up less space than VMs, can handle
more applications, and require fewer VMs and Operating systems
25
26. All contents © MuleSoft, LLC
Vs Virtualization
Containers
26
Infrastructure
(Host) Operating System
docker
App1
App2
App3
App4
App5
App6
App7
App8
App9
Infrastructure
Hypervisor
Virtual
Machine
Virtual
Machine
Virtual
Machine
Virtual
Machine
Guest
OS
Guest
OS
Guest
OS
Guest
OS
App 1 App 2 App 3 App 4
28. All contents © MuleSoft, LLC
Containerization Environments
● Alibaba: Container Service for Kubernetes + Container Service
● Amazon: ECS + EKS
● Canonical
● Google: GKE
● Gravitational: Gravity
● Heptio
● IBM: Cloud Kubernetes Service + Multicloud Manager
● Kontena Pharos
● Microsoft: AKS + ACI
● Oracle: Application Container Cloud
● Pivotal: Cloud Foundry + Container Service
● Rancher
● RedHat: OpenShift
● Suse
28
29. All contents © MuleSoft, LLC
Most Commonly used
Containerization Environments
● Cloud Platforms
○ Amazon EKS
○ Google GKE
○ Microsoft AKS
29
● On Premise
○ RedHat OpenShift
○ Pivotal Cloud Foundry
30. All contents © MuleSoft, LLC
Containerizing the Mule Runtime with
Kubernetes
31. All contents © MuleSoft, LLC
Containerizing the Mule Runtime with Kubernetes
Pods vs Standalone Server
31
Kubernetes
Server
Mule
App
Standalone Mule Runtime
App
App
App
App
App
App
...
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
32. All contents © MuleSoft, LLC
Advantages vs Standalone
Containerizing the Mule Runtime with Kubernetes
● Application Isolation
○ Dedicated CPU
○ Dedicated Memory
○ Dedicated Disk
● Configurable
○ CPU, Memory, Disk
● Horizontal Scaling
● Vertical Scaling
32
33. All contents © MuleSoft, LLC
However...
Containerizing the Mule Runtime with Kubernetes
● Application Isolation
○ Dedicated CPU
○ Dedicated Memory
○ Dedicated Disk
● Configurable
○ CPU, Memory, Disk
● Horizontal Scaling
● Vertical Scaling
33
● No Automatic Load Balancing
● No Zero Downtime re-deploy
● Build your own Image
● Apply Patches yourself
34. All contents © MuleSoft, LLC
However...
Containerizing the Mule Runtime with Kubernetes
34
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Pod
Mule
App
Proxy
Anypoint Platform
36. All contents © MuleSoft, LLC
Base Image
Building the Image
● OS
● JDK
● Mule Runtime
● License File
36
37. All contents © MuleSoft, LLC
Hardware
● 2 GHz CPU (or 1 virtual CPU in virtualized environments)
● 1 GB of RAM
● 4 GB of storage
Software
● RHEL 7
● JDK 1.8.0 or JDK 11
● Mule Runtime version 4.3
● Mule License File
Building the Image
What do you need (for Mule 4.3)?
37
39. All contents © MuleSoft, LLC
Deploying the application
Method 1: Unmanaged
● Copy the Application.zip fille into the
container
● Run mule
39
● Not Managed by Anypoint runtime
Manager
40. All contents © MuleSoft, LLC
Method 2: Managed by Anypoint Runtime Manager
Deploying the application
● Install the Runtime Manager Agent
● Upload your application (CLI/API)
● Deploy your application into the
container
40
42. All contents © MuleSoft, LLC
CI/CD
Startup
After starting the Mule Runtime, the
Runtime needs to be connected to the
Anypoint Platform.
Use the Anypoint Platform API to register
the runtime
42
Shutdown
When a container is shutdown, the
container instance (Name + IP) is
terminated.
Before Shutdown, the container needs to be
properly de-registered from the Anypoint
Platform using the Anypoint Platform API
44. All contents © MuleSoft, LLC
Anypoint Runtime Fabric
Deploy and manage runtimes across any cloud
Centrally manage all applications
Manage applications in your private cloud
(AWS, Azure) and CloudHub within Anypoint
Platform
Run Mule apps on Kubernetes
Get containerization benefits such as isolation,
horizontal scaling, and auto redeploy by
default
Move deployments in a few clicks
Pivot your deployments between CloudHub,
your private cloud, or a data center with no
impact
45. All contents © MuleSoft, LLC
Anypoint Runtime Fabric
VM
Mule
App
VM
Mule
App
Mule
App
Runtime Fabric components
Runtime Fabric appliance
Mule
App
network
Runtime Fabric
Mule
App
Mule
App
VM
Runtime Fabric orchestrates and automates the
deployment of Mule runtimes into containers in any
cloud or on-premises environment
Benefits
● Deploy consistently across any cloud or data center
● Run multiple runtime versions in the same
Runtime Fabric
● Scale horizontally and redeploy w/ zero-downtime
● Easily manage via the control plane hosted by
MuleSoft
● Flexible deployment upon existing infrastructure or
managed K8s services
46. All contents © MuleSoft, LLC
Anypoint Runtime Fabric on EKS and AKS
47. All contents © MuleSoft, LLC
Anypoint Runtime
Fabric on EKS and AKS
Available
now
Available
NOW
EKS
AKS
47
48. All contents © MuleSoft, LLC
Deploy and manage runtimes on AKS or EKS
Runtime Fabric on AKS and EKS
Extend control plane benefits to
runtimes on your AKS or EKS
● Customer gets the latest
management and monitoring
features of Anypoint Platform
Decouple Mule runtime services
from your own infrastructure
● Give your ops teams the
power to use their
infrastructure how they want48
CUSTOMER’S NETWORK
Mule
App
Mule
App
Mule
App
Runtime Fabric services
Mule
App
Mule
App
Mule
App
Mule
App
Mule
App
Mule
App
Runtime Plane
CONTROL
PLANE
Kubernetes-as-a-Service
Available now
49. All contents © MuleSoft, LLC
Deploy and manage across any K8s service
Deploy across private
clouds
Open K8s access to
managed providers
Expanded K8s access
on managed provider
Across any K8s service
EKS
AKS
GKE
● Deploy standalone cluster
within AWS, Azure, or
on-prem
● Use K8s and docker without
K8s expertise
● Install RTF services into
private K8s environment
● Open access for customers
with existing K8s expertise
● Support for OpenShift
container platform
● Run Mule and non-Mule
workloads on the same K8s
environment
● Open support for any
managed service built on
K8s
● Deploy across any K8s
environment
51. All contents © MuleSoft, LLC
Anu Vijayamohan
Customer Success Technical Architect, EMEA
From zero to Batch
MuleSoft Meetup: 18 November 2020
52. All contents © MuleSoft, LLC
About the Speaker
52
Anu Vijayamohan, Customer Success Technical Architect
● American born, based in the
UK
● Almost 20 years of field
expertise
● Focussed mostly in working
directly in IT Delivery and
Professional Services
● Worked for Appian for more
than 13 years
● Joined MuleSoft Customer
Success about 1.5 years ago
53. All contents © MuleSoft, LLC
Agenda
1. Introduction to Batch Processing
2. Batch Processing Use Cases
3. Batch Processing Features in Mule 4
4. Best Practices for Batch Processing
5. Error Handling
6. Performance Tuning Considerations
7. Certification Tips
53
57. All contents © MuleSoft, LLC
Batch Processing Scenarios
Extracting, transforming and loading (ETL) information into a
target system
Handling large quantities of incoming data from an API into a
legacy system
Integrating data sets to parallel process records Synchronising data sets between business applications
57
58. All contents © MuleSoft, LLC
API vs Batch Characteristics
58
Aspect API Batch
Availability On-demand based on user
interaction (e.g. real-time)
Scheduled copy of data-set that doesn’t
require user interaction
Profile • Definition: configurable
fields/attributes, join, filter, sort,
etc.
• Volume: control data-set size e.g.
10,000 records per request
• Type: transactional, status, etc.
• Self-service: empowerment
• Definition: static or changes
infrequently
• Volume: Payload size < 2GB, million+
records, Big Data
• Type: historical, reporting/analytics,
master
Data Change Frequency Real-time, near real-time, minutes Daily, weekly, monthly
Consumer One or more Destination/s
Publication Event oriented N/A
63. All contents © MuleSoft, LLC
Batch Flow
Load
Batch
Step
Error
Handling
Bulk
Insert
Logging
63
64. All contents © MuleSoft, LLC
Benefits of Using the Batch Job Scope
64
● Data Integrity
● Reliability
● Efficiency
65. All contents © MuleSoft, LLC 65
● Polling Connectors
● Poll Scope
● For-Each
● Watermarks
Batch and ETL Using Mule Flows
Sequential Record Processing Capabilities
67. All contents © MuleSoft, LLC
Mule 4 Batch Processing Basics
● Mule processes a batch job asynchronously
● Any event source that feeds data into a batch job must be one-way, not request-response
● Results of batch processing do not feed back to the caller
● Default settings
○ Process records in blocks of 100 (Batch Block Size)
○ 16 thread default threading profile
○ Cancel processing on record failure (Max Failed Records; default is 0)
● Check out the Mule documentation for more detail
67
68. All contents © MuleSoft, LLC
Batch Processing Strategies
● Pattern strategy
● Exception strategy
● Capture and design around NFRs
68
69. All contents © MuleSoft, LLC
Running Batch Processes
● Front load transformation
● Separation of deployments
● Co-running multiple batch jobs
● File sizes
● Watermarking
69
70. All contents © MuleSoft, LLC
Streaming Best Practices
● Use streaming batch processing
when writing to a file
● Performance implications
● Item access implications
70
72. All contents © MuleSoft, LLC
Error Logging
Mule 4 Default
When a batch step fails or raises an error, Mule logs the stack traces following this logic:
● Mule gets the exception’s full stack trace.
● Mule strips that stack trace from any messages.
● Mule verifies if the stack trace was already logged in the current step
○ If this was the first time the runtime encountered this error, it logs the error and showing a
message like this:
com.mulesoft.mule.runtime.module.batch.internal.DefaultBatchStep: Found
exception processing record on step 'batchStep1' for job instance 'Batch Job
Example' of job 'CreateLeadsBatch'.
○ Subsequent records with the same failures will not be logged for performance and log
readability reasons:
● Mule logs on a "by step" basis. If another step also raises the same error, the runtime logs
it again for that step
72
73. All contents © MuleSoft, LLC
Error Logging
Mule 4 Default
● When the batch job reaches the On
Complete phase, Mule displays an error
summary with every error type, and how
many times it happened in each batch step.
● Mule logs are only logs INFO level messages
by default
● For more verbose logs, set the batch
category to DEBUG level in the application’s
log4j configuration
73
74. All contents © MuleSoft, LLC
Error Handling at a Record Level
● Finish Processing
● Continue Processing
Configurable Options
74
76. All contents © MuleSoft, LLC
Performance Tuning Considerations
76
● Focus on each process step to look for specific code optimizations
● Focus on transformation and any external connectivity logic within
process steps
● Eliminate as much disk and network latency as possible
● Align performance tuning considerations with current customer
performance testing activities
● Consider Disk I/O
78. All contents © MuleSoft, LLC
MuleSoft Certified Integration Architect (MCIA)
Batch Topics Covered in Exam
● ForEach scope
● Batch Job scope
○ Synchronicity
○ Filters
○ Pros/Cons
● Bulk Operations in connectors
● Streaming
● Understand the concepts, don’t just memorize
● Learn by doing! Open up Anypoint Studio and debug code to understand
the behaviour
● Exam questions require a problem to be solved by weighing different
options
78