Organizations that can make sense out of massive amounts of data produced by systems, customers, or partners will have a competitive edge. Ballerina Stream Processing provides real-time event stream processing capabilities to microservices, with intuitive SQL queries allowing users to filter, aggregate and correlate data to make sense, take decisions and act in real-time in a distributed manner.
2. The Problem
○ Integration is not always about request-response.
○ Highly scalable systems use Event Driven Architecture to asynchronously
communicate between multiple processing units.
○ Processing events from Webhooks, CDC, Realtime ETLs, and notification
systems fall into asynchronous event driven systems.
3. What is a Stream?
An unbounded continuous flow of records (having the same format)
E.g., sensor events, triggers from Webhooks, messages from MQ
4. Why Stream Processing ?
Doing continuous processing on the data forever !
Such as :
○ Monitor and detect anomalies
○ Real-time ETL
○ Streaming aggregations (e.g., average service response time in last 5
minutes)
○ Join/correlate multiple data streams
○ Detecting complex event patterns or trends
5. Stream Processing Constructs
○ Projection
○ Modifying the structure of the stream
○ Filter
○ Windows & Aggregations
○ Collection of streaming events over a time or length duration
(last 5 min or last 50 events)
○ Viewed in a sliding or tumbling manner
○ Aggregated over window (e.g., sum, count, min, max, avg, etc)
○ Joins
○ Joining multiple streams
○ Detecting Patterns
○ Trends, non-occurrence of events
6. How to write Stream Processing Logic?
Use language libraries :
○ Have different functions for each stream processor construct.
○ Pros: You can use the same language for implementation.
○ Cons: Quickly becomes very complex and messy.
User SQL dialog :
○ Use easy-to-use SQL to script the logic
○ Pros: Compact and easy to write the logic.
○ Cons: Need to write UDFs, which SQL does not support.
7. Solution for Programing Streaming Efficiently
Merging SQL and native programing
1. Consuming events to Ballerina using standard language constructs
○ Via HTTP, HTTP2, WebSocket, JMS and more.
2. Generate streams out of consumed data
○ Map JSON/XML/text messages into a record.
3. Define SQL to manipulate and process data in real time
○ If needed, use Ballerina functions within SQL
4. Generate output streams
5. Use standard language constructs to handle the output or send to an
endpoint
8. “
Having lots of sensors, among all valid sensors,
detect the sensors that have sent sensor readings
greater than 100 in total within the last minute.
A Use Case
11. Ballerina Stream Processing type Alert record {
string name; int total;
};
type SensorData record {
string name; int reading;
};
Define input and output
record types
12. Ballerina Stream Processing type Alert record {
string name; int total;
};
type SensorData record {
string name; int reading;
};
function alertQuery(
stream<SensorData> sensorDataStream,
stream<Alert> alertStream) {
}
Define input and output
record types
Function with
input/output Streams
13. Ballerina Stream Processing type Alert record {
string name; int total;
};
type SensorData record {
string name; int reading;
};
function alertQuery(
stream<SensorData> sensorDataStream,
stream<Alert> alertStream) {
forever {
}
}
Define input and output
record types
Function with
input/output Streams
Forever block
14. Ballerina Stream Processing type Alert record {
string name; int total;
};
type SensorData record {
string name; int reading;
};
function alertQuery(
stream<SensorData> sensorDataStream,
stream<Alert> alertStream) {
forever {
from sensorDataStream
where reading > 0
window time(60000)
select name, sum(reading) as total
group by name
having total > 100
}
}
Define input and output
record types
Function with
input/output Streams
Forever block
Among all valid sensors, select
ones having greater than 100 reading
in total within the last minute
15. Ballerina Stream Processing type Alert record {
string name; int total;
};
type SensorData record {
string name; int reading;
};
function alertQuery(
stream<SensorData> sensorDataStream,
stream<Alert> alertStream) {
forever {
from sensorDataStream
where reading > 0
window time(60000)
select name, sum(reading) as total
group by name
having total > 100
=> (Alert[] alerts) {
alertStream.publish(alerts);
}
}
}
Define input and output
record types
Function with
input/output Streams
Forever block
Among all valid sensors, select
ones having greater than 100 reading
in total within the last minute
Send Alert
17. Joining Two Streams Over Time
// Detect raw material input falls below 5% of the rate of production consumption
forever {
from productionInputStream window time(10000) as p
join rawMaterialStream window time(10000) as r
on r.name == p.name
select r.name, sum(r.amount) as totalRawMaterial, sum(p.amount) as totalConsumed
group by r.name
having ((totalRawMaterial - totalConsumed) * 100.0 / totalRawMaterial) < 5
=> (MaterialUsage[] materialUsages) {
materialUsageStream.publish(materialUsages);
}
}
18. Detecting Patterns Within Streams
// Detect small purchase transaction followed by a huge purchase transaction
// from the same card within a day
forever {
from every PurchaseStream where price < 20 as e1
followed by PurchaseStream where price > 200 && e1.id == id as e2
within 1 day
select e1.id as cardId, e1.price as initialPayment, e2.price as finalPayment
=> (Alert[] alerts) {
alertStream.publish(alerts);
}
}
19. Building Autonomous Services
○ Process incoming messages or
locally produced events
○ Process events at the receiving
node without sending to
centralised system
○ Services can monitor themselves
throw inbuilt matric streams
producing events locally
○ Do local optimizations and take
actions autonomously
20. Stream Processing at the Edge
○ Support microservices architecture
○ Summarize data at the edge.
○ When possible, take localized decisions.
○ Reduce the amount of data transferred
to the central node.
○ Ability to run independently
○ Highly scalable
21. The Roadmap
○ Support stream processing to incorporate Ballerina’s custom functions.
○ Building Ballerina Stream Processing using Ballerina.
○ Support streams joining with tables.
○ Improve query language.
○ Support State Recovery.
○ Support High Availability.
24. Ballerina Stream Processing type Alert record {
string name; int total;
};
type SensorData record {
string name; int reading;
};
function alertQuery(
stream<SensorData> sensorDataStream,
stream<Alert> alertStream) {
forever {
from sensorDataStream
where reading > 0
window time(60000)
select name, sum(reading) as total
group by name
having total > 100
=> (Alert[] alerts) {
alertStream.publish(alerts);
}
}
}
Define input and output
record types
Function with
input/output Streams
Forever block
Among all valid sensors, select
ones having greater than 100 reading
in total within the last minute
Send Alert