Refactoring Toward Deeper Insight
DDD Findings in Batch Processing, a Case Study When I was introduced to the Domain-Driven Design (DDD) approach close to ten years ago, it provided me with some of the missing pieces I needed to implement Object-Orientation in an effective way. And over the years I've been coming back to Eric Evans' very rich and deep book many times to discover something new to help me design better software - thinking tools and practical design advice, in the small and in the large. Over the years Object-Orientation has become less important to me, but DDD is still my default starting point when I am helping teams to refactor their architectures and take control over their code bases. Many teams have already made attempts to implement DDD, but very often they don't get the effects they were hoping for. It turns out that DDD is hard to get right. In a current project I have been involved in yet another effort to implement DDD on a legacy code base. And I have made some interesting findings. Batch processing scenarios opened up my eyes to some intrinsic problems with the DDD approach. Issues that have been have been nagging me over the years became very clear. And yet again I managed to gain deeper insight in the DDD approach and come up with some quite interesting ways to implement it.
Andreas Brink, factor10
Driving Behavioral Change for Information Management through Data-Driven Gree...
Refactoring toward deeper insight java forum
1. Disclaimer!!!
.NET not Java
Work In Progress…
Refactoring Toward
Deeper Insight
DDD Findings In Batch Processing
A Case Study
Andreas Brink, factor10
2. Danica Pension
• Core financial business system.
• ASP.NET / RDBMS / Integrations
• Generations of technologies / architectures.
• Lots of financial business rules.
• Lots of Batch Processing.
• Mission: Taking Control of the Code Base
– DRY, Understandability, Testability, Automation
• DDD/DM + ORM fits well!
3. My View of the DDD Toolbox
Model Design & Impl.
Philosophy & Principles • Entity
• Ubiquitous Language • Aggregate
• Model Driven • Value Object
• Declarativity • Respository
• Distilling The Model • Service
• Specifiction
• Breaktrhough
• Side-Effect Free Functions
• … • …
Strategic Design
• Bounded Contexts
• Responsibilty Layers
• …
4. Domain Model Pattern (DM)
Basic Building Blocks Supple Design
• Entity, • Assertions
• Aggregate • Side-effect free
• Value Object Functions
• Specification • Standalone Classes
• Factory • Closure of Operations
Design Sweet-Spot
• Understanding & Communication
• Testability, Executable Specifications
But, not like the Office Object Model…
• Must scale
• Not one single file on disk
5. Implementability
Some
Layer…
Repository ORM
Service
• Object Navigation does not scale
Repositories
• DM does not scale well with composition & coupling
Services
• Problem Solved !?!?
6. Implementation Mess
Repository
Some ORM
Scenario…
Service
• Less focus on Domain Model
• Services – The Ultimate Loophole
– Touches big parts of the system – horizontally, vertically
– Side Effects
Understandability & Testing Problems
• Decentralized Flow Control & Data Access…
Global Optimization & Generic Processing hard or impossible
Performance Problems
7. Why DDD is Hard
Repository
Some ORM
Scenario…
Service
• Model Design is hard to begin with
– OO Expertise is still quite rare
• Have to be a Design Patterns / ORM / Architecture Expert
• Fail to get an Easily Consumable & Testable Model
• Same old Procedural Service Soup (+ some entities…)
8. My DDD Challenge
• Reclaiming the Domain Model
– Easy Reasoning, Consumption & Testing
• REAL Separation of Concerns
– Not just a complex web of objects and method
calls behind superficially simple interfaces
• And with Batch Processing in the mix…
IS THIS POSSIBLE??
9. Batch in the mix…
• ”ORM is not for Batch, use the right tool…”
• DDD/ORM vs Stored Procedures
• Service Chattiness Performance Problem
• Batch becomes a domain service in itself
– Business Workflow as a mini program
– Hard to decompose/compose without Service
– I want the business rules in the Domain Model…
12. Batch Observations
• High input/output entity ratio
– 11 Entity Types as Input – Often Complex
– 2 as Output (1 Create, 1 Update) – Simple
– Simple State Semantics
– Opportunities for Caching
– (Responsibility Layers Analysis…)
• Data is centered around a few central business
keys.
Potential for generalizing / streamlining the batch
processing pipeline??
16. Billing Batch – Generic Cursor Style
Master Keys Entities
Agreement Level Plan Agreement Agreement Insurance History
Category Level Advice History Premium Table Plan Category
Insurance Level Plan Insurance History … …
• Cursor Semantics
• A Set of Master Keys Drives the Cursor
• Entities Associated with Keys in Master
• Each Row Contains Entities for a Unit-Of-Work
17. Entity Level Keys
Level Keys Plan Insurance
Agreement Level = 4501 Agreement ID = 4501
Category Level = 78 Category ID = 78
Insurance Level = ”56076” ID = ”56076”
• Map of EntityLevel & Values
– Dictionary<EntityLevel, object>
• Or derived from Entity Properties
18. The Entity Level Abstraction
class AgreementLevel : EntityLevel {}
class CategoryLevel : EntityLevel {}
class InsuranceLevel : EntityLevel {}
public class PlanAgreement
{
[Level(typeof(AgreementLevel), IdentityType.Full)]
public int Id;
}
19. Entity Cursor: Master + Entities
void Initialize()
{
var cursor = EntityCursor.For(SessionFactory, MetaData);
// MASTER: IEnumerable<object[]> OR IEnumerable<TEntity>
cursor.Master = GetMyMaster();
cursor.MasterLevels(new AgreementLevel(), new InsuranceLevel());
cursor.Add(Query.For<PlanAgreement>());
// ADD MORE ENTITIES TO THE CURSOR...
while (cursor.MoveNext()) {
var currentPlanAgreement = cursor.Get<PlanAgreement>();
// PROCESS EACH ROW IN THE CURSOR...
}
}
20. IoC Style + Syntactic Sugar
class MyBatch : BaseBatch
{
PlanAgreement planAgreement;
EntityLevel[] Levels() { return ... }
object[] Master() { return ... }
void Initialize() {
// Query Defintions that are not simple
// Query.For<MyEntity>()
Add<PlanAgreement>()
.Where(pa => pa.Foo != null);
}
void ProcessRow() {
var foo = this.planAgreement.Foo ...
// PROCESS THE ROW...
}
}
22. Row Processing Chunked Data Fetch
Master Keys Agreement Insurance
Key 1: Agreement Id ChunkSize: 2 ChunkSize: 2
Key 2: Insurance No
(1, “InNo-1") Agreement(1) Insurance(1, “InNo-1")
(1, “InNo-2") -||- Insurance(1, “InNo-2")
(1, “InNo-3") -||-
(2, “InNo-4") Agreement(2)
Query<Insurance>()
(2, “InNo-5") -||- .Where ...
(3, “InNo-6")
... Query<Agreement>()
(n, “InNo-n") .Where(a => a.Id)
.IsIn(1, 2)
• Entities are fetched in Chunks
• Multiple chunk queries executed in one DB round-trip.
• NHibernate MultiCriteria (or Futures).
23. Row Processing Indexing
Master Keys Agreement Insurance
Key 1: Agreement ChunkSize: 2 ChunkSize: 2
Key 2: Insurance
(1, “InNo-1") Agreement(1) Insurance(1, “InNo-1")
(1, “InNo-2") Agreement(2) Insurance(1, “InNo-2")
(1, “InNo-3") Insurance(1, “InNo-3")
(2, “InNo-4") Insurance(2, “InNo-4")
(2, “InNo-5") Insurance(2, “InNo-5")
(3, “InNo-6") Agreement(3) Insurance(3, “InNo-6")
... ...
(n, “InNo-n")
• Each entity is indexed with the identifying level key(s).
• Entities in chunks synced with key for current row as the
cursor proceeds forward.
24. Entity Grouping
class InsuranceHistory : GroupingEntity<Insurance>
{
static readonly Grouping Grouping = Grouping.For<Insurance>()
.By(i => i.AgreementId)
.By(i => i.InsuranceNumber);
public InsuranceHistory(IList<Insurance> values) { ... }
}
Cursor.Add<PlanInsuranceHistory>();
Cursor.Add<PlanInsuranceHistory, PlanInsurance>()
.Where(...); // OK to override filter??
• Groups as a 1st class modeling concept
• Enriching the Domain Model
• “Virtual Aggregate Root” – Model Integrity
• Declarative expression (By, Where, Load)
25. Complex Grouping – PremiumTable
Row
10-22 23-30 31-45 Interval
0-20 100 120 135
Value
20-40 110 130 150
40-65 130 160 190
Column
Interval
• Rich Model Abstraction
• Complex data structure with lookup semantics
• No natural aggregate root
• Not cacheable in NHibernate session
• Fits well as a GroupingEntity
26. Querying
Conceptual API:
Cursor.Add(Query entityProvider)
Query.For<PlanInsurance>()
.Where(insurance => insurance.IsActive)
.Load(insurance => insurance.Category)
Query.For<AdviceHistory>()
Query.For(PremiumTable.ByAgreement)
.IndexBy(table => table.TableId)
• Filter per Entity – Cursor “Joins” using Shared Level Keys
• ORM-semantics: Where, Load
• Grouping Entity has query like qualities
• Level Queries are statically defined query using Entity Levels
Keys to construct underlying ORM query (yes, coupling)
27. Versioning
public class PlanInsurance
{
[Level(typeof(AgreementLevel), IdentityType.Partial)]
public int AgreementId;
[Level(typeof(InsuranceLevel), IdentityType.Partial)]
public string InsuranceNumber;
[VersionLevel(typeof(PlanInsurance), IdentityType.Partial)]
public int Version;
}
• Core to many business domains
• Has its own set of semantics
• Common in Groups – Latest<Insurance> vs InsuranceHistory
• Implemented in different ways in the DB
• Expressed declaratively
• Uniform Query Semantics
28. What About The Services?
void ProcessRow()
{
...
var premiumService = new PremiumService
{
PlanAgreement = Cursor.Get<PlanAgreement>(),
PlanInsurance = Cursor.Get<PlanInsurance>(),
Insurance = Cursor.Get<Insurance>(),
Insured = Cursor.Get<Person>(),
PriceBaseAmountTable = Cursor.Get<PriceBaseAmountTable>(),
PremiumTable = Cursor.Get<PremiumTable>(),
RiskTable = Cursor.Get<RiskTable>()
};
var premium = premiumService.CalculatePremium(advicePeriod);
...
}
• Service has pure calculation responsibility
• Dependencies are injected by client
• Coupling…? Boilerplate Smell…?
29. Conclusions
• Data Access Abstraction with Power & Ease of Use
• Declarative & Composable Entity Pipeline
• Minimizes DB Round-trips; Favors Eager Loading
• Repositories Become Redundant
• No More Unconstrained Services – “Calculators” / …???
• Richer Domain Model – Less Supporting Objects, More Domain
Related Objects
• DDD/ORM + Legacy DB == True
• Composite DB Key Thinking Essential to the Solution
• Patching the DB Model with Entity Level Abstraction…
• What’s Next? – Lots of Low Hanging Fruit…
TOWARDS AN EXECUTABLE ARCHITECTURE…???
30. What’s Next? – Entity Injection
Cursor.Add<PremiumCalculator>();
void ProcessRow()
{
...
var calculator = Get<PremiumCalculator>();
var premium = calculator.Calculate(advicePeriod);
...
}
• Cursor can inject entity dependencies automatically
• Calculators dependencies can be inferred and added
to cursor automatically
• ”Calculator” define Cursor Entities Implicitly
31. What’s Next? – Stateful Calculators?
class PremiumCalculator class PremiumCalculation
{ {
... ...
double CalculatePremium(...) {} double Premium;
... ...
} }
• What if we treated a calculation as a stateful object?
• Calculations become data flows through the system
• Stateful Objects as the Uniform Expression – Simplifies
declarative programming
• Captures Multiple / Intermediate Calculation Results
• Can be bound to a UI
• Additional state in the cursor – UI could add presentation
model/wrapper to the cursor
32. What’s Next? – Entity Pipeline
class BillingCalculation : EntityPipeline
{
void Initialize() {
Add<PlanAgreement>();
...
}
}
var monthlyBatch = new BillingCalculation();
monthlyBatch.Master = GetMasterForMonthlyBatch();
monthlyBatch.Persist<AdviceCalculation>(ac => ac.Advice).BatchSize(20);
monthlyBatch.Execute();
var singleInstance = new BillingCalculation();
singleInstance.Master = new object[]{ 24, "InNo-1"};
singleInstance.Persist<AdviceCalculation>(ac => ac.Advice);
singleInstance.Execute();
var nextUIPage = new BillingCalculation();
nextUIPage.Add<MyUIModel>();
nextUIPage.Master = GetMasterForNextPage();
myGrid.DataSource = nextUIPage.Select(cursor => cursor.Get<MyUIModel>())
33. What’s Next? – New Data Providers
• File Processing for Data Imports
– Prototyped batch framework
• Document Based Persistence
– Premium Table for example
• Hybrid Persistence
– Serialized object graphs in SQLServer
• SOA Integrations
– Loosely Coupled Bounded Contexts
• Parallel data fetch
– Multiple DBs / Data Services
34. What’s Next? – Business Events
• Entity Processing Pipeline seems to be a good
environment for triggering and/or handling
business events based on persistence events.
• Poor man’s Business Events!?!?
35. What’s Next? – Greenfield
• Search the Core Domain/Application Semantics
– Built-in Versioning from the start e.g.
– Semantic Storage…
• Streamline
– Uniform Expression
– Semantics
– Patterns
• Be Opinionted
– Constraints are Liberating
• Executable Architecture
