DROPDB Galactic story

Jarek Ratajski
@jarek000000
DROP DB

Jarek Ratajski
Developer, wizard, anarchitectDeveloper, wizard, anarchitect
Lives in LuzernLives in Luzern
Works in CSS InsuranceWorks in CSS Insurance
C64, 6502, 68000, 8086, C++, Java, JEE, Scala, Akka, JS,C64, 6502, 68000, 8086, C++, Java, JEE, Scala, Akka, JS,
ScalaJS....ScalaJS....
jratajski@gmail.comjratajski@gmail.com @jarek000000@jarek000000

I know only one history
Who cares?

10 000 Planets 3 Variants
3 Sizes
HAWAII
MARGHERITTA
VEGETARIAN
MEDIUM
LARGE
XL
Domain summary

MARGHERITTA MEDIUM
HAWAII LARGE
HAWAII LARGE
VEGERARIAN LARGE
HAWAII XL
MARGHERITTA XL
HAWAII LARGE
HAWAII LARGE
HAWAII LARGE
HAWAII LARGE
33
33
11
11
Delivery model

10 000 Pizzas / s
I want to look at number
of queued orders 10000 times / s
My wife also!
Requirements

Alien development
(in 2 „parallel“ universes)

Normal universe Other universe
++

In Normal universe
https://github.com/airomem/galakpizza/tree/master/galakpizza_n

Normal domain
public class Order implements Serializable {
private static final long serialVersionUID = 1L;
public final long id;
public final String planet;
public final Variant variant;
public final Size size;
public Order(long id, String planet, Variant variant, Size size) {
this.id = id;
this.planet = planet;
this.size = size;
this.variant = variant;
}
}

Normal domain 2
public enum Size {
MEDIUM,
LARGE,
XL
}
public enum Variant {
HAWAII,
MARGHERITA,
VEGETARIAN
}

Normal Service
public interface GalakPizzaOrders {
long placeOrder(String planet, Variant variant, Size size);
}

Normal Service
}
public interface GalakPizzaDelivery {
List<Order> takeOrdersFromBestPlanet();
long countStandingOrders();
}

Normal Service
}
}
public interface GalakPizzaService extends GalakPizzaDelivery,
GalakPizzaOrders{
}

Normal service implementation...
public class GalakPizzaCore implements GalakPizzaService, Serializable,
Storable<GalakPizzaCore> {
public long placeOrder(String planet, Variant variant, Size size) {
...
}
public List<Order> takeOrdersFromBestPlanet() {
...
}
public long countStandingOrders() {
...
}
}

private finalprivate final Map<String, List<Order>>Map<String, List<Order>> ordersorders == newnew HashMap<>();HashMap<>();
...
}
...
}
...
}
}

private finalprivate final Map<String, PlanetOrders>Map<String, PlanetOrders> ordersorders == newnew HashMap<>();HashMap<>();
...
}
...
}
...
}
}

PlanetOrders - helper
class PlanetOrders implements Serializable {
final String name;
private List<Order> orders = new ArrayList<>();
public PlanetOrders(String name) {
this.name = name;
}
...
}

final String name;
this.name = name;
}
void assignOrder(final Order order) {
this.orders.add(order);
}
...
}

final String name;
this.name = name;
}
void assignOrder(final Order order) {
this.orders.add(order);
}
public boolean isEmpty() {
return this.orders.isEmpty();
}
...
}

Normal place order
private final Map<String, PlanetOrders> orders = new HashMap<>();
private long orderSequence = 1;
final long id = orderSequence++;
final Order order = new Order(id, planet, variant, size);
assignOrderToPlanet(order);
return id;
}

Normal place order
return id;
}
private void assignOrderToPlanet(Order order) {
final PlanetOrders po = orders.computeIfAbsent(order.planet,
planetName -> new PlanetOrders(planetName));
po.assignOrder(order);
...

Normal place order
private AtomicLong ordersTotal = new AtomicLong(0);
ordersTotal.incrementAndGet();
return id;
}
...

Normal place order revisited
public class GalakPizzaCore { ...
private PriorityQueue<PlanetOrders> bestPlanets = new PriorityQueue<>(256);
return id;
}
this.bestPlanets.offer(po);
}

private PriorityQueue<PlanetOrders> bestPlanets = new PriorityQueue<>(256);
return id;
}
this.bestPlanets.offer(po); BAD BAD BAD BAD
}

private PriorityQueue<PlanetOrders.Wrapper> bestPlanets = new PriorityQueue<>(256);
return id;
}
this.bestPlanets.offer(new PlanetOrders.Wrapper(po));
}

Wrapper... yeah
public static class Wrapper implements Comparable<Wrapper>,Serializable{
final PlanetOrders porders;
final int size; //just keep it final
public Wrapper(PlanetOrders porders) {
this.porders = porders;
this.size = porders.orders.size();
}
@Override
public int compareTo(final Wrapper other) {
return other.size - this.size;
}
}

PlanetOrders – take orders
final String name;
List<Order> takeOrders() {
final List<Order> result = this.orders;
this.orders = new ArrayList<>();
return result;
}
}
}

PlanetOrders – take orders
final String name;
List<Order> takeOrders() {
final List<Order> result = this.orders;
this.orders = new ArrayList<>(); //yes we MUTATE!
return result;
}
}
}

CYPS 2(2) ZYPS 1(1)
CYPS 2(2) ZYPS (1)ZYPS 2(2)
+ Order 1 Hawaii XL to ZYPS
CYPS 2(2) ZYPS 1(0)
Take Orders
ZYPS 1(0)
Take Orders
IsEmpty => ignore

Normal take orderspublic class GalakPizzaCore implements GalakPizzaService, Serializable,
privateprivate PriorityQueue<PlanetOrders.Wrapper>PriorityQueue<PlanetOrders.Wrapper> bestPlanetsbestPlanets == newnew
PriorityQueue<>(PriorityQueue<>(256256););
final Optional<PlanetOrders> planetOpt = takeBestPlanet();
....
}
private Optional<PlanetOrders> takeBestPlanet() {
PlanetOrders.Wrapper planet = this.bestPlanets.poll();
while (planet != null && planet.porders.isEmpty()) {
planet = this.bestPlanets.poll();
}
return Optional.ofNullable(planet).map(p->p.porders);
}

if (planetOpt.isPresent()) {
final PlanetOrders planet = planetOpt.get();
List<Order> orders = planet.takeOrders();
return orders;
}
return Collections.EMPTY_LIST;
}
}
}

if (planetOpt.isPresent()) {
final PlanetOrders planet = planetOpt.get();
List<Order> orders = planet.takeOrders();
ordersTotal.addAndGet(-orders.size());
return orders;
}
}
}
}

Normal count orders
return this.ordersTotal.get();
}

We have service - but do we deal with?
●
Concurrency
●
Persistence

Normal - Alien service wrapper
public class GalakPizza implements GalakPizzaService {
final PersistenceController<GalakPizzaCore,GalakPizzaCore> controller;
public GalakPizza() {
controller = PrevaylerBuilder.newBuilder()
.withinUserFolder("pizza")
.useSupplier( () -> new GalakPizzaCore())
.disableRoyalFoodTester()
.build();
}
public long placeOrder(final String planet, final Variant variant, final Size size) {
return controller.executeAndQuery( core -> core.placeOrder(planet,variant,size));
}
return controller.executeAndQuery( core ->core.takeOrdersFromBestPlanet());
}
return controller.query( c->c.countStandingOrders());
}
...
}

In alternative universe
https://github.com/airomem/galakpizza/tree/master/galakpizza_h

Other Domain
@Entity
@Table(name="T_ORDER")
@NamedQueries( {
@NamedQuery(name = "select best planet", query = "SELECT o.planet, count(o) FROM Order o " +
" GROUP BY o.planet ORDER BY count(o) desc"),
@NamedQuery(name = "select orders", query = "SELECT o FROM Order o WHERE o.planet = :planet"),
@NamedQuery(name = "count orders", query = "SELECT count(o) FROM Order o ")
})
public class Order {
@Id
@GeneratedValue(generator="increment")
@GenericGenerator(name="increment", strategy = "increment")
final long id;
public final String planet;
public final Variant variant;
public final Size size;
protected Order() {
this("this is strange...", null, null);
}
public Order(String planet, Variant variant, Size size) {
this.size = size;
id = 0;
}
public long getId() {
return id;
}
}

Other domain (same as above)
public enum Size {
MEDIUM,
LARGE,
XL
}
public enum Variant {
HAWAII,
MARGHERITA,
VEGETARIAN
}

Other service (same)
}
}
public interface GalakPizzaService extends GalakPizzaDelivery,
GalakPizzaOrders{
}

Select best blanet...
SELECT o.planet, count(o) FROM Order o
GROUP BY o.planet ORDER BY count(o) desc

Other Planet (but almost same story)
@Entity
@Table(name="T_PLANET",
indexes = {@Index(name="cnt_idx", columnList = "count", unique = false)} )
@NamedQueries( {
@NamedQuery(name = "select best planet from table", query = "SELECT p FROM Planet p " +
" ORDER BY p.count desc")})
public class Planet {
@Id
public final String name;
private int count;
protected Planet() {
name = "default";
}
public Planet(String name) {
this.name = name;
}
public int getCount() {
return this.count;
}
public void increment() {
this.count++;
}
public void clear() {
this.count = 0;
}
}

Other service implementation
public class GalakPizza implements GalakPizzaService {
final SessionFactory sessionFactory;
public GalakPizza(SessionFactory sessionFactory) {
this.sessionFactory = sessionFactory;
}
...
}
...
}
...
}
}

Other - Place order
return this.runInSession(session -> {
final Order orderEntity = new Order(planet, variant, size);
final Long key = (Long) session.save(orderEntity);
incrementPlanetCounter(planet, session);
return key;
});
}
private void incrementPlanetCounter(String planet, Session session) {
Planet planetEntity = session.get(Planet.class, planet);
if (planetEntity == null) {
planetEntity = new Planet(planet);
session.save(planetEntity);
}
planetEntity.increment();
}
private <T> T runOnSession(Function<Session, T> dbCommand) { ...}

Other - runInSession
private <T> T runInSession(Function<Session, T> dbCommand) {
final Session session = sessionFactory.openSession();
session.beginTransaction();
try {
final T result = dbCommand.apply(session);
session.getTransaction().commit();
session.close();
return result;
} catch (ConstraintViolationException cve) {
session.getTransaction().rollback();
session.close();
return runInSession(dbCommand);
} catch (Throwable t) {
t.printStackTrace();
throw new RuntimeException(t);
}
}
So easy

Other - Take orders
final Query bestPlanetQuery = session.getNamedQuery("SELECT p FROM Planet p " +
" ORDER BY p.count desc");
bestPlanetQuery.setMaxResults(1);
final Iterator<Planet> bestPlanetsIterator = bestPlanetQuery.iterate();
if (bestPlanetsIterator.hasNext()) {
final Planet bestOne = bestPlanetsIterator.next();
final Query ordersQuery = session.getNamedQuery("SELECT o FROM Order o WHERE
o.planet = :planet");
ordersQuery.setParameter("planet", bestOne.name);
final List<Order> orders = ordersQuery.list();
orders.forEach(o -> session.delete(o));
bestOne.clear();
return orders;
}
});
}

Other – count orders
@Override
final Query ordersCount = session.getNamedQuery("count orders");
final Long cnt = (Long) ordersCount.iterate().next();
return cnt;
});
}

Other - others
● Hibernate config, init
● MySql config

That we can simulate as
●
Customers - 4 Threads doing in loop placeOrder (random
planet name, variant and size) – no pauses
●
PizzBoy - 1 Thread doing in loop takeBestOrder – no pauses
●
Owners – 2 Threads doing in loop – countStandingOrders

Galaxy simulation
public static void main(String... args) {
runSimulation(5000);
System.gc();
sleep();
Result result = runSimulation(SECONDS_TOTAL*1000);
System.out.println("Orders processed:" + result.processed);
System.out.println("p/s:" + (double)result.processed/(double)
SECONDS_TOTAL);
System.out.println("Orders seen:" + result.seen);
System.out.println("o/s:" + (double)result.seen/(double)
SECONDS_TOTAL);
}

~40 000 p/s ~600 p/s
Processed orders

~ 12 823 500 p/s ~6362 p/s
Peeping speed

GODS of SQL
HELP! ME

~60 000 p/s ~1000 p/s

After few days of operations...

Reality strickes back
I do not trust PizzaBoy
I want to know where exactly were pizzas delivered

Cool, we will not delete
orders any more,
only mark them
as deleted!
Cool, we will add a list
that will be processed
during takeOrders,
or maybe
we will store data
in DB Table (as archive)

Reality strickes back
I do not trust PizzaBoy
I want to know where were pizzas delivered
Yesterday!!!

Crap...
We will restart system
and commands
will be reprocessed

System evolves
I want to let people compose their pizza/
Like: more cheese on one half,
and no meat on the next half,
And onion on the third half.

We will map it with only 20 tables
Or do we add TEXT field?
public Order(long id, String
planet, Supplier<Variant>
pizzaDescription, Size size) {
public Order(long id, String
planet, Variant variant, Size
size) {
We will use factory

Different
+ Fast
+ Code in Java
+ Flexible
+ Testable
- Sophisticated java
- Terribly slow
- Code in Strings and
Annotations
- Limited domain
- Testing expensive
+ Easy java

Sweet spot known as 90's
●
Computers had mostly one CPU Core
●
Memory (RAM) was expensive but more as 100MB
available in single unit
●
Disks were slow but huge (like 2Gb-20Gb)
●
Time for JEE

Normal universe
●
RAM is so cheap and huge
●
Computers have multiple cores
●
Alien developers start to use full power of hardware with
Java
●
No need to store data in databases anymore
●
Because RAM is so cheap and huge

Other universe
●
RAM is cheap and huge
●
Computers have multiple cores
●
But alien developers have never realised that

Other universe is a good place
●
Database vendors earn money and are happy
●
Application Servers vendors earn money and are happy
●
Power plants are happy
●
Hardware vendors are happy
●
Only business complains – but they still pay
●
Developers have a lot more work to do – they are happy..
Gfx By pressfoto / Freepik

DB / ORM vs Prevalence Problems
Problem Prevalent (Normal) ORMs (Other)
Detached objects Yes Yes
Transaction
isolation
Yes in JAVA
(concurrency)
Yes in DB
Testing Trivial Awful
Cache It is all cache Lots of configs...
Magic Contained Everywhere
Mapping All serializable goes Limited

If you want to jump to normal universe...

Simple start
●
Prevayler
or
●
Airomem (Prevayler wrapper for Java 8 )
or
●
Nothing :-)

Airomem quick - maven
<dependency>
<groupId>pl.setblack</groupId>
<artifactId>airomem-core</artifactId>
<version>1.0.5</version>
</dependency>

Airomem - init
PersistenceController<GalakPizzaCore,GalakPizzaCore> controller =
controller = PrevaylerBuilder
.newBuilder()
.withinUserFolder("pizza")
.useSupplier( () -> new GalakPizzaCore())
.build();
GalakPizzaCore is main/plain object - Root aggregate!
Must be Serializable! (how bad)

Airomem – commands
controller.executeAndQuery( core->core.placeOrder(planet, variant, size ));
----------------------
controller.executeAndQuery( new PlaceOrderCommand(planet,variant,size));
private static final class PlaceOrderCommand implements Command<GalakPizzaCore, Long>
{
private final String planet;
private final Variant variant;
private final Size size;
public PlaceOrderCommand(String planet, Variant variant, Size size){
this.size = size;
}
public Long execute(GalakPizzaCore system) {
return system.placeOrder(planet,variant, size);
}
}

Commands are never executed concurrently

Query
controller.query( c->c.countStandingOrders());
Queries are executed concurrently - also with
one command!

Airomem – snapshots, events
jarek@ubuntu:~$ cd prevayler/
jarek@ubuntu:~/prevayler$ ls
pizza
jarek@ubuntu:~/prevayler$ cd pizza/
jarek@ubuntu:~/prevayler/pizza$ ls -al
total 188
drwxrwxr-x 2 jarek jarek 4096 May 6 14:46 .
drwxrwxr-x 3 jarek jarek 4096 Apr 26 13:40 ..
-rw-rw-r-- 1 jarek jarek 675 Apr 26 14:17 0000000000000000001.journal
-rw-rw-r-- 1 jarek jarek 3375 Apr 26 14:18 0000000000000000002.journal
-rw-rw-r-- 1 jarek jarek 3369 May 5 10:51 0000000000000000007.journal
-rw-rw-r-- 1 jarek jarek 2048 May 5 14:12 0000000000000000012.snapshot

Data migration
●
Not big deal in fact
●
Java Serialization (sth to learn)
●
readResolve, writeReplace
●
XML export option

Transaction
This is always transactional!!!
public Long execute(GalakPizzaCore system) {
return system.placeOrder(planet,variant, size);
}

Concurrency
Learn java.util.concurrent, synchronized or die!

Replication/ failover
Not in standard... but
Trivial to implement - send events to other machines, share
folder

Indices
Indices in memory do exists
CQEngine
Query<Car> query1 = or(endsWith(Car.NAME, "vic"),
lessThan(Car.CAR_ID, 2));
for (Car car : cars.retrieve(query1)) {
System.out.println(car);
}

Airomem summary
CQRS simplified - one command one event→
With RAM projection

Advice
Remember to add Thread.sleep(1000) often

Storing in Prevayler vs DB
●
Store facts
●
Shared state is a cache
●
Domain not polluted with magic
●
Clean architecture

Alternatives
●
Lagom
●
Akka persistence
●
CQRS generally

End
●
Try on something small first (not bigger then one planet)
●
Drop DB from core of your system
●
Drop your application server
●
Drop annotations
●
Drop magic
●
Learn Java and algorithms
●
Feel the power of RAM and CPU alltogether!

DROPDB Galactic story

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à DROPDB Galactic story

Similaire à DROPDB Galactic story (20)

Plus de Jarek Ratajski

Plus de Jarek Ratajski (16)

Dernier

Dernier (20)

DROPDB Galactic story