Gautier Krings and Svend Vanderveken introduce Trumania, a scenario-based data generation python library developed at Real Impact Analytics. Among other things, data generation is a great technique when we need test dataset for applications that manipulate sensitive information. Trumania source code: https://github.com/RealImpactAnalytics/trumania

1. Simulate all
the things !
Gautier Krings
Svend Vanderveken
BigData.be meetup
January 2018

2. We should talk about
random data

3. CDRs
(caller,
callee,
time,
position…)
Analytics
(RFM,
network
analysis,
mobility…)
Recommendations
(reporting,
campaigns…)
A typical pipeline for a
telecom analytics company

4. How do I know if my
code is robust
to larger sizes?
to errors in data?
to diversity in
dimensions?

5. Also, don’t forget… is coming
Using client data, are you?

6. The search for “good” fake data
Looks like real data Parameterizable Same complexity

7. Trumania
A free open-source
random data generator
Developed and
maintained by

8. Demo time!

9. Populations of customers, shops, locations, SIM cards…
Essentially, dimensional data

10. person = create_population(
name="person", size=1000000, id_generator=...)
person.create_attribute("NAME", values=...)
person.create_attribute("AGE", generator=...)
+-------------------+-----------------+---------+
| | NAME | AGE |
|-------------------+-----------------+---------|
| PERSON_0000000000 | Amy Berger | 28.588 |
| PERSON_0000000001 | Michael Curry | 28.7499 |
| PERSON_0000000002 | Robert Ramirez | 35.9242 |
| PERSON_0000000003 | Derek Gonzalez | 34.7091 |

11. Generators: any parametric distribution, empirical
distribution from real data or deterministic

12. Story: dynamic aspects of a scenario: sequence of
operations that generate logs

13. example_story = create_story(
initiating_population=person,
member_id_field="PERSON_ID", …
)
+-------+-------------------+
| | PERSON_ID |
|-------+-------------------+
| 0 | PERSON_0000000000 |
| 1 | PERSON_0000000001 |
| 2 | PERSON_0000000002 |
| 3 | PERSON_0000000003 |

14. example_story.set_operations(
clock.timestamp(named_as="TIME"),
ConstantGenerator(value="hello world").generate(named_as="MESSAGE")
)
+-------------------+---------------------+-------------+
| PERSON_ID | TIME | MESSAGE |
+-------------------+---------------------+-------------|
| PERSON_0000000000 | 2017-01-01 01:14:12 | hello world |
| PERSON_0000000001 | 2017-01-01 01:23:51 | hello world |
| PERSON_0000000002 | 2017-01-01 01:37:11 | hello world |
| PERSON_0000000003 | 2017-01-01 01:33:12 | hello world |

15. example_story.set_operations(
clock.timestamp(named_as="TIME"),
FakerGenerator(method="word").generate(named_as="MESSAGE"),
person.select_one(named_as="OTHER_PERSON") ...)
+-------------------+---------------------+-------------+-------------------+
| PERSON_ID | TIME | MESSAGE | OTHER_PERSON |
+-------------------+---------------------+-------------+-------------------+
| PERSON_0000000000 | 2017-01-01 01:14:12 | motorbike | PERSON_0000000852 |
| PERSON_0000000001 | 2017-01-01 01:23:51 | tree | PERSON_0000000429 |
| PERSON_0000000002 | 2017-01-01 01:37:11 | Sunday | PERSON_0000000925 |
| PERSON_0000000003 | 2017-01-01 01:33:12 | table | PERSON_0000000347 |

16. Simulate all the things!
Weighted relationships between elements of your
populations: social networks, mobility preferences,
hierarchical distribution networks...

17. example_story.set_operations(
clock.timestamp(named_as="TIME"),
FakerGenerator(method="word").generate(named_as="MESSAGE"),
person.get_relationship("friends").select_one(named_as="OTHER_PERSON"),
person.lookup(select={"NAME": "EMITTER_NAME"}),
person.lookup(select={"NAME": "RECEIVER_NAME"}) )
+------------+----------+---------+--------------+---------------+---------------------+
| PERSON_ID | TIME | MESSAGE | OTHER_PERSON | EMITTER_NAME | RECEIVER_NAME |
+------------+----------+---------+--------------+---------------+---------------------|
| PERSON_000 | 01:14:12 | Become | PERSON_058 | Ann Cruz | Victoria Washington |
| PERSON_001 | 01:23:51 | Month | PERSON_066 | Kimberly S | Steven Williams |
| PERSON_002 | 01:37:11 | Blue | PERSON_013 | Bethany Smith | Frances Davis |

18. A relationship relating each person to 20 friends on
average shows like this in the output dataset

19. We can add temporal patterns for user activity, shop hours...

20. 1.0
github.com/RealImpactAnalytics/trumania trumania.slack.com

21. gautier@jetpack.ai
Thank you
We’re hiring!
info@realimpactanalytics.com
Svend Vanderveken
http://svend.kelesia.com
@sv3ndk