In this talk I demonstrate and explain how I tackled the problem of importing a huge data set with many repetitive contents. The dataset was provided in JSON format.
8. Events
Information about all the
visible events in the app
Genres
All the possible genres that
are assigned to events
App Settings Remote app config
9. Events
Information about all the
visible events in the app
Genres
All the possible genres that
are assigned to events
Userdata
Information for the currently
logged in user
App Settings Remote app config
10. Events
Information about all the
visible events in the app
Genres
All the possible genres that
are assigned to events
Userdata
Information for the currently
logged in user
Prizes Favorites Profile
App Settings Remote app config
11. Events
Information about all the
visible events in the app
Genres
All the possible genres that
are assigned to events
Userdata
Information for the currently
logged in user
Prizes Favorites Profile
Homepage
Which data to display on the
homepage
App Settings Remote app config
12. Events
Information about all the
visible events in the app
Genres
All the possible genres that
are assigned to events
Userdata
Information for the currently
logged in user
Prizes Favorites Profile
Homepage
Which data to display on the
homepage
App Settings Remote app config 1
13. Events
Information about all the
visible events in the app
Genres
All the possible genres that
are assigned to events
Userdata
Information for the currently
logged in user
Prizes Favorites Profile
Homepage
Which data to display on the
homepage
App Settings Remote app config 1
2
14. Events
Information about all the
visible events in the app
Genres
All the possible genres that
are assigned to events
Userdata
Information for the currently
logged in user
Prizes Favorites Profile
Homepage
Which data to display on the
homepage
App Settings Remote app config 1
3
2
15. Events
Information about all the
visible events in the app
Genres
All the possible genres that
are assigned to events
Userdata
Information for the currently
logged in user
Prizes Favorites Profile
Homepage
Which data to display on the
homepage
App Settings Remote app config 1
3
2
4
4
27. Events
Userdata
Homepage
Genres
Information about all the
visible events in the app
All the possible genres that
are assigned to events
Information for the currently
logged in user
Prizes Favorites Profile
Which data to display on the
homepage
App Settings Remote app config 1
3
2
4
4
67. ME AND MY IMPORTERS
PROBLEM 1 SOLVED
▸ Operations simplify dependencies
68. ME AND MY IMPORTERS
PROBLEM 1 SOLVED
▸ Operations simplify dependencies
▸ Operations can execute in parallel or serially
69. ME AND MY IMPORTERS
PROBLEM 1 SOLVED
▸ Operations simplify dependencies
▸ Operations can execute in parallel or serially
▸ Operations can contain their own private queues for complex nesting
76. PROCESSING IMPLEMENTATION V1
Find event in DB /
Create new event
Find location in DB /
Create new location
Loop over events
in JSON response
77. PROCESSING IMPLEMENTATION V1
Find event in DB /
Create new event
Find location in DB /
Create new location
Loop over events
in JSON response
Find artist in DB /
Create new artist
78. PROCESSING IMPLEMENTATION V1
Find event in DB /
Create new event
Find location in DB /
Create new location
Loop over events
in JSON response
Find artist in DB /
Create new artist
Find gallery in DB /
Create new gallery
79. PROCESSING IMPLEMENTATION V1
Find event in DB /
Create new event
Find location in DB /
Create new location
Loop over events
in JSON response
Find artist in DB /
Create new artist
Find text in DB /
Create new text
Find gallery in DB /
Create new gallery
80. PROCESSING IMPLEMENTATION V1
Find event in DB /
Create new event
Find location in DB /
Create new location
Loop over events
in JSON response
Find artist in DB /
Create new artist
Find sponsor in DB /
Create new sponsor
Find text in DB /
Create new text
Find gallery in DB /
Create new gallery
81. PROCESSING IMPLEMENTATION V1
Find event in DB /
Create new event
Find location in DB /
Create new location
Loop over events
in JSON response
Find artist in DB /
Create new artist
Find sponsor in DB /
Create new sponsor
Find text in DB /
Create new text
Find gallery in DB /
Create new gallery
Persist DB
92. COLLECT ALL UNIQUE EVENTS AND
RELATIONS?
for event in events {
guard let eventId = event["id"] as? Int
else { continue }
eventIds.append(eventId)
if let location = event["location"] as? DWJSON,
let locationId = location["id"] as? Int {
locationIds.append(locationId)
}
// etc...
}
95. ME AND MY IMPORTERS
PROBLEM 2 SOLVED
▸ Collecting all unique ids for objects
96. ME AND MY IMPORTERS
PROBLEM 2 SOLVED
▸ Collecting all unique ids for objects
▸ Fetching all objects in one go
97. ME AND MY IMPORTERS
PROBLEM 2 SOLVED
▸ Collecting all unique ids for objects
▸ Fetching all objects in one go
▸ Each object is only updated once
98. ME AND MY IMPORTERS
PROBLEM 2 SOLVED
▸ Collecting all unique ids for objects
▸ Fetching all objects in one go
▸ Each object is only updated once
▸ Pulling apart the data before processing it can have great benefits
113. UI UPDATING IMPLEMENTATION V2
Hold on to
notification
Managed object
context did save
Sync helper did
start notification
114. UI UPDATING IMPLEMENTATION V2
Hold on to
notification
Managed object
context did save
Sync helper did
start notification
Sync helper did
end notification
115. UI UPDATING IMPLEMENTATION V2
Hold on to
notification
Managed object
context did save
Sync helper did
start notification
Sync helper did
end notification
Fire one notification
for each entity
117. ME AND MY IMPORTERS
PROBLEM 3 SOLVED
▸ Use multiple contexts for reading and writing
118. ME AND MY IMPORTERS
PROBLEM 3 SOLVED
▸ Use multiple contexts for reading and writing
▸ Store notifications while you’re importing
119. ME AND MY IMPORTERS
PROBLEM 3 SOLVED
▸ Use multiple contexts for reading and writing
▸ Store notifications while you’re importing
▸ Filter notifications and fire only the notifications you really need to fire
121. ME AND MY IMPORTERS
TAKEAWAYS
▸ Operations can make complex processes easier to manage
122. ME AND MY IMPORTERS
TAKEAWAYS
▸ Operations can make complex processes easier to manage
▸ You might need to pre-process imported data for performance
123. ME AND MY IMPORTERS
TAKEAWAYS
▸ Operations can make complex processes easier to manage
▸ You might need to pre-process imported data for performance
▸ Inserting a layer between your app and CoreData can help with managing an
import pipeline