https://github.com/NYTimes/Store

1. StoreDataLoadingMadeEasy-ish
Mike Nakhimovich - NY Times

2. We ❤Making Apps

3. We ❤Open Source

4. Why?Open Source Makes Life Easier

5. Networking is easier:
HTTPURLCONNECTION
Volley
Retrofit/ Okhttp

6. Storage is Easier
SharedPreferences
Firebase
Realm
SqlBrite/SqlDelight

7. Parsing is Easier
JSONObject
Jackson
Gson
Moshi

8. Fetching, Persisting &
Parsing datahas become
easy

9. What's NOT Easy?
DATALOADINGEveryone does itdifferently

10. Whatis DataLoading?
TheActofgetting
datafroman external
systemtoauser'sscreen

11. Loading is
complicated
Rotation Handling isa special
snowflake

12. NewYorkTimes builtStoreto
simplifydataloading
github.com/NYTimes/store

13. OUR GOALS

14. Datashould survive
configuration changes
agnostic ofwhere itcomes from or
howitis needed

15. Activitiesand
presenters should
stopretaining MBs
ofdata

16. Offline as configuration
Cachingas standard,
notan exception

17. API should be
simple enough for
an internto use,yet
robust enough
for everydataload.

18. Howdowework
with DataatNY
Times?

19. 80% case:
Need data, don'tcare
iffresh or cached

20. Need fresh data
background
updates &
pull to refresh

21. Requests needto be
async & reactive

22. Datais dependenton
each other,
DataLoading should
betoo

23. Performance is
importantNewcalls should hook into in flight
responses

24. Parsing should be
done efficientlyand
minimallyParse onceand cachethe resultfor future calls

25. Let's Use Repository
PatternBy creating reactiveand persistent
DataStores

26. Repository Pattern
Separatethe logicthatretrievesthe
dataand maps ittothe [view] model
fromthe [view] logicthatacts onthe
model.
The repositorymediates betweenthe
datalayerandthe [view] layer ofthe
application.

27. WhyRepository?
Maximizetheamountof
codethatcan betested
withautomation by
isolatingthe datalayer

28. Why
Repository?
Makes iteasierto have
multiple devsworking on
same feature

29. Why
Repository?
Datasource from many
locationswillbe centrally
managedwith consistent
access rulesand logic

30. Our Implementation
https://github.com/NYTimes/Store

31. WhatisaStore?
Aclassthatmanagesthe
fetching, parsing,and
storage ofaspecific data
model

32. Tell a Store:
Whatto fetch

33. Tell a Store:
Whatto fetch
Whereto cache

34. Tell a Store:
Whatto fetch
Whereto cache
Howto parse

35. Tell a Store:
Whatto fetch
Whereto cache
Howto parse
The Store handles flow

36. Storesare Observable
Observable<T> get( V key);
Observable<T> fetch(V key)
Observable<T> stream()
void clear(V key)

37. Let's see howstores
helped usachieve
our goalsthrougha
PizzaExample

38. Getis for Handling our 80% Use Case
public final class PizzaActivity {
Store<Pizza, String> pizzaStore;
void onCreate() {
store.get("cheese")
.subscribe(pizza ->.show(pizza));
}
}

39. Howdoes itflow?

40. On Configuration Change
You onlyneedto Save/Restoreyour !to get
your cached "
public final class PizzaActivity {
void onRecreate(String topping) {
store.get(topping)
.subscribe(pizza ->.show(pizza));
}
}
Please do notserializeaPizza
Itwould beverymessy

41. Whatwe gain?:Fragments/Presenters don'tneedto be
retained
NoTransactionTooLargeExceptions
Onlykeys needto be Parcelable

42. Efficiencyis Important:
for(i=0;i<20;i++){
store.get(topping)
.subscribe(pizza -> getView()
.setData(pizza));
}
ManyconcurrentGetrequestswillstillonly
hitnetwork once

43. Fetching NewDatapublic class PizzaPresenter {
Store<Pizza, String> store;
void onPTR() {
store.fetch("cheese")
.subscribe(pizza ->
getView().setData(pizza));
}
}

44. Howdoes Store routethe request?
Fetchalsothrottles multiple requestsand
broadcastresult

45. Stream listens for events
public class PizzaBar {
Store<Pizza, String> store;
void showPizzaDone() {
store.stream()
.subscribe(pizza ->
getView().showSnackBar(pizza));
}
}

46. How do We build a
Store?
By implementing interfaces

47. Interfaces
Fetcher<Raw, Key>{
Observable<Raw> fetch(Key key);
}
Persister<Raw, Key>{}
Observable<Raw> read(Key key);
Observable<Boolean> write(Key key, Raw raw);
}
Parser<Raw, Parsed> extends Func1<Raw, Parsed>{
Parsed call(Raw raw);
}

48. Fetcher defines how a Store will get new data
Fetcher<Pizza,String> pizzaFetcher =
new Fetcher<Pizza, String>() {
public Observable<Pizza> fetch(String topping) {
return pizzaSource.fetch(topping);
};

49. Becoming Observable
Fetcher<Pizza,String> pizzaFetcher =
topping ->
Observable.fromCallable(() -> client.fetch(topping));

50. Parsers helpwith
Fetchersthatdon't
returnViewModels

51. Some ParsersTransform
Parser<Pizza, PizzaBox> boxParser = new Parser<>() {
public PizzaBox call(Pizza pizza) {
return new PizzaBox(pizza);
}
};

52. Others read Streams
Parser<BufferedSource, Pizza> parser = source -> {
InputStreamReader reader =
new InputStreamReader(source.inputStream());
return gson.fromJson(reader, Pizza.class);
}

53. MiddleWare is forthe commonJSON cases
Parser<BufferedSource, Pizza> parser
= GsonParserFactory.createSourceParser(gson,Pizza.class)
Parser<Reader, Pizza> parser
= GsonParserFactory.createReaderParser(gson,Pizza.class)
Parser<String, Pizza> parser
= GsonParserFactory.createStringParser(gson,Pizza.class)
'com.nytimes.android:middleware:CurrentVersion'
'com.nytimes.android:middleware:jackson:CurrentVersion'
'com.nytimes.android:middleware-moshi:CurrentVersion'

54. Data FlowwithaParser

55. Adding Offline with Persisters

56. File System Record Persister
FileSystemRecordPersister.create(
fileSystem,key -> "pizza"+key, 1, TimeUnit.DAYS);

57. File System is KISS storage
interface FileSystem {
BufferedSource read(String var) throws FileNotFoundException;
void write(String path, BufferedSource source) throws IOException;
void delete(String path) throws IOException;
}
compile 'com.nytimes.android:filesystem:CurrentVersion'

58. Don'tlike our persisters?
No Problem! You
can implement your
own

59. Persister interfaces
Persister<Raw, Key> {
Observable<Raw> read(Key key);
Observable<Boolean> write(Key key, Raw raw);
}
Clearable<Key> {
void clear(Key key);
}
RecordProvider<Key> {
RecordState getRecordState( Key key);

60. DataFlowwithaPersister

61. We have our components!
Let's buildand openastore

62. MultiParser
List<Parser> parsers=new ArrayList<>();
parsers.add(boxParser);
parsers.add(GsonParserFactory.createSourceParser(gson, Pizza.class));

63. Store Builder
List<Parser> parsers=new ArrayList<>();
parsers.add(boxParser);
parsers.add(GsonParserFactory.createSourceParser(gson, Pizza.class));
StoreBuilder<String,BufferedSource,Pizza> parsedWithKey()

64. Add Our Parser
List<Parser> parsers=new ArrayList<>();
parsers.add(boxParser);
parsers.add(GsonParserFactory.createSourceParser(gson, Pizza.class));
StoreBuilder<String,BufferedSource,Pizza> parsedWithKey()
.fetcher(topping -> pizzaSource.fetch(topping))

65. NowOur Persister
List<Parser> parsers=new ArrayList<>();
parsers.add(boxParser);
parsers.add(GsonParserFactory.createSourceParser(gson, Pizza.class));
StoreBuilder<String,BufferedSource,Pizza> parsedWithKey()
.fetcher(topping -> pizzaSource.fetch(topping))
.persister( FileSystemRecordPersister.create( fileSystem,
key -> "prefix"+key, 1, TimeUnit.DAYS))

66. And Our Parsers
List<Parser> parsers=new ArrayList<>();
parsers.add(boxParser);
parsers.add(GsonParserFactory.createSourceParser(gson, Pizza.class));
StoreBuilder<String,BufferedSource,Pizza> parsedWithKey()
.fetcher(topping -> pizzaSource.fetch(topping))
.persister( FileSystemRecordPersister.create( fileSystem,
key -> "prefix"+key, 1, TimeUnit.DAYS))
.parsers(parsers)

67. Timeto OpenaStore
List<Parser> parsers=new ArrayList<>();
parsers.add(boxParser);
parsers.add(GsonParserFactory.createSourceParser(gson, Pizza.class));
Store<Pizza, String> pizzaStore =
StoreBuilder<String,BufferedSource,Pizza> parsedWithKey()
.fetcher(topping -> pizzaSource.fetch(topping))
.persister( FileSystemRecordPersister.create( fileSystem,
key -> "prefix"+key, 1, TimeUnit.DAYS))
.parsers(parsers)
.open();

68. Configuring caches

69. Configuring MemoryPolicy
StoreBuilder
.<String, BufferedSource, Pizza>parsedWithKey()
.fetcher(topping -> pizzaSource.fetch(topping))
.memoryPolicy(MemoryPolicy
.builder()
.setMemorySize(10)
.setExpireAfter(24)
.setExpireAfterTimeUnit(HOURS)
.build())
.open()

70. Refresh On Stale - BackFillingthe Cache
Store<Pizza, String> pizzaStore = StoreBuilder
.<String, BufferedSource, Pizza>parsedWithKey()
.fetcher(topping -> pizzaSource.fetch(topping))
.parsers(parsers)
.persister(persister)
.refreshOnStale()
.open();

71. Network Before Stales Policy
Store<Pizza, String> pizzaStore = StoreBuilder
.<String, BufferedSource, Pizza>parsedWithKey()
.fetcher(topping -> pizzaSource.fetch(topping))
.parsers(parsers)
.persister(persister)
.networkBeforeStale()
.open();

72. Stores inthewild

73. Intern Project: BestSellers List
public interface Api {
@GET
Observable<BufferedSource>
getBooks(@Path("category") String category);

74. @Provides
@Singleton
Store<Books, BarCode> provideBooks(FileSystem fileSystem,
Gson gson, Api api) {
return StoreBuilder.<BarCode, BufferedSource, Books>
parsedWithKey()
.fetcher(category -> api.getBooks(category))
.persister(SourcePersisterFactory.create(fileSystem))
.parser(GsonParserFactory.createSourceParser(gson, Books.class))
.open();
}

75. public class BookActivity{
...
onCreate(...){
bookStore
.get(category)
.subscribe();
}

76. Howdo books getupdated?
public class BackgroundUpdater{
...
bookStore
.fresh(category)
.subscribe();
}

77. DataAvailablewhen screen needs it
UI calls get, background services call fresh

78. DependentCalls

79. Simple case Using RxJavaOperators
MapStore resulttoanother Store Result
public Observable<SectionFront> getSectionFront(@NonNull final String name) {
return feedStore.get()
.map(latestFeed -> latestFeed.getSectionOrBlog(name))
.flatMap(section -> sfStore.get(SectionFrontId.of(section)));
}

80. More Complicated Example
Video Store Returns singlevideos,
PlaylistStore returns playlist,
Howdowe combine?

81. Relationships by overriding
Get/Fetch

82. Step1: CreateVideo Store
Store<Video, Long> provideVideoStore(VideoFetcher fetcher,
Gson gson) {
return StoreBuilder.<Long, BufferedSource, Video>parsedWithKey()
.fetcher(fetcher)
.parser(GsonParserFactory
.createSourceParser(gson, Video.class))
.open();
}

83. Step 2: Create PlaylistStore
public class VideoPlaylistStore extends RealStore<Playlist, Long> {
final Store<CherryVideoEntity, Long> videoStore;
public VideoPlaylistStore(@NonNull PlaylistFetcher fetcher,
@NonNull Store<CherryVideoEntity, Long> videoStore,
@NonNull Gson gson) {
super(fetcher, NoopPersister.create(),
GsonParserFactory.createSourceParser(gson, Playlist.class));
this.videoStore = videoStore;
}

84. Step 3: Override store.get()
public class VideoPlaylistStore extends RealStore<Playlist, Long> {
final Store<CherryVideoEntity, Long> videoStore;
@Override
public Observable<Playlist> get(Long playlistId) {
return super.get(playlistId)
.flatMap(playlist ->
from(playlist.videos())
.concatMap(video -> videoStore.get(video.id()))
.toList()
.map(videos ->
builder().from(playlist)
.videos(videos)
.build()));
}

85. Listening for changes

86. Step1: Subscribeto Store, Filterwhatyou need
sectionFrontStore.subscribeUpdates()
.observeOn(scheduler)
.subscribeOn(Schedulers.io())
.filter(this::sectionIsInView)
.subscribe(this::handleSectionChange);
Step2: Kick offafresh requesttothatstore
public Observable<SectionFront> refreshAll(final String sectionName) {
return feedStore.get()
.flatMapIterable(this::updatedSections)
.map(section->sectionFrontStore.fetch(section))
latestFeed -> fetch(latestFeed.changedSections));
}

87. Bonus: Howwe made Store

88. RxJavaforAPIand FunctionalNiceties
public Observable<Parsed> get(@Nonnull final Key key) {
return Observable.concat(
cache(key),
fetch(key)
).take(1);
}
public Observable<Parsed> getRefreshing(@Nonnull final Key key) {
return get(key)
.compose(repeatOnClear(refreshSubject, key));
}

89. GuavaCaches For MemoryCache
memCache = CacheBuilder
.newBuilder()
.maximumSize(maxSize())
.expireAfterWrite(expireAfter(),timeUnit())
.build();

90. In FlightThrottlerToo
inflighter.get(key, new Callable<Observable<Parsed>>() {
public Observable<Parsed> call() {
return response(key);
}
})
compile 'com.nytimes.android:cache:CurrentVersion'
Why GuavaCaches?
Theywillblockacrossthreads saving precious MBs ofdownloads
when making same requests in parallel

91. FileSystem:
Builtusing OKIOtoallowstreaming
from Networkto disk & diskto parser
On Fresh InstallNYTimesappsuccessfullydownloadsand caches
dozens ofmbs ofdata

92. Would love
contributions &
feedback!thanks for listening