Mobile Devices come in many forms and shapes. Most of them work with data - either collecting, presenting or editing data. This data is usually retrieved from or eventually uploaded to a back end server. However, in order to allow users to work smoothly and quickly - even with limited or sometimes even no bandwidth - it becomes imperative to offer local storage on the device. To act as a cache for pre-fetched data as well as for modified and newly gathered data. And to negotiate with the back-end systems in a way that is transparent for users and apps.
This presentation introduces the need for a local, on device data store. It discusses the requirements for such a mobile database. Then it introduces the key concepts in Service Oriented Architecture and finally it marries the mobile database and the SOA with its Enterprise Service Bus together - explaining how SOA concepts as well as a typical SOA implementation using an ESB is a perfect fit with and foundation for the Mobile Database.
Apidays New York 2024 - The value of a flexible API Management solution for O...
Mobile Database and Service Oriented Architecture
1. Lucas Jellema (AMIS, The Netherlands)
MOBILE DATABASES AND SOA
Workshop “Big Data and Its Impact” – 16 & 17 March 2013
for the Military College of Telecommunication Engineering
1
2. THE PRESENTER:
LUCAS JELLEMA
• Lives in The Netherlands
(close to Amsterdam)
• 1994-2002 Oracle Consulting Global Center of
Excellence for Internet Development
• Joined AMIS in 2002 – now working as
CTO, Consultant (Architect & Technical Lead) &
Trainer
• Oracle ACE (2005) & ACE Director (2006)
• Author of „Oracle SOA Suite 11g Handbook‟
(Oracle Press, 2010)
• Presenter at Oracle OpenWorld, JavaOne, OTN
Yathra & more User Group Conferences
• Frequent blogger at http://technology.amis.nl
• Active with SQL & PL/SQL, Java EE &
2 ADF, SOA, BPM & more Fusion Middleware
3. OVERVIEW
• Overview of mobile devices
• Data is presented, collected and edited in devices
• Handle „challenged server data connection‟
• Introduce the Mobile Database
• Mobile Database challenges
– Big Data, Fast Data, Fresh Data, Distributed Database
System, Security, Notifications
• Overview of core SOA concepts and objectives
• How SOA helps to enable the Mobile Database
3
4. MY CAR – IS A DATA COLLECTOR
• Collecting metrics
– Some for big data analysis for all cars (of this type)
– Some for car analysis of my car when serviced
– Some for real time (traffic jam, local temperatures)
4
14. DATA AND MOBILE DEVICES
• Some data requirements depend on the actual location
of the device
• Some data has to be on the device even when no data
connection is available
• Some data to be presented has to be fetched in (near)
real time from a central service
• Some (big) data is collected for analysis at a much
later point in time
• Some data is unstructured
• Some data needs to be sent from the device to a
central service in real time
• Most data will need to be refreshed at some point in
time – although the refresh rate differs
• Sometimes the user needs to be (very) aware when
data is not recently refreshed
14
15. DATA USED ON MOBILE DEVICES
• Originates from/to be sent to a back end (“the server”)
• Through limited bandwidth
• Sometimes entirely disconnected
– No reception
– Not permitted to use device
– Server side unavailable
15
16. NEED FOR ON-DEVICE-DATABASE
• Provide quick response for user
• Work when off-line
• Handle bursts of fast-data and piles of big data
• Take load off back end servers
16
17. REQUIREMENTS FOR MOBILE DATABASE
• Able to handle right types of data – including
unstructured data
• Easily accessible to apps
– Support for SQL
• Right scalability, right performance
• Right security
– Encrypted, Recoverable, Remote Wipe
• Maintainable/upgradable
– Database software and database model
– Pre-populated data (“firmware”)
• Able to fetch and process data from back-end
– Compact, secure, smart (incremental data updates)
• Smart upload to server, also Fast and Big Data
– pre-aggregated, pre-filtered, compacted
17
18. TRANSPARENT TO USER AND APPS
• The local database should be „invisible‟ to parties:
they interact with data service to query or report data
– However: it should be clear when data cannot
guaranteed to be fresh or complete
18
19. SECURITY
• Confidentiality – no unauthorized access
– Encryption of data in storage and transport
– Authorization on server access
• Integrity – no tampering, constraint compliant
• Availability – no data loss, right-time-access and
freshness
19
20. HOW DOES DATA ENTER THE MOBILE
DATABASE?
• Initial load when app is first installed
• Fetched from back end
• Entered by mobile app(s)
• When
– During install or upgrade of app
– During use of the application (by the user or device
features such as GPS, Bar Code
scanner, Camera or Recorder)
– When cache freshness is in doubt
– When cache completeness
is in doubt
– Synch upon re-connect:
submit collected new and
changed data
20
21. SOME GENERIC CHARACTERISTICS
• Pre loading
• Lazy loading (background)
• Write behind (lazy posting)
• Data Freshness and Completeness management
• Local fallback (next best thing)
21
22. DATA FRESHNESS AND COMPLETENESS
MANAGEMENT
• Based on actual time and geo-location…
– And possibly user, running app(s) and other context
• … assess if locally available data is fresh & complete
• When data is no longer fresh
– Try to negotiate a smart refresh with the server
– Depending on type of data: hide stale data, show stale
data with warning, continue to show stale data
• Incomplete data should be complemented
– Different location/area, zoom level, …
22
23. PUSH NOTIFICATIONS
• The server side can process messages, updates and
events from many directions
• Sometimes a mobile device should be informed of this
new information
• The server needs to know about the relevance of
information to a device (subscription)
• The server needs to be able to
push information (or poll-based
semi-push)
• The mobile database & app has to
process the pushed information
push
23
27. MOBILE DATABASES DO NOT SUPPORT
DISTRIBUTED TRANSACTIONS
• Server side updates happen independent from clients
– Refresh of Mobile Database may take place at any
point in the future
• Data manipulation in Mobile Database is committed
locally on the device
– Upload to server happens later: „write (far) behind‟
• This means that at some point after the transaction is
complete, the transaction
is „replicated‟ to other
databases
• During this replication,
conflicts may occur
27
28. DATA REPLICATION – STAGE ONE
• Updates on the client are made through local apps an
stored in the mobile database
• Local validations apply – based on the information
available in the mobile device at the time of the
transaction
– Without consulting the back end
28
29. DATA REPLICATION („WRITE BEHIND‟)
CONFLICTS – STAGE TWO
• Updates on client violate server side integrity rules
– Invalid data is not tolerated by the back end
– The mobile client has previously accepted the
transaction
– How is the app/user engaged to correct/resolve?
29
30. DATA REPLICATION („WRITE BEHIND‟)
CONFLICTS
• Updates on client collide with changes in the server
– The client attempts to update a record that has also
been updated in some other way
– How are conflicts detected?
– How should conflicts be resolved?
– How should parties be notified
about „lost conflicts‟?
30
39. TRANSFORMATION/ADAPTION OF
PROTOCOL AND FORMAT
XML/SOAP
HTTP
WS-Security EJB over RMI
B
D
B
ESB
A
E
SQL
JSON
JDBC over SQL*Net
HTTP
No encryption
SSL XML according to
canonical model
39
44. SOA AND MOBILE DATABASE
• Services that hide underlying complexity
– Technology, formats, protocols, data structure of back
end systems
• Services that handle two-way data synchronization
– Possibly conflicting changes on both ends
– Violation of server side data constraints
– Smart refresh of data on mobile device
• ESB that supports subscriptions and push to send
alerts/notifications/updates to mobile devices
• ESB that exposes services in appropriate protocol and
format (e.g. http/json)
• ESB that handles
security in transport
and message contents
44
46. SOA AND MOBILE DATABASE
• ESB ensuring the right availability – possibly
exceeding the service window of back-end systems
• ESB dealing with potentially large number of devices
– Scalability, Throttling (DoS protection)
• (Simulated) Device-to-Device-Push
• Upgrade of meta-data, apps and data-structure
• Bulk upload of Big Data from mobile device
• Initialization/Update of
pre-populated data on device
46
47. SUMMARY
• Mobile Database is a necessity to ensure apps can run
– With acceptable performance
– At all (in off-line situations)
• Mobile Database is a local cache that contains a
snapshot of the „back end data‟ at a certain timestamp
– Cache is not complete and is not current
– Cache contains data collected and edited on device
• Mobile Database is part of distributed database system
– Without real-time distributed transactions
– Periodic synchronization between Mobile Database and
Server is required (to really complete the transaction)
– Data conflicts need to be resolved
• SOA can help to hide complexity of back end and offer
available services with fitting protocol and data format
– Providing security, handling peak loads, supporting
push, understanding delta-based updates, …
47
Different zoom level, different informationMaps are updated once a year when my car is serviced
Semi liveTransmitted along with radio signalUpdate with changing zoom levelImportant to know when information is not fresh
No traffic jams indicated: are there none or is the connection unavailable?
Semi liveTransmitted along with radio signalUpdate with changing zoom levelImportant to know when information is not fresh
Types and usagesData collectionPhoto cameraSound recorderWork anytime anywhereeReaderNavigationInspection sheetTour guide3d overlayEmailChatAct on tasks in business process (approval, decision)