What if all members of your software development team from Project Managers, Business Analysts, Testing and documentation members could create and modify web applications and web services? With traditional SQL solutions this was difficult because of the need to convert web pages to objects, objects to tables as well as the reverse functions. But now with native XML databases and drag-and-drop forms builders, data can flow from the XML model of a web form to the database and back again without translation. This radically simpler process combined with standardized query languages makes it easier for non-programmers to build and maintain their own applications and web services.
08448380779 Call Girls In Civil Lines Women Seeking Men
Agile NoSQL With XRX
1. Agile NoSQL with XRX
Dan McCreary
President
M Dan McCreary & Associates
dan@danmccreary.com
D
(952) 931 9198
931-9198
Metadata Solutions
2. Session Description
What if all members of your software development team from
Project Managers, Business Analysts, Testing and Documentation
Managers Analysts
members could create and modify web applications and web
services? With traditional SQL solutions this was difficult because
of th need t convert web pages t objects, objects t t bl as
f the d to t b to bj t bj t to tables
well as the reverse functions. But now with native XML databases
and drag-and-drop forms builders, data can flow from the XML
model of a web form to the database and back again without
translation. This radically simpler process combined with
standardized query languages makes it easier for non-
non
programmers to build and maintain their own applications and web
services.
M
D 2 of
Copyright 2011 Dan McCreary & Associates
N
3. During this session viewers will:
• Understand the challenges associated with traditional four-
translation web/object/RDBMS systems
• Understand the elegant simplicity of zero-translation systems
• Understand the role of web standards in the XRX environment
• See how each component of XRX plays key roles in the
application
• See how organizations have leveraged XRX for agile web
development
• See variations of XRX architectures
• Create and objective p
j process to evaluate the benefits of XRX
• Get resources for building XRX pilot projects
M
D 3
Copyright 2011 Dan McCreary & Associates
4. Executive Summary
• Schema-free, zero translation NoSQL
systems have the ability to have a big
impact on overall system agility
p y g y
• XRX systems dramatically increase
overall agility and can also empower
non-progammers to build and maintain
NoSQL systems
M
D 4
Copyright 2011 Dan McCreary & Associates
5. Background for Dan McCreary
• Circuit Designer at Bell Labs
• UNIX/Supercomputers
• NeXT Computer (Steve Jobs)
• Owner of 75-person software consulting firm
with a focus on Object-Oriented systems and
Object Oriented
Object-relational mapping frameworks
• US Federal data integration (National
Information Exchange Model NIEM.gov)
• NativeXML/XQuery for metadata management
since 2006
• Advocate of web standards, OpenSource,
NoSQL d
N SQL and XRX systems t
• W3C invited expert (forms working group)
• Likes functional programming
M
D 5
Copyright Kelly-McCreary & Associates, LLC
6. Origins: The Humble Data Dictionary
M
6
D Copyright 2011 Kelly-McCreary & Associates
7. Electronic Certificate of Real Estate
Summer 2006
1 Document
= 44 SQL inserts
M
D 7
Copyright 2011 Kelly-McCreary & Associates
8. 250 Data Elements
XForms
Mockup
M
D 8
Copyright 2011 Kelly-McCreary & Associates
9. Four Translations
T1 T2
T4 T3
Relational
Web Browser Object Middle
Database
Tier
• T1 – HTML into Java Objects
• T2 – Java Objects into SQL Tables
• T3 – Tables into Objects
• T4 – Objects into HTML
M
9
D Copyright 2011 Kelly-McCreary & Associates
10. Kurt's Suggestion
Use a
A Native XML
Database!
Web Form
Save
Web Browser
Kurt Cagle
store($collection, $file-name, $data)
M
eXist
D 10
Copyright 2011 Kelly-McCreary & Associates
11. Zero Translation
XForms
Web Browser XML database
• XML lives in the web browser (XForms)
• REST interfaces
• XML in the database (Native XML, XQuery)
( , y)
• XRX Web Application Architecture
• No translation!
M
D Copyright 2010 Dan McCreary & Associates 11
12. Key Question: Impact on Agility
• What impact do zero translation
NoSQL
N SQL systems h
t have on
system agility?
• Agility: the ability to quickly
react to changing business
requirements at any stage of
the software development
lifecycle
• Question: Big impact or little
impact?
M
• Answer: Big impact
D 12
Copyright 2011 Kelly-McCreary & Associates
13. No-Shredding!
My Form
Data
• Relational databases take a single hierarchical
document and shred it into many pieces so it will fit in
tabular structures
• Many document-oriented NoSQL databases prevent
this shreddingg
M
D 13
Copyright 2008 Dan McCreary & Associates
14. Is Shredding Really Necessary?
• Every time you take
hierarchical data and
put it into a traditional
database you have to
put repeating groups in
separate tables and
use SQL “joins” to
joins
reassemble the data
M
D 14
Copyright 2008 Dan McCreary & Associates
15. Many Processes Today Are Driven By…
The constraints of yesterday…
Challenge:
Ask ourselves the question…
Do
D our current method of solving problems with t b l d t
t th d f l i bl ith tabular data…
Reflect the storage of the 1950s…
Or our actual business requirements?
What structures best solve the actual business problem?
p
M
D 15
Copyright 2008 Dan McCreary & Associates
16. "Schema Free"
• Systems that automatically determine
how to index data as the data is loaded
into the database
• No a priori knowledge of data structure
• N need f up-front logical d t modeling
No d for f t l i l data d li
– …but some modeling is still critical
• Adding new data elements or changing
g g g
data elements is not disruptive
• Searching millions of records still has sub-
second response time
M
D 16
Copyright 2010 Dan McCreary & Associates
19. Finding the Right Match
Schema-Free
Standards Compliant
Mature Query Language
M Use CMU's Architectural Tradeoff and Modeling (ATAM) Process
D 19
Copyright 2010 Dan McCreary & Associates
20. Architectural Summary
Four Translation Zero Translation
T T
T T web browser XML database
web browser
database
• HTML web pages
eb • XForms Client
• Object middle tier • Native XML Database
• RDBMS database
Which system more agile and by how much?
M How can this help us manage enterprise metadata?
D 20
Copyright 2010 Dan McCreary & Associates
21. Origins: The XML Data Dictionary
M
D 21
Copyright 2010 Dan McCreary & Associates
22. Electronic Certificate of Real Estate
Summer 2006
1 Document
= 44 SQL inserts
M
D 22
Copyright 2010 Dan McCreary & Associates
23. 250 Data Elements
XForms
Mockup
M
D 23
Copyright 2010 Dan McCreary & Associates
24. The NO-SQL Universe
Key-Value Stores Document Stores
XML
Graph Stores
Object Stores
M
24
D Copyright 2010 Dan McCreary & Associates
25. Three Core Processes
1. Add new data to the database
1. Use XML example but JSON and other
formats could also be used
2. Query the d
2 Q h data
3. Create an XML web service
• Analyze the effort for each step
• Compare SQL and NoSQL XRX systems
NoSQL-XRX
• Analyze impact on participation of non-programmers
• Rate the relative agility of each system
M
D 25
Copyright 2010 Dan McCreary & Associates
26. It is Easy to Import Data
SQL XQuery
1. Analyze data for all parent child 1. Drag XML files into folder
relationships and repeating g p
p p g groups
2. Design logical and physical ER
diagrams
3. For each table create a Data Definition
File using a data definition language
(DDL)
4. Create indexes using DDL
5. Create one table for each set of
repeating set of data
6. Run DDL on database creating tables
using the appropriate data types
7. Create indexes
8. Create Insert statements
9. Create separate insert statements for
each repeating group
10. Run Insert statements on primary
structures in database
11. Use primary keys of the first data
inserts as foreign keys of dependant
M data structures
D 26
27. The XML File system
• XML File system – a way of
storing information in XML that
can be quickly searched
• You can drag-and-drop almost
any fil onto thi fil system
files t this file t
• You access it by using a
WebDAV connector
• Microsoft Windows “My Network
Places” function
M
D 27
28. Functional Programming
y = f(x)
• Computer programs are like mathematical functions
• Developers do not manipulate states and variables (things that
change value), but focus entirely on constants and functions
g ) y
(things that never change) and transformations of data
• Makes it very easy to build modular programs
• Software written in functional programming languages tend to
be very concise and easy to port to highly parallel systems
M http://en.wikibooks.org/wiki/Computer_programming/Functional_programming
D 28
Copyright 2011 Kelly-McCreary & Associates
29. It's Easy to Query XML Data
SELECT COL1, Col2
for $r in doc(‘t.xml’)//row
FROM TABLE where col1=1
col1 1
WHERE COL1=1 return $r/col1, $r/col2
<root>
<row>
Col1 Col2 <col1>1</col1><col2>A</col2>
</row>
1 A <row>
<col1>1</col1><col2>B</col2>
1 B </row>
<row>
1 C <col1>1</col1><col2>C</col2>
</row>
1 D <row>
<col1>1</col1> <col2>D</col2>
M </row>
</root>
D 29
30. It is Easy to Create A Web Service
The Java/JDBC/SQL Way The XRX Way
1. All XQuerys are web services
1.
1 Learn Java or find a Java Developer
2. Install TomCat Web Server
3. Install Java AXIS Web Server
4. Write a JDBC program that sends
SQL queries to a database
5. Get the results back in Java Result
Object structures
6. Go through the Java Results
Structues and use print statements
to wrap XML tags aro nd the strings
rap around
in the result objects
7. Rename your class files to .jws files
8. Add the .jws files to the TomCat
deploy folders
9. The WSDL files will automatically be
generated
M
D 30
31. Insert/Select/Publish Comparison
SQL
SQL
Java
logical Tomcat
data AXIS
modeling JDBC
SQL XQuery
XQuery XQuery
Insert Query Web Service NoSQL
M
D Total Effort 31
32. Steps for Adding a New Element
• Add the element to the "new
instance
instance" a form uses and set
Takes about
one minute the default value there
• Add the UI control
• Optionally – run an XQuery
update to update existing data
Takes about 10
minutes
or add an insert to the
"enricher" when you get an
y g
instance
M
D 32
Copyright 2011 Kell-McCreary & Associates
33. Six Translation – Web Service
Web Service
T5 T6
T1 T2
T3 T4
Relational
Web Browser Object Middle
Database
Tier
• T1 – HTML into Java Objects
• T2 – Java Objects into SQL Tables
• T3 – Tables into Objects
• T4 – Objects into HTML
• T5 – Objects to XML
M
• T6 – XML to Objects
D Copyright 2011 Dan McCreary & Associates 33
34. XML Stored in XForms Model
Browser Database
model save
update
view
M
D Copyright 2011 Kelly-McCreary & Associates 34
35. XRX Core Process
Browser Database
model
save/edit
update
view
M
35
D Copyright 2011 Kelly-McCreary & Associates
36. Code Table Services
Client Server
model Form Data
Code Table Service
all-codes.xq
view
Code
C d
Tables
M
Code tables are separated from form instance data
D 36
Copyright 2011 Kelly McCreary & Associates
37. XRX Dynamic Forms Generation
Application Server
Client Application
XForms Model
Session
Form Data Form Data
User Team Document Collection
Status
Code Tables
Role Group
Views DataElement
Registry
Binding Rules
Code Table Services
Required Context
filters
Read-only Suggest Services
Data Types
Business Rules Editor
Calculations
Calculations Inference
Submissions
Constraints
XForms View
XF Vi
XML Schema Registry
Static Controls
Subschema Service
Dynamic Controls
Constraint Schemas Semantic Schemas
M
Design Time
D Run Time
38. Model Driven
• XForms enables the
developer to reuse business
XForms rules encapsulated in XML
Application
Schemas (xsd) and XML
Transforms (XQuery
Typeswitch)
• XForms reduces duplication
Meta
XML
Data and ensures that a change in
Schema
Registry the underlying business logic
does not require rewriting in
M another language
D Copyright 2011 Kelly-McCreary & Associates 38
39. View and Model are Trees
Model • The view is a tree of a presentation
data element
• Models are comprised of one or
more trees
• XForms supplies the control layer
Control (Bind) that moves data elements to and
from the model
View (Presentation) • Users don’t have to worry about
moving things to and from the
g g
screen
M
D 39
Copyright 2011 Kelly McCreary & Associates
40. Models and View Are Linked with "Bind"
HTML
head
h d
xf:model body
Person
P form
f
Name fieldset
label
label
first last input
<bind> input
• Both the model and the views are trees of
M data l
d t elements
t
D 40
Copyright 2011 Dan McCreary & Associates
41. Just “Do The Right Thing”
HTML
head
body
xf:model
Person form
PersonCurrentOnTaxes type="xs:boolean" fieldset
label
PersonBirthDate type="xs:date" label
input
input
<bind>
• Data types from the model just do the right thing
• Boolean variables become checkboxes
M • Dates have d t selectors
D t h date l t
41
D Copyright 2008 Dan McCreary & Associates
42. Example of Automatic UI Generation
• All true/false data types
(xs:boolean) automatically
become a checkbox
• All dates (xs:date) have a
date selector to the right of
the date field
• All codes can be selected
M from lists
42
D Copyright 2008 Dan McCreary & Associates
43. Structure of a XForms File
• XForms tags are just XML
Namespaces
tags imbedded in a
CSS Imports (View) standard XHTML file with
Model a different namespace
Constraints (Bindings)
( g ) • Most HTML form tags are
g
UI (View)
exactly the same but some
attributes have been
Submit Controls
promoted to be full
MyForm.xhtml
y elements
M
43
D Copyright 2008 Dan McCreary & Associates
44. REST
• REpresentation State Transfer
• Create applications based on well
designed URLs
• Take advantage of web caching
• Migrate toward Resource Oriented
Resource-Oriented
Computing (ROC)
• REST evangelists: RESTifarians
M
D Copyright 2011 Kelly-McCreary & Associates 44
45. Five RESTFull Friends
1. In-resident memory cache in your
browser
2. You local hard drive cache
3. Your local enterprise cache
4.
4 The cache on the web server farm
5. The cache on the database
Please make sure to check with your RESTfull
friends BEFORE you bother the database.
M
D 45
Copyright 2011 Kelly-McCreary & Associates
46. Shallow REST vs. Deep REST
• You can start taking advantage of ReST
buy just doing
b j t d i well th ll thought-out URL
ht t
design
• To take advantage of deep ReST you
must consider the subtleties of the
HTTP protocol
– GET vs POST vs PUT
– DELETE
M
D 46
Copyright 2008 Dan McCreary & Associates
47. Benefits of REST
• Provides improved response time
• Reduced server load
• Improves server scalability
p y
• Requires less client-side software
• Depends less on vendor dependencies
• Promotes discovery
• Provides better long-term compatibility
• y
Better and evolvability
M
D 47
Copyright 2008 Dan McCreary & Associates
48. Sample XForms
M
D 48
Copyright 2011 Dan McCreary & Associates
49. Requirements Editor
Code
Table
Selection
Lists
Repeating
R ti
Elements
M
D 49
50. Drag-and-Drop XForms Builders
• Several options for
drag-and-drop f
d dd forms
builder
• Open Source
– Orbeon
– BetterFORM
• Commercial
– IBM Forms
M
D 50
Copyright 2011 Kelly-McCreary & Associates
55. XRX Apps Demo Site
• Demo site for
danmccreary.com
• Some programs written
under an hour with
student data
• Several utility programs
that start with template
and add
transformations to
other formats
• Focus on metadata
management
M
D 55
Copyright 2011 Kelly-McCreary & Associates
56. Structured Retrieval is Better
Introduction to
Information Retrieval
by Christopher D. Manning,
y p g
Prabhakar Raghavan and
Hinrich Schütze
Cambridge University
C b id U i i
Press, 2008
M
http://nlp.stanford.edu/IR-book/information-retrieval-book.html
http://nlp stanford edu/IR-book/information-retrieval-book html
D 56
57. Table 10.1 - Revised
unstructured structured
RDB search
retrieval retrieval
unstructured trees with text at
objects records
documents leaves
vector space &
model relational model XML hierarchy
others
trees with node-
main data
table inverted index ids for
structure
t t
document ids
queries SQL free text queries XQuery fulltext
XML - Table 10.1 and structured information retrieval. SQLRDB (relational database) search,
unstructured information retrieval
M
D 57
58. Retain Document Structure in Search
"Bag of Words" "Retained Structure"
keywords
doc-id
keywords
'love' keywords
keywords
'hate' keywords
'new'
new
keywords
'fear'
• All keywords in a single container • Keywords associated with each
• Only count frequencies are stored sub-document component
with each word • Assign higher weight for titles and
M
names
D • Set by non-programmer 58
59. Empower the Non Programmer!
Before XRX After XRX
SUPER PM, BA!
,
Sorry, we have no idea Let me just search our XRX system…
what code 47 means. I'll have your answer in 150 milliseconds.
M
D 59
Copyright 2011 Kelly-McCreary & Associates
60. Using the Right Architecture
Start Finish
Find ways to remove barriers to empowering
the non programmers on your team
team.
M
D 60
Copyright 2011 Kelly-McCreary & Associates
61. Six "S"s of XRX Agility
1. Semantics – build around shard metadata
registry services
i t i
2. Search – structured search
3. Standards – CSS, XML, XPath, XQuery,
XForms, XML Schemas
4. Services – all XQ i are REST services
4 S i ll XQueries i
5. Solutions - that are quickly customized
6. Super – Empower the non-programmers
M
61
D Copyright 2010 Kelly-McCreary & Associates
62. XRX Resources
• XRX Wikipedia Page
– http://en.wikipedia.org/wiki/XRX(web_application_architecture)
p p g ( _ pp _ )
• XRX Resources
– http://www.danmccreary.com/xrx/
• LinkedIn XRX Group
– http://www.linkedin.com/groups/XRX-Web-Application-Architecture-1056777
• Beginner's Guide to XRX
– http://www danmccreary com/xrx/beginners-guide
http://www.danmccreary.com/xrx/beginners-guide
• XRX Wikibook
– http://en.wikibooks.org/wiki/XRX
• Google Code
– http://code.google.com/p/xrx/
M
D 62
Copyright 2011 Kelly-McCreary & Associates
63. References
XForms
XQuery
XQ
XRX
A Beginner's Guide to XRX
Send e-mail to dan@danmccreary com for extended
e mail dan@danmccreary.com
M
list of "getting started" resources.
D 63
Copyright 2011 Kelly-McCreary & Associates
64. Thank You!
Dan McCreary, President
Kelly-McCreary & Associates
dan@danmccreary.com
(952) 931-9198
eXist Meeting Prague March 12th, 2010
M
D 64
Copyright 2007 Dan McCreary & Associates