Building a large eCommerce application with Extbase, Fluid and Apache Solr
1. October 2, 2010 1
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
2. October 2, 2010 2
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
3. IBC SOLAR: Selling 857 million turnover
everything from a in 2009
solar module to
4-figure number of
complete solar plants
products x product
Working with retail variants x retail
partners in Germany partners
and throughout the
world
October 2, 2010 3
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
4. „Build yet another Shop
System for TYPO3 – are you
insane?“
All of us & the customer (before the start of the project)
October 2, 2010 4
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
5. Previous system based on Commerce
• scalability issues
tt_products? Hell no! ;-)
Specialized functionality lacking in all systems
Magento looked fairly nice – but not integrated in
TYPO3
October 2, 2010 5
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
6. October 2, 2010 6
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
7. October 2, 2010 7
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
8. October 2, 2010 8
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
9. October 2, 2010 9
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
10. All data comes from the company‘s ERP system
Custom import functionality for complete orders
Individual prices for customers
No need for retail functions like online payment
processing
October 2, 2010 10
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
11. standing on the shoulders of giants
October 2, 2010 11
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
12. How do you design a generic domain model for a
shop system?
October 2, 2010 12
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
13. You don‘t …
(unless you have too much time on your hands)
October 2, 2010 13
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
14. …because it already exists:
October 2, 2010 14
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
16. Proven & open standard for product catalogues
Very well documented (specification >600 pages)
Originally an XML data exchange format
October 2, 2010 16
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
17. The Extbase kickstarter:
a great way to create lots of model objects
October 2, 2010 17
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
18. October 2, 2010 18
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
19. October 2, 2010 19
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
20. The kickstarter is not an editor!
October 2, 2010 20
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
21. Using SCM to cherry-pick changes:
1. Save in kickstarter
2. svn revert <fileThatShouldNotBeTouched>
3. Eclipse diff view to manually merge files
October 2, 2010 21
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
22. Apache Solr as a Storage Backend
October 2, 2010 22
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
23. Java based search
server
Searches millions of
documents (products,
text entries, prices…)
in very little time
Provides faceted
search facilities
October 2, 2010 23
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
24. The naive way:
writing direct querys to the Solr server from the
application code
October 2, 2010 24
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
25. NO
Faceted Extbase
Repository Query
FACETED persistence
Query
QUERIES layer
Background
storage Solr storage
Storage
Backend
backend
(DB, Solr…)
October 2, 2010 25
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
26. Tx_Extbase_Persistence_Query
$query->equals(‘foobar’, $foobar)
$query->logicalAnd($statement1,
$statement2)
$query->in(‘foobar’, array($item1,
$item2))
October 2, 2010 26
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
28. Tx_Nxsolrbackend_Persistence_FacetedQuery
extends Tx_Extbase_Persistence_Query
$query->facet($facetConfiguration)
$query->execute():
Tx_Nxsolrbackend_Persistence_
FacetedQueryResult
$facetedQueryResult->getObjects()
$facetedQueryResult->getFacets()
October 2, 2010 28
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
29. Structural differences: database vs. index
Data duplication: database and Solr index
Extbase knows only one StorageBackend
October 2, 2010 29
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
31. How to feed data into the database
from external sources?
October 2, 2010 31
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
32. (Photo courtesy of trenttsd/flickr)
October 2, 2010 32
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
33. Looking for the right tool:
Integrate different data sources homogenously
Source data from Microsoft Dynamics NAV
(MSSQL)
CSV file to specify additional files for products
like datasheets and the like
Need to understand it 3 years later
October 2, 2010 33
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
34. Pentaho Data Integration
Extract, Transform and Load
JDBC + plugins for different data sources/targets
October 2, 2010 34
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
35. Configurability: import into different sysfolders,
configure parameters and default values
Where needed: Use JS and external code for
special transformations
October 2, 2010 35
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
36. How fast is Extbase?
October 2, 2010 36
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
37. First implementations were painfully slow due to…
Working with thousands of objects
Bug in Extbase: most objects considered dirty at
any time
October 2, 2010 37
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
38. Build first, optimize later!
(Might be scary at times, but the approach does actually work…)
October 2, 2010 38
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
39. Optimization #1:
Make heavy use of lazy loading, load objects only
when needed
October 2, 2010 39
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
40. Optimization #2:
Fix Extbase: Objects no longer dirty ;-)
October 2, 2010 40
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
41. Optimization #3:
Not everything needs to be an object:
If you don‘t need domain logic,
you might just render raw query results
October 2, 2010 41
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
42. Optimization #4:
Use the good old TYPO3 caching mechanisms,
but be especially careful
October 2, 2010 42
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
43. October 2, 2010 43
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
44. October 2, 2010 44
Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de