2. Social Commerce: An Introduction The last few years have seen tremendous growth in social networks Some estimates place Facebook above Google Even if not, we’re talking millions of daily unique visitors So the obvious question is… where’s the money? 2
4. Social Commerce: Business Case What’s wrong with traditional e-commerce? Discovery/recommendation features are extremely hard to get right Overly broad market targeting means lost sales and disgruntled, ad-weary customers The trust model is inherently broken Impossible to gauge truth and accuracy in customer reviews “Wisdom of the masses” does not always apply Not fun! Shopping is a social experience (going to the mall, holiday shopping sprees) This does not translate to existing e-commerce sites! 4
5. Social Commerce: Business Case “Social commerce” aims to address these deficiencies Correlating interests and products is more accurate and significantly easier when based on social context Social circles are inherently constructed on shared interests and perspectives A customer’s social network is much smaller in scope than generating a global, statistical recommendation model More accurate personalized data exposes new opportunities Personalized discovery allows more opportunity to tap the long tail Social interaction makes it easy to identify domain experts A single opinion provided by a friend, family member or acquaintance is more trustworthy than dozens of unrelated product reviews/ratings 5
6. Social Commerce: Business Case Most crucially, social commerce is all about user engagement and collaboration: Should I buy an iPhone, Blackberry or Android phone? Which wedding dress looks best? Which video games are suitable for a preschooler? 6 Ask your friends!
7. Social Commerce: The Axiom Social features increase user engagement Increased conversion Profit! 7
9. Enter: Delver The Delver team has two products on the market Two sides of the same coin, really: sears.com is a traditional e-commerce website with a social twist delver.com is a traditional social website with an e-commerce twist 9
10. The Technical Challenge sears.com is a fully blown commercial retail site Over 1 million page-views daily Over 270,000 visitors daily Traffic can easily spike up to ten times in the holiday season! 10
11. The Technical Challenge Processing social networks is not an easy proposition Massive amounts of branching data No data locality Very few assumptions can be made about the data Let’s address each of these in turn 11 Source: NetworkWeaver
12. The Technical Challenge Massive amounts of branching data: Imagine every Facebook user (500 million) Imagine each person is only connected to 100 others (conservative estimate) How is user X connected with Y? X has 100 friends Each of them has 100 friends 10,001 nodes visited! 101 reads from the underlying storage system! 12 X Y
13. The Technical Challenge No data locality: Any object may be connected to any other object in no particular order How to split the data? Some research is being done in the area (SPAR) 13
14. The Technical Challenge No easy assumptions: No “typical user” Not enough data to draw archetypes Significant, unavoidable long tail Difficult to pre-tune data structures 14
15. The Technical Challenge The crux of the problem: High branch factor necessitates many loads to serve even a simple request No data locality + high branch factor means very high random I/O Traditional storage models (RDBMS, flat files etc.) are a poor fit Serious research into graph storage, social network composition etc. only dates back a few years No best practices or “accepted truths” to build on 15
16. Use Case for GigaSpaces To solve the graph storage and traversal problem, we arrived at the following requirements: Completely in-memory storage No data locality means caching is inefficient Massive amounts of random I/O cannot scale vertically, and hardware (basically, spindle count) cost quickly becomes prohibitive If data access is sufficiently fast, data can be randomly partitioned Horizontal scaling with a well-known scale-up strategy Add more memory or more nodes to handle data growth Add more CPUs or additional nodes to handle load growth 16
17. Use Case for GigaSpaces Additional requirements include: Map/Reduce execution framework Graph traversal and data analysis requirements lend well to the map/reduce paradigm Code execution on the data nodes Because of the massive amounts of data involved, the network interface will be quickly saturated by retrievals Memory retrieval is at least two orders of magnitude faster than network throughput (DDR2-800 on a dual channel memory controller has a theoretical throughput maximum of 102.4Gb/s) 17
18. Use Case for GigaSpaces As an operations tech I had a few things to add to the list, namely… Nonfunctional requirements: Built-in fault tolerance and high availability Zero-configuration (or as close to it as it gets) setup; in particular, component discovery and assignment must be automated Well-documented deployment, configuration and tuning process Monitoring API Administrative client for diagnosis, trouble resolution and manual intervention 18
19. Use Case for GigaSpaces GigaSpaces features map well to our requirements Data grid Compute grid High availability Horizontal data and load scaling Management API Very few viable alternatives: Hadoop, neo4j are disk-based Terracotta is overly simplistic and has no execution framework Oracle Coherence is expensive and has a limited feature set 19
20. Delver Architecture We ended up with a hybrid platform: GigaSpaces for graph storage, traversal and analysis MySQL for traditional, “simple” data as well as a backing store for GigaSpaces .NET-based front-end, Java-based back-end We had to factor our organization accordingly Data access team provides abstracted interfaces on top of GigaSpaces and MySQL Back-end “heavy lifting” services (e.g. recommendation engine) work directly against GigaSpaces Most other components either use the abstracted DAL or are simple enough to work directly against MySQL using (N)Hibernate 20
22. Key Benefits Significantly reduced integration costs GigaSpaces does a lot of what we need out of the box An alternative solution would require integrating several products, incurring significant integration and development overhead Broad feature set Social commerce is an emerging, dynamic market requiring rapid experimentation and adaptation The large feature set allows us to introduce new features into the system at a furious pace While primarily intended for graph storage, we also use GigaSpaces as a message queue, distributed lock server and distributed scheduler 22
23. Now is a good time for… Questions?COMMENTS? 23
24. Endgame Experience our work! Visit Delver at http://www.delver.com/in?invite=friends-and-family Visit Sears Social at http://catalog.sears.com Read about our work at http://blog.delver.com Have anything to discuss? Contact me at tomer@delver.com Visit my blog at http://www.tomergabel.com Follow me on Twitter at http://www.twitter.com/tomerg Thank you for your time! 24
Editor's Notes
Image sources (also linked):* Facebook US traffic estimates (http://www.insidefacebook.com/2011/01/03/november-2010-facebook-traffic/)* Top 20 visited websites (http://www.hitwise.com/us/datacenter/main/)
Case in point:Discovery and recommendation features: when was the last time YOU “might be interested in this product”? How accurate are the typical recommendation systems for you?Just how relevant is the typical ad or marketing campaign? When was the last time you went into Amazon and got a coupon for a truly relevant occasion or product? How tired are you of flashing banners?
Public data source: http://bizinformation.ca/www.sears.com#visitors
Massive amounts of data:* Imagine modeling every person on the planet (say, 6 billion). Now say each person is connected to just 100 others (a conservative estimate)Image source: http://networkweaver.blogspot.com/2010/03/overlapping-boards.html
Time distribution image source: http://blog.nielsen.com/nielsenwire/global/social-norms-twitter-users-follow-the-797-rule-in-the-u-k/Twitter following distribution image source: http://www.personalizemedia.com/twitter-long-tail-broadcastization-pre-twitter-reputation/
Inevitably, someone will ask: what are the problems you encountered?Barrier of entry:Ops: setting up a GigaSpaces cluster is not a hassle-free affair. Lots of work went into a robust, efficient bootstrapping procedure and we had to content with quite a few unexpected snags. I believe things are a lot better with the current version than they were a while ago. Furthermore, the overall cost of setting up and deploying GigaSpaces is significantly less than the total overhead for using specific products to tackle our various needs (compared to a traditional system, the cost of setting up e.g. MySQL+RHCS+client configuration; more likely we’d have had to use some sort of 3rd party graph storage, clustering and persistence solution)Devs: working against GigaSpacesis considerably harder than vanilla, commonplace RDBMS. To counter the barrier of entry we modeled our organization so that a core team of developers handle graph storage and data analysis, with most other teams either integrating with this subsystem or handling their own requirements with regular Hibernate/NHibernate over MySQL.Hard to handle migration paths, zero-time deployment and schema evolution. Features in 8.0 should help remedy the situation (cue Nati Shalom)