Adobe Behance Scales to Millions of Users at Lower TCO with Neo4j

Scaling Creative Social Graph
to Millions of Users
David Fox | Sr Software Engineer

© 2021 Adobe. All Rights Reserved. Adobe Confidential.
Behance
§ Leading online platform to showcase and discover creative work
§ 25 million members
§ Millions of monthly visitors
§ 40 million+ user-created projects
§ 320 million ”follows” relationships
§ Other users
§ Curated galleries
§ 700 million+ misc. actions, ex.
§ User liked project
§ User viewed project

Behance “For You” feed: logged-in homepage

The Beginning: Activity Feed Problem
(4 years ago)

Cassandra-powered feed
§ Rigid data format, hard to add new features
§ Not able to help product team carry out product vision
§ Expensive (lots of hardware)
§ Difficult to manage
§ At this time, deleting data was very tricky, cluster grew indefinitely!

Cassandra (48 instances)

Neo4j (3 instances)

How?
§ Replaced heavily repeated data in Cassandra with an optimized Neo4j graph model
§ Feeds generated dynamically (on the fly) instead of stored for every user
§ No more fanouts!

Cassandra fanout model
John

John
Published

John
Published
write

John
Published
write
write

John
Published
write
write
write

John
Published
write
write
write
write

John
Published
write
write
write
write
x 10,000

John
Published
write
write
write
write
x 10,000
item 1, item 2, item 3, …
stored
stored
stored
stored

Neo4j graph-based model
17
x 10,000

18
x 10,000
Neo4j graph-based model

Neo4j-powered feed
§ Easy to manage
§ 3 instances at the time powering millions of activity feed loads per day
§ > 200% reduction in devops human-resource time
§ Better performance and user experience
§ Extremely flexible
§ Enables us to execute on product vision
§ Use-cases and scale we could never imagine at the time

Phase 2: Recommendations
(2 years ago)

Apache Spark-powered recommendations (For You feed)
§ Similar to activity feed – difficult to manage (AWS EMR cluster)
§ Slow to develop and iterate on algorithms
§ Slow to update recommendations for users
§ Model recalculation
§ User recommendations recalculation
§ Hours of delay before a user sees new recommendations
§ Expensive to power and maintain

Neo4j-powered recommendations (For You feed)
§ Quick to develop (all our algos are just Cypher queries)
§ Easy to deploy
§ Easy to test different algos

Transformation results (Spark -> Neo4j)
§ Real-time recommendations (millions generated per day)
§ Business value – major uplift in KPIs
§ 18% increase in views of recommended projects
§ 36% increase in appreciations (“likes”) of recommended projects
§ Hundreds of thousands of $ in yearly savings (no more Spark/EMR cluster!)
§ Very quick to iterate on
§ Algorithms built and tuned by developers, no specialized skills required
§ Easy to build new recommendation features on top of graph

All of those features, live in production, user-facing!
§ Added 2 additional instances (read replicas) to help us serve more recommendations
§ Millions generated and served per day, over 7+ user-facing features powered
§ 5 instance cluster now (3 core, 2 read replicas)

New focus on recommendations for Behance
§ Product able to develop a strategy around more recommendation-based features
§ Opened opportunities to external teams wanting to explore
§ Internal research teams using graph
§ Capstone teams

Beyond Behance:
Adobe-wide Creative Social Graph

How can we bring benefits of graph to all of Adobe?
§ Creative Social Graph initiative
§ One social graph for all adobe products (Behance graph)
§ Millions of additional users and pieces of data
§ Power activity feeds and recommendations based on social data

Lightroom (activity feed for new community feature)
§ Lightroom built new community feature on top of Behance user data
§ Short timeline for feature dev before Adobe MAX
§ Thanks to flexibility of graph, we were able to generate a feed and surface through API in just a few
days
§ Lightroom devs were able to set up in-app feed in matter of days, feature ready long before deadline
§ Scale:
§ Millions of MAU
§ Lightroom for iOS is top 20 Photo/Video app in iOS App Store (U.S.)

Photoshop Express (activity feed for community)
§ Had a community for long time built on Behance user system, but no way to see activity from people
you follow
§ Wanted feed similar to Lightroom’s
§ Easy to produce additional feeds quickly for other products
§ Scale:
§ Millions of MAU
§ Photoshop Express for iOS top 20 ranked Photo/Video app in iOS App Store (U.S.)

Keys to success of Creative Social Graph project
§ Get more Adobe products to utilize the graph
§ Made possible by pretty effortless scalability of our graph
§ Continue to build out solid architecture to power a growing social graph
§ 0 downtime (knock on wood!) since we’ve started powering features in Lightroom and Photoshop Express

Thanks everyone!
§ Please feel free to ask any questions!
§ Can also email me: dafox@adobe.com

Adobe Behance Scales to Millions of Users at Lower TCO with Neo4j

Adobe Behance Scales to Millions of Users at Lower TCO with Neo4j

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Adobe Behance Scales to Millions of Users at Lower TCO with Neo4j

Similaire à Adobe Behance Scales to Millions of Users at Lower TCO with Neo4j (20)

Plus de Neo4j

Plus de Neo4j (20)

Dernier

Dernier (20)

Adobe Behance Scales to Millions of Users at Lower TCO with Neo4j