Personalization allows Stitch Fix to style its clients and provide recommendations to help them find what they love. To do this, the company gathers information about a client’s preferences up front when they sign up from the service and learns more about them as they become longer-term customers. This information is important for making recommendations but also must be protected and managed with care. The data science team at Stitch Fix is the primary owner of the recommendation systems. Backing them up is the data platform team, who maintain the data infrastructure, data warehouse, and supporting tools and services. This data warehouse has several different data sources that read and write into it. This includes a logging pipeline for events, every Spark-based ETL, and daily snapshots of structured data from Stitch Fix applications. Neelesh Srinivas Salian explains Stitch Fix’s process to better understand the movement and evolution of data within its data warehouse, from the initial ingestion from outside sources through all of its ETLs. Neelesh also details how Stitch Fix built a service that helps the company understand the lineage information that is associated with each table in the data warehouse. This service helps the company understand the source, parentage, and journey of all data in the warehouse. Although Stitch Fix makes sure to anonymize and filter out sensitive information from this data, the company needs a more flexible long-term solution as the business expands.

1. Tracking Data Lineage at Stitch Fix
Neelesh Srinivas Salian
Strata Data Conference - New York
September 12, 2018

2. Stitch Fix
Personalized styling service serving Men, Women, and Kids
Founded in 2011, Led by CEO & Founder, Katrina Lake
Employ more than 5,800 nationwide (USA)
Algorithms + Humans

3. About Me

4. This talk
● Data Ecosystem
● Data Lineage
● The Need
● Challenges
● Approach
● Architecture
● Questions

5. Data Ecosystem

7. Data Lineage

11. The Need and Challenges

12. Key Terminology
Resource
● Structured Data - Hive Table
● Postgres Database
ID - Unique identifier
● Service generated
● Synthesised
Job
● Service defined batch jobs
● Performs read/write on resources
Event
● Read Resource
● Write Resource

13. Managing a Resource
● Visibility - Data Scientists need to know what could break.
○ Upstream and Downstream to a Resource
● Effects of Change - If a resource is modified what does it affect?
○ Schema change
○ Data type modification
● Tracing - How did we get to this resource - source to destination?
○ Journey of a resource
● Debugging - How can you reliably debug a large pipeline?
● History - What has been writing to this resource?
○ Historical information

14. Upstream and Downstream

15. Traceability

16. Challenges - Consistency
● Multiple services
● Different Job Representations
● Different points of concern
● Extractable information needs to be identified

17. Approach

18. Simplifying the Data Model
Owner (User/ Team)
Job
Parent Job
Read Resource / Write Resource

20. Augmenting Code
● Avoid breaking API Changes
○ If any, there needs to be better communication
● Augment with necessary information to pass to Data
Ingestion pipeline
● Most of the changes are backend libraries
● Idempotency in workflows
○ Behavior
○ Function

21. Architecture

24. Data Acquisition
Event Driven
● Using the Data Ingestion
pipeline
● A Custom S3 Sink to write to
Hive table
● Clients can send lineage
information
Scheduled
● Ad-hoc usage
● Use only if additional
information is needed
● Harder to maintain

25. Event Driven

26. Intermediate Data Collection
Resource Attributes
● database
● table
● batchId
Service Data Attributes
● owner
● jobId
● serviceName
● parentId
Hive Tables

27. Presto Data Lineage
● Extract information from Queries
● Currently implemented
● Missing pieces
○ Parent- Child relationship
○ Augmenting various clients

28. Spark Data Lineage
● Adding ability to log reads and
writes as the happen
● Move over to Parquet as the
default FileFormat
● Augmenting library + clients to
pass parentage information

29. Data Refinement
● Regular cadence of ETLs extracting
Lineage information
● Output into clean Postgres Tables
● ETLs for
○ Aggregated Metric Extraction
○ Resource Relationships
ETL
Postgres DB

30. User Interaction
● Dashboards for Resource Views
○ Showing Upstream and Downstream
dependencies
● Static Views
○ Metrics from the Warehouse
● Dynamic Views
○ In-flux changes to Resources
● Custom dashboards can be built

31. neeleshssalian@gmail.com
Reach Out

32. Thank you!
https://multithreaded.stitchfix.com/