apidays LIVE Paris 2021 - APIs and the Future of Software December 7, 8 & 9, 2021 Data Deletion at Facebook Benoît Reitz, Privacy Engineer manager at Meta

1. Deletion Framework
How Facebook upholds its commitments towards data
deletion
Benoît Reitz - 2021

2. Introduction

3. Introduction
- Deletion Framework is responsible for graph deletions at Facebook
- Deletion Framework in numbers:
- 500B new graph deletions per day
- ~1T individual objects deleted per day
- Handles a dozen of interconnected datastores
- A birds eye view of many systems rather than a deep dive

4. Outline
I. Design Overview
II. Guarantees
III. Monitoring Guarantees
IV. Conclusion

5. Deletion Framework

6. Outline
I. Design Overview
A. Relying on a Data Definition Language
B. Walking the graph
C. Batching
D. Scheduling deletions in the future
II. Guarantees
III. Monitoring Guarantees
IV. Conclusion

7. Relying on a Data Definition Language
- Code generation is a critical component
- Deletion logic is generated from annotations present on the schema
- Storage configuration
- Deletion Edge annotations
- Shallow / Deep / Refcounted
- Deletion Constraints

8. User
Constraint: Never
deep deleted
Storage: Tao
Post
Constraint: Deep
deleted by any other
entity
Storage: Tao
Comments
Constraint: Deep
deleted by Post
Storage: Tao
FromUserToPost: deep
FromPostToComment: deep
FromCommentToPost: shallow
FromPostToAuthor: shallow

9. {UserDeleter}
{Storage
Deleter}
{PostDeleter}
{...}
Delegates to primitive
deleters for objects
and associations
...
Follows deep assocs

10. Outline
I. Design Overview
A. Relying on a Data Definition Language
B. Walking the graph
C. Batching
D. Scheduling deletions in the future
II. Guarantees
III. Monitoring Guarantees
IV. Conclusion

11. Walking the graph
- We use a DFS
- We need to be re-entrant
- At least once semantics

12. Post
Comment
Comment
Reply
Alice
Bob
Delete Request
Author
Author
To Comment
To Comment
To Reply

13. Post
Actions:
1. Read
Delete Request

14. Post
Persistent Stack:
- Post
Actions:
1. Read
2. Checkpoint on stack
Delete Request

15. Post
Persistent Stack:
- Post
Actions:
1. Read
2. Checkpoint on stack
3. Log restoration logs
Delete Request
Restoration Logs:
Post

16. Post
Persistent Stack:
- Post
Actions:
1. Read
2. Checkpoint on stack
3. Log restoration logs
4. Self Delete
Delete Request
Restoration Logs:
Post
X

17. Post
Persistent Stack:
- Post
Actions:
1. Read
2. Checkpoint on stack
3. Log restoration logs
4. Self Delete
5. Graph Delete
Delete Request
Restoration Logs:
Post
X

18. Post
Persistent Stack:
- Post
Actions:
1. Read
Delete Request
Restoration Logs:
Post
Author X

19. Post
Persistent Stack:
- Post
Actions:
1. Read
2. Log restoration logs
Delete Request
Restoration Logs:
Post, Assoc From author
Author X

20. Post
Persistent Stack:
- Post
Actions:
1. Read
2. Log restoration logs
3. Delete
Delete Request
Restoration Logs:
Post, Assoc From author
Author X
X
X

21. Post
Persistent Stack:
- Post
Actions:
1. Read
Delete Request
Restoration Logs:
Post, Assoc From author
Author X
X
X

22. Post
Persistent Stack:
- Post
Actions:
1. Read
2. Traverse
Delete Request
Restoration Logs:
Post, Assoc From author
Author X
X
X
To Comment
To Comment

23. Post
Persistent Stack:
- Post
Actions:
1. Read
Delete Request
Restoration Logs:
Post, Assoc From author
Author X
X
X
To Comment
To Comment
Comment

24. Post
Persistent Stack:
- Post
- Comment
Actions:
1. Read
2. Checkpoint on stack
Delete Request
Restoration Logs:
Post, Assoc From author
Author X
X
X
To Comment
To Comment
Comment

25. Post
Persistent Stack:
- Post
- Comment
Actions:
1. Read
2. Checkpoint on stack
3. Log restoration logs
Delete Request
Restoration Logs:
Post, Assoc From author,
Comment
Author X
X
X
To Comment
To Comment
Comment

26. Post
Persistent Stack:
- Post
- Comment
Actions:
1. Read
2. Checkpoint on stack
3. Log restoration logs
4. Self Delete
Delete Request
Restoration Logs:
Post, Assoc From author,
Comment
Author X
X
X
To Comment
To Comment
Comment
X

27. Post
Persistent Stack:
- Post
- Comment
Actions:
1. Read
2. Checkpoint on stack
3. Log restoration logs
4. Self Delete
5. Graph Delete
Delete Request
Restoration Logs:
Post, Assoc From author,
Comment
Author X
X
X
To Comment
To Comment
Comment
X

28. Post
Persistent Stack:
- Post
- Comment
Actions:
1. Read
2. Checkpoint on stack
3. Log restoration logs
4. Self Delete
5. Graph Delete
Delete Request
Restoration Logs:
Post, Assoc From author,
Comment, Assoc From
author
Author X
X
X
To Comment
To Comment
Comment
X
Author
X
X

29. Post
Persistent Stack:
- Post
- Comment
Actions:
1. Pop from stack
Delete Request
Restoration Logs:
Post, Assoc From author,
Comment, Assoc From
author
Author X
X
X
To Comment
To Comment
Comment
X
Author
X
X

30. Post
Persistent Stack:
- Post
Actions:
1. Read
2. Traverse
3. Log Restoration Log
Delete Request
Restoration Logs:
Post, Assoc From author,
Comment, Assoc From
author, Assoc to Comment
Author X
X
X
To Comment
To Comment
Comment
X
Author
X
X

31. Post
Persistent Stack:
- Post
Actions:
1. Read
2. Traverse
3. Log Restoration Log
4. Delete
Delete Request
Restoration Logs:
Post, Assoc From author,
Comment, Assoc From
author, Assoc to Comment
Author X
X
X
To Comment
To Comment
Comment
X
Author
X
X
X X

32. Post
Persistent Stack:
- Post
- Comment
Delete Request
Restoration Logs:
Post, Assoc From author,
Comment, Assoc From
author, Assoc to Comment,
Comment
Author X
X
X
To Comment
To Comment
Comment
X
Author
X
X
X X Comment

33. Post
Persistent Stack:
- Post
- Comment
Delete Request
Restoration Logs:
Post, Assoc From author,
Comment, Assoc From
author, Assoc to Comment,
Comment
Author X
X
X
To Comment
To Comment
Comment
X
Author
X
X
X X Comment
X

34. Post
Persistent Stack:
- Post
- Comment
Delete Request
Restoration Logs:
Post, Assoc From author,
Comment, Assoc From
author, Assoc to Comment,
Comment
Author X
X
X
To Comment
To Comment
Comment
X
Author
X
X
X X Comment
X
To Reply

35. Post
Persistent Stack:
- Post
- Comment
- Reply
Delete Request
Restoration Logs:
Post, Assoc From author,
Comment, Assoc From
author, Assoc to Comment,
Comment
Author X
X
X
To Comment
To Comment
Comment
X
Author
X
X
X X Comment
X
Reply
To Reply

36. Post
Persistent Stack:
- Post
- Comment
- Reply
Delete Request
Restoration Logs:
Post, Assoc From author,
Comment, Assoc From
author, Assoc to Comment,
Comment, Reply
Author X
X
X
To Comment
To Comment
Comment
X
Author
X
X
X X Comment
X
Reply
To Reply
X

37. Post
Persistent Stack:
- Post
- Comment
- Reply
Delete Request
Restoration Logs:
Post, Assoc From author,
Comment, Assoc From
author, Assoc to Comment,
Comment, Reply, ...
Author X
X
X
To Comment
To Comment
Comment
X
Author
X
X
X X Comment
X
Reply
To Reply
X
X X

38. Post
Persistent Stack:
- Post
- Comment
- Reply
Delete Request
Restoration Logs:
Post, Assoc From author,
Comment, Assoc From
author, Assoc to Comment,
Comment, Reply, ...
Author X
X
X
To Comment
To Comment
Comment
X
Author
X
X
X X Comment
X
Reply
To Reply
X
X X

39. Post
Persistent Stack:
- Post
- Comment
- Reply
Delete Request
Restoration Logs:
Post, Assoc From author,
Comment, Assoc From
author, Assoc to Comment,
Comment, Reply, ...
Author X
X
X
To Comment
To Comment
Comment
X
Author
X
X
X X Comment
X
Reply
To Reply
X
X X
X
X

40. Post
Persistent Stack:
- Post
Delete Request
Restoration Logs:
Post, Assoc From author,
Comment, Assoc From
author, Assoc to Comment,
Comment, Reply, ...
Author X
X
X
To Comment
To Comment
Comment
X
Author
X
X
X X Comment
X
Reply
To Reply
X
X X
X
X

41. Post
Delete Request
Restoration Logs:
Post, Assoc From author,
Comment, Assoc From
author, Assoc to Comment,
Comment, Reply, ...
Author X
X
X
To Comment
To Comment
Comment
X
Author
X
X
X X Comment
X
Reply
To Reply
X
X X
X
X
Done!

42. Outline
I. Design Overview
A. Relying on a Data Definition Language
B. Walking the graph
C. Batching
D. Scheduling deletions in the future
II. Guarantees
III. Monitoring Guarantees
IV. Conclusion

43. Post
Comment
Comment
Reply
Alice
Bob
Delete Request Actions:
1. Read
2. Checkpoint on stack
3. Log restoration logs
4. Self Delete
5. Graph Delete
Author
Author
To Comment
To Comment
To Reply

44. Post
Delete Request Actions:
1. Read

45. Post
Delete Request Actions:
1. Read
Author

46. Post
Delete Request Actions:
1. Read
Author
To Comment
To Comment

47. Post
Comment
Delete Request Actions:
1. Read
Author
To Comment
To Comment

48. Post
Comment
Delete Request Actions:
1. Read
Author
Author
To Comment
To Comment

49. Post
Comment
Delete Request Actions:
1. Read
Author
Author
To Comment
To Comment
Comment

50. Post
Comment
Delete Request Actions:
1. Read
2. Checkpoint on stack
Author
Author
To Comment
To Comment
Persistent Stack:
- Post
- Comment
- Comment
Comment

51. Post
Comment
Delete Request Actions:
1. Read
2. Checkpoint on stack
3. Log restoration logs
Author
Author
To Comment
To Comment
Persistent Stack:
- Post
- Comment
- Comment
Comment
Restoration Logs:
Batched Restoration Record

52. Post
Comment
Delete Request Actions:
1. Read
2. Checkpoint on stack
3. Log restoration logs
4. Delete
Author
Author
To Comment
To Comment
Persistent Stack:
- Post
- Comment
- Comment
Comment
Restoration Logs:
Batched Restoration Record
X
X
X
X X
X
X
XX

53. Post
Comment
Delete Request Actions:
1. Read
2. Checkpoint on stack
Author
Author
To Comment
To Comment
Persistent Stack:
- Post
- Comment
Comment
Restoration Logs:
Batched Restoration Record
X
X
X
X X
X
X
XX
Reply
To Reply

54. Post
Comment
Delete Request Actions:
1. Read
2. Checkpoint on stack
3. Log restoration logs
Author
Author
To Comment
To Comment
Persistent Stack:
- Post
- Comment
Comment
Restoration Logs:
Batched Restoration
Record, other batched
record
X
X
X
X X
X
X
XX
Reply
To Reply

55. Post
Comment
Delete Request Actions:
1. Read
2. Checkpoint on stack
3. Log restoration logs
4. Delete
Author
Author
To Comment
To Comment
Persistent Stack:
- Post
- Comment
Comment
Restoration Logs:
Batched Restoration
Record, other batched
record
X
X
X
X X
X
X
XX
Reply
To Reply
XX
X
X X

56. Outline
I. Design Overview
A. Relying on a Data Definition Language
B. Walking the graph
C. Batching
D. Scheduling deletions in the future
II. Guarantees
III. Monitoring Guarantees
IV. Conclusion

57. Scheduling Deletions in the Future
- Essential to enable ephemerality at Facebook
- Like stories, or the account deletion grace period
- Supports scheduling years in the future
- Supports custom TTL logic
- "Delete this post 9 days after the last comment"
- 160B events processed per day

59. Outline
I. Design Overview
II. Guarantees
A. Scheduling
B. Eventual Completion
C. Eventual Completeness
D. Restorations
III. Monitoring Guarantees
IV. Conclusion

60. Scheduling
- Deletions run in two phases
- The "Sync part", running synchronously in the web request
- The "Async part", which iterates over the graph

61. Post
Comment
Comment
Reply
Alice
Bob
Author
To Comment
To Reply
Sync
Async
X
Steps:
1. Write the deletion
metadata
2. Log the deletion
request
3. Delete the top level
object
4. Schedule the async job

62. Scheduling
- Deletions run in two phases
- The "Sync part", running synchronously in the web request
- The "Async part", which iterates over the graph
- The sync part effectively hides the sub-graph

63. Post
Comment
Comment
Reply
Alice
Bob
Author
To Comment
To Reply
Sync
Async
X

64. Scheduling
- Deletions run in two phases
- The "Sync part", running synchronously in the web request
- The "Async part", which iterates over the graph
- The sync part effectively hides the sub-graph
- The privacy layer is distinct from the deletion graph, so we have checks in place

65. Post
Comment
Actions:
1. Read
2. Checkpoint on stack
Author
Author
To Comment
To Comment
Persistent Stack:
- Post
- Comment
Comment
Restoration Logs:
Batched Restoration Record
X
X
X
X X
X
X
XX
Reply
To Reply
Sync
Async
Can we load this object?

66. Outline
I. Design Overview
II. Guarantees
A. Scheduling
B. Eventual Completion
C. Eventual Completeness
D. Restorations
III. Monitoring Guarantees
IV. Conclusion

67. Eventual Completion
- Deletions encounter infrastructure issues
- Deletions encounter bugs
- Deletions get dropped by dependencies
- Deletions fail halfway through their scheduling
- Every deletion started must complete

68. Deletion
Executions
Deletion
History
Logs: Start, Exceptions,
Timeouts, and Completions
Deletion
Status
Maintains a daily status for all
deletions in flight
Idle Deletions
Stuck
Deletions
Reschedules Deletions
Escalation

69. Outline
I. Design Overview
II. Guarantees
A. Scheduling
B. Eventual Completion
C. Eventual Completeness
D. Restorations
III. Monitoring Guarantees
IV. Conclusion

70. Eventual Completeness
- Issues can lead to orphaned data
- Bugs in the deletion logic
- Race conditions
- Deletion settings mis-configurations
- We need to cleanup such data retroactively
- The solution: Object re-deletions

71. Post
X Share
X
Post to
Comment:
Deep
Share to
post:
Shallow

72. Post
Comment
Rest of the
subgraph
X
X X
Share
Post
Rest of the
subgraph
X
X X
X
X
X
X
Post to
Comment:
Deep
Share to
post:
Shallow

73. Eventual Completeness
- We run migrations on object types
- Migrations start every other week
- They iterate over all our data, and schedule the deletions

74. Outline
I. Design Overview
II. Guarantees
A. Scheduling
B. Eventual Completion
C. Eventual Completeness
D. Restorations
III. Monitoring Guarantees
IV. Conclusion

75. Restorations
- Data losses are inevitable
- Product bugs
- Deletion misconfiguration
- We log restoration logs before any delete we issue
- Graph indexed, different from the data stores backups

76. Deletion
Executions
Logs: Restoration logs
Write Ahead Log
Restoration
Index
Payloads
Tailer

77. Deletion
Executions
Logs: Restoration logs
Write Ahead Log
Restoration
Index
Payloads
Tailer
What if those have backups?

78. Deletion
Executions
Logs: Restoration logs
Write Ahead Log
Restoration
Index
Payloads
Tailer
What if those have backups?
Encryption
Service
Provides a daily rotating
encryption key

79. Preventing data loss
- Static analysis on the deletion graph
- Dynamic checks based on Deletion Constraints
- Predictions on the edges' deletion behaviours

80. Outline
I. Design Overview
II. Guarantees
III. Monitoring Guarantees
IV. Conclusion

81. Monitoring Guarantees
- Scheduling
- Success rate
- Amount of object detected as still visible during a deletion
- Eventual Completion
- Amount of deletions getting rescheduled
- Amount of deletions not getting executed for more than a day
- Eventual Completeness
- Amount of object remediations
- Restorations
- Oldest encryption key stored

82. The Happy path
The safety net(s)
Measure the happy path's reliability
Measure how much falls through the gaps
Measure the safety net's reliability
Bonus points:
- An orthogonal way to measure success
- A second safety net

83. Conclusion

84. Conclusion
- Deletion is a hard problem
- The happy path isn't enough
- We need to make it easier for developers to do the right thing than the wrong
things

85. Thanks!

87. nebryum@fb.com