Доповідь присвячена рішенню проблеми зміни структури даних сутності в REST API на Node.js, а також міграції даних в MongoDB Як елегентно не поламати сумісність з існуючими користувачами API Як підтримувати сумісність кількох версії структур даних одної сутності Які варіанти міграції даних в MongoDB з і без downtime Історія втрати даних при міграції даних в MongoDB Які інструменти і підходи забезпечать максимальну виживаємість даних при неуспішних міграціях

1. Data migration in Node.js
REST API and MongoDB
Andrii Shumada
@eagleeye_s
Sr. Software Developer at

3. frontend
REST API
MongoDB
database
JSON
JSON
3rd party clients &&
Mobile apps
JSON

4. Description
ID
Line1: amount, taxRate
Line2: amount, taxRate
ID
Line1: amount, taxRate
description
Line2: amount, taxRate
description
Yesterday
Document v1
Tomorrow
Document v2

5. REST API
v1
V2
or v1
You can’t release new
version of mobile app
as fast your frontend
app
v1
Your partners can’t
switch to new version
instantly

9. Because documents usually are independent!
/document/v1
/document/v2
/document/v3

10. GET
/document/:id/v2
Find document in
DB
Return this
document
GET
/document/:id/v1
Assuming for now that document has v2
structure in DB/business logic
Find document in
DB
Transform v2 -> v1
Return this
document

11. transformObject(document, 'v2', 'v2|v1', options, (err, documentVx) => {
return res.json(documentVx);
});
Return ‘downgraded’ version of document

12. POST/PUT
document/:id/v1
Save document in
DB v2
Transform v1 -> v2
Return this
document in v1
Validate document
v2
Transform v2 -> v1

13. transformObject(documentVX, 'v1|2', 'v2', options, (err, documentV2) => {
//business logic with documentV2
//maybe save in db with V2
transformObject(document, 'v2', 'v2|v1', options, (err, documentVx) => {
res.json(documentVx) // same version as we receive
});
});
First “Upgrade”, then “Downgrade”

14. GET
/documents/v2
Find documents
in DB
Return all
documents v2
(via transform stream)
(Stream all documents to res)
(get readable stream)
Transform all documents
v2 -> v1

15. Readable
stream
Transform
stream
writableStream
MongoDB response

16. const through2 = require('through2');
let stream = through2.obj((obj, encoding, callback) => {
//async transform single object here
through.push();
callback();
});
Transform stream

17. function getTransformStream(options) {
return through2.obj(function(obj, encoding, callback){
var through = this;
if(!preparedDataSets) {// Do all async stuff once
prepareTransform(options, (err, dataSets) => {
through.push(syncTransformReadyData(obj, dataSets, options));
callback();
});
} else {
through.push(syncTransformReadyData(obj, dataSets));
callback();
}
});

18. POST and PUT/api/entity/v1
/api/entity/v2
Transform v1 to v2
data format
/api/entity/v3
Transform v2 to v3
data format
Business logic (only accepts
v3 data format in)
API version
layer
GET and result of POST and PUT
/api/entity/v1 /api/entity/v2
Transform v1 to v2
data format
/api/entity/v3
Transform v2 to
v3 data format
Business logic (only returns
v3 data format out)
API version
layer
v2, v3Handle different
versions in DB
v2, v3
Handle different
versions in DB
v3
v3

19. https://github.com/debitoor/sir-transformalot

21. var transformConfig = {
v2: {
V1toV2: { },
V2toV1: { },
},
v3: {
V2toV3: {
transform: function (data, preparedData) {
//sync code
return data;
},
prepareTransform: function(options, callback) {
//async code
return callback(error, asyncData);
}
},
V3toV2: {
transform: function (data) {
//sync code
return data;
},
prepareTransform: function(options, callback) {
//async code
return callback(error, asyncData);
}
}
}
};

22. Possible categories
of solutions:
With downtime Without downtime

23. Loosingmoneyhere

24. API request
(POST, PUT,GET document)
Patch user
data
Document is
“patched”?
Proceed with business logic
No
documents.findOne({id, schema: ‘v2’}) ?
Yes
documets.update({...,id, schema: ‘v2’})

25. Any API REQUEST
Patch user
data
User data is
“patched”?
Proceed with business logic
No
settings.findOne({id, patched: true})
Yes
invoices.update({..., patched: true})
transactions.update({..., patched: true})
expenses.update({..., patched: true})

26. But there could be something
better and more secure

27. https://github.com/debitoor/mongopatch

30. -
-
-

31. module.exports = setup((patch) => {
patch.version('2.4.3');
patch.update('settings', search, (settings, callback) => {
return callback(null, modifier);
});
patch.after(whiteList({
'fieldThatAllowedToBeChanged': ['added|removed|modified'],
}));
});

32. @eagleeye_s
Andrii Shumada
Write good code!