Undelete (and more) rows from the binary log

Hacking Session: undelete (and more) rows
from the binary log
Percona Live Europe Amsterdam 2015 Frédéric -lefred- Descamps

Who am I ?
●
Frédéric Descamps
●
@lefred - follow me on twitter if you want ;-)
●
http://about.me/lefred
●
Working for Percona since 2011
●
Managing MySQL since 3.23
●
devops believer

Why ?
Because of Scott Noyes' blog post :
http://thenoyes.com/littlenoise/?p=307
●
Because it's faster than point-in-time
recovery
●
Because I didn't remember the awk script
in Scott's blog post

Requirements
●
Have the binary logs in ROW format
●
Have binlog_row_image set to FULL
●
Have access to the binary logs
●
Have access to MySQL
●
Have access to the MySQL source code [not mandatory]
●
Have a brain and being able to use it ;-)
Percona Live Europe Amsterdam 2015

Point-in-Time Recovery
Usually when we need to «undo» even one single statement, we
need to perform a point-in-time recovery
●
restore the last backup (full or several incrementals)
●
then replay some binary logs (and it can be a bunch of them!)
●
this is a very slow operation and while the binary logs are
replayed, usually everything needs to be stopped.

Let's hack together a quicker process
Ready ?

Let's delete a row
mysql> select @@version, @@version_comment;
+------------+------------------------------+
| @@version | @@version_comment |
+------------+------------------------------+
| 5.6.22-log | MySQL Community Server (GPL) |
+------------+------------------------------+
mysql> select * from fosdem.community_dinner;
+----+--------+--------+-------+
| id | name | pizzas | beers |
+----+--------+--------+-------+
| 1 | dim0 | 99 | 99 |
| 2 | Liz | 2 | 24 |
| 3 | Kenny | 12 | 32 |
| 4 | lefred | 10 | 10 |
+----+--------+--------+-------+
4 rows in set (0.00 sec)

Let's delete a row (2)
mysql> delete from community_dinner where id=2;
Query OK, 1 row affected (0.05 sec)
+----+--------+--------+-------+
+----+--------+--------+-------+
| 1 | dim0 | 99 | 99 |
| 3 | Kenny | 12 | 32 |
| 4 | lefred | 10 | 10 |
+----+--------+--------+-------+

Let's find the event in the binary log
mysql> show master statusG
*************************** 1. row ***************************
File: mysql-bin.000018
Position: 907
Binlog_Do_DB:
Binlog_Ignore_DB:
Executed_Gtid_Set:
1 row in set (0.00 sec)
mysql> show binlog events in 'mysql-bin.000018';
+------------------+-----+-------------+-----------+-------------+---------------------------+
| Log_name | Pos | Event_type | Server_id | End_log_pos | Info |
+------------------+-----+-------------+-----------+------------+----------------------------+
| mysql-bin.000018 | 4 | Format_desc | 1 | 120 | Server ... |
| mysql-bin.000018 | 120 | Query | 1 | 220 | create ... |
| mysql-bin.000018 | 220 | Query | 1 | 404 | use `fosdem`; |
| mysql-bin.000018 | 404 | Query | 1 | 478 | BEGIN |
| mysql-bin.000018 | 478 | Table_map | 1 | 544 | table_id: 76 (fosdem.commu|
| mysql-bin.000018 | 544 | Write_rows | 1 | 653 | table_id: 76 flags: STMT_E|
| mysql-bin.000018 | 653 | Xid | 1 | 684 | COMMIT /* xid=32 */ |
| mysql-bin.000018 | 824 | Delete_rows | 1 | 876 | table_id: 76 flags: STMT_E|
+------------------+-----+-------------+-----------+------------+====------------------------+

Let's find the event in the binary log
mysql> show master statusG
*************************** 1. row ***************************
File: mysql-bin.000018
Position: 907
Binlog_Do_DB:
Binlog_Ignore_DB:
Executed_Gtid_Set:
1 row in set (0.00 sec)
+------------------+-----+-------------+-----------+-------------+---------------------------+
+------------------+-----+-------------+-----------+------------+----------------------------+
+------------------+-----+-------------+-----------+------------+====------------------------+
The event we are looking for

binary log's event content
# mysqlbinlog mysql-bin.000018 -j 824 --stop-position=876
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/;
/*!40019 SET @@session.max_insert_delayed_threads=0*/;
/*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/;
DELIMITER /*!*/;
# at 4
#141221 21:20:55 server id 1 end_log_pos 120 CRC32 0xff8913a3 Start: binlog v 4, server v
5.6.22-log created 141221 21:20:55 at startup
# Warning: this binlog is either in use or was not closed properly.
ROLLBACK/*!*/;
BINLOG '
pyuXVA8BAAAAdAAAAHgAAAABAAQANS42LjIyLWxvZwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAACnK5dUEzgNAAgAEgAEBAQEEgAAXAAEGggAAAAICAgCAAAACgoKGRkAAaMT
if8=
'/*!*/;
# at 824
#150103 16:55:49 server id 1 end_log_pos 876 CRC32 0xaac329ee Delete_rows: table id 76 flags:
STMT_END_F
BINLOG '
BRGoVCABAAAANAAAAGwDAAAAAEwAAAAAAAEAAgAE//ACAAAAA0xpegIAAAAYAAAA7inDqg==
'/*!*/;
DELIMITER ;
# End of log file
ROLLBACK /* added by mysqlbinlog */;
/*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/;
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/;

binary log's event content decoded
...
BINLOG '
'/*!*/;
...
●
Let's check from the source the type code:
https://github.com/mysql/mysql-server/blob/5.6/sql/log_event.h
WRITE_ROWS_EVENT = 30,
UPDATE_ROWS_EVENT = 31,
DELETE_ROWS_EVENT = 32,

...
BINLOG '
'/*!*/;
...
●
Let's check from the source the type code:
https://github.com/mysql/mysql-server/blob/5.6/sql/log_event.h
WRITE_ROWS_EVENT = 30,
UPDATE_ROWS_EVENT = 31,
DELETE_ROWS_EVENT = 32,
MySQL 5.5 and MariaDB use V1:
WRITE_ROWS_EVENT_V1 = 23,
UPDATE_ROWS_EVENT_V1 = 24,
DELETE_ROWS_EVENT_V1 = 25,

# python -c "print hex(32)"
0x20
●
Again from the source we can find the event type's offset
#define EVENT_TYPE_OFFSET 4
# python -c "import base64; print ord(base64.b64decode(b'BRGoVCABA
AAANAAAAGwDAAAAAEwAAAAAAAEAAgAE//ACAAAAA0xpegIAAAAYAA
AA7inDqg==')[4])"
32
●
This is exactly the event we were looking for !!

rebuild a new binlog event
●
Now we can rebuild a new event by replacing x32 by x30
# python
>>> import base64
>>>
data=base64.b64decode(b'BRGoVCABAAAANAAAAGwDAAAAAEwAAAAAAAEAAgAE//ACAAAAA0xpegIAAAAYA
AAA7inDqg==')
>>> data
'x05x11xa8T x01x00x00x004x00x00x00lx03x00x00x00x00L
x00x00x00x00x00x01x00x02x00x04xffxf0x02x00x00x00x03
Lizx02x00x00x00x18x00x00x00xee)xc3xaa'
>>> hex(30)
'0x1e'
>>> data2=base64.b64encode("x05x11xa8Tx1ex01x00x00x004x00x00x00lx03x00
x00x00x00Lx00x00x00x00x00x01x00x02x00x04xffxf0x02x00x00x00x03
Lizx02x00x00x00x18x00x00x00xee)xc3xaa")
>>> data2
'BRGoVDABAAAANAAAAGwDAAAAAEwAAAAAAAEAAgAE//ACAAAAA0xpegIAAAAYAAAA7inDqg=='
●
Let's compare the two lines:
BRGoVB4BAAAANAAAAGwDAAAAAEwAAAAAAAEAAgAE//ACAAAAA0xpegIAAAAYAAAA7inDqg==

rebuild a new binlog event & replay it
●
ready to replay ?
●# mysqlbinlog mysql-bin.000018 -j 824 --stop-position=876 |
> sed s/BRGoVCAB/BRGoVB4B/ | mysql
+----+--------+--------+-------+
+----+--------+--------+-------+
| 1 | dim0 | 99 | 99 |
| 3 | Kenny | 12 | 32 |
| 4 | lefred | 10 | 10 |
+----+--------+--------+-------+

rebuild a new binlog event & replay it
●
ready to replay ?
sed s/BRGoVCAB/BRGoVB4B/ | mysql
+----+--------+--------+-------+
+----+--------+--------+-------+
| 1 | dim0 | 99 | 99 |
| 3 | Kenny | 12 | 32 |
| 4 | lefred | 10 | 10 |
+----+--------+--------+-------+
uuh ? did it fail ?

find what to replay
+------------------+-----+-------------+-----------+-------------+---------------------------+
+------------------+-----+-------------+-----------+------------+----------------------------+
+------------------+-----+-------------+-----------+------------+====------------------------+
●
We need to replay more, from the BEGIN to the COMMIT

replay it, 2nd try
●
ready to replay, again ?
> sed s/BRGoVCAB/BRGoVB4B/ | mysql
+----+--------+--------+-------+
+----+--------+--------+-------+
| 1 | dim0 | 99 | 99 |
| 2 | Liz | 2 | 24 |
| 3 | Kenny | 12 | 32 |
| 4 | lefred | 10 | 10 |
+----+--------+--------+-------+

More ?
●
With the same kind of technique, we can also:
– delete rows that were inserted (sql injection?)
– un-update modified rows (more complicated)
●
I've created a script that automates all that: MyUndelete

MyUndelete : «un-insert»
●
We can delete an INSERT
# ./MyUndelete.py -s 41989 -e 42207 -i -b mysql-bin.000004
*** WARNING *** USE WITH CARE ****
Binlog file is /var/lib/mysql/mysqld-bin.000004
Start Position file is 41989
End Postision file is 42207
We also look to undo INSERTs
Event type ('x1e') is an insert v2
Ready to revert the statement ? [y/n]
y
Done... I hope it worked ;)

MyUndelete : «un-update»
●
This is a bit more difficult as we need to find the lenght of the
record to be able to split both records and rebuild a working
binary event
mysql> select * from community_dinner;
+----+-------+--------+-------+
+----+-------+--------+-------+
| 1 | fred | 3 | 3 |
| 2 | lizz | 2 | 10 |
| 3 | dimi | 99 | 99 |
| 4 | kenny | 20 | 20 |
+----+-------+--------+-------+

MyUndelete : «un-update» (2)
mysql> update community_dinner set beers=0 where id=3;
Query OK, 1 row affected (0.01 sec)
Rows matched: 1 Changed: 1 Warnings: 0
+----+-------+--------+-------+
+----+-------+--------+-------+
| 1 | fred | 3 | 3 |
| 2 | lizz | 2 | 10 |
| 3 | dimi | 99 | 0 |
| 4 | kenny | 20 | 20 |
+----+-------+--------+-------+

mysql> show binlog events in 'mysqld-bin.000018';
...
| mysqld-bin.000018 | 1145 | Query | 1 | 1219 | BEGIN |
| mysqld-bin.000018 | 1219 | Table_map | 1 | 1285 | table_id: 70
(fosdem.community_dinner) |
| mysqld-bin.000018 | 1285 | Update_rows | 1 | 1357 | table_id: 70 flags: STMT_END_F
| mysqld-bin.000018 | 1357 | Xid | 1 | 1388 | COMMIT /* xid=44 */
...
# ./MyUndelete.py -s 1145 -e 1388 -u -b /var/lib/mysql/mysqld-bin.000018
*** WARNING *** USE WITH CARE ****
Binlog file is /var/lib/mysql/mysqld-bin.000018
Start Position file is 1145
End Postision file is 1357
Event type ('x1f') is an update v2
We got an update!!
Ready to revert the statement ? [y/n]
y
Sending to mysql...
Done... I hope it worked ;)

+----+-------+--------+-------+
+----+-------+--------+-------+
| 1 | fred | 3 | 3 |
| 2 | lizz | 2 | 10 |
| 3 | dimi | 99 | 99 |
| 4 | kenny | 20 | 20 |
+----+-------+--------+-------+

Resources
●
Oracle MySQL Source code — https://github.com/mysql/mysql-server
●
MariaDB Source Code — https://github.com/MariaDB/server
●
Scott Noyes Blog Post - http://thenoyes.com/littlenoise/?p=307
●
MyUndelete on github — https://github.com/lefred/MyUndelete

Related topic
●
MySQL released «Binary Log API» that could be use to decode
in a better way the binary logs
– http://mysqlhighavailability.com/mysql-binlog-events-reading-and-handling-
information-from-your-binary-log/
– http://mysqlhighavailability.com/mysql-binlog-events-use-case-and-examples/
●
Jeremy Cole wrote a Ruby Library to also parse binary logs
– https://github.com/jeremycole/mysql_binlog

Thank you
Questions ?

Undelete (and more) rows from the binary log

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Undelete (and more) rows from the binary log

Similar to Undelete (and more) rows from the binary log (20)

More from Frederic Descamps

More from Frederic Descamps (20)

Recently uploaded

Recently uploaded (20)

Undelete (and more) rows from the binary log