2. Neptune Distributed Data Storage semi-structured data store(not file system) Use Distributed File System for Data file Supports real time and batch processing Google Bigtable clone Data Model, Architecture, Features Open source http://dev.naver.com/projects/neptune(korean) http://www.openneptune.com Goal 500 nodes 200 GB 이상/per node, Peta bytes
3. Features Schema Management Create, drop, modify table schema Real-time Transaction Single row operation(no join, group by, order by) Multi row operation: like, between Batch Transaction Scanner, Direct Uploader, MapReduce Adapter Scalability Automatic table split & re-assignment Reliability Data file stored in Distributed File System Commit log stored in ChangeLog Cluster Failover Tablet takeover time: max 1 min. Utility Web Console, Shell(simple query), Data Verifier
7. Index Root Index M.T1.1000:M1 … Max:mn M.T1.2000:M2 index of Meta Index Meta Tablet m2 m1 mn T1.100:U1 … xx xx … n T1.200:U2 T1.1000:UN … T1.2000:UN T1.1100:U1 T1.1200:U2 … index of User Tablet User defined Tablet U1 U2 10 20 … 100 110 120 … 200 … Index of TableMapFile’sblock(max-key, file-offset) Index Record format: . Key - TableName.MaxRowKey . Value – Tablet Name, assigned host scan 64KB TableMapFile(Physical file,sortedby rowkey, columnkey)
9. TabletServer Minor Compaction MemoryTable ChangeLogServer Data Operation put(key, value) ChangeLog Searcher get(key) Merged MapFile (HDFS) MapFile#2 (HDFS) MapFile#1 (HDFS) MapFile #n (HDFS) Major Compaction
10. Failover Master fail disabled only Table Schema Management and Tablet Split can execute Multi-Master TabletServer fail assign to other TabletServer by master within 2 minutes
12. Client Client API Single row operation: put/get Multi row operation: like, between Batch operation: scanner/uploader MapReduce: TabletInputFormat Command line Shell NQL(Neptune Query Language) JDBC support Web Console
13. Client API Example TableShematableSchema = new TableSchema(“T_TEST”, new String[]{“col1”, “col2”}); NTable.createTable(tableSchema); NTablentable = Ntable.openTable(“T_TEST”); Row row = new Row(new Row.Key(“RK1”)); Row.addCell(“col1”, new Cell(new Cell.Key(“CK1”), “test_value”.getBytes())); ntable.put(row); Row selectedRow = ntable.get(new Row.Key(“RK1”)); System.out.println(selectedRow.getCellList(“col1”).get(0)); TableScanner scanner = ntable.openScanner(ntable, new String[]{“col1”}); Row scanRow = null; while( (scanRow = scanner.next()) = null) { System.out.println(selectedRow.getCellList(“col1”).get(0)); } scanner.close();
14. Neptune Shell Data Definition CREATE TABLE DROP TABLE SHOW TABLES DESC Data Manipulation SELECT DELETE INSERT TRUNCATE COLUMN TRUNCATE TABLE Cluster Monitoring PING TABLETSERVER REPORT TABLE