2. Background
• Sharded MySql ETL
Platform on EC2 (EBS)
• Database Size - Up to
1TB
• Write latencies
exponentially
proportional to data size
3. Background
• Cassandra Thrift Loading on EC2
(Ephemeral RAID0)
• Database Size - ∑ available node space
• Write latencies ~linearly proportional to
number of nodes
• 12 XL node cluster (RF=3): 6-125x
Improvement over EBS backed MySQL
systems
4. Thrift ETL
• Thrift overhead: Converting to/from
internal structures
• Routing from coordinator nodes
• Writing to commitlog
• Internal structures -> on-disk format
Source: http://wiki.apache.org/cassandra/BinaryMemtable
5. Bulk Load
• Core functionality
• Existing ETL
Nodes for bulk
loading
• Move data file &
index generation
off C* nodes
6. BMT Bulk Load
• Requires StorageProxy API (Java)
• Rows not live until flush
• Wiki example uses Hadoop
Source: http://wiki.apache.org/cassandra/BinaryMemtable
7. Streaming Bulk Load
• Cassandra as Fat Client
• BYO SSTables
• sstableloader [options] /path/to/
keyspace_dir
• Can ignore list of nodes (-i)
• keyspace_dir should be name of keyspace
and contain generated SSTable Data &
Index files
15. for (File : files)
{
ETL JAR
importer = new CBLI(...);
importer.open();
// Processing omitted
importer.close()
}
16. ETL JAR
CassandraBulkLoadImporter.initSSTableWriters():
File tempFiles = new File(“/path/to/Prefs”);
tempFiles.mkdirs();
for (String cfName : COLUMN_FAMILY_NAMES) {
SSTableSimpleUnsortedWriter writer = new
SSTableSimpleUnsortedWriter(
tempFiles, Model.Prefs.Keyspace.name, cfName,
Model.COLUMN_FAMILY_COMPARATORS.get(cfName),
null, bufferSizeInMB);// No Super CFs
writers.put(name, writer);
}
17. ETL JAR
CassandraBulkLoadImporter.processSuppressionRecips():
for (User user : users) {
String key = user.getUserId();
SSTSUW writer = tableWriters.get(Model.Users.CF.name);
// rowKey() converts String to ByteBuffer
writer.newRow(rowKey(key));
o.a.c.t.Column column = newUserColumn(user);
writer.addColumn(column.name, column.value,
column.timestamp);
...
// Repeat for each column family
}
18. ETL JAR
CassandraBulkLoadImporter.close():
for (String cfName : COLUMN_FAMILY_NAMES) {
try {
tableWriters.get(cfName).close();
}
catch (IOException e) {
log.error(“close failed for ”+cfName);
throw new RuntimeException(cfName+” did not close”);
}
String streamCmd = "sstableloader -v --debug"
+ tempFiles.getAbsolutePath();
Process stream = Runtime.getRuntime().exec(streamCmd);
if (!stream.waitFor() == 0)
log.error(“stream failed”);
}
19. cassandra_bulk_load.py
import random
import sys
import uuid
from java.io import File
from net.grinder.script.Grinder import grinder
from net.grinder.script import Statistics
from net.grinder.script import Test
from com.mycompany import App
from com.mycompany.tool import SingleColumnBulkImport
20. cassandra_bulk_load.py
input_files = [] # files to load
site_id = str(uuid.uuid4())
import_id = random.randint(1,1000000)
list_ids = [] # lists users will be loaded to
try:
App.INSTANCE.start()
dao = App.INSTANCE.getUserDAO()
bulk_import = SingleColumnBulkImport(
dao.prefsKeyspace, input_files, site_id,
list_ids, import_id)
except:
exception = sys.exc_info()[1]
print exception.message
print exception.stackTrace
22. cassandra_bulk_load.py
# Import and record stats
def import_and_record():
bulk_import.importFiles()
grinder.statistics.forCurrentTest.setLong(
"userLong0", bulk_import.totalLines)
grinder.statistics.forCurrentTest.setLong(
"userLong1",
grinder.statistics.forCurrentTest.time)
# Create an Import Test with a test number and a
description
import_test = Test(1, "Recip Bulk Import").wrap(
import_and_record)
# A TestRunner instance is created for each thread
class TestRunner:
# This method is called for every run.
def __call__(self):
import_test()
23. Stress Results
• Once Data and Index files generated,
streaming bulk load is FAST
• Average: ~2.5x increase over Thrift
• ~15-300x increase over MySQL
• Impact on cluster is minimal
• Observed downside: Writing own SSTables
slower than Cassandra