Hadoop Conference Japan 2013 Winter の午後のライトニングトークの３番目のスライドです。

1. メインフレームと
Hadoopで
大量のPDF帳票を
出力してみた

2. 名前：三木 大知
所属：JSOL
職業：通りすがりの
システムエンジニア
twitter：@DaichiMiki

3. メインフレーム
ICON: Ravenna 3D By Double-J DesignAuthor Website: http://www.doublejdesign.co.uk/products-page/icons/ravenna-3d/

4. ICON: Ravenna 3D By Double-J DesignAuthor Website: http://www.doublejdesign.co.uk/products-page/icons/ravenna-3d/

5. PDFでペーパーレス
ICON: Ravenna 3D By Double-J DesignAuthor Website: http://www.doublejdesign.co.uk/products-page/icons/ravenna-3d/

6. ICON: Ravenna 3D By Double-J DesignAuthor Website: http://www.doublejdesign.co.uk/products-page/icons/ravenna-3d/

7. 1ページ = 0.1秒
4,000,000ページ
？秒
ICON: Ravenna 3D By Double-J DesignAuthor Website: http://www.doublejdesign.co.uk/products-page/icons/ravenna-3d/

8. 呼んだ？
Yes, Yes, Yes!

9. CSV
BLOB
ICON: Ravenna 3D By Double-J DesignAuthor Website: http://www.doublejdesign.co.uk/products-page/icons/ravenna-3d/

10. BLOBの情報
ICON: Ravenna 3D By Double-J DesignAuthor Website: http://www.doublejdesign.co.uk/products-page/icons/ravenna-3d/

11. Ma
pp er
InputFormat
Ma
pp er
InputFormat
Ma
pp er
InputFormat
ICON: Ravenna 3D By Double-J DesignAuthor Website: http://www.doublejdesign.co.uk/products-page/icons/ravenna-3d/

12. public class BLobRecordReader
implements RecordReader<RecordDataWritable, BytesWritable> {
private long start;
private long end;
private TreeMap<BLobInfo, RecordData> infoMap;
public BLobRecordReader(JobConf job, FileSplit split) throws Exception {
this.start = split.getStart();
this.end = start + split.getLength();
this.infoMap = new TreeMap<BLobInfo, RecordData>();
BufferedReader in = ... //from distributed cache
String line;
while( (line = in.readLine()) != null) {
BLobInfo infoKey = new BLobInfo(line);
if(start <= infoKey.getPos() && end > infoKey.getPos()) {
infoMap.put(infoKey, new RecordData(line));
}
}
...
}
public boolean next(RecordDataWritable key, BytesWritable value) throws IOException {
BLobInfo infoKey = new BLobInfo(pos);
if(!infoMap.containsKey(infoKey))return false;
RecordData infoData = infoMap.get(infoKey);
key.set(infoData);
int newSize = infoData.getFileSize();
in.readLine(value, newSize);
pos += newSize;
return true;
}
}

13. ?
ICON: Ravenna 3D By Double-J DesignAuthor Website: http://www.doublejdesign.co.uk/products-page/icons/ravenna-3d/

14. 60時間
元の処理

15. 1時間
20分
5 10 15 20

16. 400万ページのPDFも
20分で出力できるよ
...そう、Hadoopならね

17. Thanks!