Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Hacking images for faster OpenCV(iOS)

Tomoki Yamaguchi
LINE Fukuoka / Development Team F

先日、Grabcutと呼ばれる画像を半自動で切り抜く機能を、LINEスタンプの作成を行えるアプリ「LINE Creators Studio」に追加しました。

本アプリは主婦の方々などライト層が中心のため、丁寧なUIを作る必要があり、

そのためにはいろいろな画像を同時に作らなければなりません。

C++で書かれたOpenCVの実装とSwiftで書かれたUIの実装との間でのやり取りが多く発生してしまうため、画像オブジェクトの変換・コピーにかかるオーバーヘッドを無視できません。

そこでこれらのメモリレベルでの中身を知ることで余計な処理を省き、スムーズなユーザー体験を作ることができました。そのテクニックをご紹介します。

  • Soyez le premier à commenter

Hacking images for faster OpenCV(iOS)

  1. 1. Hacking images for Faster OpenCV / LINE Fukuoka(@_ha1f)
  2. 2. ● 2017’s new graduate, iOS Developer ● Current Project: LINE ● Previous Project: LINE Creators Studio (@_ha1f)
  3. 3. LINE Creators Studio Create & Publish LINE Stickers
  4. 4. LINE Creators Studio Create & Publish LINE Stickers
  5. 5. As-Is ● The same path drew ● We can modify but…
  6. 6. New feature: Grabcut Interactive foreground extraction
  7. 7. ● Estimate segmentation from input marker ● OpenCV has the implementation, so it’s easy to integrate! Grabcut
  8. 8. ● Estimate segmentation from input seed ● OpenCV has the implementation, so it’s easy to integrate! Grabcut
  9. 9. Convenience ● We’re using application instead of WEB for “convenience” ● Performance is essential C++ ● We cannot catch errors (crashes immediately) ● We cannot share types (Only Obj-C can bridge) Difficulties
  10. 10. MatToUIImage, UIImageToMat Copy memory every time… https://github.com/opencv/opencv/blob/master/modules/imgcodecs/src/ios_conversions.mm
  11. 11. Grabcut flow Original Image Input Image Input Mask Grabcut Initial Path Marker convert
  12. 12. Grabcut flow Mask Image Original Image Input Image Input Mask Grabcut Initial Path Marker We have to copy every time we add markers convert convert
  13. 13. Hacking images What is “image”?
  14. 14. var bits = [UInt8](repeating: 0, count: width * height * 4) let context = CGContext( data: &bits, width: width, height: height, bitsPerComponent: 8, bytesPerRow: 8 * width, space: CGColorSpaceCreateDeviceRGB(), bitmapInfo: CGImageAlphaInfo.premultipliedLast.rawValue) image.draw() print(bits) We can read as [UInt8], without copying Hacking CGContext
  15. 15. Mat( int _rows, int _cols, int _type, void* data ) Needless to say, it uses pointer Hacking cv::Mat
  16. 16. Similar format.
  17. 17. Why not share the memory?
  18. 18. @property cv::Mat rawImage; - (unsigned char *)data { return self.rawImage.data; } - (size_t)bufferSize { return self.rawImage.total() * self.rawImage.elemSize(); } Expose as Swift-compatible types In Objective-C++
  19. 19. CGContext( data: image.data, width: image.cols(), height: image.rows(), bitsPerComponent: image.elemSize1() * 8, bytesPerRow: image.step[0], space: CGColorSpaceCreateDeviceGray(), bitmapInfo: CGImageAlphaInfo.none.rawValue ) ※depends on color mode, this is case for Mask Create CGContext
  20. 20. let bufferSize = context.bytesPerRow * context.height let dataProvider = CGDataProvider(data: Data(bytesNoCopy: data, count: bufferSize, deallocator: .none) as CFData)! let maskImage = CGImage(maskWidth: context.width, height: context.height, bitsPerComponent: context.bitsPerComponent, bitsPerPixel: context.bitsPerPixel, bytesPerRow: context.bytesPerRow, provider: dataProvider, decode: nil, shouldInterpolate: false)! ※depends on color mode, this is case for Mask Read from CGContext as CGImage
  21. 21. We can share memory between cv::Mat, CGContext, CGImage Conclusion
  22. 22. ● We can clone CGImage using `pointer.copyMemory(from:byteCount:)` ● We can reset mask using `assign(repeating:count:)` Easy undo/redo/initialize Simpler image operations
  23. 23. ● Performance was not so improved… ● We had to build masks to show ● We had to copy to push undo stack ● Simple & Clear data-flow made us happy! Result
  24. 24. THANK YOU

×