Swift & Fika talk discussing an overview of machine learning tools for Apple platforms, covering short examples using Vision, Create ML, Turi Create, & Core ML.
12. ML approach
Label: "Border"
1. Detect game box (rectangle
detection)
2. Classify Agricola piece within
rectangle (image classification)
13. Capabilities
... built right into the Vision
framework, no model training
needed
→ detection: rectangles, face,
barcode, text
→ object tracking
→ image alignment
16. // 2. Create request
let request = VNDetectRectanglesRequest(completionHandler: self.handleDetectedRectangles)
17. // 3. Send request to handler
do {
try handler.perform([request])
} catch let error as NSError {
// handle error
return
}
18. // 4. Handle results
func handleDetectedRectangles(request: VNRequest?, error: Error?) {
if let results = request?.results as? [VNRectangleObservation] {
// Do something with results [*bounding box coordinates*]
}
}
19. ML approach
Label: "Border"
1. Detect game box (rectangle
detection)
2. Classify Agricola piece within
rectangle (image classification)
20. Why restrict input image to the box?
→ easier to train an accurate model
→ faster to collect image data
21. Capabilities
In Xcode playground, train custom
model for:
→ image classification
→ text classification
→ classification & regression of
column data
22. Collect Data
→ Collect images representative of real world use
cases
→ Vary angle & lighting
→ >10 images per label, but ideally more
→ Equal # images for each label
→ Recommended: >299x299 pixels
23. Collecting Data Quickly
// Extract .jpg frames from .mov @ 5 frames/sec
ffmpeg -i stone.mov -r 5 data/stone/stone_%04d.jpg
34. Capabilities
→ perform predictions using
model
→ quantized weights (32 bit -> 16,
8, 4... bit)
→ perform batch predictions
→ create custom model layer
35. Vision + Core ML
1. create Vision Core ML model
2. create handler
3. create task specific request
4. send request to handler
5. handle results
36. // 1. Create Vision Core ML model
let model = AgricolaPieceClassifier()
guard let visionCoreMLModel = try? VNCoreMLModel(for: model.model) else { return }