1. The document discusses using Swift with various media frameworks like AV Foundation and Audio Toolbox that are commonly used for audio and video processing but were designed for Objective-C. 2. It notes challenges like the frameworks using C APIs, working with real-time constraints, and capturing objects from Swift that could cause performance issues. 3. The author recommends strategies like using AV Foundation if possible, learning to balance Swift and C idioms, and aiming for idiomatic Swift rather than Swift that looks like translated Objective-C.

1. Media Frameworks and
Swift:
This is Fine
Chris Adamson • @invalidname
Forward Swift, March 2017

2. Who the what, now?
@invalidname

16. import Cocoa
import AVFoundation
import CoreMediaIO
if let devices = AVCaptureDevice.devices(),
let avDevices = devices.filter(
{$0 is AVCaptureDevice}) as? [AVCaptureDevice] {
for device in avDevices {
print("(device.description)")
}
}

17. <AVCaptureHALDevice: 0x100b16ab0 [Loopback Simulator][com.rogueamoeba.Loopback:E8577B20-0806-4472-A5E6-426CABCD6C8E]>
<AVCaptureHALDevice: 0x100c1a7c0 [Loopback Line-In][com.rogueamoeba.Loopback:A00F38FD-C2B6-43FD-98B7-23BAA6FACB03]>
<AVCaptureHALDevice: 0x100c16910 [iMic USB audio system][AppleUSBAudioEngine:Griffin Technology, Inc:iMic USB audio system:220000:2,1]>
<AVCaptureHALDevice: 0x100d13900 [Loopback Keynote][com.rogueamoeba.Loopback:1936D2A3-6D0B-428E-899E-0ABE46628EA4]>
<AVCaptureHALDevice: 0x100a26850 [Soundflower (64ch)][SoundflowerEngine:1]>
<AVCaptureHALDevice: 0x100a26310 [HD Pro Webcam C920][AppleUSBAudioEngine:Unknown Manufacturer:HD Pro Webcam C920:1218B05F:3]>
<AVCaptureHALDevice: 0x100d13660 [Soundflower (2ch)][SoundflowerEngine:0]>
<AVCaptureDALDevice: 0x100a348f0 [iGlasses][iGlasses]>
<AVCaptureDALDevice: 0x100a28d00 [HD Pro Webcam C920][0x244000046d082d]>
Program ended with exit code: 0

19. CMIOObjectPropertyAddress prop =
{ kCMIOHardwarePropertyAllowScreenCaptureDevices,
kCMIOObjectPropertyScopeGlobal,
kCMIOObjectPropertyElementMaster };
UInt32 allow = 1;
CMIOObjectSetPropertyData( kCMIOObjectSystemObject,
&prop, 0, NULL, sizeof(allow), &allow );

20. var prop = CMIOObjectPropertyAddress(
mSelector: CMIOObjectPropertySelector(
kCMIOHardwarePropertyAllowScreenCaptureDevices),
mScope: CMIOObjectPropertyScope(kCMIOObjectPropertyScopeGlobal),
mElement: CMIOObjectPropertyElement(
kCMIOObjectPropertyElementMaster))
var allow : UInt32 = 1
CMIOObjectSetPropertyData(CMIOObjectID(kCMIOObjectSystemObject),
&prop,
0,
nil,
UInt32(MemoryLayout<UInt32>.size),
&allow)

21. CMIOObjectPropertyAddress prop =
{ kCMIOHardwarePropertyAllowScreenCaptureDevices,
kCMIOObjectPropertyScopeGlobal,
kCMIOObjectPropertyElementMaster };
UInt32 allow = 1;
CMIOObjectSetPropertyData( kCMIOObjectSystemObject,
&prop, 0, NULL, sizeof(allow), &allow );
var prop = CMIOObjectPropertyAddress(
mSelector: CMIOObjectPropertySelector(
kCMIOHardwarePropertyAllowScreenCaptureDevices),
mScope: CMIOObjectPropertyScope(kCMIOObjectPropertyScopeGlobal),
mElement: CMIOObjectPropertyElement(
kCMIOObjectPropertyElementMaster))
var allow : UInt32 = 1
CMIOObjectSetPropertyData(CMIOObjectID(kCMIOObjectSystemObject),
&prop,
0,
nil,
UInt32(MemoryLayout<UInt32>.size),
&allow)
This
is
fine

23. var prop = CMIOObjectPropertyAddress(
mSelector: CMIOObjectPropertySelector(
kCMIOHardwarePropertyAllowScreenCaptureDevices),
mScope: CMIOObjectPropertyScope(
kCMIOObjectPropertyScopeGlobal),
mElement: CMIOObjectPropertyElement(
kCMIOObjectPropertyElementMaster))
var prop = CMIOObjectPropertyAddress(
mSelector: CMIOObjectPropertySelector(
kCMIOHardwarePropertyAllowScreenCaptureDevices),
mScope: CMIOObjectPropertyScope(
kCMIOObjectPropertyScopeGlobal),
mElement: CMIOObjectPropertyElement(
kCMIOObjectPropertyElementMaster))

26. public typealias CMIOObjectPropertySelector = UInt32
public typealias CMIOObjectPropertyScope = UInt32
public typealias CMIOObjectPropertyElement = UInt32
public struct CMIOObjectPropertyAddress {
public var mSelector: CMIOObjectPropertySelector
public var mScope: CMIOObjectPropertyScope
public var mElement: CMIOObjectPropertyElement
public init()
public init(mSelector: CMIOObjectPropertySelector,
mScope: CMIOObjectPropertyScope,
mElement: CMIOObjectPropertyElement)
}

27. extension CMIOObjectPropertySelector {
static let allowScreenCaptureDevices = CMIOObjectPropertySelector(
kCMIOHardwarePropertyAllowScreenCaptureDevices)
}
extension CMIOObjectPropertyScope {
static let global = CMIOObjectPropertyScope(kCMIOObjectPropertyScopeGlobal)
}
extension CMIOObjectPropertyElement {
static let master = CMIOObjectPropertyElement(kCMIOObjectPropertyElementMaster)
}

28. var prop = CMIOObjectPropertyAddress(
mSelector: .allowScreenCaptureDevices,
mScope: .global,
mElement: .master)
var prop = CMIOObjectPropertyAddress(
mSelector: CMIOObjectPropertySelector(
kCMIOHardwarePropertyAllowScreenCaptureDevices),
mScope: CMIOObjectPropertyScope(
kCMIOObjectPropertyScopeGlobal),
mElement: CMIOObjectPropertyElement(
kCMIOObjectPropertyElementMaster))

30. Demo
http://github.com/invalidstream/audio-reverser

31. Reversing Audio
1. Decode the MP3/AAC to LPCM2. Grab a buffer from the end3. Reverse its samples in memory4. Write it to the front of a new file5. Repeat until fully baked

32. API Needs
• Convert from MP3/AAC to LPCM
• Write sequentially to audio file (.caf, .aif, .wav)
• Random-access read from audio file

33. Plan A (Swift)
• AV Foundation
• AVAssetReader/Writer can do format conversion
while reading/writing audio files
• Can’t (easily) read from arbitrary packet offsets;
meant to process everything forward

34. Plan B (C, Swift?)
• Audio Toolbox (part of Core Audio)
• ExtAudioFile can do format conversions while
reading/writing audio files
• AudioFile can read from arbitrary packet offset

36. // declare LPCM format we are converting to
AudioStreamBasicDescription format = {0};
format.mSampleRate = 44100.0;
format.mFormatID = kAudioFormatLinearPCM;
format.mFormatFlags = kAudioFormatFlagIsPacked |
kAudioFormatFlagIsSignedInteger;
format.mBitsPerChannel = 16;
format.mChannelsPerFrame = 2;
format.mBytesPerFrame = 4;
format.mFramesPerPacket = 1;
format.mBytesPerPacket = 4;

37. // declare LPCM format we are converting to
var format = AudioStreamBasicDescription(
mSampleRate: 44100.0,
mFormatID: kAudioFormatLinearPCM,
mFormatFlags: kAudioFormatFlagIsPacked +
kAudioFormatFlagIsSignedInteger,
mBytesPerPacket: 4,
mFramesPerPacket: 1,
mBytesPerFrame: 4,
mChannelsPerFrame: 2,
mBitsPerChannel: 16,
mReserved: 0)

38. // open AudioFile for output
AudioFileID forwardAudioFile;
err = AudioFileCreateWithURL(forwardURL,
kAudioFileCAFType,
&format,
kAudioFileFlags_EraseFile,
&forwardAudioFile);
IF_ERR_RETURN
#define IF_ERR_RETURN if (err != noErr) { return err; }

39. // open AudioFile for output
var forwardAudioFile: AudioFileID?
err = AudioFileCreateWithURL(forwardURL,
kAudioFileCAFType,
&format,
AudioFileFlags.eraseFile,
&forwardAudioFile)
if err != noErr { return err }

41. // open AudioFile for output
var forwardAudioFile: AudioFileID?
err = AudioFileCreateWithURL(forwardURL,
kAudioFileCAFType,
&format,
AudioFileFlags.eraseFile,
&forwardAudioFile)
if err != noErr { return err }

42. // open AudioFile for output
var forwardAudioFile: AudioFileID?
err = AudioFileCreateWithURL(forwardURL,
kAudioFileCAFType,
&format,
AudioFileFlags.eraseFile,
&forwardAudioFile)
if err != noErr { return err }
1. Uses a free function, rather than a method on AudioFile

43. // open AudioFile for output
var forwardAudioFile: AudioFileID?
err = AudioFileCreateWithURL(forwardURL,
kAudioFileCAFType,
&format,
AudioFileFlags.eraseFile,
&forwardAudioFile)
if err != noErr { return err }
2. Errors are communicated via the return value, rather than
throws

44. // open AudioFile for output
var forwardAudioFile: AudioFileID?
err = AudioFileCreateWithURL(forwardURL,
kAudioFileCAFType,
&format,
AudioFileFlags.eraseFile,
&forwardAudioFile)
if err != noErr { return err }
3. Some parameters are UInt32 constants, some are enums

45. // open AudioFile for output
var forwardAudioFile: AudioFileID?
err = AudioFileCreateWithURL(forwardURL,
kAudioFileCAFType,
&format,
AudioFileFlags.eraseFile,
&forwardAudioFile)
if err != noErr { return err }
4. Audio format is passed as an
UnsafePointer<AudioStreamBasicDescription>

46. // open AudioFile for output
var forwardAudioFile: AudioFileID?
err = AudioFileCreateWithURL(forwardURL,
kAudioFileCAFType,
&format,
AudioFileFlags.eraseFile,
&forwardAudioFile)
if err != noErr { return err }
5. Created object is returned via an in-out parameter

47. To say nothing of…

48. Pointer arithmetic!
// swap packets inside transfer buffer
for i in 0..<packetsToTransfer/2 {
let swapSrc = transferBuffer.advanced(by: Int(i) * Int(format.mBytesPerPacket))
let swapDst = transferBuffer.advanced(by: transferBufferSize -
(Int(i+1) * Int(format.mBytesPerPacket)))
memcpy(swapBuffer, swapSrc, Int(format.mBytesPerPacket))
memcpy(swapSrc, swapDst, Int(format.mBytesPerPacket))
memcpy(swapDst, swapBuffer, Int(format.mBytesPerPacket))
}

50. Couldn’t you just…

51. extension AudioFileID {
init? (url: URL, fileType: UInt32,
format: AudioStreamBasicDescription, flags: AudioFileFlags) {
var fileId : AudioFileID?
var format = format
let err = AudioFileCreateWithURL(url as CFURL,
fileType,
&format,
flags,
&fileId)
guard err != noErr, let createdFile = fileId else { return nil }
self = createdFile
}
}

52. Been there, done that
• The Amazing Audio Engine 💀
• Novocaine (💀?)
• EZAudio 💀
• AudioKit
• Superpowered
• etc…

54. /**
Convert a source audio file (using any Core Audio-supported codec) and create LPCM .caf
files for its forward and backward versions.
- parameter sourceURL: A file URL containing the source audio to be read from
- parameter forwardURL: A file URL with the destination to write the decompressed (LPCM) forward file
- parameter backwardURL: A file URL with the destination to write the backward file
*/
OSStatus convertAndReverse(CFURLRef sourceURL, CFURLRef forwardURL, CFURLRef backwardURL);
AudioReversingC.h
//
// Use this file to import your target's public headers that you would like to expose to Swift.
//
#import <CoreFoundation/CoreFoundation.h>
#import <AudioToolbox/AudioToolbox.h>
OSStatus convertAndReverse(CFURLRef sourceURL, CFURLRef forwardURL, CFURLRef backwardURL);
AudioReverser-Bridging-Header.h

55. if USE_SWIFT_CONVERTER {
err = convertAndReverseSwift(sourceURL: source as CFURL,
forwardURL: self.forwardURL as! CFURL,
backwardURL: self.backwardURL as! CFURL)
} else {
err = convertAndReverse(source as! CFURL,
self.forwardURL as! CFURL,
self.backwardURL as! CFURL)
}

56. C APIs on iOS/macOS
• Core Foundation
• Core Audio
• Core Media
• Video Toolbox
• Keychain
• IOKit
• OpenGL
• SQLite
• Accelerate
• XPC
• BSD, Mach
• etc…

57. Going deeper…

59. Audio Units
• Discrete software objects for working with audio
• Generators, I/O, Filters/Effects, Mixers,
Converters
• Typically combined in a “graph” model
• Used by Garage Band, Logic, etc.

62. Wait, what?

63. -(instancetype)initWithComponentDescription:(AudioComponentDescription)componentDescription
options:(AudioComponentInstantiationOptions)options
error:(NSError **)outError {
self = [super initWithComponentDescription:componentDescription
options:options
error:outError];
if (self == nil) {
return nil;
}
// ...
return self;
}
MyAudioUnit.m

65. XPC
(macOS only)

66. Swift 3Swift 4

67. https://github.com/apple/swift/blob/master/docs/
ABIStabilityManifesto.md

68. “Given the importance of getting the core ABI and the
related fundamentals correct, we are going to defer the
declaration of ABI stability out of Swift 4 while still
focusing the majority of effort to get to the point where
the ABI can be declared stable.”
—Ted Kremenek, Feb. 16, 2017
“Swift 4, stage 2 starts now”
https://lists.swift.org/pipermail/swift-evolution/Week-of-
Mon-20170213/032116.html

72. // Block which subclassers must provide to implement rendering.
- (AUInternalRenderBlock)internalRenderBlock {
// Capture in locals to avoid Obj-C member lookups.
// If "self" is captured in render, we're doing it wrong. See sample code.
return ^AUAudioUnitStatus(AudioUnitRenderActionFlags *actionFlags,
const AudioTimeStamp *timestamp,
AVAudioFrameCount frameCount,
NSInteger outputBusNumber,
AudioBufferList *outputData,
const AURenderEvent *realtimeEventListHead,
AURenderPullInputBlock pullInputBlock) {
// Do event handling and signal processing here.
return noErr;
};
}

73. Don’t do this
• An audio unit’s render block is called on a
realtime thread
• Therefore it cannot perform any action that could
block:
• I/O (file or network)
• Waiting on a mutex or semaphore

74. Also, don’t do this
• Call objc_msg_send()
• Capture any Objective-C or Swift object
• Allocate memory
Basically, if you touch anything in the block other than a pre-
allocated C struct, you’re asking for trouble.

75. https://github.com/apple/swift/blob/master/docs/
OwnershipManifesto.md

76. Certain kinds of low-level programming
require stricter performance guarantees.
Often these guarantees are less about
absolute performance than predictable
performance. For example, keeping up with
an audio stream is not a taxing job for a
modern processor, even with significant per-
sample overheads, but any sort of
unexpected hiccup is immediately noticeable
by users.
—“Swift Ownership Manifesto”,
February 2017

77. We believe that these problems can be addressed with an opt-in set of
features that we collectively call ownership. […]
Swift already has an ownership system, but it's "under the covers": it's an
implementation detail that programmers have little ability to influence. What
we are proposing here is easy to summarize:
• We should add a core rule to the ownership system, called the Law of
Exclusivity […]
• We should add features to give programmers more control over the
ownership system […]
• We should add features to allow programmers to express types with
unique ownership […]

78. And yet…

79. “[Swift] is the first industrial-quality systems
programming language that is as expressive and
enjoyable as a scripting language.”
https://developer.apple.com/library/content/documentation/
Swift/Conceptual/Swift_Programming_Language/

80. So… when?

81. Waiting…
• ABI stability — will not be in Swift 4
• Ownership — unclear
• Are these traits sufficient?

84. Strategies
• Use AV Foundation if you can
• Learn to balance C and Swift
• “Render undo Caesar what is Caesar’s…”
• The goal is to have idiomatic Swift, not Swift
that may work but looks like C

85. Media Frameworks and
Swift:
This is Fine
Chris Adamson • @invalidname
Forward Swift, March 2017