media frameworks versus swift (swift by northwest, october 2017)

Media Frameworks Versus Swift

Chris Adamson • @invalidname Swift by Northwest, October 2017

Slides available at slideshare.net/invalidname Code available at github.com/invalidstream

Who the what, now?

@invalidname

import Cocoa import AVFoundation import CoreMediaIO

if let devices = AVCaptureDevice.devices(), let avDevices = devices.filter( {$0 is AVCaptureDevice}) as? [AVCaptureDevice] { for device in avDevices { print("\(device.description)") } }

[Loopback Simulator][com.rogueamoeba.Loopback:E8577B20-0806-4472-A5E6-426CABCD6C8E] [Loopback Line-In][com.rogueamoeba.Loopback:A00F38FD-C2B6-43FD-98B7-23BAA6FACB03] [iMic USB audio system][AppleUSBAudioEngine:Griffin Technology, Inc:iMic USB audio system:220000:2,1] [Loopback Keynote][com.rogueamoeba.Loopback:1936D2A3-6D0B-428E-899E-0ABE46628EA4] [Soundflower (64ch)][SoundflowerEngine:1] [HD Pro Webcam C920][AppleUSBAudioEngine:Unknown Manufacturer:HD Pro Webcam C920:1218B05F:3] [Soundflower (2ch)][SoundflowerEngine:0] [iGlasses][iGlasses] [HD Pro Webcam C920][0x244000046d082d] Program ended with exit code: 0

CMIOObjectPropertyAddress prop = { kCMIOHardwarePropertyAllowScreenCaptureDevices, kCMIOObjectPropertyScopeGlobal, kCMIOObjectPropertyElementMaster }; UInt32 allow = 1; CMIOObjectSetPropertyData( kCMIOObjectSystemObject, &prop, 0, NULL, sizeof(allow), &allow );

var prop = CMIOObjectPropertyAddress( mSelector: CMIOObjectPropertySelector(

kCMIOHardwarePropertyAllowScreenCaptureDevices), mScope: CMIOObjectPropertyScope(kCMIOObjectPropertyScopeGlobal), mElement: CMIOObjectPropertyElement(

kCMIOObjectPropertyElementMaster)) var allow : UInt32 = 1 CMIOObjectSetPropertyData(CMIOObjectID(kCMIOObjectSystemObject), &prop, 0, nil, UInt32(MemoryLayout<UInt32>.size), &allow)

CMIOObjectPropertyAddress prop = { kCMIOHardwarePropertyAllowScreenCaptureDevices, kCMIOObjectPropertyScopeGlobal, kCMIOObjectPropertyElementMaster }; UInt32 allow = 1; CMIOObjectSetPropertyData( kCMIOObjectSystemObject, &prop, 0, NULL, sizeof(allow), &allow );

var prop = CMIOObjectPropertyAddress( mSelector: CMIOObjectPropertySelector(

kCMIOHardwarePropertyAllowScreenCaptureDevices), mScope: CMIOObjectPropertyScope(kCMIOObjectPropertyScopeGlobal), mElement: CMIOObjectPropertyElement(

kCMIOObjectPropertyElementMaster)) var allow : UInt32 = 1 CMIOObjectSetPropertyData(CMIOObjectID(kCMIOObjectSystemObject), &prop, 0, nil, UInt32(MemoryLayout<UInt32>.size), &allow)

This is

fine

var prop = CMIOObjectPropertyAddress( mSelector: CMIOObjectPropertySelector(

kCMIOHardwarePropertyAllowScreenCaptureDevices), mScope: CMIOObjectPropertyScope(

kCMIOObjectPropertyScopeGlobal), mElement: CMIOObjectPropertyElement(

kCMIOObjectPropertyElementMaster))

var prop = CMIOObjectPropertyAddress( mSelector: CMIOObjectPropertySelector(

kCMIOHardwarePropertyAllowScreenCaptureDevices), mScope: CMIOObjectPropertyScope(

kCMIOObjectPropertyScopeGlobal), mElement: CMIOObjectPropertyElement(

kCMIOObjectPropertyElementMaster))

public typealias CMIOObjectPropertySelector = UInt32

public typealias CMIOObjectPropertyScope = UInt32

public typealias CMIOObjectPropertyElement = UInt32

public struct CMIOObjectPropertyAddress { public var mSelector: CMIOObjectPropertySelector

public var mScope: CMIOObjectPropertyScope

public var mElement: CMIOObjectPropertyElement

public init()

public init(mSelector: CMIOObjectPropertySelector, mScope: CMIOObjectPropertyScope, mElement: CMIOObjectPropertyElement)

}

extension CMIOObjectPropertySelector { static let allowScreenCaptureDevices = CMIOObjectPropertySelector( kCMIOHardwarePropertyAllowScreenCaptureDevices) }

extension CMIOObjectPropertyScope { static let global = CMIOObjectPropertyScope(kCMIOObjectPropertyScopeGlobal) }

extension CMIOObjectPropertyElement { static let master = CMIOObjectPropertyElement(kCMIOObjectPropertyElementMaster) }

var prop = CMIOObjectPropertyAddress( mSelector: .allowScreenCaptureDevices, mScope: .global, mElement: .master)

var prop = CMIOObjectPropertyAddress( mSelector: CMIOObjectPropertySelector(

kCMIOHardwarePropertyAllowScreenCaptureDevices), mScope: CMIOObjectPropertyScope(

kCMIOObjectPropertyScopeGlobal), mElement: CMIOObjectPropertyElement(

kCMIOObjectPropertyElementMaster))

Demo

http://github.com/invalidstream/audio-reverser

Reversing Audio

1. Decode the MP3/AAC to LPCM2. Grab a buffer from the end3. Reverse its samples in memory4. Write it to the front of a new file5. Repeat until fully baked

API Needs

• Convert from MP3/AAC to LPCM

• Write sequentially to audio file (.caf, .aif, .wav)

• Random-access read from audio file

Plan A (Swift)

• AV Foundation

• AVAssetReader/Writer can do format conversion while reading/writing audio files

• Can’t (easily) read from arbitrary packet offsets; meant to process everything forward

Plan B (C, Swift?)

• Audio Toolbox (part of Core Audio)

• ExtAudioFile can do format conversions while reading/writing audio files

• AudioFile can read from arbitrary packet offset

// declare LPCM format we are converting to AudioStreamBasicDescription format = {0}; format.mSampleRate = 44100.0; format.mFormatID = kAudioFormatLinearPCM; format.mFormatFlags = kAudioFormatFlagIsPacked | kAudioFormatFlagIsSignedInteger; format.mBitsPerChannel = 16; format.mChannelsPerFrame = 2; format.mBytesPerFrame = 4; format.mFramesPerPacket = 1; format.mBytesPerPacket = 4;

// declare LPCM format we are converting to var format = AudioStreamBasicDescription( mSampleRate: 44100.0, mFormatID: kAudioFormatLinearPCM, mFormatFlags: kAudioFormatFlagIsPacked + kAudioFormatFlagIsSignedInteger, mBytesPerPacket: 4, mFramesPerPacket: 1, mBytesPerFrame: 4, mChannelsPerFrame: 2, mBitsPerChannel: 16, mReserved: 0)

// open AudioFile for output AudioFileID forwardAudioFile; err = AudioFileCreateWithURL(forwardURL, kAudioFileCAFType, &format, kAudioFileFlags_EraseFile, &forwardAudioFile); IF_ERR_RETURN

#define IF_ERR_RETURN if (err != noErr) { return err; }

// open AudioFile for output var forwardAudioFile: AudioFileID? err = AudioFileCreateWithURL(forwardURL, kAudioFileCAFType, &format, AudioFileFlags.eraseFile, &forwardAudioFile) if err != noErr { return err }

// open AudioFile for output var forwardAudioFile: AudioFileID? err = AudioFileCreateWithURL(forwardURL, kAudioFileCAFType, &format, AudioFileFlags.eraseFile, &forwardAudioFile) if err != noErr { return err }

// open AudioFile for output var forwardAudioFile: AudioFileID? err = AudioFileCreateWithURL(forwardURL, kAudioFileCAFType, &format, AudioFileFlags.eraseFile, &forwardAudioFile) if err != noErr { return err }

1. Uses a free function, rather than a method on AudioFile

// open AudioFile for output var forwardAudioFile: AudioFileID? err = AudioFileCreateWithURL(forwardURL, kAudioFileCAFType, &format, AudioFileFlags.eraseFile, &forwardAudioFile) if err != noErr { return err }

2. Errors are communicated via the return value, rather than throws

// open AudioFile for output var forwardAudioFile: AudioFileID? err = AudioFileCreateWithURL(forwardURL, kAudioFileCAFType, &format, AudioFileFlags.eraseFile, &forwardAudioFile) if err != noErr { return err }

3. Some parameters are UInt32 constants, some are enums

// open AudioFile for output var forwardAudioFile: AudioFileID? err = AudioFileCreateWithURL(forwardURL, kAudioFileCAFType, &format, AudioFileFlags.eraseFile, &forwardAudioFile) if err != noErr { return err }

4. Audio format is passed as an UnsafePointer<AudioStreamBasicDescription>

// open AudioFile for output var forwardAudioFile: AudioFileID? err = AudioFileCreateWithURL(forwardURL, kAudioFileCAFType, &format, AudioFileFlags.eraseFile, &forwardAudioFile) if err != noErr { return err }

5. Created object is returned via an in-out parameter

To say nothing of…

Pointer arithmetic!

// swap packets inside transfer buffer for i in 0..<packetsToTransfer/2 { let swapSrc = transferBuffer.advanced(by: Int(i) * Int(format.mBytesPerPacket)) let swapDst = transferBuffer.advanced(by: transferBufferSize - (Int(i+1) * Int(format.mBytesPerPacket))) memcpy(swapBuffer, swapSrc, Int(format.mBytesPerPacket)) memcpy(swapSrc, swapDst, Int(format.mBytesPerPacket)) memcpy(swapDst, swapBuffer, Int(format.mBytesPerPacket)) }

Couldn’t you just…

extension AudioFileID { init? (url: URL, fileType: UInt32, format: AudioStreamBasicDescription, flags: AudioFileFlags) { var fileId : AudioFileID? var format = format let err = AudioFileCreateWithURL(url as CFURL, fileType, &format, flags, &fileId) guard err != noErr, let createdFile = fileId else { return nil } self = createdFile } }

Been there, done that• The Amazing Audio Engine 💀

• Novocaine (💀?)

• EZAudio 💀

• AudioKit

• Superpowered

• etc…

/** Convert a source audio file (using any Core Audio-supported codec) and create LPCM .caf files for its forward and backward versions. - parameter sourceURL: A file URL containing the source audio to be read from - parameter forwardURL: A file URL with the destination to write the decompressed (LPCM) forward file - parameter backwardURL: A file URL with the destination to write the backward file */ OSStatus convertAndReverse(CFURLRef sourceURL, CFURLRef forwardURL, CFURLRef backwardURL);

AudioReversingC.h

// // Use this file to import your target's public headers that you would like to expose to Swift. //

#import <CoreFoundation/CoreFoundation.h> #import <AudioToolbox/AudioToolbox.h>

OSStatus convertAndReverse(CFURLRef sourceURL, CFURLRef forwardURL, CFURLRef backwardURL);

AudioReverser-Bridging-Header.h

if USE_SWIFT_CONVERTER { err = convertAndReverseSwift(sourceURL: source as CFURL, forwardURL: self.forwardURL as! CFURL, backwardURL: self.backwardURL as! CFURL) } else { err = convertAndReverse(source as! CFURL, self.forwardURL as! CFURL, self.backwardURL as! CFURL) }

A Minor Mea Culpa…

// open AVAudioFile for URL input let sourceAudioFile = try AVAudioFile(forReading: sourceURL as URL, commonFormat: .pcmFormatInt16, interleaved: true)

// open AudioFile for output var forwardAudioFile = try AVAudioFile(forWriting: forwardURL as URL, settings: [:], commonFormat: .pcmFormatInt16, interleaved: true)

// convert to a flat file let outputBufferSize: size_t = 0x8000 // 32 KB buffer let framesPerBuffer: UInt32 = UInt32(outputBufferSize) / 4

let transferBuffer = AVAudioPCMBuffer(pcmFormat: sourceAudioFile.processingFormat, frameCapacity: framesPerBuffer) var totalFrameCount: Int64 = 0

while(true) { do { try sourceAudioFile.read(into: transferBuffer) } catch { // FIXME: discern between empty read and genuine error. // this could happen if last read(from:) exactly filled the // buffer, and this one gets 0 frames. // empty read should break, real error should throw throw error } // this never happens, actually. earlier try fails instead guard transferBuffer.frameLength > 0 else { print ("done reading file") break } // increment frame count totalFrameCount += Int64(transferBuffer.frameLength) // write to forwardFile try forwardAudioFile.write(from: transferBuffer) if transferBuffer.frameLength < transferBuffer.frameCapacity { // didn't fill buffer; we're done break } }

C APIs on iOS/macOS• Core Foundation

• Core Audio

• Core Media

• Video Toolbox

• Keychain

• IOKit

• OpenGL

• SQLite

• Accelerate

• OpenCV

• BSD, Mach

• etc…

Going deeper…

Audio Units

• Discrete software objects for working with audio

• Generators, I/O, Filters/Effects, Mixers, Converters

• Typically combined in a “graph” model

• Used by Garage Band, Logic, etc.

Demo

R(t) = C(t) x M(t)

https://github.com/invalidstream/ring-modulator-v3audiounit

Ring Modulator

• Multiplication of two signals

• One is usually a long-period sine wave

• Originally implemented as a ring-shaped circuit

0 0.8 1.6 2.4 3.2 4 4.8 5.6 6.4 7.2

-2.4

-1.6

-0.8

0.8

1.6

2.4

0 0.8 1.6 2.4 3.2 4 4.8 5.6 6.4 7.2

-2.4

-1.6

-0.8

0.8

1.6

2.4

0 0.8 1.6 2.4 3.2 4 4.8 5.6 6.4 7.2

-2.4

-1.6

-0.8

0.8

1.6

2.4

0 0.8 1.6 2.4 3.2 4 4.8 5.6 6.4 7.2

-2.4

-1.6

-0.8

0.8

1.6

2.4

R(t) = C(t) x M(t)

Modulate! Modulate!

• Ring modulator best known as the “Dalek” voice effect on Doctor Who (circa 1963)

• Also used in early electronic music

Wait, what?

-(instancetype)initWithComponentDescription:(AudioComponentDescription)componentDescription options:(AudioComponentInstantiationOptions)options error:(NSError **)outError { self = [super initWithComponentDescription:componentDescription options:options error:outError]; if (self == nil) { return nil; } // ... return self; }

MyAudioUnit.m

XPC

(macOS only)

Swift 4Swift 5

https://github.com/apple/swift/blob/master/docs/ABIStabilityManifesto.md

“First, ABI stability is the center focus of Swift 5 — and we will pivot much of our prioritization of efforts for Swift 5 around it. With Swift 4, ABI stability was a strong goal. In Swift 5, it is a *requirement* of the release. Whatever ABI we have at the end of Swift 5 is the ABI that we will have. ABI stability is an important inflection point for the maturity of the language, and it cannot be delayed any longer.”

—Ted Kremenek, Aug 8, 2017 “Swift 5: start your engines”

https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20170807/038645.html

// Block which subclassers must provide to implement rendering.

- (AUInternalRenderBlock)internalRenderBlock { // Capture in locals to avoid Obj-C member lookups. // If "self" is captured in render, we're doing it wrong. See sample code. return ^AUAudioUnitStatus(AudioUnitRenderActionFlags *actionFlags, const AudioTimeStamp *timestamp, AVAudioFrameCount frameCount, NSInteger outputBusNumber, AudioBufferList *outputData, const AURenderEvent *realtimeEventListHead, AURenderPullInputBlock pullInputBlock) {

// Do event handling and signal processing here. return noErr; }; }

Don’t do this

• An audio unit’s render block is called on a realtime thread

• Therefore it cannot perform any action that could block:

• I/O (file or network)

• Waiting on a mutex or semaphore

Also, don’t do this

• Call objc_msg_send()

• Capture any Objective-C or Swift object

• Allocate memory

Basically, if you touch anything in the block other than a pre-allocated C struct or numeric type (or a pointer to those types),

you are asking for trouble.

// Block which subclassers must provide to implement rendering. - (AUInternalRenderBlock)internalRenderBlock { // Capture in locals to avoid Obj-C member lookups. // If "self" is captured in render, we're doing it wrong. See sample code. AUValue *frequencyCapture = &frequency; AudioStreamBasicDescription *asbdCapture = &asbd; __block UInt64 *totalFramesCapture = &totalFrames; AudioBufferList *renderABLCapture = &renderABL; return ^AUAudioUnitStatus(AudioUnitRenderActionFlags *actionFlags, const AudioTimeStamp *timestamp, AVAudioFrameCount frameCount, NSInteger outputBusNumber, AudioBufferList *outputData, const AURenderEvent *realtimeEventListHead, AURenderPullInputBlock pullInputBlock) {

// Do event handling and signal processing here.

// BLOCK IMPLEMENTATION ON NEXT SLIDE

return noErr; };

❌

// pull in samples to filter pullInputBlock(actionFlags, timestamp, frameCount, 0, renderABLCapture);

// copy samples from ABL, apply filter, write to outputData size_t sampleSize = sizeof(Float32); for (int frame = 0; frame < frameCount; frame++) { *totalFramesCapture += 1; for (int renderBuf = 0; renderBuf < renderABLCapture->mNumberBuffers; renderBuf++) { Float32 *sample = renderABLCapture->mBuffers[renderBuf].mData + (frame * asbdCapture->mBytesPerFrame); // apply modulation Float32 time = totalFrames / asbdCapture->mSampleRate; *sample = *sample * sinf(M_PI * 2 * time * *frequencyCapture); memcpy(outputData->mBuffers[renderBuf].mData + (frame * asbdCapture->mBytesPerFrame), sample, sampleSize); } }

return noErr;

Obj-C Instance Variables!

❌AUValue *frequencyCapture = &frequency; AudioStreamBasicDescription *asbdCapture = &asbd; __block UInt64 *totalFramesCapture = &totalFrames; AudioBufferList *renderABLCapture = &renderABL;

https://github.com/apple/swift/blob/master/docs/OwnershipManifesto.md

Certain kinds of low-level programming require stricter performance guarantees. Often these guarantees are less about absolute performance than predictable performance. For example, keeping up with an audio stream is not a taxing job for a modern processor, even with significant per-sample overheads, but any sort of unexpected hiccup is immediately noticeable by users.

—“Swift Ownership Manifesto”,February 2017

We believe that these problems can be addressed with an opt-in set of features that we collectively call ownership. […]

Swift already has an ownership system, but it's “under the covers”: it's an implementation detail that programmers have little ability to influence. What we are proposing here is easy to summarize:

• We should add a core rule to the ownership system, called the Law of Exclusivity […]

• We should add features to give programmers more control over the ownership system […]

• We should add features to allow programmers to express types with unique ownership […]

And yet…

“[Swift] is the first industrial-quality systems programming language that is as expressive and enjoyable as a scripting language.”

https://developer.apple.com/library/content/documentation/Swift/Conceptual/Swift_Programming_Language/

Well, now that you mention it…

“systems prog

ramming language”

removed

So… when?

???Big Bang Heat Death of the Universe

Waiting…

• ABI stability — Will be in Swift 5 (2018)

• Ownership — unclear

• Are these traits sufficient?

Strategies

• Use AV Foundation if you can

• It does most of what anyone needs at this point

• Learn to balance C and Swift

• “Render undo C-sar what is C-sar’s…”

• The goal is to have idiomatic Swift, not Swift that may work but looks like C

Media Frameworks versus Swift

Chris Adamson • @invalidname Swift by Northwest, October 2017

Slides available at slideshare.net/invalidname Code available at github.com/invalidstream