Reverse Playback of Audio Streams in iOS

Abel DominguesAudio Tools / iOS

!sretsieM oiduA SOi, sgniteerG

In our last post, we looked at how to access iPod Library tracks and stream them from disk in real time, using Apple’s Extended Audio File Services and Audio Unit APIs. We also speculated about the possibility of modifying our code to allow for reverse playback of those streams – you know, so you can hear what stuff sounds like when you play it backwards.

These days, of course, reversing an audio file – or some section of it – is pretty much a snap: just open the file in an audio editor (Audacity is very capable – and free), apply whatever reverse effect is available to flip the samples around, and save the result as a new file. Online services that perform this service for you are quite common: you upload a file, wait a little bit, and back it comes, nice and backwards.

But that’s not playing a file in reverse; that’s playing a previously reversed file in the usual (forward) manner and its not what we want, No. What we want is to stream audio files from disk, in real time, and reverse direction as playback is occurring. In other words: we want the ability to read through the file, from disk, in reverse.

Now, technically that’s not exactly practical: most audio file-reading APIs (Extended Audio File Services, included) don’t read samples from a file one-by-one – that would be far too inefficient. Instead, they read them in blocks of (for example) 1024 samples, then they process the whole block in one go before shipping it out to the hardware. And the samples in those blocks are usually acquired using the same strategy: you open the file, set the ‘playhead’ to a specific location and grab the next block of samples from that point forward.

So: you can’t just ‘go the other way.’

This makes reading an audio file in reverse something of a challenge, but it also suggests a solution: if we could read our forwardly-arranged sample blocks in reverse order, and at the same time reverse the order of sample frames within each block, we might be able to achieve on-the-fly reverse playback. Let’s have a look at how we might do that.

The Demo Project

If you’d like to follow along, download the starter demo project and open it up. (Alternatively, a completed version of the demo can be gotten here, on Github.)

This is essentially the same player we set up last time around, but with a few modifications:

  • Swift: This time we’re using Swift as our project language (but note we’ve retained the use of Objective-C for our reading and rendering code, for reasons we touched on here.)
  • FileReader Class:
    • we’ve moved our file reading methods (openFileAtURL: and readFrames:audioBufferList:bufferSize:) into their own FileReader class, where they belong;
    • we’ve refactored those methods by breaking them into smaller, more tightly-focused sub-methods; and
    • we’ve given our view controller the necessary property and method for creating a fresh reader object each time a new file is loaded from the iPod Library;
  • Bridging Header: We’ve added a bridging header our project, in order to make our Objective-C classes available to Swift.
  • Scrubbing: we’ve added
    • a UISlider for scrubbing through the audio file.
    • a seekToFrame: method to FileReader (so it can perform the scrubbing);
    • an IBAction to our view controller (so our slider can talk to FileReader); and
    • a FileReaderDelegate protocol (in FileReader) to which our view controller conforms, so it can be notified of changes to the current playback location and update the slider control accordingly.)
  • Continuous Looping of the File: We’ve set things up so when playback reaches the end of the file, it jumps back to the beginning and starts all over again.
  • UI: Finally, we’ve gussied up the UI (well, just a smidgen) with:
    • the aforementioned UISlider for scrubbing through the audio file;
    • a UISegmentedControl for toggling playback direction;
    • a pair of labels for displaying title/artist information for the currently loaded track;
    • a handful of descriptive labels for each of the above.
    • a little splash of color for some added stimulation!

Like so:

Starter Demo UI

Run the project (on a device, not the simulator) and you’ll see it behaves much as before: we can browse the device’s iPod Library, choose a file, play it and pause it. In addition, we can now scrub to random locations within the file, and when playback reaches the end of the file, it loops back to the beginning.

As we pointed out last time, if you check the app’s memory footprint in the Debug navigator you’ll find that it is in fact quite small, and for good reason: we’re streaming our audio file directly from disk rather than loading the entire thing into memory.

The Reverse segmented control doesn’t do anything, of course, since we haven’t implemented reverse playback yet – that’s what we’re here to talk about!

Reversing The Audio Stream

Like I said, we’re going to tackle this in two steps: first, by attempting to read our forwardly-arranged sample blocks in reverse order, and second, by attempting to reverse the order of samples within each block.

The first thing we need is a way to indicate what the current playback direction should be. Add a new property to FileReader.h:


@property (assign, nonatomic) BOOL reversePlayback;

Then, in FileReader.m, add the following to initWithFileURL:, below the existing _isReading = NO; line:


_reversePlayback = NO;

Next we need an @IBAction for setting the reversePlayback flag when the UISegmentedControl is toggled, so let's add an action method to ViewController:


  @IBAction func reversePlayback(sender: UISegmentedControl)
  {
    if reader != nil {
      if sender.selectedSegmentIndex == 0 {
        reader.reversePlayback = true
      } else {
        reader.reversePlayback = false
      }
    }
  }

Don't forget to wire this action up to the UISegmentedControl's "Value Changed" event in the Storyboard!

Finally, head down to the readFrames: method and replace the line _frameIndex += frames; with the following:


    // Reverse Playback
    if (self.reversePlayback) {
      NSLog(@"Reversing playback");
      for (int buff = 0; buff < audioBufferList->mNumberBuffers; buff++) {
        Float32* revBuff = audioBufferList->mBuffers[buff].mData;
        [self reverseContentsOfBuffer:revBuff numberOfFrames:frames];
      }
      _frameIndex -= frames; // decrement frame index
    } else {
      _frameIndex += frames; // increment frame index
    }

This is actually fairly straightforward:

  • First we check if our reversePlayback flag is set. If its not - i.e. we're currently performing normal playback - the only thing we do is what we were doing before: we allow our buffers to continue unmolested on their journey back to the render callback, and we increment _frameIndex for the next read cycle.
  • If reversePlayback is set, we loop through our incoming buffers (we're using stereo interleaved here, so there's only going to be one), set up a new pointer variable and pass that to something called reverseContentsOfBuffer. We'll write that method in a moment.

If, at this point, you were to comment out everything in that code block except for the decrement of _frameIndex, you'd get a nice demonstration of why merely reversing the index is only a partial solution: the sequence of reads does indeed move backwards through the file, but the sample frames in each read block remain arranged in forward sequential order. (If you decide to try this, be sure to first scrub to some location other than the very beginning of the file, before reversing playback.)

Let's take care of that now by adding the following method to FileReader.m:


- (Float32*)reverseContentsOfBuffer:(Float32*)audioBuffer numberOfFrames:(UInt32)frames
{
  Float32* reversedBuffer = audioBuffer;
  Float32 tmp;
  int i = 0;
  int j = frames - 1;
 
  while (j > i) {
    tmp = reversedBuffer[j];
    reversedBuffer[j] = reversedBuffer[i];
    reversedBuffer[i] = tmp;
    j--;
    i++;
  }
  
  return reversedBuffer;
}

So much power in so few lines of code:

  • First, we're passing in a pointer to our buffer - the one we declared a few lines back - along with the size of that buffer. Remember, this pointer is really pointing to our render callback's outgoing buffer(s), which ExtAudioFileRead has just filled with fresh audio samples from the file on disk.
  • Next, we set up a pair of int counter variables, one pointing to the first element in the buffer and another pointing to the last element.
  • We also set up a third variable, tmp, to help us play Three Card Monte with the buffer's samples.
  • Finally, we set up a while loop to perform some magic: on each iteration of the loop, the counters i and j increment in opposite directions - i going up and j going down. When they meet in the middle - j > i - all the samples in the buffer will have been shuffled into reverse order - which is how the render callback will now ship them out to the hardware.

The last thing we need to do is modify checkCurrentFrameAgainstLoopMarkers - which currently handles looping around when playback reaches the end of the file - to perform a similar check when playback is reversed and we reach the beginning of the file:


- (void)checkCurrentFrameAgainstLoopMarkers:(SInt64)currentFrame inFrames:(UInt32)frames
{
  if (self.reversePlayback) {
    // If we're approaching startOfFile and have fewer than a buffer's worth of samples to go, reduce the size of the read
    if (currentFrame - (SInt64)frames < self.lMarker) {
      frames = (UInt32)currentFrame;
    }
    // If we're *exactly* at startOfFile, seek to (endOfFile - frames)
    if (currentFrame - (SInt64)frames == self.lMarker) {
      [self seekToFrame:self.rMarker - frames];
    }
  } else { // forward
    // If we've reached endOfFile, seek to startOfFile
    if (currentFrame > self.rMarker) {
      [self seekToFrame:self.lMarker];
    }
  }
}

Try running the app, but with the following caveat: depending on the file you've loaded for playback - whether its compressed or not and, if so, how it was encoded and at what bit rate - you may (or may not) hear disappointing results when you reverse playback.

This means we have one more challenge to overcome.

Dealing With Compressed Formats

Reversing playback on the fly like this works great for PCM formats where each sample frame consists of a single set of samples, spread across all channels at that point in time. It even works well for robust compressed formats like AAC and well-encoded, high bit rate MP3s.

But compressed formats present an inherent challenge: unlike uncompressed formats, the frames of a compressed file consist of blocks of encoded samples plus a header (at the beginning of each block) indicating how the samples in that particular block were encoded - information a decoder needs in order to decode!

So what happens ordinarily, when you're playing a compressed file and the next read lands in the middle of a block, without the header? The MP3 codec has a mechanism for retaining the header and samples of incomplete blocks, so they can be reunited with the rest of the block on the next read. Clever!

But you can see where this is going: read those sample blocks in reverse order and you've clearly subverted the entire mechanism! The result: buffers coming back from ExtAudioFileRead with entire swaths of their samples set to zero. Nasty!

A Solution

What to do? One strategy is to seek to a location in the file prior to the data you're actually interested in, read in a greater number of samples (in the hopes of capturing the complete block of target samples including the header) and, after conversion, simply drop the extra frames.

This actually works surprisingly well, primarily because ExtAudioFileRead is optimized to be fast. Still, given the nature of compressed formats, it involves something of a gamble, which is: how can you be sure you've gone back far enough in the file to get the header data you need for the frames you actually want to read?

You can't. But by reading a fairly conservative number of extra frames, we can get pretty far with this idea. I'm only going to focus on detecting MP3 files because, honestly, I haven't had much trouble reversing other compressed formats.

Begin by adding a define macro to the top of FileReader.m


#define kExpandedBufferSize 4096

Typically, reading is done in blocks of 512 or 1024 sample frames so 4096 makes for a nice, comfortable multiple. If it seems excessive to be grabbing 4-8x as many samples as we would normally need, keep in mind that not only is ExtAudioFileRead very fast, but the way we're going to 'crop' our buffer is by simply reassigning its pointer.

Next, there should be a flag we can set when we're dealing with an .MP3 file. Add another property to FileReader.m:


@property (assign, nonatomic) BOOL isMP3;

And add this line to initWithFileURL, so we always begin life with isMP3 turned OFF:


_isMP3 = NO;

For this to be useful, we need some way of ascertaining if the current file is an MP3 or not. Add the following method, which decodes the file's formatID flag (and then tests whether that flag indicates an MP3) to the bottom of FileReader.m:


- (BOOL)fileIsMP3:(UInt32)formatID
{
  char formatIDString[5];
  UInt32 ID = CFSwapInt32HostToBig (formatID);
  bcopy (&ID, formatIDString, 4);
  formatIDString[4] = '\0';
//  NSLog (@"Format ID: %10s", formatIDString);
  NSString* fileExtension = [NSString stringWithCString:formatIDString encoding:NSUTF8StringEncoding];

  if ([fileExtension isEqual: @".mp3"]) {
    return YES;
  }

  return NO;
}

Now, in openFileAtURL we can check (after we've retrieved the file's native format, of course) if we've got an MP3 and, if we have, set the flag accordingly::


self.isMP3 = [self fileIsMP3:_fileFormat.mFormatID] ? YES : NO;

With our flag and helper method in place, we can now add the necessary conditional code for reversing an MP3 file stream. Start by modifying createFileReadingBufferList as follows:


- (void)createFileReadingBufferList
{
  // AudioBufferList defines a single buffer, so we need to allocate additional buffers ourselves
  fileReadingBufferList = (AudioBufferList*)malloc(sizeof(AudioBufferList) + (sizeof(AudioBuffer) * (_clientFormat.mChannelsPerFrame)));
  fileReadingBufferList->mNumberBuffers = _clientFormat.mChannelsPerFrame;
  
  for ( int i=0; i < fileReadingBufferList->mNumberBuffers; i++ ) {
    fileReadingBufferList->mBuffers[i].mNumberChannels = 1;
    UInt32 bufferSize;
    if (self.isMP3 == YES) {
      bufferSize = kExpandedBufferSize; // for reversed MP3 files
    } else {
      bufferSize = 1024;
    }
    fileReadingBufferList->mBuffers[i].mDataByteSize = bufferSize * sizeof(float);
    fileReadingBufferList->mBuffers[i].mData = malloc(bufferSize * sizeof(float));
  }
}

This sets a larger buffer size (using our define macro) for reading in the extra frames. Next, add the following to readFrames:, above the code that currently handles reversing:


    UInt32 framesToRead;
    if (self.isMP3 && self.playbackIsReversed) {
      // Larger reads for reversed MP3s
      framesToRead = kExpandedBufferSize;
      CheckError(ExtAudioFileSeek(_audioFile, _frameIndex - (framesToRead - frames)), "MP3 ExtAudioFileSeek FAILED");
      CheckError(ExtAudioFileRead(_audioFile, &framesToRead, fileReadingBufferList),
                 "Failed to read audio data from audio file");
      for (int buff = 0; buff < fileReadingBufferList->mNumberBuffers; buff++) {
        Float32* croppedBuffer = fileReadingBufferList->mBuffers[buff].mData;
        croppedBuffer = &croppedBuffer[kExpandedBufferSize - frames];
        audioBufferList->mBuffers[buff].mData = croppedBuffer;
      }
    } else {
      // Normal reads otherwise
      framesToRead = frames;
      CheckError(ExtAudioFileSeek(_audioFile, _frameIndex), "Non-MP3 ExtAudioFileSeek FAILED");
      CheckError(ExtAudioFileRead(_audioFile, &framesToRead, audioBufferList),
                 "Failed to read audio data from audio file");
    }

All we're doing here is checking if we're currently reversing playback and, if we are, is the file we're reversing is an MP3? If we are (and it is), we (a) set our read size (framesToRead) to the kExpandedBufferSize macro we set up earlier; (b) have ExtAudioFileRead read framesToRead number of sample frames to a new float buffer; and (c) 'crop' the returned (and filled) buffer by resetting its pointer (which normally points to the first element in the buffer) to point instead to kExpandedBufferSize minus frames - which gives us the sample frames we were looking for in the first place. Thanks to the expanded read size, however, ExtAudioFileRead will have been able to completely fill those frames with properly decoded samples!

With that, we should give it a test run... check it out...

And there you have it: On-the-fly reversible playback, streamed right out of your iPod Library!

Post to Twitter