The present and future of post production business and technology

Introducing AV Foundation and the future of QuickTime [Updated]

Introduction to AV Foundation To be honest I don’t know why this isn’t hidden behind an NDA, but it’s not and until someone has it taken down, and asks me to do the same, I’ll consider it public knowledge.

Now, AV Foundation is the iOS media system, so we’re not talking about QuickTime per se but I have to wonder.

QuickTime – the real OS-centric media framework, not the little sub applications that function as players – is transitioning from C APIs (Carbon) to Cocoa via QTKit. Trouble is, QTKit got a lot of work around QuickTime 7’s release, but not so much in recent years. And yet Final Cut Pro needs a lot of what’s not written, before it can release a Cocoa version of Final Cut Pro.

Actually, Apple could do what Adobe have done for Premiere Pro CS5. In rewriting their core media handling engine, Adobe retained QuickTime support by spinning it off into a 32 bit thread, but that’s a complex workaround that does nothing for performance, nothing positive anyway.

When you consider slide 9… Even though it was only introduced in iOS 2.2, extended in iOS 3 and “completed” in iOS 4 (consider the reference framework growth in slides 6, 7 and 8), AV Foundation has 56 Classes and 460 Methods (the more you have of these, the more you can do with it). QTKit has 24 Classes (less than half) and 360 Methods. Compare that with the (very mature) QuickTime for Java with 576 Classes and more than 10,000 Methods. Something tells me that QTKit is not in favor at Apple.

Not that I think QuickTime is going away, at least not as a brand for their media players and the overall technology. I say that because, although the code that’s in iPhone OS shows a simplified player, that was all that was originally released and it shared no “QT Classes or Frameworks”. So, the QuickTime brand is likely to be retained.

If I was extrapolating from this presentation, and I am extrapolating wildly from a small amount of data, I’d guess that the direction within Apple was toward the more modern Classes and Methods of AV Foundation, and that, eventually, AV Foundation, Core Audio, Core Animation and Core Media will replace what we currently have under QuickTime on OS X: Core Audio, Core Video (well, just a subclass of Core Image) and a lot of deprecated (do not use) C APIs.

If you consider slide 14, and the similarity of Classes between QTKit and AV Foundation it makes no sense to build two technologies in the company that were essentially doing the same thing.  Slide 29 shows how similar an AVAsset is to a QTMovie. The other Classes all seem to duplicate functionality that’s in QuickTime now, but in efficient, new, modern code. Capture, editing, playback, media formats… they all seem to be in AV Foundation duplicating work done (or not yet done) in QuickTime’s QTKit.

Importantly Core Media Time is in “n’ths of a second” not “ticks” or “events”. Media based on time will be better for video frame rate uses than one based on ticks or events, which caused the “Long Frames” problems of earlier versions of Final Cut Pro.

In support of my hypothesis I offer slide 42: specific references to AVAssetExportSession.h being available in OS X with 10.7 and likewise CMTime.h has a reference to becoming available in 10.7.

So, I’ll go on a limb and suggest that QuickTime as we’ve known it is somewhat dead; long live a new QuickTime. QuickTime will continue being the branding, but everything “below that” will transition to new architectures essentially ported from iOS to OS X.

This would be a very good thing. A completely new, modern, efficient (you see what it does on the iPhone) underpinning for QuickTime down below that QTKit layer.

Who wouldn’t want to use that in an modern NLE, even if it means waiting for OS X 10.7, which hasn’t been announced yet? It would make it much easier for the Final Cut Pro team to create a much more powerful media engine than it has now; one that really understands time and not events and one that mimics the power of Adobe’s Mercury Engine. Let’s face it, media performance on a 1 GHz A4 chip is in some ways better than the performance on 8 core processors. iMovie for iOS, built on these frameworks (if slide 24 is to be believed) can edit Long GOP H.264, which Final Cut Pro can’t! (And in both cases the H.264 playback is accelerated by hardware: dedicated chips in the iPhone, on the graphics card in OS X.)

As always, conjecture on my part, and this time based solely on what I’ve learnt from the quoted slide show. Chris Adamson does not work for Apple but he does claim expertise in iOS and QuickTime. Other posts on his blog indicate some differences between AV Foundation and QuickTime; and Classes still missing from AV Foundation that are in the current version of QuickTime. That shakes my confidence in the hypothesis a little, but given how little work has been done on QTKit in the last two years, and the need to have the foundations for QuickTime modernized, it still seems like the most likely path Apple will take.

Another data point is that the QuickTime X player was promoted thusly:

Using media technology pioneered in OS X iPhone™, Snow Leopard introduces QuickTime X, which optimizes support for modern audio and video formats resulting in extremely efficient media playback. Snow Leopard also includes Safari® with the fastest implementation of JavaScript ever, increasing performance by 53 percent, making Web 2.0 applications feel more responsive.*

Pioneering the technology under iOS, and then porting it to Mac OS X has happened already.

UPDATE: Chris Adamson, who did the presentation I referred to, clarified many of the points I get wrong or wrongish, including the fact that AV Foundation is not under NDA. His Connecting the Dots post is essential reading if you’ve got this far!







6 responses to “Introducing AV Foundation and the future of QuickTime [Updated]”

  1. Terence Curren

    So, are we going out on a limb even further and guessing that Jobs is going announce an actual Apple television this week? You know, a giant iPad for your living room.

  2. Philip Hodgetts

    I’d be surprised if the iTV or whatever it will be called, will show up at a music focused event. More likely a special event for that, but who knows, until Wednesday.


  3. Tony Williams


    A cogent argument.

    In further support I’d remind you and your readers that the differences between iOS and Mac OS are almost entirely at the user interface level. The underlying operating system is built on the same core and getting a framework available for the iPhone working under Mac OS would be absurdly easy.

    // Tony

  4. This is an interesting argument but really, is it very bad news? If you are right Phil, we are looking at a VERY VERY long time before FCP the next generation will become available. Wait for 10.7, not even announced, Build a new FCP re-write on top. Wait 6 months after release before any production grade user would even consider using a product with so much new code in its core. (Hundreds of Bug upgrades will come)
    Will Adobe just run away with it for a while?? (I must admit, they are only tools to me and I am looking at moving for my projects)

    On iTV, I am skeptical about it as there is a HUGE problem with iOS4. Its a TOUCH SCREEN OS.
    A TV is not a touch screen, it is a remote control. A platform they have already tried at with the silly 5 button control.. A proxy control to a iPhone or iPad is not the answer. (so you need a 99 iTV and a 300 iTouch remore to use it. Really??) I think this product will need considerable interface research before Steve lets it out.


  5. Philip Hodgetts

    I’ve been saying 2012 before the next version of FCP for quite a while now. I expect a re-imagined and rebuilt product, not a “faster horse” that Adobe and Avid have undoubtedly delivered. (Look for an upcoming post titled “Should Apple build a faster horse?).

    Apple will lose some market share in the short term, but the inertia that prevents people changing will work in their favor. Most people dislike learning new interfaces, workflows, software and “quirks”. I agree, if Apple isn’t offering a tool you need, go somewhere else. PPro CS5 and MC5 are excellent applications.

    As for iTV, see Of course it’s a touch screen, but the touch screen won’t be on the device, it’ll be on you iPhone, iTouch or iPad. Keep everyone locked in their ecosystem. I think it is the answer btw, so I disagree with you. Particularly as I use this configuration now with the old Apple TV.


  6. Sorry Philip, I disagree.
    The way you watch TV is nothing like the average consumer.
    A solution needs to integrate Terrestrial, it will never go away, will always be a need, optional DVR, and a $20 remote. not a minimum iPodTouch at $200.

    iPod Touch, plus iTV = $300, XBOX 360 is cheaper and FAR better.(1080p, Real Games) And has all this “Amazing new features” NOW that the iTV will bring for Apple.

    Plus the Xbox uses a $20 remote control you can loose, or destroy, cost nothing to replace, is easy to use even my parents can use it..

    I get a real feeling that the Technorati are really underestimating the Microsoft second and usually decisive attach on this market.

    Many of the big media events of late are calling Microsoft to become the dominant phone and media platform in the next few years. Exactly the opposite of what all the crap coming out of Blogs etc based in Western US.

    They have good reasoning behind this. Basically goes along the line, Microsoft are not competing in the same space like Apple and Google, so all the leading media companies are likely to push it.

    Papers, TV stations, PayTV. Microsoft is the enabler without the conflict of interest.

    But back to the topic. Yes the iTV/iTouch would be nice, but so would a Ferrari.

    Apples CLOSED nature will ensure its back seat position, just like it did in the Mac vs PC days.

    But then again, thats what some people want… For now. You may be playing in the best garden currently available, but its a walled garden that you can never leave.

    No walls may mean more risk, but also more freedom.

    a Choice.