The present and future of post production business and technology

What is Apple doing with QuickTime?

As expected, AV Foundation from iOS 4 will be added to Lion. My take is that signals the end of QuickTime as we’ve known it. But it’s not only that there’s a new Framework for working with time-based audiovisual media – there’s a lot more to QuickTime than that, and it’s all the interactive and additional technologies in QuickTime that don’t appear to have a future. Features that were important when QuickTime MOVs were the preferred (at Apple) distribution format.

It’s been a very long time since QuickTime was seen in Apple’s view as a distribution format. Apple threw its weight behind MP4 (largely based on the MOV container, but NOT the same) some time before the release of QuickTime 4 in 2002. Since then, distribution has been via MP4: originally using the MP4 Simple Profile, and at QuickTime 7.0.2 with the H.264 codec.

The focus on QuickTime spun around to the use in the OS X ecosystem. There’s unlikely to ever be a 64 bit “QuickTime for Windows”: MP4 is for distribution.

Most people think of the QuickTime Player as “QuickTime” but if you look carefully it’s a very small application for the power it has. That’s because “QuickTime” is really a series of system-wide event-based media handling routines. Your Mac won’t even boot without QuickTime!

The other thing that most people don’t know is that QuckTime handles way more than just “video” playback. It was planned as a very powerful multi-media tool. In many ways it took Flash until version 8 to reach feature parity with QuickTime 3. Most of the features I’m talking about – beyond playback of diverse codecs and formats – are for interactive media:

  • QuickTime Vector Graphics (yes like Flash vectors)
  • Wired Sprites (clickable interactivity – used extensively in our “Intelligent Assistants” of early this century)
  • Sprite tracks (that can be animated around the screen)
  • Flash Media (since removed for security reasons when Adobe didn’t “fix” security on older versions)
  • Transitions (still available in Final Cut Pro 7)
  • Filters (still available in Final Cut Pro 7) including filters like Clouds and Fire that required no media
  • Text tracks (although most developers used Flash for text because QT Text didn’t cross platforms well)
  • HREF tracks
  • Media free color source
  • SMIL – the basis of our 2001 technology demo that led eight years later to First Cuts.
  • Streaming Track (RTSP)
  • VR – both objects and panoramas
  • Movie within Movie loading, which we used extensively in the Intelligent Assistant for Adobe After Effects – when you drilled down for more detail, that movie loaded within the parent movie.

It was all this functionality that attracted me to QuickTime as a technology, and as a company we certainly took advantage of these features in our Intelligent Assistants. For part of 1998-99 my email sig line included “Accidental QuickTime Evangelist”.

All in all, “QuickTime” – the original 32 bit C API (Application Programming Interface) – had more than 575 Classes and more than 10,000 Methods. (Although you don’t need to know what that means, for simplicity think of Class as a related collection of Methods, while Methods are the actual functions you would use when writing code.) But sadly, 32 bit APIs have no future in Apple’s world, and Apple changed their mind on developing 64 bit Carbon APIs at in 2007.

During the development of Tiger, Apple started work on a Cocoa version of QuickTime with a new Framework (a collection of Classes) to replace the old C API. There have been few additional Classes and Methods added to the QTKit Framework, which allowed QuickTime 7 to talk directly to Core Audio and Core Video. (These are the very low level Frameworks to handle video and audio media.) The primary benefit was to allow graphic card acceleration through Core Video.

Even with the additional Methods added to QTKit in Leopard and Snow Leopard (with apparently more to be added in Lion) the QTKit Framework has 24 Classes with around 360 methods, mostly covering relatively simple media playback. Anything not covered uses the older C APIs for backward compatibility.

(Many thanks to Chris Adamson’s excellent Mastering Media with AV Foundation for the comparison of Classes and Frameworks. I’m a regular reader of his blog, and follow him as @InvalidName on Twitter.)

I’ve covered before why Final Cut Pro couldn’t continue on the C APIs as it needed to go to 64 bit, and that the QTKit Framework didn’t yet have all the functions that a 64 bit Final Cut Pro would need, particularly since so little work had been done.

Then I discovered that Apple were building a new framework on the iOS for handling media called AV Foundation, which seemed to handle the requirements of a professional NLE in 64 bit quite well. This seems to be the basis of the announced-but-not-seen Final Cut Pro 8 (or Final Cut Pro 2011), and that’s good. But even AV Foundation, as of iOS 4.2, has 56 Classes and 460 Methods. That’s more than double those in QTKit but a long way short of the QuickTime C API’s 10000 plus Methods!

Most importantly for our purposes in professional video, AV Foundation is time based. QuickTime is event based. It’s an important distinction for when you media playback timing has to be within video tolerances. (On a computer 29.97 or 29.99 doesn’t much matter!)

I was resigned to the fact that all the interactivity, filtering, transitions, etc, etc, from QuickTime were gone. At least those parts not appearing in AV Foundation. Then I discovered Quartz Composer!

I had been vaguely aware of Quartz Composer in the past – after all Noise Industry’s FX Factory and CHV’s QC Integration FX are for the sole purpose of wrapping a Quartz Composer Composition inside an FXPlug plug-in for Final Cut Pro or Motion.  But beyond that, I didn’t know much.

Then, as part of our regular evening entertainment of watching Apple Developer videos (yes, we are that geeky) there were some introductions to Quartz Composer! That’s when I realized that yes, those QuickTime technologies were dead, but that there was a much better replacement in Quartz Composer. Even better, Apple supplies an authoring tool.

There was a time when choosing the preferred interactive authoring environment was a sufficiently close call that GoLive had an Interactive QuickTime authoring tool when Adobe purchased it. I believe that one of the reasons Flash got traction over Interactive QuickTime was because Macromedia had a business model in selling the tools. For Interactive QuickTime, it was left to Totally Hip with their LiveStage Pro, Electrifier Pro and GoLive. (All of which we used.) There wasn’t a business model for QuickTime within Apple, and no authoring tool officially sanctioned by Apple.

So Interactive QuickTime clearly faded in Apple’s attention during the transition to OS X, getting enough development to maintain compatibility but no new features. On Apple’s website, Interactivity is via Javascript in the browser, not via QuickTime Sprites and Wired Sprites.

But with Quartz Composer, Apple supplied a development tool. You do need to download the free Developer Tools. Quartz Composer is amazingly powerful. It’s node based and reasonably accessible to non-programmers. (You don’t have to write any code of any kind, except for custom expressions.) It can take data input and manipulate visuals. It can mix all types of media, including text (and generate text dynamically based on data inputs). It can be used in Applications or stand-alone compositions or saved as a movie from QuickTime Player 7.

There are standard compositions that are shared system-wide for use in any application. Pixelmator’s filters are Quartz Composer Filters in the standard System location.

In iMovie ’11, all the filters, transitions and templates are all Quartz Compositions! Each Trailer Template is a single Quartz Composition. They are very cool.

So, two conclusions. Clearly the features not already added to QTKit for Lion will never be part of QuickTime moving forward. In its place is a much more flexible and powerful technology called Quartz Composer, integrated at the System level.

My other conclusion. I would be very, very surprised if – in addition to FXPlug – Final Cut Pro 8 (or 2011) did not support native Quartz Compositions as filters and transitions.

So, goodbye the QuickTime I fell in love with in 1997, and hello to a new world populated by QTKit, AV Foundation, Javascript and Quartz Composer.








30 responses to “What is Apple doing with QuickTime?”

  1. My head hurts. But sounds like good news!

    1. It probably needs to carry a “nerdy” warning. 🙂

  2. Great piece, Quartz Composer rocks!

  3. This is a fantastic article, Phillip. Quartz Composer really is the bomb, and this article gives it some context I had been lacking. Thank you!

  4. Philip. This is a heady read. What do you see as the practical implication for editors? Is this going to change how the Effects architecture works, or is this more granular than that?

    I take it this change doesn’t affect any pre-Lion iteration of FCS coming our way, or can we not take ANYTHING for granted at this point. Based on Larry Jordan’s comments, I think all bets are off on features until we get more info.

    Though everyone is looking forward to a more open-format FCP, it’s clear that professional finishing codecs are still going to need to be part of the equation. I think the division between people pinning for DSLR h264 native editing and those who don’t mind transcoding, are those who aren’t pushing the image around much. If I was a wedding video editor, I’m sure I wouldn’t want to be transcoding 20 hours of 7D footage for every job- that makes sense. But if you’re doing any kind of intensive colour correction, or effects work, you’d be nutty not to push the material to ProRes.

    1. I don’t think “pre-Lion” has context this point. If (and I think it is) FCPx is based on AV Foundation then they’re bundling AV Foundation in the app package for this release. Quartz Composer is already well established in the OS now, so could easily be included as a FX source in FCPx. It is likely that it will change how the effects architecture works.

      And I agree, just because you can edit Long GOP natively, it would probably be unwise even then.

  5. Greg

    Practical implication for editors? If Quartz Composer filters/transitions are natively supported in FCP it will open up the market. And would even allow editors to “roll their own” filters! To see how easy Quartz Composer is to use, check out this free tutorial from DV Creators: Note that the tutorial is making a piece of media that they’re using in FCP, but with support for Quartz Composer that would then become a filter that could be applied in FCP. Noise Industries have a way of wrapping a Quartz Composer composition in an FXPlug which allows them to work with FCP right now.

  6. ToolsOnAir uses Quartz extensively in their products, and they built a very slick artists’ GUI called Composition Builder that spits out Quartz Compositions that are used as live graphics in their playout systems. Very slick.

    1. I didn’t know that. Thanks.

  7. Graeme Herwig

    A great detailed read – thanks Philip

  8. Thanks for the insight, Philip. AV Foundation along with Quartz Composer sounds like a worthy replacement for our beloved QuickTime.

    My concern involving the death of QuickTime, though, would be the loss of the useful QuickTime Player Pro application. This $30 piece of software has been a valuable tool in my multimedia workflow for years. While I’ve heard that the updated QuickTime X Player in Lion adds a few “Pro” features, will we ever be gaining a utility app that will provide interface access to the AV Foundation capabilities for when simple editing, transcoding, exporting, testing and other functions are needed?

    1. If not QT Player, then something from a third party could do it.

  9. This has some pretty profound implications – Phillip you’re as great as ever at connecting the dots for us.

    I do a fair bit of VJ work, and have have tinkered with Quartz Compositions a bit. Never really dived in fully, but there’s lots of great examples. What I find exciting about FCPx supporting it fully is the opportunity not to roll my own filters (there’s just too many good ones out there), but to plug in data or variables into the comps. I have my mind wrapped around that for live performances (say, with VDMX plugging in audio beats into a visualizer), but haven’t considered the possibilities of what it might mean for an NLE.

  10. @ Brian Satchfield – I’ve been using MPEG Streamclip for those “Pro” features you describe. It’s free, solid, very flexible.

    It’s ugly, but trust me, you’ll never miss the Quicktime 7 app again.

    The only thing I might miss is being able to access discrete tracks from QT movies; this invariably breaks stuff anyway so probably better to avoid it.

  11. Very interesting, Philip. Quartz Composer might also be the source of last year’s rumor that the next generation of Final Cut Pro would be based on iMovie.

  12. Great article (very heady stuff). One point though about the history of Flash. Flash owes a tremendous bit of its web-domination success to the fact that Microsoft paid Macromedia to develop it – to destroy Quicktime. Microsoft did not want Apple and Quicktime to dominate the on-line distribution of content and they acted on it with their wallets.

    1. I didn’t know about the MS/Flash connection. Got a link?

  13. Regarding MpegStreamclip making QTPro a moot point, I disagree. I use Mpegstreamclip as my swiss army knife when someone sends me corrupted, crappy, or oddly-encoded video files; but it does not enhance my export capabilities from other pieces of software. QT Pro allows me several more encode choices directly from other programs. I hope that does not go away with the upcoming changes.

    Philip, will Quartz Composer or AV Foundation allow me to integrate with other pieces of software as well as FCP?

    1. AV Foundation has been announced as part of 10.7 Lion, so other apps can easily build on it (or you can via Cocoa programming) but Quartz Composer Compositions are already widely used in apps and OS.

  14. @stan – never heard the MS story before, a prooflink would be helpful.

    If MS wanted Flash to destroy QT, why didn’t they buy it when it was FutureSplash? Hard to remember a time when Flash wasn’t about video. As a former flash dev who was there at the beginning of SWF, I’m skeptical. MS had a hard time seeing the value of the Internet, much less Flash.

  15. Regarding AV Foundation, so I’ll have to geek out and get into the developer side of things for the applications for which I need it to work? Interesting. Something tells me that I need some plaid pants and suspenders for this task. Any advanced info you can direct me to, such as the Apple Developers Network, etc? Or do we have to wait until June 7th?

  16. The link to AV Foundation – the best that’s known, is by Chris Adamson at the link in the article. Other than that, there’s the developer page at apple (You may need a free developer account, but probably not)

    But AV Foundation is for programmers writing applications. OTOH Quartz Composer would be much more accessible for non-programmers.

  17. I have a developer acct. Wow, I’m not scared of programming, but that’s over my head. My next step is the array of Quartz Composer tutorials. Thanks for the tips.

  18. Objective C is an acquired taste – one I hope never to personally acquire! The AV Foundation stuff is for people writing video apps or audio apps, like Final Cut Pro. 🙂

  19. Bill

    Perhaps as a distribution format QT will fade, but what of it’s internal uses? File mobility with reference files is a big deal for those of us that work in multiple applications. Moving a file from Avid to After Effects would be a pain without.

    1. I think the .MOV format for interchange in production won’t go away, but be interpreted by AV Foundation instead.

  20. Chris WIlby

    I know we all love FCP but… what will happen to the other apps in the Studio? A new Motion, Color, Compressor etc.?

    1. Absolutely no word, but we have to presume they’re being worked on as well.

  21. Super unlikely.

    1. What is unlikely George? The changes I mentioned aren predictions, they’re reporting of what’s already changed.