The present and future of post production business and technology

I probably have mentioned that we’re working on a documentary about Bob Muravez/Floyd Lippencotte Jnr in part because we wanted demo footage we “owned” (so we could make tutorials available down the line) but also because I wanted to try it in action on a practical job.

I start work in prEdit shortly – nearly started today, so it looks like Friday now – but already we discovered some ideas that have now been implemented in the 1.1. release.

Along the way I’ve learnt a lot about how well (or not) Adobe’s Speech Analysis works. (Short answer: it can be very, very good, or it can be pretty disappointing.) As prEdit is really designed to be used with transcriptions I also tested the Adobe Story > OnLocation > Premiere method, which always worked.

Well, from that workflow it became obvious that speakers (Interviewer, Subject) were correctly identified so wouldn’t it be nice if prEdit automatically subclipped on speaker changes. And now it does.

If multiple speakers have been identified in a text transcript, prEdit will create a new subclip on import at each change of speaker

It also became obvious as I was planning my workflow that we needed a way to add new material to an existing prEdit project, and to be able to access the work from historic projects to add to a new one.

New Copy Clips… button so you can copy clips from one prEdit project to another open project

Now that I’m dealing with longer interviews than when we tested, I needed search in the Transcript view.

Search popup menu added to Transcript View.

That led to one problem: adding metadata to multiple subclips at a time. Previously I’d advocated adding common metadata to the clip before subclipping in prEdit (by simply adding returns to the transcript) but if it comes in already split into speakers, that wasn’t going to work!

Logging Information and Comments can be added for multiple selected subclips if the field doesn’t already have an entry for any of the selected subclips

Because you never, ever want to run the risk of over-writing work ready done.

And some nice enhancements:

Faster creation of thumbnails

Bugfix for Good checkbox in Story View

prEdit 1.1 is now available. Check for updates from within the App itself. And if you work in documentary, you should have checked it out already.

Ogg: The “Intelligent Design” of digital media http://bit.ly/bUYo7B

The only thing Ogg is good for, is being open source, which isn’t relevant to professional media producers.

People who actually work in media don’t mind paying for stuff, and don’t mind not owning/sharing the IP. Video production professionals are so accustomed to standardizing on commercial products, many of them become generic nouns in industry jargon: “chyron” for character generators, “grass valley” for switchers, “teleprompters”, “betacam” tape, etc. Non-free is not a problem here. And if your argument for open-source is “you’re free to fix it if it doesn’t do what you want it to,” the person who has 48 shows a day to produce is going to rightly ask “why would I use something that doesn’t work right on day one?”

The open source community doesn’t get media. Moreover, it doesn’t get that it doesn’t get media. The Ogg codecs placate the true believers, and that’s the extent of their value.

Introduction to AV Foundation http://slidesha.re/aYEJfR To be honest I don’t know why this isn’t hidden behind an NDA, but it’s not and until someone has it taken down, and asks me to do the same, I’ll consider it public knowledge.

Now, AV Foundation is the iOS media system, so we’re not talking about QuickTime per se but I have to wonder.

QuickTime – the real OS-centric media framework, not the little sub applications that function as players – is transitioning from C APIs (Carbon) to Cocoa via QT Kit. Trouble is, QT Kit got a lot of work around QuickTime 7’s release, but not so much in recent years. And yet Final Cut Pro needs a lot of what’s not written, before it can release a Cocoa version of Final Cut Pro.

Actually, Apple could do what Adobe have done for Premiere Pro CS5. In rewriting their core media handling engine, Adobe retained QuickTime support by spinning it off into a 32 bit thread, but that’s a complex workaround that does nothing for performance, nothing positive anyway.

When you consider slide 9… Even though it was only introduced in iOS 2.2, extended in iOS 3 and “completed” in iOS 4 (consider the reference framework growth in slides 6, 7 and 8), AV Foundation has 56 Classes and 460 Methods (the more you have of these, the more you can do with it). QT Kit has 24 Classes (less than half) and 360 Methods. Compare that with the (very mature) QuickTime for Java with 576 Classes and more than 10,000 Methods. Something tells me that QT Kit is not in favor at Apple.

Not that I think QuickTime is going away, at least not as a brand for their media players and the overall technology. I say that because, although the code that’s in iPhone OS shows a simplified player, that was all that was originally released and it shared no “QT Classes or Frameworks”. So, the QuickTime brand is likely to be retained.

If I was extrapolating from this presentation, and I am extrapolating wildly from a small amount of data, I’d guess that the direction within Apple was toward the more modern Classes and Methods of AV Foundation, and that, eventually, AV Foundation, Core Audio, Core Animation and Core Media will replace what we currently have under QuickTime on OS X: Core Audio, Core Video (well, just a subclass of Core Image) and a lot of deprecated (do not use) C APIs.

If you consider slide 14, and the similarity of Classes between QT Kit and AV Foundation it makes no sense to build two technologies in the company that were essentially doing the same thing.  Slide 29 shows how similar an AVAsset is to a QTMovie. The other Classes all seem to duplicate functionality that’s in QuickTime now, but in efficient, new, modern code. Capture, editing, playback, media formats… they all seem to be in AV Foundation duplicating work done (or not yet done) in QuickTime’s QT Kit.

Importantly Core Media Time is in “n’ths of a second” not “ticks” or “events”. Media based on time will be better for video frame rate uses than one based on ticks or events, which caused the “Long Frames” problems of earlier versions of Final Cut Pro.

In support of my hypothesis I offer slide 42: specific references to AVAssetExportSession.h being available in OS X with 10.7 and likewise CMTime.h has a reference to becoming available in 10.7.

So, I’ll go on a limb and suggest that QuickTime as we’ve known it is somewhat dead; long live a new QuickTime. QuickTime will continue being the branding, but everything “below that” will transition to new architectures essentially ported from iOS to OS X.

This would be a very good thing. A completely new, modern, efficient (you see what it does on the iPhone) underpinning for QuickTime down below that QT Kit layer.

Who wouldn’t want to use that in an modern NLE, even if it means waiting for OS X 10.7, which hasn’t been announced yet? It would make it much easier for the Final Cut Pro team to create a much more powerful media engine than it has now; one that really understands time and not events and one that mimics the power of Adobe’s Mercury Engine. Let’s face it, media performance on a 1 GHz A4 chip is in some ways better than the performance on 8 core processors. iMovie for iOS, built on these frameworks (if slide 24 is to be believed) can edit Long GOP H.264, which Final Cut Pro can’t! (And in both cases the H.264 playback is accelerated by hardware: dedicated chips in the iPhone, on the graphics card in OS X.)

As always, conjecture on my part, and this time based solely on what I’ve learnt from the quoted slide show. Chris Adamson does not work for Apple but he does claim expertise in iOS and QuickTime. Other posts on his blog indicate some differences between AV Foundation and QuickTime; and Classes still missing from AV Foundation that are in the current version of QuickTime. That shakes my confidence in the hypothesis a little, but given how little work has been done on QT Kit in the last two years, and the need to have the foundations for QuickTime modernized, it still seems like the most likely path Apple will take.

Another data point is that the QuickTime X player was promoted thusly:

Using media technology pioneered in OS X iPhone™, Snow Leopard introduces QuickTime X, which optimizes support for modern audio and video formats resulting in extremely efficient media playback. Snow Leopard also includes Safari® with the fastest implementation of JavaScript ever, increasing performance by 53 percent, making Web 2.0 applications feel more responsive.*

Pioneering the technology under iOS, and then porting it to Mac OS X has happened already.

UPDATE: Chris Adamson, who did the presentation I referred to, clarified many of the points I get wrong or wrongish, including the fact that AV Foundation is not under NDA. His Connecting the Dots post is essential reading if you’ve got this far!

Japan develops ‘touchable’ 3D TV images http://bit.ly/cCu9kB

Touchable in the sense that the images react to being touched, poked, distorted etc, by using cameras to determine where hands are. They are not claiming (as far as I can see) that the viewer can “feel the image”

‘Interoperable Master Format’ Aims to Take Industry Into a File-based World http://bit.ly/bvF6Vk

A group working under the guidance of the Entertainment Technology Center at USC is trying to create specifications for an interoperable set of master files and associated metadata. This will help interchange and automate downstream distribution based on metadata carried in the file. The first draft of the specification is now done based on (no surprises) the MXF wrapper. (Not good for FCP fans, as Apple has no native support for MXF, without third party help).

Main new items: dynamic metadata for hinting pan-scan downstream and “Output Profile List”:

“The IMF is the source if you will, and the OPL would be an XML script that would tell a transcoder or any other downstream device how to set up for what an output is on the other side,” Lukk explained.

The intention is to bring this draft spec to SMPTE, but first, ETC@ USC is urging the industry to get involved. “We need industry feedback and input on the work that the group has done thus far,” said ETC CEO and executive director David Wertheimer. “Anyone who has interest in this topic should download the draft specification and provide feedback to the group.”

The Future of Picture Editing http://bit.ly/aNRLVA

I’ve had the pleasure of meeting Zak Ray when I travelled to Boston. I like people who have an original take on things and Zak’s approach to picture editing – and his tying it to existing technologies (that may ned improvement) – is an interesting one.

And yet, despite such modern wonders as Avid Media Access and the Mercury Playback Engine, modern NLEs remain fundamentally unchanged from their decades-old origins. You find your clip in a browser, trim it to the desired length, and edit it into a timeline, all with a combination of keys and mouse (or, if you prefer, a pen tablet). But is this process really as physically intuitive as it could be? Is it really an integrable body part in the mind’s eye, allowing the editor to work the way he thinks? Though I can only speak for myself, with my limited years of editing experience, I believe the answer is a resounding “no”. In his now famous lecture-turned-essay In the Blink of an Eye, Walter Murch postulates that in a far-flung future, filmmakers might have the ability to “think” their movies into existence: a “black box” that reads one’s brainwaves and generates the resulting photo-realistic film. I think the science community agrees that such a technology is a long way off. But what untilthen? What I intend to outline here is my thoughts on just that; a delineation of my own ideal picture-editing tools, based on technologies that either currently exist, or are on the drawing board, and which could be implemented in the manner I’d like them to be. Of course, the industry didn’t get from the one-task, one-purpose Moviola to the 2,000 page user manual for Final Cut Pro for no reason. What I’m proposing is not a replacement for these applications as a whole, just the basic cutting process; a chance for the editor to work with the simplicity and natural intuitiveness that film editors once knew, and with the efficiency and potential that modern technology offers.

It’s a good article and a good read. Raises the question though – if Apple (or Adobe/Avid) really innovated the interface would people “hate it” because it was “different”?

Powerful new transcript workflow tool – paper cuts without the pain – from Intelligent Assistance (my day job). http://bit.ly/9nQv07

We just launched prEdit, our pre-editing tool for developing paper cuts (a.k.a. radio cut) from transcripts. prEdit:

• Lets producers or editors cut transcripts into selects in seconds
• Adds and updates log notes with auto-complete logging fields
• Previews the video for any clip, subclip, paper cut or section of paper cut
• Exports to Excel spreadsheets and Final Cut Pro, or Premiere Pro Sequences

“prEdit marks a new generation of postproduction tools,”  say I. “Video editing by text is a whole new way of working that will take weeks out of developing a paper cut.”

prEdit is available now from AssistedEditing.com and carries an MSRP of $395, discounted for an introductory special to $295 until August 31st. The prEdit workflow is described at http://assistedediting.com/prEdit/workflow.html and a video overview is available at http://assistedediting.com/prEdit. The first 80 seconds provide an overview.

The video is now available at YouTube  http://youtu.be/3fV388QsVVA?a

Thanks to an upcoming piece of software we’re working on I’ve been spending a lot of time within the CS5 workflow environment, particularly looking at metadata and the Story workflow, and I’ve come to the conclusion that we’ve been so blinded by the Mercury Engine’s performance that we might not have seen where they’re heading. And I like what I see.

Most readers will likely be aware of the Transcription ability introduced with CS4 and updated in CS5. Either in Soudbooth, or in Adobe Media Encoder (AME) via Premiere Pro for batches, the technology Adobe builds on from Autonomy will transcribe the spoken word into text. Our initial testing wasn’t that promising, but we’ve realized we weren’t sending it any sort of fair test. With good quality audio the results are pretty good: not perfect but close, depending on the source, of course.

We first explored this early in the year when we built and released Transcriptize, to port that transcription metadata from the Adobe world across to Apple’s. That’s what set us down our current path to the unreleased software, but more of that in a later post.

Now we’re back in that world, it’s a pretty amazing “story”. There’s three ways they get it that I see:

1. Good metadata tracking at the file level
2. Flexible metadata handling
3. Metadata-based workflows built into the CS applications (and beyond).

Balancing that is the serious miss of not showing source metadata from non-tape media that doesn’t fit into pre-defined schema. At least that seems to be the case: I can’t find a Metadata Panel that shows the Source Metadata from P2, AVCCAM/AVCHD, or RED to display. Some of the source metadata is displayed in existing fields, but they are only the fields that Adobe has built into Premiere Pro, which miss a lot of information from the source. For example, none of the exposure metadata from RED footage is displayed, nor Latitude and Longitude from P2 and AVCCAM footage.

That’s the downside. To be fair, Final Cut Pro doesn’t display any of the Source Metadata either (although you can access it via the XML file.)  Media Composer can show all the Source if desired.

Good Metadata Tracking at the file level

Apple added QuickTime Metadata to Final Cut Pro 5.1.2 where they retain and track any Source Metadata from files imported via Log and Transfer. This is a flexible schema but definitely under supported. Adobe’s alternative is XMP metadata. (Both XMP and QuickTime metadata can co-exist in most media file formats.)

XMP metadata is file based, meaning it is stored in, and read from, the file. There are seven built-in categories, plus Speech Analysis, which is XMP metadata stored in the file (for most formats) but considered as a different category in the Premiere Pro CS5 interface. I believe that the Source metadata should show in the XMP category because it is file-based even if its not XMP.

On the other plus side XMP metadata is very flexible. You don’t need third party applications to write to the XMP metadata. Inside Premiere Pro CS5 you simply set up the schema you want and the data is written to the file transparently. If the data is in a file when it’s added to a project, it’s read into the project and immediately accessible.

This metadata travels with the file to any and all projects. This provide a great way of sending custom metadata between applications. Speed Analysis metadata is also carried in the file, so it can be read by any Adobe application (and an upcoming one from us, see intro paragraph) direct from the file.

Flexible Metadata Handling

Not only is the XMP file-based metadata incredibly flexible, but you can also apply any metadata scheme to a clip within a Premiere Pro project, right into Clip metadata. For an example of how this is useful, let’s consider what we had to do in Final Cut Pro for First Cuts. Since Final Cut Pro doesn’t have a flexible metadata format, we had to co-opt Master Comments 1-4 and Comment A to carry our metadata. In Premiere Pro CS5 we could simply create new Clip-based fields for Story Keywords, Name, Place, Event or Theme and B-roll search keywords.

(Unfortunately this level of customization in Premiere Pro CS5 does not extend to Final Cut Pro XML import or export.)

An infinitely flexible metadata scheme for clips and for media files (independently) is exactly what I’d want an application to do.

Metadata-based Workflows in the CS5 Applications

To my chagrin I only recently discovered how deeply metadata-based workflows have become embedded in the Adobe workflow. (Thanks to Jacob Rosenberg’s demonstration at the June Editor’s Lounge for turning me on to this.) Adobe have crafted a great workflow for scripted productions that goes like this:

1. Collaboratively write your script in Adobe Story, or import a script from most formats, including Final Draft. (Story is a web application.)
   • Adobe Story parses the script into Scenes and Shots automatically.
2. Export from Adobe Story to a desktop file that is imported into OnLocation during shooting.
   • In OnLocation you have access to all the clips generated out of the Adobe Story file. Clips can be duplicated for multiple takes.
   • Clips are named after Scene/Shot/Take.
3. During shooting you do not need to have a connection to the camera because some genius at Adobe realized that metadata could solve that problem. All that needs to be done during shooting of any given shot/take is for a time stamp to be marked against the Clip:
   • i.e. this clips was being taken “now”.
   • Marking a time stamp is a simple button press with the clip selected.
4. After footage has been shot, the OnLocation project is “pointed” to the media where it automatically matches the shot with the appropriate media file, based on the time stamp metadata in the media file with the time mark in the OnLocation Clip.
   • The media file is renamed to match the clip. Ready for import to Premiere Pro CS5.

Now here’s the genius part in my opinion (other than using the time stamp to link clips). The script from Adobe Story has been embedded in those OnLocation clips, and is now in the clip. Once Speech Analysis is complete for each clip, the script is laid-up against the analyzed media file so each word is time stamped. The advantage of this workflow over using a guide script directly imported is that the original formatting is used when the script comes via Story.

All that needs to be done is to build the sequence based on the script, with the advantage that every clip is now searchable by word. Almost close to, but not quite, Avid’s ScriptSync based on an entirely different technology (Nexidia).

It’s a great use of script and Speech Analysis and a great use of time-stamp metadata to reduce clip naming, linking and script embedding. A hint of the future of metadata-based workflows.

All we need now, Adobe, is access to all the Source Metadata.

During a recent thread here where I “infamously” suggested Apple should drop Log and Capture for the next version of FCP, one of the topics that came up was the use of metadata. Most commenters (all?) appeared – to my interpretation – to feel that reel name and TC were the “essence” of metadata.

And yet, if we look at the most recent work of the Chief Video Architect (apparently for both pro and consumer applications) Randy Ubilos we see that Location metadata is a requirement for the application. According to Apple’s FAQ for iMovie for IPhone if you don’t allow iMovie for iPhone to access your location metadata:

Because photos and videos recorded on iPhone 4 include location information, you must tap OK to enable iMovie to access photos and videos in the Media Library.

If you do not allow iMovie to use your location data, then the app is unable to access photos and videos in the Media Browser.

You can still record media directly from the camera to the timeline but, without the Location metadata, you’re pretty much locked out of iMovie for iPhone for all practical purposes.

There is no location metadata from tape capture! There’s not much from non-tape media right now, although some high end Panasonic cameras have an optional GPS board. However P2 media (both DVCPRO HD and AVC-I) as well as AVCCAM all have metadata slots for latitude and longitude.

Now, I’m NOT saying that Apple should force people to use metadata – particularly if it’s non existent – and this type of restriction in a Pro app would be unconscionable. I merely point out that this shows the type of thinking within Apple. In iMovie for iPhone they can create a better user (consumer) experience because they use Location metadata for automatic lower third locations in the themes.

Where I think it’s a little relevant is in counterpoint to some of my commentors: building an app that’s reliant on metadata is a different app than one relying on simple reel name and TC numbers.

I was finally watching the Steve Jobs Keynote from WWDC on June 7. (I know, but this was our second try – we get talking about stuff, what can I say?) I got to the iMovie for iPhone 4 demo and was blown away by the creative use of source metadata.

At 58 minutes into the keynote, Randy Ubillos is demonstrating adding a title to the video he’s editing in iMovie and iMovie automatically ads the location into the title. Not magic, but it’s simply reading the location metadata stored with images and videos shot with an iPhone and using that to generate part of the title. This is exactly how metadata should be used: to make life easier and to automate as much of the process as possible.

Likewise the same metadata draws a location pin on the map in one of the different themes. Exactly like the same metadata does in iPhoto.

In a professional application, that GPS data – which is coming to more and more professional and consumer video camcorders – could not only be used to add locations, but also to read what businesses are at the address. From that source and derived metadata (address and business derived from location information) we can infer a lot.

Check out my original article on metadata use in post production and for a more detailed version, with some pie-in-the-sky predictions of where this is going to lead us, download the free Supermeet Magazine number 4 and look for the article (featured on the cover) The Mundane and Magic future of Metadata.