The present and future of post production business and technology

As reported previously in the Pro Apps Hub, JVC’s ProHD strategy is a marketing catch-all for all their HD offerings based on MPEG-2 transport streams. Included in the stragegy is the HDV KY-HD100U and the HDV+ GY-HD7000U.

Available in “early summer” the KY-HD100 is a professional-level HDV camcorder with solid camera-operator-friendly features that justify the ProHD moniker. Three 1/3″ CCDs sit behind a removable lens, although standard is a Fujinon 16x lens developed with JVC. The GY-HD100 records at 30p and 24P at 1280 x 720 resolution in HD and at 29.97 in DV. 24P is accommodated within the 60i standard framework by repeating a 2:3:3:2 pulldown like the only used by Pansonic in the AG-DVX100using MPEG2 “flags” to flag certain whole frames need to be duplicated, so it does 3 frames and then 2 frames (not fields like the DVX would do) and hence embeds 24p in a 60p video stream, but only records 24p frames of data to tape - quite clever really. This allows JVC to offer 24P even though it was not part of the original HDV specification, without deviating from the specification. [Thanks to Graeme Nattress for the correction.]

See Hub news on March 10 for more details on both cameras. At US$6295 JVC have come in well “under $10,000″.

Panasonic generated a lot of buzz at NAB with the announcement of the AG-HVX 200 multi-format camcorder expected to sell for $5949?? without media. The AG-HVX 200 camcorder is a small-form-factor unit with a built in DV deck for DV25 recording and P2 solid state media support for DVCPRO 50 and DVCPRO 100 recording. Reportedly the FireWire output is also active for recording DVCPRO 50 or HD to a tethered FireWire deck. Panasonic are talking with Focus to develop support for the FireStore HD.

The camera itself is an impressive, multi-format, multiple frame rate device. In DVCPRO HD it supports 1080 i at 30i (60 fields), 1080 p at 24 frames/sec, 720 P at 60 or 24 Progressive frames. In 720 P mode is supports variable frame rates like the Varicam to the P2 media. It has 3 x 1/3″ native 16:9 CCDs.

Panasonic plan a bundle with two 8 GB P2 memory cards for US$9999 - an indication of just how far we have to go before solid state media becomes a viable proposition outside news gathering and other niche markets. While P2 media can be used directly as a source in many NLEs- Final Cut Pro adds native support for this media - it’s not viable to retain the P2 memory cards during editing. Most commonly the card’s contents is immediately dumped to hard drive. Panasonic announced a unit specifically for the purpose recently: the AJ-PCS060 portable hard drive with a P2 card slot. [Hub news February 14th]

Having the media on hard drive makes it instantly available for editing, but does not address the need for archive. Either the hard drives need to be permanently retained for archive or the media needs to be copied to another format for archive. This is more handling than most people are used to.

The AG-HVX 200 won’t ship until some time in the fourth quarter of the year, leaving JVC and Sony a long lead time for the competing HDV to become established. With 37,000 FX1 and Z1U units sold, according to the Apple presentation, in just the first six months, that’s a huge lead for Panasonic to catch up with, particularly since JVC will be shipping their KY-HD100U nearly six months ahead.

Although no new applications were announced, Apple upgraded all the Pro Video Apps with new versions of Final Cut Pro, Soundtrack Pro, Compressor 2, Motion 2, LiveType 2, DVD Studio Pro 4 and Shake 4.

In their Sunday morning presentation at Paris, Las Vegas, Apple announced upgrades across the their Pro Video line, consolidating the tools in the $1299 Final Cut Pro Studio. With Final Cut Pro alone priced at $999, the Studio becomes the purchase option of choice if you want Final Cut Pro and any of the other applications. In depth articles will follow, but here’s the 20,000 ft view.

The suite features improved integration across the suite with automatic asset updating from application to application but no dramatic changes to workgroup editing.

Final Cut Pro 5
Key new features are Multicam, Multichannel audio input and support for HDV and P2 media natively. Multicam allows up to 128 angles to be switched in a Multiclip. 4, 9 or 16 angles can be displayed and switched at a time. Final Cut Pro 5 supports tapeless media from Panasonic’s P2 and native IMX support (and keep an eye out for Panasonic’s new camera - P2 media and DV tape for the best of both worlds). MXF media from XDCAM is supported with a 3rd party plug-in from Flip4Mac (Telestream). Final Cut Pro HD works seamless with almost any type of media. HDV media is supported natively. It’s not clear whether or not media can be mixed in a Sequence without rendering. Since it’s not featured, probably not.
RTExtreme has been extended with a new Dynamic RT architecture that adjusts the amount of real-time according to the processor and graphics card speeds - as speeds increase, more real-time will become available. During playback Dynamic RT looks ahead in the timeline and dynamically adapts rather than suddenly stopping playback. Real-time speed change with frame blending is new to version 5.
Final Cut Pro now allows simultaneous import of up to 24 channels of audio. Final Cut Pro audio can now be controlled on any control surface that supports the Mackie Control Protocol meaning that Final Cut Pro mixing can be done a hardware mixer.!
Motion 2
Motion had the most dramatic update with new features that bring the application up to a truly material application for motion graphic design. New interaction techniques - including controlling parameters with a MIDI controller (did anyone say VJ?) - and Replicator for building patterns of repeating objects like flocks of birds. Replicator gives more control than a particle generator and comes with 150 patters with controllable parameters.
Rendering depth has been beefed up to 16 and 32 bits per channel float for those who need it. 32 bit processing is done on the CPU. Motion on Tiger supports more than 4 GB of RAM.
Motion also gains the third dimension with a new 3D distortion filter that allows pseudo 3D with beautiful transparency and effects in real time. A new GPU accelerated architecture lets 3rd parties access the GPU acceleration so Boris, Zaxwerks and DV Garage plug-ins now display in real time.
Soundtrack Pro
Although it shares part of a name with Soundtrack, Soundtrack Pro is far more positioned for a "regular editor" replacement for Pro Tools than simply for scoring music for video. Soundtrack Pro retains the loop editing functionality of Soundtrack, but adds waveform editing, sound design (including a library of sound effects) and includes more than 50 effects from Logic.
Soundtrack Pro comes complete with "search and destroy" tools for most common audio flaws - clicks & pops, AC hum, DC offset, phase and Clipped signal, plus tools for ambient noise reduction and automatically fill gaps with natural sound.
DVD Studio Pro
With an upgrade to version 4, Shake is HD ready with built-in support for H.264 encoding (adopted by both Blu-ray and HD DVD camps) and direct encoding from HDV without intermediate format conversion. Distributed processing using Qmaster for encoding and built-in AC3 encoding (no need to use A.Pack) and enhanced transition support headline DVD Studio Pro’s new features.
On the technical side, DVD Studio Pro 4 supports VTS editing for greater playback performance by allocating menus throughout VTS folders to overcome 1GB menu limitations. GPRM partitioning enhances the scripting options for highly interactive DVDs, for example jumps to motion menu loops to avoid repeating introduction transitions.
LiveType remains part of the Final Cut Pro package and is at version 2. Visually the interface does not appear to have changed. Most of the changes are under the hood with changes to the LiveFont format to support Unicode and vector fonts.

More soon.

Before we determine whether or not a computer is likely to replace an editor, we need to discuss just exactly what is the role of an editor - the human being that drives the software (or hardware) that edits pictures and sound? What do they bring to the production process? Having determined that, perhaps we can consider what it is that a piece of computer software might be capable of now or in the future.

First off I think we need to rid ourselves of any concept that there is just one “editor” role even though there is only one term to cover a vast range of roles in post production. Editing an event video does not use the same skills and techniques as editing a major motion picture; documentary editing is different from episodic television; despite the expectation of similarity, documentary editing and reality television require very different approaches. There is a huge difference in the skills of an off-line editor (story) and an on-line editor (technical accuracy) even if the two roles are filled by the same person.

So let’s start with what I think will take a long time for any computer algorithm to be able to do. There’s no project from current technology - in use or in the lab - that would lead to an expectation that an algorithm would be able to find the story in 40 hours of source and make an emotionally compelling, or vaguely interesting, program of 45 minutes. Almost certainly not going to happen in my lifetime. There’s a higher chance of an interactive storytelling environment à la Star Trek’s Holodeck (sans solid projection). Conceptually that type of environment is probably less than 30 years away, but that’s another story.

If a computer algorithm can’t find the story or make an emotionally compelling program, what can it do? Well, as we discovered earlier, not all editing is the same. There is a lot of fairly repetitive and rather assembly line work labeled as editing: news, corporate video, event videography are all quite routine and could conceivably be automated, if not completely at least in part. Then there is the possibility of new forms of media consumption that could be edited by software based on metadata.

In fact, all use of computer algorithms to edit rely on metadata - descriptions of the content that the software can understand. This is analogous to human logging and log notes in traditional editing. The more metadata software has about media the more able it is to create some sort of edit. Mostly now that metadata will come from the logging process. (The editor may be out of a job, but the assistant remains employed!) That is the current situation but there’s reason to believe it could change in the future - more on that later in the piece.

If we really think about what it is we do as editors on these more routine jobs, we realize that there are a series of thought processes that we go through and underlying “algorithms” that determine why one shot goes into this context rather than anther shot.

To put it at the most basic level, an example might be during editing content from an interview. Two shots of the same person have audio content we want in sequence but the effect is a jump cut. [If two shots in sequence feature same person, same shot…] At this point we choose between putting another shot in there - say from another interview or laying in b-roll to cover the jump cut. [… then swap with alternate shot with same topic. If no shot with same topic available, then choose b-roll.]

That’s a rudimentary example and doesn’t take into account the value judgment that the human editor brings as to whether another interview conveys the story or emotion as well. Most editors are unfamiliar with their underlying thought processes and not analytical about why any given edit “works” - they just know it does but ultimately that judgment is based on something. Some learned skill, some thought process, something. With enough effort that process can be analyzed and in some far distant time and place, reproduced in software. Or it could except for that tricky emotional element - the thing that makes our storytelling interesting and worth watching.

The more emotion is involved in your storytelling output, the safer your job - or the longer it might be before it can be replaced.

Right now, the examples of computerized editing available now- Magic iMovie and Muvee Auto Producer use relatively unsophisticated techniques to build “edited” movies. Magic iMovie essentially adds transitions to avoid jump-cut problems and builds to a template; Muvee Auto Producer requires you to vet shots (thumbs up or down) then uses a style template and cues derived from audio to “edit” the program. This is not a threat to any professional or semi-professional editor with even the smallest amount of skill.

However, it is only a matter of time before some editing functions are automated. Event videography and corporate presentations are very adaptable to a slightly more sophisticated version of these baby step products. OK, a seriously more sophisticated version of these baby-step products, but the difference between slightly and seriously is about 3 years of development!

In the meantime, there are other uses for “automated” editing. For example, I developed a “proof of concept” piece for QuickTime Live! in February 2002 that used automated editing as a means of exploring the bulk of material shot for a documentary but not included in the edited piece. Not intended to be direct competition for the editor (particularly as that was me) it was intended as a means of creating short edited videos that were customized in answer to a plain language query of a database. The database contained metadata about the Clips - extended logging information really. In addition to who, where and when, there are fields for keywords, a numeric value for relative usefulness of the clip, a field for keywords to search for for b-roll [If b-roll matches this, search for more than one clip in the search result, edit them together and lay b-roll over all the clips that use this b-roll.]

So, right now, computer editing can be taught rudimentary skills. This particular piece of software knows how to avoid jump cuts and cut to length based on the quality criteria. It is, in fact, a better editor than many who don’t know the basic grammar of video editing. Teaching the basic grammar is relatively easy. Teaching software to take some basic clips and cut into a news item or even basic template-based corporate video is only a matter of putting in some energy and effort.

But making something that is emotionally compelling - not any time soon.

Here’s how I see it could pan out over the next couple of year. Basic editing skills from human-entered metadata - easy. Generating that metadata by having the computer recognize the images - possible now but extremely expensive. Having a computer edit an emotionally compelling piece - priceless.

It’s not unrealistic to expect, probably before the end of the decade, that a field tape could be fed into some future software system that recognizes the shots as wide, medium, close-up etc; identifies shots in specific locations and with specific people (based on having been shown examples of each) and transcribes the voice content and the text in signs and other places in the image. Software will recognize poor exposure, loss of contrast and loss of focus, eliminating shots that do not stand up technically. Nothing here is that difficult - it’s already being done to some degree in high end systems that are > $300,000 right now. From there it’s only a matter of time before the price comes down and the quality goes up.

Tie that together with a template base for common editing formats and variations and an editing algorithm that’s not that much further on than where we are now and it’s reasonable to expect to be able to input one or more source tapes into the system in the afternoon, and next morning come back to review several edited variations. A couple of mouse-clicks to choose the best of each variation and the project’s done, output to a DVD (or next generation optical disc), to a template-based website, or uploaded to the play-out server.

Nothing’s far fetched. Developing the basic algorithm was way too easy and it works for its design goals. Going another step is only a matter of time and investment. Such is the case with anything that is repetitive in nature: ultimately it can be reproduced in a “good enough” manner. It’s part of a trend I call the “templatorization” of the industry. But that’s another blog discussion. For now, editors who do truly creative, original work need not be worried, but if you’re hacking together video in an assembly-line fashion start thinking of that fall-back career.