Most readers will likely be aware of the Transcription ability introduced with CS4 and updated in CS5. Either in Soudbooth, or in Adobe Media Encoder (AME) via Premiere Pro for batches, the technology Adobe builds on from Autonomy will transcribe the spoken word into text. Our initial testing wasn’t that promising, but we’ve realized we weren’t sending it any sort of fair test. With good quality audio the results are pretty good: not perfect but close, depending on the source, of course.

We first explored this early in the year when we built and released Transcriptize, to port that transcription metadata from the Adobe world across to Apple’s. That’s what set us down our current path to the unreleased software, but more of that in a later post.

Now we’re back in that world, it’s a pretty amazing “story”. There’s three ways they get it that I see:

1. Good metadata tracking at the file level
2. Flexible metadata handling
3. Metadata-based workflows built into the CS applications (and beyond).

Balancing that is the serious miss of not showing source metadata from non-tape media that doesn’t fit into pre-defined schema. At least that seems to be the case: I can’t find a Metadata Panel that shows the Source Metadata from P2, AVCCAM/AVCHD, or RED to display. Some of the source metadata is displayed in existing fields, but they are only the fields that Adobe has built into Premiere Pro, which miss a lot of information from the source. For example, none of the exposure metadata from RED footage is displayed, nor Latitude and Longitude from P2 and AVCCAM footage.

That’s the downside. To be fair, Final Cut Pro doesn’t display any of the Source Metadata either (although you can access it via the XML file.)  Media Composer can show all the Source if desired.

Good Metadata Tracking at the file level

Apple added QuickTime Metadata to Final Cut Pro 5.1.2 where they retain and track any Source Metadata from files imported via Log and Transfer. This is a flexible schema but definitely under supported. Adobe’s alternative is XMP metadata. (Both XMP and QuickTime metadata can co-exist in most media file formats.)

XMP metadata is file based, meaning it is stored in, and read from, the file. There are seven built-in categories, plus Speech Analysis, which is XMP metadata stored in the file (for most formats) but considered as a different category in the Premiere Pro CS5 interface. I believe that the Source metadata should show in the XMP category because it is file-based even if its not XMP.

On the other plus side XMP metadata is very flexible. You don’t need third party applications to write to the XMP metadata. Inside Premiere Pro CS5 you simply set up the schema you want and the data is written to the file transparently. If the data is in a file when it’s added to a project, it’s read into the project and immediately accessible.

This metadata travels with the file to any and all projects. This provide a great way of sending custom metadata between applications. Speed Analysis metadata is also carried in the file, so it can be read by any Adobe application (and an upcoming one from us, see intro paragraph) direct from the file.

Flexible Metadata Handling

Not only is the XMP file-based metadata incredibly flexible, but you can also apply any metadata scheme to a clip within a Premiere Pro project, right into Clip metadata. For an example of how this is useful, let’s consider what we had to do in Final Cut Pro for First Cuts. Since Final Cut Pro doesn’t have a flexible metadata format, we had to co-opt Master Comments 1-4 and Comment A to carry our metadata. In Premiere Pro CS5 we could simply create new Clip-based fields for Story Keywords, Name, Place, Event or Theme and B-roll search keywords.

(Unfortunately this level of customization in Premiere Pro CS5 does not extend to Final Cut Pro XML import or export.)

An infinitely flexible metadata scheme for clips and for media files (independently) is exactly what I’d want an application to do.

Metadata-based Workflows in the CS5 Applications

To my chagrin I only recently discovered how deeply metadata-based workflows have become embedded in the Adobe workflow. (Thanks to Jacob Rosenberg’s demonstration at the June Editor’s Lounge for turning me on to this.) Adobe have crafted a great workflow for scripted productions that goes like this:

1. Collaboratively write your script in Adobe Story, or import a script from most formats, including Final Draft. (Story is a web application.)
   • Adobe Story parses the script into Scenes and Shots automatically.
2. Export from Adobe Story to a desktop file that is imported into OnLocation during shooting.
   • In OnLocation you have access to all the clips generated out of the Adobe Story file. Clips can be duplicated for multiple takes.
   • Clips are named after Scene/Shot/Take.
3. During shooting you do not need to have a connection to the camera because some genius at Adobe realized that metadata could solve that problem. All that needs to be done during shooting of any given shot/take is for a time stamp to be marked against the Clip:
   • i.e. this clips was being taken “now”.
   • Marking a time stamp is a simple button press with the clip selected.
4. After footage has been shot, the OnLocation project is “pointed” to the media where it automatically matches the shot with the appropriate media file, based on the time stamp metadata in the media file with the time mark in the OnLocation Clip.
   • The media file is renamed to match the clip. Ready for import to Premiere Pro CS5.

Now here’s the genius part in my opinion (other than using the time stamp to link clips). The script from Adobe Story has been embedded in those OnLocation clips, and is now in the clip. Once Speech Analysis is complete for each clip, the script is laid-up against the analyzed media file so each word is time stamped. The advantage of this workflow over using a guide script directly imported is that the original formatting is used when the script comes via Story.

All that needs to be done is to build the sequence based on the script, with the advantage that every clip is now searchable by word. Almost close to, but not quite, Avid’s ScriptSync based on an entirely different technology (Nexidia).

It’s a great use of script and Speech Analysis and a great use of time-stamp metadata to reduce clip naming, linking and script embedding. A hint of the future of metadata-based workflows.

All we need now, Adobe, is access to all the Source Metadata.

And yet, if we look at the most recent work of the Chief Video Architect (apparently for both pro and consumer applications) Randy Ubilos we see that Location metadata is a requirement for the application. According to Apple’s FAQ for iMovie for IPhone if you don’t allow iMovie for iPhone to access your location metadata:

Because photos and videos recorded on iPhone 4 include location information, you must tap OK to enable iMovie to access photos and videos in the Media Library.

If you do not allow iMovie to use your location data, then the app is unable to access photos and videos in the Media Browser.

You can still record media directly from the camera to the timeline but, without the Location metadata, you’re pretty much locked out of iMovie for iPhone for all practical purposes.

There is no location metadata from tape capture! There’s not much from non-tape media right now, although some high end Panasonic cameras have an optional GPS board. However P2 media (both DVCPRO HD and AVC-I) as well as AVCCAM all have metadata slots for latitude and longitude.

Now, I’m NOT saying that Apple should force people to use metadata – particularly if it’s non existent – and this type of restriction in a Pro app would be unconscionable. I merely point out that this shows the type of thinking within Apple. In iMovie for iPhone they can create a better user (consumer) experience because they use Location metadata for automatic lower third locations in the themes.

Where I think it’s a little relevant is in counterpoint to some of my commentors: building an app that’s reliant on metadata is a different app than one relying on simple reel name and TC numbers.

The project is about a young triple threat – singer, dancer, actor – Tim Draxl, discovered in Sydney when he was just short of his 18th birthday.

He played Rolf in a professional touring production in Australia in 2000 and his career has blossomed from there, and the three CD deal he has with Sony Universal: the first when he was 18!

Remember, these edits were done in seconds, from selects using Assisted Editing’s First Cuts software. And yes, this is my baby (along with Dr Greg Clarke).

The Sound of Music Edit

Without limits – about 13 minutes of material.

Four minute limit set. Edit is tighter and only the best material makes it to the edit.

Growing Up

Tim grew up partly in Australia and partly in Austria as his father worked as a ski instructor. This is the unlimited version of the “Growing Up” edit.

[Update: I forgot the 10 minute limit so one of the movies was too long and YouTube can't distinguish between a 6 minute cut and a 4 minute cut, thinking they're the same. Fortunately the videos are also available on our site. The Growing up unlimited  and six minute versions are available at http://assistedediting.com/FirstCuts/results.html]

And finally with a 4 minute limit.

At 58 minutes into the keynote, Randy Ubillos is demonstrating adding a title to the video he’s editing in iMovie and iMovie automatically ads the location into the title. Not magic, but it’s simply reading the location metadata stored with images and videos shot with an iPhone and using that to generate part of the title. This is exactly how metadata should be used: to make life easier and to automate as much of the process as possible.

Likewise the same metadata draws a location pin on the map in one of the different themes. Exactly like the same metadata does in iPhoto.

In a professional application, that GPS data – which is coming to more and more professional and consumer video camcorders – could not only be used to add locations, but also to read what businesses are at the address. From that source and derived metadata (address and business derived from location information) we can infer a lot.

Check out my original article on metadata use in post production and for a more detailed version, with some pie-in-the-sky predictions of where this is going to lead us, download the free Supermeet Magazine number 4 and look for the article (featured on the cover) The Mundane and Magic future of Metadata.

I love being a small independent software developer: it’s great to be able to respond to customer requests promptly – and it makes the software better. Incidents like this one today make me also appreciative of the communication tools we now have

Some time, overnight our time, we had a new customer buy a copy of Sync-N-Link to sync rushes for 8 episodes of a new drama series: in Belgium! A few hours later he emailed to say that it was doing everything he expected, but their sound guy entered metadata (log notes) into the sound clips and Sync-N-Link (like Final Cut Pro itself) discards audio metadata in favor of the video metadata. (In a merged clip there is only room for one of each type of log note/metadata). The feature request was that the metadata from the audio could be preserved instead of from that from the video.

A good request. The ever efficient Greg Clarke, after morning coffee, got to work. At around 1:30 pm (Pacific) an update was published, ready for download, with the feature added. Not quite seven hours from feature request to released software.

I love that we can do that.

If you use any of our software let us know what more you want it to do. We can be very responsive!

Note: Apparently I believed I’d remember the people in my pictures for ever! While I remember the people, names that are not noted somewhere have evaporated over the 30 or so years in between.

The first, and most common, source of metadata was “Added” metadata: those notes that were made on the slide, a carrier or slide sheet carrier heading. Apparently in my early 20’s I could write in 6 point text, which I’m having trouble reading with much older eyes. Regardless, Added metadata has proven to be the most valuable.

The brief added metadata is usually combined with some “Analytical” metadata. Analytical metadata is metadata we get from analyzing visual content in the image. For example, this picture:

was one of a group of 20 on the same sheet with the single word ‘Burbank’ in the heading. In fact only two of the dozen images were in Burbank. But using Analytical metadata – the general location and the street sign near the traffic lights – the shot is clearly on Burbank Blvd, on the rise to the overpass over the 5 Freeway and railway lines. (The sign is for Front St.)

Co-incidence number 1: this is less than a mile from where i now live.

The other very useful metadata was stamped on the slides themselves: the month and year of processing, which locks down an approximate time scale. Also useful was the fact that I’d numbered all my files sequentially from the beginning of 1973 (my year in Japan). That sequential metadata made it much easier to identify specific times, in combination with the stamped-on date.

A combination of Added and Analytical metadata led me to the discovery that most of my 1976 trip was spent in the West San Fernando Valley. Identified was the location of an awards ceremony (by Church name) and a street shot, that was clearly looking across the SF Valley, was also named. Both turn out to be incredibly close to our Tarzana office and Woodland Hills home (2001-2005 before we moved to Burbank).

This jogged my memory that we had spent a lot of time in a school hall for rehearsal/training and enjoyed a close-by Denny’s. Taft High school (Ventura Blvd at Winnetka) has a Denny’s across the road.

So it is entirely possible that during my 1976 trip to the US, I spent the majority of my time around the area that was to become home a quarter of a century later; and some of the rest of my time near my 30-years-in-the-future home in Burbank.

Crazy. And without metadata I’d have never remembered.

Here’s my interview with Larry Jordan where we announced it and the press release is below.

“Announcing Transcriptize on the Digital Production BuZZ”

Transcriptize expands the usefulness of Adobe Speech Transcription

Take transcriptions from Adobe Production Bundle to Media Composer, Excel and Final Cut Pro.

Burbank, CA (February 12, 2010) – Intelligent Assistance, Inc has introduced a new software tool that takes Transcription XML from Adobe Premiere Pro CS4 or Soundbooth CS4 and converts it text, Excel Spreadsheet or Final Cut Pro clip markers.

“Late last year, Larry Jordan asked if we could create something to make the Adobe Speech Transcription more available”, says Intelligent Assistance’ CEO Philip Hodgetts. “We thought that was a great idea and Transcriptize is the result, less than two months later.”

Transcriptize imports the transcription XML from the Adobe Production Bundle and allows editors and producers to name the speakers – something not possible in the Production Bundle. From there users have the option to:

Export a plain text file, suitable for the needs of a producer or to import into Media Composer’s Script Sync engine.

Export an Excel spreadsheet with a variable number of words per row – Perfect for a producer.

Open the XML from a Final Cut Pro clip and add the transcription to Markers where:

There are a variable number of words per Marker (including one Marker per speaker)

The speaker name is placed in the Marker name

Transcription appears in the clip Marker comment

Marker colors are used to identify each speaker (FCP 7 onward).

The transcription can be searched within Final Cut Pro.

Markers can be easily subclipped based on transcription content.

Transcriptize is available now from www.assistedediting.com/Transcriptize/. MSRP is US$149 with an introductory offer of $99 until the end of February 2010. NFR versions for review are available, contact Philip Hodgetts, details below.

The pause has been caused by a couple of reasons: number one of which is that this week (and the next two) I’m looking after myself. Partner Greg is in Australia for a Visa renewal and I’m once again realizing how much he does to make our lives easier. (Mine particularly).

Also, we’ve been releasing new software, updating older products and revising earlier books. In fact we’ve been doing so much that I can’t announce stuff in press releases yet!

About a month back I finished completely revising Simple Encoding Recipes for the Web 2009 edition. Anyone who purchased in 2009 should have received a download link. Announcements to everyone else are coming or you can buy the update for $4.95. (It’s a complete rewrite).

Last week the revision of The HD Survival Handbook 2009-2010 was finished and, again, those who purchased in 2009 will have received an email with an update link. All other previous purchasers will have received a $4.75 upgrade offer. It’s been about 30% rewritten, almost an additional 20 pages, so the upgrade price represents the value add that’s gone into it. The “upgrade” is the full new version, not changed pages. Also this year we went with Avid support – codecs, hardware and workflow. Given that’s now a 233 page US Letter book, it’s a huge project to revise. So much has changed in a year.

In between, Greg’s been working hard to release an updated First Cuts to First Cuts Studio by adding in the functionality of one of our new applications, exceLogger. Have I mentioned we love customer feedback? It’s made Sync-N-Link a much stronger product. Naturally we want the same feedback from customers of our other products. Good, bad or feature request, all feedback is welcomed. (Begged for!) exceLogger was a feature request for First Cuts for Final Cut Pro, and is available as a stand-alone application for those who just want to log in Excel but merge with captured media in Final Cut Pro.

BTW this now makes  First Cuts Studio great value: At $295 it includes Finisher (US$149) and exceLogger (US$69) – so the Auto-edit functionality of First Cuts is just $77!

Greg also developed two additional applications that fit perfectly in our metadata focus. miniME (Metadata Explorer) when we discovered (just four years after Apple told us!) that the QuickTime metadata from IT-based digital video sources (non-tape) is preserved in FCP but only visible in exported XML. So, Greg wrote me a simple tool to view the hidden metadata and export to an Excel spreadsheet. (That functionality is free in the demo version.) The paid version lets you remap that metadata into visible fields in Final Cut Pro.

Finally, the night we demonstrated miniME and exceLogger a friend of mine again suggested an idea for software that would report clips used in a Sequence – video or audio – as he has to provide reports to his clients, but equally useful for music reports. Greg worked on it for a while and this week we released Sequence Clip Reporter. (Yeah, we tried to find a better name but that’s descriptive and stuck.)

Now there’s a lot of work goes into writing software. There’s the work on the actual functions of the software, but then there’s questions about interface and how functions should work. Then there’s software updating to be added, serial number support to be added and feedback mechanisms added. All beyond the actual functionality.

Me, I get to design a new logo for each piece of software, write website and postcard copy, write a press release and send it out. Plus Help files need to be written so people can actually use the software. So, around any new software there’s a lot of work that doesn’t actually involve much software writing!

And that’s why posting has been sparse.

Before I get to that though, let me ask (beg) for feedback on any of our software products. We want to keep making them better and love feedback, feature requests and especially problems. We respond quickly – this particular feature request was received on Friday 26th, discussed briefly during a Hollywood Bowl concert on Saturday night and was a preliminary feature by Wednesday!

Anyhow, in discussing this particular tool with Jerry (you’ll find out what it is soon enough!) I asked how much metadata from RED is imported to Final Cut Pro via Log and Transfer. Jerry, who uses RED a whole lot more than me (i.e. he uses it!) said “not very much”, which pretty much matched my understanding working with a whole bunch of RED clips with Sync-N-Link and never seen any of the color temperature, date or other information that’s in the RED metadata.

In sharing this conversation with my smart partner, and our main code writer, Greg Clarke, he commented “Oh, I do think Mr Hofman is mistaken!” (or words to that effect). Turns out Greg has been scrolling past this metadata for most of the last year. The difference is that Greg works with FCP XML exports, while Jerry and I were looking through the Final Cut Pro interface.

OMG! What a treasure-trove of metadata there is. And why didn’t we know of this? Surely someone from all the conversations we’ve had with people developing Sync-N-Link must know about this? (You’ll all come out of the woodwork into the comments and let me know you’ve known about it for years!)

So this morning Greg’s built me a tool for exploring this hidden (I prefer “secret” because it makes it seem more mysterious) metadata, turning it into an excel spreadsheet. I already had XDCAM EX media and P2 media along with RED clips and I was able to download some AVCCAM media shot with Panasonic’s HMC-150 camera.

There’s an enormous amount of Source metadata there. A lot of fields that seems to be unused even in the camera. Clearly, the current version of Final Cut Pro doesn’t have the flexibility to display items like ‘whiteBalanceTint’ or ‘digitalGainBlue’ settings in the original file. I guess this type of metadata is going to be challenging for Apple and Avid to deal with, as they don’t (currently) have displays in their application for the enormous amount of metadata that are generated with tapeless cameras. I’m just very thankful that it’s being retained, and that it’s available via XML (and associated with a Final Cut Pro clip ID).

There’s definitely metadata already  being produced that we can use to improve First Cuts – at least for non-tape media sources. But it’s also interesting to explore fields that are available but not being used.

Show all columns and you'll be surprised at what's available, or going to become available.

BTW, you can explore yourself using Log and Transfer. Open any type of media that Log and Transfer supports and then, right click on the column header (like you would in Final Cut Pro) and select “Show all Columns”. The columns displayed will change according to the type of media selected.

So far, Sony’s XDCAM EX has the least amount of metadata and the least interesting metadata – barely more than the basic video requirements and information on the device: model and serial number.  RED footage has a lot of metadata, although most is focused on the technical aspects of the shot as you would expect for a digital cinema camera.

But take a peak at the source metadata from P2 Media! All the goodness like the date of the shoot (which FCP otherwise does not export) and time (as does RED) but also fields for ‘Reporter Name’ (awesome for a First Cuts – News product) and Latitude and Longitude. While they’ve been blank in every instance because I don’t think Panasonic are shipping any cameras with GPS built in yet, it does suggest that future Panasonic cameras are likely to contain GPS and store that data in with the media file. Anyone who’s a regular reader will know that means Derived Metadata! There are also fields for ‘Location Source’, ‘Location Name’, ‘Program Name’, ‘Reporter’, ‘Purpose’and ‘Object’ (??).

AVCCAM carries all the fields of P2, more or less, with the addition of a “memo” and “memo creator” fields.

It’s been fun exploring this ’secret’ metadata. Now to find a way to make some use of it, or make it practical. Would anyone be interested in a tool that would not only read and explore this metadata, but allow some of it to be mapped to existing Final Cut Pro fields?

The “iPhoto disaster of May 09″ is actually turning out to be good for my thinking! In earlier versions, iPhoto created a copy of the image whenever any adjustments were made. The original was stored, which explains why my iPhoto folder was almost twice the size of my actual library as reported in iPhoto. iPhoto 09 (and maybe 08, I skipped a version) does things a little differently.

When I changed images while the processor was under load, the image came up in its original form and then – a second or so later – all the corrections I’d made would be applied. It was obvious that the original image was never changed. All my color balance, brightness, contrast and even touch up settings were being stored as metadata, not “real changes”.

The original image (or “essence” in the AAF/MXF world) is untouched but there is metadata as to how it should be displayed. Including, as I said, metadata on correcting every image blemish. (The touch up tool must be a CoreImage filter as well, who knew?)

So, I’m thinking this is a different type of metadata than the five types of metadata previously identified. My first instinct was to call this Presentation Metadata – information on how to present the raw image. Greg (my partner) argued strongly that it should be Aesthetic Metadata because decisions on how to present an image or clip or scene, but I was uncomfortable with the term. I was uncomfortable because there are instances of this type of metadata that are compulsory, rather than aesthetic.

Specifically, I was thinking about Raw images (like those from most digital cameras, including RED). Raw images really need a Color Lookup Table (CLUT) before they’re viewable at all. A raw Raw file is very unappealing to view. Since not all of this type of metadata is aesthetic I didn’t feel the title was a good fit.

Ultimately, after some discussion – yes, we really spend our evenings discussing metadata while the TV program we were nominally watching was in pause – we thought that Transform Metadata was the right name.

Specifically not “Transformative” Metadata, which would appear to be more grammatically correct, because Transformative has, to me, a connotation of the transform being completed, like when a color look is “baked” into the files, say after processing in Apple’s Color or out of Avid Symphony. Transform Metadata does not change the essence or create new essence media: the original is untouched and Transfomed on presentation.

Right now we’re a long way from being able to do all color correction, reframing and digital processing in real time as metadata on moving images as iPhoto does for still images, but in a very real sense an editing Project file is really Transform Metadata to be applied to the source media (a.k.a essence).

This is very true in the case of Apple’s Motion. A Motion project is simply an XML file with the metadata as to how the images should be processed. But there’s something “magic” going on because, if you take that project file and change the suffix to .mov, it will open and play in any application that plays QuickTime movies. (This is how the Project file gets used in FCP as a Clip.) The QuickTime engine does its best to interpret the project file and render it on playback. A Motion Project file is Transform Metadata. (FWIW there is a Motion QuickTime Component installed that does the work of interpreting the Motion Project as a movie. Likewise a LiveType QuickTime Component does the same for that application’s Transform Metadata, a.k.a. project file!)

I think Dan might be right – there could well be seven or eight distinct types of metadata. It will be interesting to discover what they are.