Firstly, the distinction between facial detection and facial recognition is the difference between recognizing “oh, that’s a face” compared with “oh that’s Philip Hodgetts’ face”. That’s a huge distinction. Most digital still cameras sold today recognize faces in the image and attempt to lock focus on them. Heck, Smile Click recognizes that a subject is smiling before taking a picture, and that’s a 99c app! Facial detection is a developer framework in iOS 5, making it easy to add facial detection to any app – for tracking, changing or modifying! Sanyo have facial detection (they call if Face Chaser) in a video camera. In professional cameras, Fujifilm have released a lens with its TRACE technology that tracks faces in shot to maintain focus and exposure – up to 12 faces in a shot.

In postproduction software both Premiere Pro CS 5 and later and Final Cut Pro X attempt facial detection. Neither attempts to identify the individual, just that there is one or more people in the image. Subjectively, it’s still a work in progress. When it works, it’s great but even facial detection isn’t perfect with moving video. Final Cut Pro X further takes this useful metadata (how many people are in a picture) and attempts to infer the shot based on the size of the detected faces: big faces = closeup; many small faces = wide. (I’m sure that’s a gross simplification of someone’s very hard work, but you get the point.)

Facial recognition is valuable because it allows us to quickly group shots with the same person (or character) in them without any additional work. Taking the boring out of post, as I like to say. We’d have to identify the face once and probably deal with some false identifications, but if it gets more accurate than iPhoto currently is, it could be more useful.

I’d also suggest that it’s likely to be more useful in a video project, and more accurate, than in a general iPhoto library because of the more limited set of examples for each person. iPhoto (pre-Polar Rose technology) does a good job where it’s presented with a set of images of the same person in roughly the same time. The more examples in photos across a lifetime, and the accuracy is reduced. So, with the more limited time scale of the typical video project, I’d expect better accuracy.

Facial recognition is still very much a work in progress. In September 2010 Apple purchased Swedish facial recognition company Polar Rose, presumably to boost the facial recognition technology in iPhoto and across their entire product range (I hope!). Full facial recognition probably won’t make it to Final Cut Pro X this year, but you do have to wonder what they have planned for a “full revision” release (one we might pay for) after all the catch-up features are added. (I have absolutely no ideas, this is pure hypothesis/wishful thinking on my part.)

However, if we broaden out a little and consider the research that’s being done – the existing integration of facial recognition in social media and photo sharing sites, and how object detection/computer understanding of what it’s seeing is developing – then it’s obvious that computer recognized metadata will start to be a viable alternative.

Facial recognition

I mentioned Polar Rose, now integrated into Apple where I’d expect it will be used to improve the accuracy of facial recognition in iPhoto and Aperture, but also likely to be added to the facial detection framework now in iOS 5. As iOS and OS X are merging, I would expect the same frameworks to become available to OS X developers in due course. From GigOm - Apple’s iOS facial recognition could lead to Kinect-like interaction:

The unearthed APIs are described as “highly sophisticated,” and can determine where a user’s mouth, and left and right eyes are located, as well as process images taken by the iPhone for face detection. Aside from providing Apple an easy way to introduce Faces (which recognizes specific people in iPhoto) to both its own Photos app and any third-party apps that access that library, it should also open the door for much more advanced facial recognition applications.

Beyond Apple, facial recognition is becoming ubiquitous, even to the point where Slate’s Farhad Manjoo wrote in his “Smile, You’re on everyone’s camera“:

Soon, face recognition will be ubiquitous. While the police may promise to tread lightly, the technology is likely to become so good, so quickly that officers will find themselves reaching for their cameras in all kinds of situations. The police will still likely use traditional ID technologies like fingerprinting—or even iris scanning—as these are generally more accurate than face-scanning, but face-scanning has an obvious advantage over fingerprints: It works from far away. Bunch of guys loitering on the corner? Scantily clad woman hanging around that run-down motel? Two dudes who look like they’re smoking a funny-looking cigarette? Why not snap them all just to make sure they’re on the up-and-up?

This is absolutely a technology who’s time is coming. Further in the Slate article:

In 2006, Google acquired the biometric recognition company Neven Vision, and Hartmut Neven, one of the world’s experts in computer vision, is a respected engineer at the company. A Microsoft research team in Israel has built a fantastic app that uses face-recognition systems to search the Web for pictures of people who are in your photo album. And last year Facebook rolled out a tool that automatically suggests names of people to tag in your pictures.

The Google acquisition has now rolled out as facial recognition in Picassa.

Late 2011 the New York Times had a feature article Face Recognition Makes the Leap From Sci-Fi in their business section. In the article they list these examples:

• SceneTap, a new app for smart phones, uses cameras with facial detection software to scout bar scenes. Without identifying specific bar patrons, it posts information like the average age of a crowd and the ratio of men to women, helping bar-hoppers decide where to go. More than 50 bars in Chicago participate.
• Immersive Labs, a company in Manhattan, has developed software for digital billboards using cameras to gauge the age range, sex and attention level of a passer-by.
• Those endeavors pale next to the photo-tagging suggestion tool introduced by Facebook this year. When a person uploads photos to the site, the “Tag Suggestions” feature uses facial recognition to identify that user’s friends in those photos and automatically suggests name tags for them. And apparently you don’t need a person’s permission to tag them on Facebook.
• “Millions of people are using it to add hundreds of millions of tags,” says Simon Axten, a Facebook spokesman. Other well-known programs like Picasa, the photo editing software from Google, and third-party apps like PhotoTagger, from face.com, work similarly.

Powered by technology from Face.com (one of the key players in white label face detection) Facebook processed 400 million photographs in 30 days!

So my dream of having all the people in my source video recognized and grouped. Perhaps for manual naming (once) but more likely we’ll be able to use existing resources to match the face somewhere and associate the name with it. In a study reported in the Cloud-Powered Facial Recognition Is Terrifying article at The Atlantic:

Unlike Groucho Marx, unfortunately, the cloud never forgets. That’s the logic behind a new application developed by Carnegie Mellon University’s Heinz College that’s designed to take a photograph of a total stranger and, using the facial recognition software PittPatt, track down their real identity in a matter of minutes. Facial recognition isn’t that new — the rudimentary technology has been around since the late 1960s — but this system is faster, more efficient, and more thorough than any other system ever used. Why? Because it’s powered by the cloud.

….

With Carnegie Mellon’s cloud-centric new mobile app, the process of matching a casual snapshot with a person’s online identity takes less than a minute. Tools like PittPatt and other cloud-based facial recognition services rely on finding publicly available pictures of you online, whether it’s a profile image for social networks like Facebook and Google Plus or from something more official from a company website or a college athletic portrait. In their most recent round of facial recognition studies, researchers at Carnegie Mellon were able to not only match unidentified profile photos from a dating website (where the vast majority of users operate pseudonymously) with positively identified Facebook photos, but also match pedestrians on a North American college campus with their online identities.

At most you need one photograph of the individual, and good matching algorithms. CNET News has a background article that gives another take on the experiment identifying random people in public spaces.

It’s getting quite common:

• FTC Calls For Comments On Facial Recognition Technology
• Ubiquitous facial-recognition software is coming
• FBI rolling out nationwide face search and recognition system
• London Police Use Flickr to Identify Looters

Facial Recognition Technologies

Before moving on to other ways computers are “seeing” images, here’s a short summary of the primary technology providers.

• Google purchased PittPatt and are integrating the technology into Picassa
• Apple purchase Polar Rose and are integrating the technology widely through their OS and applications.
• Facebook relied on Face.com technology. In fact Face.com are one of the major providers of facial recognition technology with 7 Billion Photos in the last year in its Facebook apps, Face.com is now available for developers (such as Facebook, and well, even Assisted Editing) to use:

• Developers who are interested in building their own facial recognition apps can now get full access to the open Face.com API, free of charge. That basically means developers can tap into Face.com’s face detection and recognition technology and create brand new ways for friends to engage through photos at zero cost. Hard to beat that offer.
• It is also well funded: “Yandex, operator of Russia’s largest search engine, has invested in Tel-Aviv based facial recognition technology startup Face.com, marking its first investment in an Israeli company. In total, Face.com has raised$4.3 million in Series B funding in a round led by previous investor Rhodium.”

I want to draw special attention to Affectiva who use the tag line “Respectful emotional measurement and communication.” Yes, they read emotion from video.

The software “makes it possible to measure audience response with a scene-by-scene granularity that the current survey-and-questionnaire approach cannot,” Mr. Hamilton said. A director, he added, could find out, for example, that although audience members liked a movie over all, they did not like two or three scenes. Or he could learn that a particular character did not inspire the intended emotional response.

I fully expect facial recognition to come into the postproduction world rapidly and provide useful metadata. Automatically identifying and labeling people in video content will be very empowering. I do foresee an issue with *automatic* identification with actors and roles in narrative, but the software would surely have the ability to manually tag every instance of “this identified face” with a character name instead of the actor’s name sourced from IMDB!

If software can detect emotional responses to movies, it can detect emotional performances and – for documentary/reality/news – detect the emotion in the face to help drive editing.

Oh, if you really want to avoid facial detection? Social Beat has a few tips.

Metadata for asset management

Both DigitalSmiths and Asterpix have tools specifically intended to create visual metadata automatically for asset management and exploitation.

Asterpix

The actual process of machine-tagging involves pulling in imaging data from the video clip and matching it up to whatever text was included by the video’s creator. Nat Kausik, CEO of Asterpix tells me the process is a little similar to Google’s search algorithm in creating relevancy based on what bits of parts of the video get the most screen time. For example, in a video of a someone walking through a grocery store there would be a wealth of information about other products and people, but if you’re focused on that one person for the majority of the clip the engine will pick on it and react accordingly.

DigitalSmiths

Digitalsmiths is the technology leader in the rapidly growing segment of video search and recommendation. Digitalsmiths Seamless Discovery™ has revolutionized the accuracy and ease with which end-users find relevant, personalized entertainment across multiple channels and devices.

From powering tablet-enabled set-top-boxes, to live integration of sporting events, to first-run premium web-content, Digitalsmiths has deep experience and proven results for increasing engagement and viewership though our unique, holistic solutions. Our solutions serve customers that span across all media channels and devices and reach a combined audience of millions of consumers.

Generating other visual metadata

While we’re only just seeing facial detection and recognition rolling into useful applications, but already computers are being taught to read the text in images, and process the images themselves to recognize people, place and things.

Text recognition isn’t exactly new, but:

Google Goggles Can Now Read Print Ads. Oh, And Play Freaking Sudoku!

OCRKit 1.2 – The simplest Text Recognition for the Mac

Beyond Text

Computers That See You and Keep Watch Over You

A New York Times article outlining how computer vision is being used:

Perched above the prison yard, five cameras tracked the play-acting prisoners, and artificial-intelligence software analyzed the images to recognize faces, gestures and patterns of group behavior. When two groups of inmates moved toward each other, the experimental computer system sent an alert — a text message — to a corrections officer that warned of a potential incident and gave the location.

“Machines will definitely be able to observe us and understand us better,” said Hartmut Neven, a computer scientist and vision expert at Google. “Where that leads is uncertain.”

The applications are quite amazing. From observing prisoners, to reminding hospital staff to wash their hands if it’s detected they haven’t, to a host of other uses, smart computer software is observing us and making accurate observations.

Google Experiments With Next Generation Image Search

Notably, the new image search technology doesn’t just index text associated with an image in determining what’s in it. Google is now talking about using computers to analyze the stuff in photos, and using that to associate it in a ranked way with keyword queries. In effect, they’re talking about something similar to PageRank for images (but without the linking behavior).

Teaching Google To See Images

Nuno Vasconcelos, a professor of electrical engineering at the UCSD Jacobs School of Engineering, discusses the approach, called Supervised Multiclass Labeling (SML), in a recentnews release from the school (hat tip to Threadwatch for the pointer). Though SML sounds like a mouthful of jargon, what it really amounts to is systematically training a computer to recognize statistically similar objects, and teaching it to differentiate them from other objects that have similar characteristics.

Google Researchers Teach Computers Out How To Recognize Images Of Famous Landmarks

In the experiment, the researchers fed “an unnamed, untagged picture of a landmark” found on the Internet and the system would spit back the name and location of the landmark, such as the Acropolis in Greece. Each untagged photo was be compared to 40 million GPS-tagged images on Picasa and Panoramio (both owned by Google), as well as related photos found through Google Image Search. Using clustering and new image indexing techniques, the Google researchers were able to identify untagged photos of the same landmarks from different angles and under various lighting conditions.

The researchers report that their system can identify 50,000 landmarks with 80 percent accuracy. I’m not sure that’s quite good enough to even roll that out in a beta product, but if Google can get it to 90 percent or 95 percent that would start to be consumer-friendly.

Researchers from MIT’s CSAIL teach computers to recognize objects

The system uses a modified version of a so-called motion estimation algorithm, a type of algorithm common in video processing. Since consecutive frames of video usually change very little, data compression schemes often store the unchanging aspects of a scene once, updating only the positions of moving objects. The motion estimation algorithm determines which objects have moved from one frame to the next. In a video, that’s usually fairly easy to do: most objects don’t move very far in one-30th of a second. Nor does the algorithm need to know what the object is; it just has to recognize, say, corners and edges, and how their appearance typically changes under different perspectives.

The MIT researchers’ new system essentially treats unrelated images as if they were consecutive frames in a video sequence. When the modified motion estimation algorithm tries to determine which objects have “moved” between one image and the next, it usually picks out objects of the same type: it will guess, for instance, that the 2006 Infiniti in image two is the same object as the 1965 Chevy in image one.

SeeIT.com

 SeeIT.com is in beta while the company is scaling the index from millions to hundreds of millions of images. You can try it by clicking here, then entering the user name picture and password picture93AE (exclusive access for Search Engine Land readers). See thisinformation for new users for more information, including some of the limitations of the current beta release.

Riya

Riya started out focusing primarily on facial recognition, but now has a beta visual search that lets you find similar faces and objects on many images across the web and then refine your results, using color, shape and texture.

Riya also powers the visually oriented product search service Like.com that lets you find clothing and a few home furnishing items based on visual similarity. Like also has a “celebrity” search that lets you see what the stars are currently wearing and find similar accoutrements for your own adornment.

DARPA building search engine for video surveillance footage

According to a prospectus written in March but released only this month, the Video and Image Retrieval and Analysis Tool (VIRAT) will enable intel analysts to “rapidly find video content of interest from archives and provide alerts to the analyst of events of interest during live operations,” taking both conventional video and footage from infrared scanners as input. The VIRAT project is an effort to cope with a growing data glut that has taxed intelligence resources because of the need to have trained human personnel perform time- and labor-intensive review of recorded video.

Check out the “simple’ diagram accompanying the article. This stuff isn’t simple.

VideoSurf: New, Genuinely Radical Video Search

VideoSurf is a computer vision search engine that processes all of the kinds of information most video search services do, but then goes a step further, applying a proprietary process using “multigrid fast computation” and some heavy-duty computer processing power to analyze videos, identify people, and extract all kinds of additional information directly from the video itself. Until I saw the demo, I thought this type of technology was still years away.

Apple wins patent on 3D object-recognition technology

 The USPTO has awarded Apple a patent on 3D object-recognition technology that goes well beyond the current face recognition already included in apps such as iPhoto and the iOS 5 camera application, allowing a device to “build” a 3D face or object by analyzing the curves, contours and shadows of a 2D image. Such technology would give Kinect-like detection and recognition capabilities to cameras such as those found in iOS and Mac devices.

Another benefit of the Polar Rose acquisition, where the technology behind this patent was developed.

Developer APIs

Kooaba

The Swiss company aims to unlock its library of over 10 million images, ranging from album covers to books and movie posters, and provide access to all that precious data via the cloud.

Kooaba hopes that the launch of the API will trigger third-party developers to develop more mobile applications –iPhone and Android versions exist already – or tools that tap into social networking services like Facebook and Twitter, etcetera.

OpenCV

OpenCV (Open Source Computer Vision) is a library of programming functions for real time computer vision.  It has C++, C, Python and soon Java interfaces running on Windows, Linux, Android and Mac. The library has >2500 optimized algorithms.

Out at the Leading Edge of technology

New Image-Recognition Software Could Let Computers ‘See’ Like Humans Do

Using such small amounts of data per image makes it possible to search for similar pictures through millions of images in a database, using an ordinary PC, in less than a second, Torralba says. And unlike other methods that require first breaking down an image into sections containing different objects, this method uses the entire image, making it simple to apply to large datasets without human intervention.

Developing artificial intelligence systems that can interpret images

 Torralba is also attempting to develop systems that can scan a short video clip and predict what is likely to happen next, based on what people or objects are in the scene. To do this, the systems will need to understand what actions each object or person in the scene is capable of making, and what their limitations are. This will allow the systems to make predictions about what each of these entities is likely to do in the near future.

We already have facial detection in our software, and identifying the person is definitely coming. There are technologies to recognize a smile in a cheap iPhone app, or recognize human emotion currently exploited for focus group work. There are computers patrolling prison yards and making sure doctors and nurses wash their hands between patients. There is no doubt in my mind that pre-edit processing will give editors name, context and emotion metadata. And smart companies, like us, will exploit that as input for automating certain editing tasks.

Essentially Prelude is an ingest, logging and rough cut tool. It’s designed to make it easy to add log notes (a.k.a. metadata) quickly and easily, then perform a rough first string out to send to a craft (skilled) editor for the real work. Now obviously, any tool that encourages the entry of metadata is good by me!

So far, there’s not page for Prelude I can link to but from what I remember the key features are:

• Works directly off non-tape media
• Media can be previewed and selects trimmed before ingest to Prelude
• Icons can be “skimmed” (I think Adobe has a different term) FCP X style.
• Will copy media from the card to a specified location
• Will (optionally) do proxy (or really any specified format) generation in parallel with ingest
• Has great tools for subclipping and adding log notes easily
• Can build simple cuts-only timelines
• Exports direct to Premiere Pro or to Final Cut Pro 7 XML.

That’s about what’s known right now. Given that Al finished off by saying “See you at NAB” I can only guess we’ll learn more then (or perhaps before).

Adobe's Al Mooney previews Prelude. Shown here is the simple timeline assembly tool.

Now, before I start let me be clear. I am NOT saying timecode is unimportant. I’m NOT saying that timecode is passé and suddenly irrelevant. Timecode remains incredibly important for any tape based access.

What I am saying is that text search – or phonetic search derived from text – is becoming a highly viable, and in many ways superior, way to search and find content. Timecode’s primary role was in being able to identify any given frame from a tape by tape and frame number. There’s nothing wrong with that approach, but as humans we don’t think in “reel and Timecode”, which is why text is a superior option.

In this technology summary, I’m going to consider:

• What tools are at our finger tips right now and their relative merits,
• Why speech transcription is ultimately more valuable than phonetic search (long term),
• Developments in speech transcription, and
• Transcription technologies.

Software and tools available now

There are two broad approaches to text search in the marketplace today: those that transcribe the speech into text and those that use phonetic search. In the first category we have Adobe Premiere/Audition/Soundbooth using Autonomy’s technology to transcribe speech to text. In the second we have Avid and Boris using Nexidia’s phonetic search technology in Media Composer (for Phrasefind and Script Sync) and Soundbite (formerly Get!) respectively. Our own prEdit edits video using a text transcript.

Phonetic search makes no attempt to understand meaning. Essentially, Nexidia’s technology “understands” what an audio waveform would look like when the text is input. Nexidia scans all the audio files ahead of search and indexes them for search. When you input a word, it estimates the waveform then finds matches for that waveform by comparing the index with the target waveform. This technique is great when you don’t have a transcript, or have a transcript without any time stamps.

This technology is also used to align a text file (script) with the audio in the audio and video files in Avid’s ScriptSync with the optional ScriptSync package. This also works great if you have a transcript of interviews to align the audio in the video with the transcript. However, if you have a transcript available, working with prEdit may be faster, manipulating the audio and video by editing and modifying text.

Going the other way – from speech to text – provides an additional advantage in that we have meaning associated with the text, and we have a transcript that carries through production into distribution. If you want searchable text for distribution then Adobe’s professional tools are the only automated tools: phonetic search isn’t viable because it would require distributing the Nexidia engine into distribution (and with their licensing model, I don’t see that happening any time soon).

However, the speech to text engine used by Adobe (licensed from Autonomy) is still a work in progress: results can be quite good but the average result is less spectacular, sometimes completely useless. That is why Adobe have added, and strongly pushed, the ability to provide the speech transcript engine a guide script. Surprisingly, a guide script alone – even one where it is an exact transcript – does not transcribe perfectly. The best results require a trip via Adobe Story. This has the advantage of keeping all punctuation and paragraph breaks  (and automatic subclipping into paragraph subclips in prEdit). For a comparison on accuracy you might be interested in Colin Brougham’s comparisons.

The big disadvantage to this approach is that it requires a transcript, the very thing most people want to automate because of the cost.

Why speech transcription is more valuable than phonetic search

Speech transcription carries meaning. Phonetic search does not carry meaning. This is an important distinction, because it means that speech transcription is valuable metadata as well as a production tool, while phonetic search is a useful production tool, but has no value as metadata away from the Nexidia engine.

Speech transcription can be carried with the media throughout its life, even into edited versions. Indeed, this is Adobe’s intent. They tend to focus on the speech transcript metadata as part of a distribution strategy more than its use in postproduction. The speech transcript metadata is carried inside media files as XMP metadata. (There are other alternatives for speech transcript metadata in distribution files, but I’ll discuss that further down.)

Speech transcription can be searched by anyone, at any stage, without needing a proprietary engine.

Speech transcription can be used to derive keywords and other expressions of meaning, which is valuable not only for automating some types of production (some types folks, only some) but extremely valuable as metadata for later finding content.

Transcribed speech is the input to prEdit. Briefly prEdit allows you to easily add metadata (log notes), break interviews into thought segments and eliminate less useful material, before searching and building a story by dragging and dropping text blocks. Editing can continue through the story building process and narration added (converted to voice instantly). At any time you can preview a clip or clips, the full story, or any selected part of the story before exporting to Final Cut Pro 7 or Premiere Pro CS 5.5 or later.

So, while phonetic search is a great post-production tool, transcription into real text has a wider range of uses and uses outside post-production. The only trouble is, speech transcription is expensive!

Developments in speech transcription

It’s a great time to live if you’re interested in speech transcription. The most significant developments have been hybrid computer-human approaches used by 3PlayMedia, SpeakerText and SpeechPad to reduce the cost and time of transcription. These companies particularly are focused on the need for transcription for video in distribution, but are excellent choices for transcriptions for prEdit or other postproduction needs.

Until we get a fully automatic speech transcription with adequate accuracy, these will help. Even with human correction some uncommon words or names will not be transcribe accurately.

Transcription technologies

I’ve talked about Nexidia and the phonetic search technology and why my long term preference is for speech transcription, so it’s no surprise that I spend some time following what’s happening with the technology. What is interesting to me is that the two companies who are generally recognized as having the most accurate under the widest range of conditions are not (yet) available for postproduction work.

Google has been amassing huge numbers of examples of speech for recognition – an important first step to accurate speech recognition – via the (now defunct) Goog411 initiative. Google have been using this technology for automated captioning for YouTube videos and for voicemail transcriptions within Google Voice. As a Google voice customer I’d say that the results are definitely much better than the attempts by Vonage (laughable) and comparable or better than Premiere Pro’s use of Autonomy.

More information on Google’s voice recognition plans can be found in this Techcruch article with Mike Cohen, head of Google’s Voice recognition efforts. Also interesting to note is that Google is slowly opening up an API for their speech recognition efforts, starting with Chrome version 11. How open that is for third party developers to use, remains unknown, but it’s an interesting direction from the search giant. If the API became open, and this is one of the two most accurate speech transcription technologies, why wouldn’t savvy developers like us, start to use it and integrate it into our software?

Equally prominent in the speech transcription/recognition community is Nuance, probably best known for powering Dragon Dictate, Dragon Naturally Speaking (and the variations) and the speech recognition component of Apple’s Siri technology. (Siri adds a lot of powerful tools on top of this basic recognition layer, but if the speech isn’t recognized accurately nothing good can come from it downstream.) There is no public API for any of Nuance’s technology (nor Siri for that matter). Nuance tends to do direct deals with companies who want to license its technology – a fairly standard practice in the technology world.

My fondest hope is that Apple’s license from Nuance will be extended to OS X and a speech recognition framework be included in OS X for developers. It’s a fond hope, not anything real!

Google and Nuance have the most accurate technologies that do not require speech training. I’m a Dictate user but that product uses training to obtain high (very high) accuracy in transcription. To be able to be accurate without needing training is what we need for interview transcriptions for post.

Tip: If you do have Dictate or any of the PC variants of Nuance’s products, one technique is to listen to the interview and speak it with your own (trained) voice. I’m not yet there, but it is possible to speak-as-you-hear (the basis of the common ear-bud presenter trick) for fast, accurate transcription.

Beyond the giants, there are many other technology companies, or open source projects, in the speech recognition field that are worth mentioning. One thing that should be noted is that most of these technologies are cloud based (as is Apple’s Siri). They work as long as they have a connection to their primary servers.

I’ll let these folk speak for themselves. Whichever technologies prevail, we’re definitely seeing a surge in the accuracy and flexibility of speech recognition, that is going to factor in post production in the coming years, beyond where we are at right now.

SeedVocal SDK

Speech Recognition for your iPhone application. The claim is that the technology is:

• currently unavailable from Apple APIs
• easy to add to your application
• convenient for all types of apps: games, fun and promo applications or utilities
• suitable for the iPhone and the iPod Touch
• cheaper than you might expect.

Sphinx4

Sphinx-4 is a state-of-the-art speech recognition system written entirely in the Java^TMprogramming language. It was created via a joint collaboration between the Sphinx group at Carnegie Mellon University, Sun Microsystems Laboratories, Mitsubishi Electric Research Labs (MERL), and Hewlett Packard (HP), with contributions from the University of California at Santa Cruz (UCSC) and the Massachusetts Institute of Technology (MIT).

Sphinx-4 started out as a port of Sphinx-3 to the Java programming language, but evolved into a recognizer designed to be much more flexible than Sphinx-3, thus becoming an excellent platform for speech research.

Speech Recognition with Javascript; speechapi.com

With speechapi.com’s javascript API, it is possible to build interesting speech-web mashups that include both speech-to-text as well as text-to-speech.

A combination of several technologies and open source tools make this possible. In the browser, Flash is used to access the microphone and stream the audio to an RTMP server. Red5 is used because its a versatile media server that has the benefit of being open source and free.

Open-Source Large Vocabulary CSR Engine Julius

“Julius” is a high-performance, two-pass large vocabulary continuous speech recognition (LVCSR) decoder software for speech-related researchers and developers. Based on word N-gram and context-dependent HMM, it can perform almost real-time decoding on most current PCs in 60k word dictation task. Major search techniques are fully incorporated such as tree lexicon, N-gram factoring, cross-word context dependency handling, enveloped beam search, Gaussian pruning, Gaussian selection, etc. Besides search efficiency, it is also modularized carefully to be independent from model structures, and various HMM types are supported such as shared-state triphones and tied-mixture models, with any number of mixtures, states, or phones. Standard formats are adopted to cope with other free modeling toolkit such as HTK, CMU-Cam SLM toolkit, etc.

I have no idea what that means either!

Scripto

Scripto is a light-weight, open source, tool that will allow users to contribute transcriptions to online documentary projects.  The tool will include a versioning history and full set of editorial controls, so that project staff and manage public contributions.  The design and development of the tool is being supported by grant funding from the National Endowment for the Humanities, Office Digital Humanities, and the National Historical Publication and Records Commission.

Encoding All

From an audio or video media, transcription can automatically generate a file ofsubtitles, the keyword list in XML format and the entire plain text.  (That’s the translation from the French – speech recognition is not limited to English!)

And interesting, although not currently speech recognition:

Soundhound

Currently, SoundHound’s specialty is delivering information about music. Users can sing or hum a tune into its SoundHound app and the app returns the song name, as well as other information. Last week, the company released its Hound app, which can identify when a user says the name of an artist or album.

The slightly frivolous-seeming “name that tune” aspects of SoundHound’s applications belie the seriousness of the technology and business underneath it all. SoundHound has raised $16 million in venture capital and currently has 55 full-time employees. Investors have been attracted to the company by the future potential of SoundHound’s core technology, Mohajer told me. “We own all of our technology, while a lot of other apps in this space license their core technology,” he said. “We built everything in-house and we own all of our intellectual property.

Many thanks to my friend Don Berube for pointing this out.  While the headline slightly overstates the case, it’s clear we’re heading for an era when computers in general will understand meaning and the content of images. 

Face Detection and Recognition is now prevalent in digital cameras which can detect many faces and even recognize those you tag, giving them preference as to focus and exposure. But Antonio Torralba is more concerned with what’s not the face in a photo: his Artificial Intelligence (A/I/) software tags all objects in an image and makes the image more easily searchable so as to be able to place the photo in an appropriate context. This ability will eventually mean that robots can recognize their surroundings, say in a house or office building, based on what furniture and objects they see around them.

Going one step further with this research, Torralba is also developing systems that can scan a short video clip and predict what is likely to happen next, based on the people and objects in the scene. This should eventually allow robots to anticipate how their actions will influence future events.

Seems like magic to me, but I seen enough of these type of articles – fully acknowledging they’re at the cutting edge of research and many years from practical application in an editing situation – to know that the future will be computer aided way beyond what we have now.

I’ve long said that the application of at least basic editing algorithms (such as in our First Cuts software) isn’t that difficult being largely a matter of modeling the behavior of human edits and what drives their decisions. The problem is having the source information – metadata if you will – to feed the algorithm. Speech transcription, facial detection, emotion detection, and now software to tag all objects in the image and predict what will happen next all point to a much more computer-assisted editing future than most expect.

Facial Recognition App Detects, Captures Smiles Technology intrudes more & more into “human” territory

Meet Swivl, The Motion Tracking iPhone Dock That Always Keeps You On Camera More and more automatics!

Now, it would be really cool if Swivl tracked you and kept you on camera using facial detection but it does not: instead it uses a hand held transmitter/controller to “know” where to point the camera.  Even with that it will make a great addition to a video blogger, web episode producer as the producer/talent can move and have the camera follow them as they do.

The SmileClick App is much more interesting:

SmileClick is a newly launched, unique application that uses proprietary facial recognition software to detect how big a photo subject is smiling. Once the app determines that the person to be photographed is sporting a large, genuine smile, it automatically snaps the picture at the right time, helping to create happy memories.

Waltham MA based  Affectiva takes it one step further:

…has facial recognition software that can accurately determine the difference between a happy smile, an embarrassed smile and a smirk!

As well as other emotions. Currently used only in focus groups and the like, wouldn’t it make a great source of metadata if, as well as the person’s face being (accurately) identified, there was another tag “smiling”, “embarrassed”, “annoyed”, et. al. I see it as inevitable but a few years off.

Following on the heels of yesterday’s post about facial recognition in the cloud here’s information on how Apple are applying the technology they gained when they acquired Polar Rose last September, at least within iOS frameworks.

When coders dug through Apple’s beta versions of iOS5 they found what were deemed to be “highly sophisticated” API systems that let an iPhone automatically track eye positions and mouth positions (so the angle to the user, and possibly where their attention is being directed could be calculated) as well as passing key data on to a face recognition algorithm that would be accessible to all apps…not just Apple’s own.

Combine this with the Nuance-licensed voice recognition technology in Siri – also new with iOS 5 and iPhone 4S – and we have the foundation of a very powerful metadata generation system that would automate naming people in clips and form the basis of speech transcription and then keyword extraction.

In my dreams these are technologies that will come to Final Cut Pro X 10.2 or 10.3 in future years.

Apple teases us with Audio Role metadata that seems to have no current use within Final Cut Pro X.

If you open the Info Pane and select the Info tab, you will see the pop-up menu in Figure 20.1. showing the Roles

One magic way for this to be useful would be as a solution to the missing audio output options. In the near-magic, near-future I expect that a future version of Final Cut Pro X will use this Audio Role metadata to route audio outputs. At version 1 Final Cut Pro X’s audio output options are very basic, and there’s improvement coming, for sure.

So you can imagine how excited I was to open Final Cut Pro X 10.0.1 and see that those fairly limited audio roles (no doubt implemented specifically to enable export to OMF/AAF) had evolved into not only customizable Audio Roles, but to have the same option available for Video clips as well. And a Title Role comes rolled in!

Display the role, or the clip name.

Commenter Andrew Richards nailed it! Second commenter Marcus R. Moore also called out a real need. Well in the new release, not only can you use Roles in a Project to enable or disable clips assigned that Role, but selecting a Role highlights all Clips in the Project that have that Role assigned (either Audio or Video). Marcus kind of also gets his wish, because there’s a new option in the Project View settings to display the Role on clips, as an alternate to displaying the Clip name. It’s a non-destrcutive toggle.

There can be different Roles assigned for Audio and Video on the same clip and Roles can have sub-roles.

When selected all shots tagged with B-roll highlight in the Project. They can be deactivated temporarily with a single checkbox. (click to enlarge)

In fact, I think the implementation of Roles in Final Cut Pro X really completes the metadata story, and makes the Magnetic Timeline more valuable.

Here’s some of the ideas we’ve been thinking of:

• Multi-language Titles. Each language is a Role that is applied to the Title. With a single click on a checkbox, all titles from a language set can be enabled or disabled.
• Identifying A-roll and B-roll so one can turn off all B-roll to see the underlying A-roll story.
• Being able to selectively turn on or off the tracks that make up the audio mix, because now we can not just assign a Role of “SFX” or Dialog, but audio could be tagged with a Role that was, for example “Background Farm noise submix”. When (if) Apple implement a mixer in Final Cut Pro X then those clips could be assigned a single fader by Role name. Much smarter than track number mixing.

Roles now appear in the Event Browser’s list view as an optional column. This also means they are now available as a filterable field in the Filters window, or for Smart Collections.

I think (i.e. my opinion) that Roles provide all the advantages of using Tracks as de-facto metadata by making the job explicit and allowing us to use that metadata in a Smart Collection.

In other Final Cut Pro X update news

I truthfully have been focused on only part of the new version – the XML import and export specifically – so that we can have a new product to be announced at LAFCPUG @ DVExpo on Wednesday night, so I haven’t explored the other changes too much. I did go through every menu and dialog to find differences, but I’m sure this is not comprehensive.

Xsan is now officially supported for shared storage. At this point it’s shared storage of Events or Projects, not shared Events or Projects. Each Event or Project can only have one user active at a time.

Trial version. Check it out for yourself without commitment or cost.  Apparently fully functional but time limited, so you can even finish something for yourself.

Improvements to Share with better movie export options (from memory)

API for camera manufacturers to add their own format support. So, if we expect or want Alexa RAW or R3D native, then those companies apparently will be stepping up to make it happen. Definitely an improvement though, because users should not have to wait for a Final Cut Pro X release in order to get support for a new format. The manufacturer can develop it themselves and have it available when their camera is launched.

This release was pretty much what I expected: XML and at least one other banner feature, a lot of tweaking and (presumably) bug fixes.  They’ve knocked off a good number of features promised in the earlier FAQ, and Roles are an adequate (I say vastly better) way of achieving the same goal as track layouts did.

So, it seems they’re on the right path.

UPDATE: Turns out there’s more to Roles than meets the eye. Not only can you enable/disable particular roles in the Final Cut Pro X timeline, but there are new export options. The Export Media dialog now has options for exporting “Roles as Multitrack QuickTime Movie”, “Roles as Separate Files”, “Video Roles Only as Separate Files” and “Audio Roles Only as Separate Files”.

UPDATE 2: Here’s the knowledge base article listing the new features and bug fixes.

I think the use of tracks has evolved since then. One of the reasons that people became concerned when Final Cut Pro X took away the traditional track layout was simply because those tracks have become valuable metadata. A typical audio example might be

• Audio 1-4  Dialog allowing for stereo channel overlap
• Audio 5-8 Music, allowing for overlapping stereo
• Audio 9-12 (or more) Spot Effects.

That’s an indicative example, not prescriptive.

Similarly in the video tracks I’ve used layouts that had:

• On-camera footage on Video 1
• B-roll on Video 2 (and 3)
• Titles on Video 4 (and if there were more than one language, one track for titles in each language).

With that configuration one could easily turn off b-roll, or titles, to view the underlying video. Or turn on only one language track of titles at a time.

Some usage was as a workaround for the lack of something like Final Cut Pro X’s Auditions feature, but that’s not really a metadata-focused use and it doesn’t count here.

Simply because they were there, tracks evolved from being there, without specific purpose, to being used to carry very valuable metadata.

As expected, metadata is going to be a dominant topic at IBC, whether identified as such, or under the heading of Media Asset Management (MAM), because all media management is done using metadata, which is why I’ve combined them in my Using Metadata For Production and Asset Management day at DV Expo. (Check out my other sessions at DV Expo.)

Philip Hunter previews where metadata is heading, with a surprise resurgence of interest in MPEG-7 or the principles involved, at least.

MPEG-7 evolved from ideas developed in the IT world of object-oriented programming, where an objective was to describe different types of data including images and sound. Video did not really figure then, so MPEG-7 picked up some of the object-oriented ideas and took them further by adding a temporal dimension for the moving image. This led to the creation of nine subclasses to categorize content within an AV sequence. Spatial and temporal segments are categorised separately and then combined as necessary

I promise, I will translate these concepts into language people can understand – at least people who aren’t high level geeks, but still need to understand and use these ideas.

Hunter also reiterates the differing needs between metadata in the production phase, and what’s needed for distribution. He also notes the cost of producing metadata and therefore the necessity for moving metadata between use cases.

Finally he previews the vendors who will have new metadata or MAM related products at IBC.

This is a long, and not new, article that rambles through a fascinating story of how a lawyer, “Mr Pink”, “Mr Brown” and “Mr Bootstrap” collectively cracked the code for predicting the profitability of movies, TV shows and (separately) another team shows the likelihood of whether a song is going to be a hit.

The specifics of how they achieved both breakthroughs is interesting: have the computer software (usually some sort of neural net) analyze existing successes – music or movie.  It then analyzes new music or movie proposals to determine whether it is likley be be a hit (music) or how much money it will make at the box office (movie).

The head of hit-predictor Platinum Blue, Mike McCready claims:

Record executives have tended to be Humean: though they can tell you how they feel when they listen to a song, they don’t believe anyone can know with confidence whether a song is going to be a hit, and, historically, fewer than twenty per cent of the songs picked as hits by music executives have fulfilled those expectations. Platinum Blue thinks it can do better. It has a proprietary computer program that uses “spectral deconvolution software” to measure the mathematical relationships among all of a song’s structural components: melody, harmony, beat, tempo, rhythm, octave, pitch, chord progression, cadence, sonic brilliance, frequency, and so on. On the basis of that analysis, the firm believes it can predict whether a song is likely to become a hit with eighty-per-cent accuracy.

The emphasis is mine in case you didn’t pick that the software was four times more likely to predict a hit song than experienced record company executives – the people who make final decisions on these things! (Quite separately, how does anyone with a 20% success rate keep their job? Just asking.)

What is interesting is that it takes no “industry knowledge”:

McCready didn’t care about who the artist was, or the cleverness of the lyrics. He didn’t even have a way of feeding lyrics into his computer. He cared only about a song’s underlying mathematical structure.

Similarly the movie profitability prediction software was built without regard to “artistic” elements at all. It picks up on plot points, settings, and other aspects to determine – based entirely on what’s come before – how successful a movie will be.

Both technologies can be used to fine tune and re-predict the likelihood of a success. Now, I have all sorts of concerns about the application of the technology and how it might “force out” small, niche movies or music that wouldn’t make the hit status but would otherwise still be a satisfactory story and a profitable project.

In both instances it is metadata at the foundation of the input to the software: for music metadata about mathematic relationships within the music; for movies the details of plot points, setting, story, characters, etc. It’s another way that metadata makes the data more valuable by fine tuning the music or the plot/characters to make the project more profitable. Not exactly classic metadata!

By the way, the first half of the article I linked to above is the most salient – mostly in the middle. The end is a verbatim account of a meeting and doesn’t really go anywhere. I kept looking for the conclusion.

And don’t forget, if you’re interested in Metadata, I have a book – Conquering the Metadata Foundations of Final Cut Pro X – and an upcoming full day seminar at DV Expo – Using Metadata for Production and Asset Management coming up on September 21.

9-5 September 20 Basic Tech for Producers (and recent Film School Graduates)

In this session, technology expert and DV magazine contributor Philip Hodgetts will cover the technological choices in production and post in a non-geeky way to help producers — and others without a technical background — make good technology choices for their productions. From formats to software choices; selecting cameras to creating Web video; designing graphics that will work and much more.  PRICE: $195 ($245 after Aug 31) Click here to register now.

9-5 September 21 Using Metadata For Production and Asset Management

Metadata is becoming increasingly important throughout the production cycle–from camera to asset management. In this session learn about the types of metadata in use; how each major NLE (Final Cut Pro 7, Final Cut Pro X, Premiere Pro CS 5.5 and Media Composer 5.5) handles metadata and how we can use that metadata to speed postproduction and VFX. Once post is done, assets need to be management through through distribution and repurposing. What tools are available, how are they used and how do they fit into the metadata structures promoted by SMPTE and other standards bodies.  PRICE: $195 ($245 after Aug 31) Click here to register now.

9-5 September 22 Avoiding Postproduction Nightmares

Post expert and DV magazine contributor Philip Hodgetts details the most common (and costly) problems inadvertently created during production that will be “fixed in post.” From color correction to audio, and editing to the final QC pass on deliverables, he’ll not only reveal the tricks of the trade that he’d use to save your production, but also explain how you can avoid these costly issues in the first place. PRICE: $195 ($245 after Aug 31) | Click here to register now.

In the light of full disclosure, I certainly expect to be paid but I always deliver good value. There will be some overlap between the Basic Tech and Avoiding Postproduction Nightmares sessions as they both seek to make the technology understandable, but with a different focus to each day’s class.

The article is about ubiquitous facial recognition spurred by a report of a new app for police that allows facial recognition at five feet away. To be clear, many consumer still cameras, and some software, does facial detection: that is there is one or more faces in the picture. Some even recognize when people are smiling, but they do not identify the individual. Apple’s iPhoto and others try and do facial recognition but my experience to date is that it’s been very hit or miss. Apple purchased a Swedish company last year to improve it’s facial recognition technology.

Clearly others already have better technology and it has been pitched for law enforcement work for a long time. However, it’s the postproduction implications that interest me. If we can have a software tool identify all the people in our footage, at lest to the stage of identifying each instance of the individual. Reading through the article it is likely the name could be discovered or derived from Facebook or other social network or public record. At worst the person would need to be manually named only once.

For a metadata-based application each clip could be tagged with the person’s ID for as long as they’re in the shot.

We would end up with ‘bins” for each individual.

Identifying people in shots is Derived Metadata and then can be used as input into other smart algorithms to take more of the boring out of post.

There are a lot of other interesting applications and implications of this increasingly popular (and capable) technology.

Apple provided me with early access preview to Final Cut Pro X in a private meeting on June 13 – in the morning.  That afternoon and next morning I read the manual cover to cover (while this shiny new loaner Four Core i7 laptop that Apple had loaned me laughed at me).  Then I explored it as much as I could that day and next morning.

Thursday I outlined the chapters and worked out how to integrate what I already had from the Mastering Metadata webinar I taught for Larry Jordan a few months back. Revised it became the second half of the book – the deeper background foil to the first half’s very practical approach.

Friday morning I started writing; continued Saturday, Sunday and Monday and finished late afternoon Monday. A cover still needed to be designed, and some graphic work done, and of course it needed to be proof read, a task ably performed by my partner in most things, Greg Clarke.

What’s interesting is the parallels what seems like a design philosophy behind Final Cut Pro X. From what I’ve done in Final Cut Pro X, it seems to me to encourage a much more polished project along the way. It’s as easy to add an animated, high production value title as it would be to add a placeholder to remind you to do it later, as I’ve always done.

With the way clips (and titles) stick to other clips, I feel we’re encouraged to add polish as we go, by making ti no extra pain to add the polish: a little color touch up; maybe a reposition; animate some clips. These are all so easy in Final Cut Pro X that I have to think this is part of the design philosophy.

What’s that got to do with the book? Every other book I’ve written or attempted to write, the draft was written in a Word processor and careful track had to be kept, with strict formatting guidelines, of what images were being added later in layout. It’s not until much later in the process, after the text is finalized, that it goes to layout.

That process wasn’t going to work in this case, so as I wrote on my computer, I would have Apple’s loaner beside me, snap a screen shot with the built in tools that are not perfect but functional. Rename it, drag it to iChat and dropped it via Bonjour to my own laptop. Then dragged the imaged directly from iChat to Pages and into the layout. It turned out to be a very efficient workflow.

When I needed to take a break from that writing, Greg would review what I’d written, and it’s much more precise and readable because of it. Those would be the times I wrote last week’s blog posts, among other things.

Greg also, in parallel set up the book page, and all the consequent eCommerce stuff.

After I finished on Monday, I started on the cover design, which took a couple of hours while Greg gave the new material a revision before we both had one final read through and check for layout issues.

I left the final version as a PDF with cover for Greg when I fell asleep around 1:30 Tuesday morning, and it was on sale by 9:30 am.

Then the cover design got a slight revision and it was off to upload to Amazon Createspace and prep it for printing. And today, five working days after I finished writing, printed copies are on my doorstep.

It’s a wonderful time to be alive, don’t you think?

Given that Final Cut Pro X is so heavily built on metadata, I thought this article was relevant. Although Godin is talking about the metadata we leave behind as we work or play across the Internet (and our purchase history), it helps with the understanding of what metadata is, in a broader sense.

In any case the conclusion is very relevant:

Data about data is more important than ever, and being on the side of the person creating that data is a smart place to be.

Better media and metadata management.

Right in with 64 bit, all processors used and the use of the GPU. I immediately qualified myself:

Ok, there probably aren’t that many people clamoring for better metadata management, but it’s a significant part of better media management, and crucially important for the future of automation in post production.

Then toward the end of that article, under the heading of what I thought that Apple should do, other than what everyone expected, I said:

More metadata automation. Well, part of me hopes they won’t because that’s my field, but it would be nice to see source metadata being used to auto-populate Titles or Master Templates (like iMovie for iPhone does).

Truthfully, I was indulging in some wishful thinking. I still don’t think we’ll get – at least not with Final Cut Pro X v1 – auto-populating titles or Master Templates, but I am very pleasantly surprised how far Apple have “come around to my way of thinking”.

OK, let’s call it parallel development then, as I’m fairly sure that Apple had their metadata-centric rewrite well under way by the time I was writing, but it is gratifying to have one’s position validated. For a company that didn’t really show much sign of “getting” metadata with Final Cut Pro 1-7, they have certainly embraced it for Final Cut Pro X.

Final Cut Pro X is entirely metadata based. You find clips initially via their date/format/source metadata. Each metadata category gets its own smart collections. From there, each time a keyword is applied to a clip, a new Smart Collection is created. Add the keyword to another clip  (they autocomplete even for multiple keywords at once, just like our prEdit does) and it goes into the same bin (whether the keyword is applied to the full clip or a range). Or drag the clip or range to a Smart Collection and have the keyword applied.

Media organization/location, Bins and Subclips are all metadata functions in Final Cut Pro X. There’s no need to assign a Bin for imported footage (a.k.a. Log and Capture/Transfer target bin) because all media finds its way into Keyword Collections and Smart Collections based on metadata.

I also suspect that audio routing to replicate the functionality of fixed tracks will also use metadata, so that similarly tagged clips are always output to the same channels, i.e. dialog, music, effects, etc.

To be more precise, we’re talking about two different types of metadata. I’ve written extensively about the Six Types of Metadata we use in postproduction. Initially identifying four: Source, Added, Derived and Inferred; and then later adding Analytical and Transform. You can find out more in my Mastering Metadata for Postproduction webinar that I did for Larry Jordan and Associates back in March.

In Final Cut Pro X Apple are using Source, Added and Derived Metadata. They also use Source and Derived for automation, beyond simply automatically creating Bins (as Smart Collections).

The information that comes with the file from the camera is clearly Source Metadata: camera, format, frame rate, codec, date shot, transfer date, and other information comes from the camera with no additional work. Interestingly there is more camera metadata available than apparently there are columns that can be displayed in Final Cut Pro X. In the NAB sneak peek we caught a brief look at the optional columns for the Browser and there are missing columns for metadata I know is available from the camera. Maybe our Metadata Explorer isn’t dead after all?

Obviously keywords are Added Metadata: it has to be added by someone. Keywords and especially Range-based Keywords, provide much more flexible media sorting than Clips and Subclips ever could.

More interesting is that Apple have gone beyond those two types of metadata: something I’d assume any modern NLE to embrace one way or the other. Apple have already embraced Derived Metadata. My definition:

Derived Metadata is calculated using a non-human external information source and includes location from GPS, facial recognition, or automatic transcription.

In Final Cut Pro X Apple derive:

• Facial Detection;
• Shot detection (plausibly derived from the size of any faces, and amount of sky area in the picture, among other Source criteria);
• Stabilization and rolling shutter correction based on detecting and removing the artifacts (saved purely as metadata);
• Automatic neutral white balance, based essentially on the auto-white balance technologies that are available (saved purely as metadata);
• Audio cleanup based on the detection of common source problems and applying a correction.

These functions, being metadata driven, are optional. Turn them off or have them performed only when you need them (but I would let them be done in the background during ingest, and just turn off automation when it doesn’t work to my favor).

Metadata driven automation is something that highly established editors well into their career tend to not want, while those in short term, faster turnaround jobs tend to appreciate it more. When demonstrating our First Cuts software the response of editors with traditional experience is generally kind of negative; but the response from those earlier in their career, or who have tight turnaround, is more “Thanks for providing these great tools”.

You see, this metadata stuff isn’t just theory to me. I live in it every day. Every one of our Assisted Editing applications manipulates or uses metadata to make editing easier: to take the boring out of post.

We believe in metadata. I believe that the automations Apple have already built into Final Cut Pro X are just the beginning. The focus on metadata is good for us as one of the problems we’ve faced in the past is to have our metadata needs force-fitted into Final Cut Pro’s earlier inflexible metadata structures.

Now, with metadata embraced so comprehensively in Final Cut Pro X, perhaps it will make it easier for people to understand our goal of taking the boring out of post by automating as much as possible using the metadata provided or derived/inferred.

prEdit 1.0.x required the transcript to be generated in Adobe Premiere Pro or Soundbooth; or be processed through either of those applications to lock text to video. While prEdit 1.5 still supports that workflow, it now also works with:

Text, DOC and RTF formats that have timecode at (least) the start of each speaker. More frequent Timecode entries will make the text/video match more accurate.

3Play Media’s JSON format, also available from Media Silo, is supported and has absolute word accuracy. We recommend the JSON format for new transcripts in the future, for now prEdit works with what you’ve already got. (3Play Media also offer more conventional formats with the JSON format, so you can have multiple options for the same transcription fee.)

We tested with every type of text, rtf and Microsoft Word Doc format that we could find and made sure that prEdit accurately read in the transcript accurately identifying the timecodes regardless of how that timecode is formatted.

If you’re working with transcripts you can swap over to prEdit at any time because any log notes entered in Final Cut Pro or Premiere Pro will be editable in prEdit. New log notes can be added in prEdit’s more efficient interface – with autocomplete – and merged with work done in the NLE.

Since I last wrote about  prEdit we added the ability to write in narration to link the interview segments. Narration is immediately turned into an audio file using the current (at the time) system voice for a temp audio file used during previews and for when the project is sent to Final Cut Pro or Premiere Pro CS5 or later.

As well as working with any transcript prEdit 1.5:

• Let’s producers or editors cut transcripts into selects in seconds
• Add and update log notes with auto-complete logging fields
• Uses notes already added in Final Cut Pro or Premiere Pro
• Previews the video for any clip, subclip, paper cut or section of paper cut
• Exports to Excel spreadsheets and Final Cut Pro, or Premiere Pro Sequences.

There’s a free demo and prEdit is only $395 for a massive boost in productivity. As Lee Schneider said:

Thanks for prEDIT!  It’s a wonderful application that “thinks” exactly like an editor does.  The workflow is smooth and intuitive and it speeds the process of logging by theme and topic that makes it much easier to assemble sequences.

One of the technologies I’ve been following, because I think it’s relevant to my goals with Assisted Editing (to take the boring out of postproduction). One piece of the “boring” is deriving keywords and concepts from spoken word (transcribed, of course).

Technologies like this, and others developed for the Library and Archivist industries, are becoming very sophisticated.

In an Assisted Editing context, the extraction of keywords (particularly) from a “chunk” of transcribed spoken word (let’s say an interview for a documentary), removes the need for a human to enter the keywords.

Having keywords is valuable because you can search for all instances of the keyword (to find common themes), which is something prEdit really does well, whether you’re going to build the initial outline manually in a tool like prEdit or Final Cut Pro, or use an Assisted Editing tool to get to a rough first assemble.

It’s repeated during the day.

“Master Metadata for Post-Production.”  http://www.larryjordan.biz/sem/webinar_2010b.php#mar02 Metadata can be used for tracking media content, usage and rights management, but metadata – log notes – can also drive post production by automating tasks that aren’t a part of the creative process. Metadata driven post production frees editors to focus on creative outcomes. Nowhere is this more important than in Final Cut Pro.

Most editors avoid creating and updating metadata because it is viewed as boring and unessential to editing. However, metadata has taken on an increasing importance in production and post production with the shift to tapeless media. Today, metadata is being acquired on set to pass through to post and VFX departments. Metatdata is essential to tracking existing assets. And, metadata is required for distribution and rights management control.  Knowing how to organize, enter, and search on metadata is essential.

My webinar will explain why metadata is so important and provide some simple ways to begin adding metadata to your own projects. In addition, he will explain the concept of metadata, outline the various types of metadata in use today, such as Dublin Core, XMP, MPEG-7, MPEG-2, and SMPTE-2021 (BXF) – and tell you which ones matter. As well as these metadata standards, the presentation will outline the six types of metadata used in post production to help you take the boring parts out. He’ll consider which commonly used applications supports metadata, and how to preserve metadata when moving between applications.

I will show how to efficiently enter metadata in Final Cut Pro, reducing the amount of work needed, and how to use Intelligent Assistance’s prEdit to even more competently enter metadata and work with text transcript metadata, before the final section of the webinar when he will consider what are the future possibilities of metadata in post production.

Did I mention it’s FREE?

The examples in the article are surprisingly “high end”, pitting humans choosing potential University entrants against a simple algorith, and the algorithm wins.

Could a computer do your job as a ‘creative’ individual? An editor, writer or producer?

If your role is truly creative probably not. But as we try and take the boring out of post, we’ve realized a lot of so-called ‘creative work’ isn’t. It’s not creative synchronizing separate system audio and video. It’s not creative stringing out a first assemble of scripted content. That’s grunt work, to get to a point where the final story can be crafted out of the script and shoot. (The first assemble bears little relationship to the final edit.)

We’ve even established that, with the right metadata a basic first cut can be pulled out of log notes – that’s First Cuts, which we released more than two years ago.

So, can a computer do your job? If someone can analyze what you do, it can be replicated in software. Fortunately most “creative’ tasks are hard to analyze as to why something “works” – intuition is hard to replicate (at least until we understand what mechanisms happen beneath the intuition).

Anyway, with hindsight, I found most of the facade put forth by various departments (eg, auto lending, health care lending), was very misleading. Everyone made the simple complicated. I think deep down, no one likes to think a computer can do their job, and there are many instances where exceptions matter, so a great deal is made out of these special cases. Yet the false positives made them great anecdotes, but horrible for generalizations. Thus, simple rules dominate their much more costly, confusing, and non-quantitative product created by teams of analysts.

Our longest show ever!

In this episode Terence and Philip, with Greg Huson from Secret HQ, look forward to what we can expect in 2011.  You might want to pace yourself on this one as we’ve set a new record for show length.

What will the Microsoft Kinect be used for? Who’ll be releasing new software this year – will we see new versions of Media Composer, Final Cut Pro or Adobe Creative Suite? Will Avid open up to 3rd party hardware? What will be in those releases? Is this the year Metadata (finally) takes off?

How many movies have to not make money from 3D before the fad is over? Or will 3D TV spark 3D production? Will we see RED Epic this year and will it be a success? What will develop with large sensor cameras?

How will the collapse of State Governments affect production subsidies? Will runaway production come back to LA?

When will the tipping point come when distribution breaks out of broadcast and cable channel models? Is ivi going to be ruled legal? What’s the future of Netflix? Is a social network a replacement for channel guides?

What do we wish we could predit for this year? More use of metadata for production automation and where it comes from? What if we didn’t do a first string-out manually? This leads to a discussion of the philosophy of editing.

What will be this year’s surprise? Another DSLR? Another daVinci/Smoke on Mac?

What will happen in distribution? What’s the future of DVD Extras?

What isn’t going to happen that needs to happen?