Firstly, the distinction between facial detection and facial recognition is the difference between recognizing “oh, that’s a face” compared with “oh that’s Philip Hodgetts’ face”. That’s a huge distinction. Most digital still cameras sold today recognize faces in the image and attempt to lock focus on them. Heck, Smile Click recognizes that a subject is smiling before taking a picture, and that’s a 99c app! Facial detection is a developer framework in iOS 5, making it easy to add facial detection to any app – for tracking, changing or modifying! Sanyo have facial detection (they call if Face Chaser) in a video camera. In professional cameras, Fujifilm have released a lens with its TRACE technology that tracks faces in shot to maintain focus and exposure – up to 12 faces in a shot.

In postproduction software both Premiere Pro CS 5 and later and Final Cut Pro X attempt facial detection. Neither attempts to identify the individual, just that there is one or more people in the image. Subjectively, it’s still a work in progress. When it works, it’s great but even facial detection isn’t perfect with moving video. Final Cut Pro X further takes this useful metadata (how many people are in a picture) and attempts to infer the shot based on the size of the detected faces: big faces = closeup; many small faces = wide. (I’m sure that’s a gross simplification of someone’s very hard work, but you get the point.)

Facial recognition is valuable because it allows us to quickly group shots with the same person (or character) in them without any additional work. Taking the boring out of post, as I like to say. We’d have to identify the face once and probably deal with some false identifications, but if it gets more accurate than iPhoto currently is, it could be more useful.

I’d also suggest that it’s likely to be more useful in a video project, and more accurate, than in a general iPhoto library because of the more limited set of examples for each person. iPhoto (pre-Polar Rose technology) does a good job where it’s presented with a set of images of the same person in roughly the same time. The more examples in photos across a lifetime, and the accuracy is reduced. So, with the more limited time scale of the typical video project, I’d expect better accuracy.

Facial recognition is still very much a work in progress. In September 2010 Apple purchased Swedish facial recognition company Polar Rose, presumably to boost the facial recognition technology in iPhoto and across their entire product range (I hope!). Full facial recognition probably won’t make it to Final Cut Pro X this year, but you do have to wonder what they have planned for a “full revision” release (one we might pay for) after all the catch-up features are added. (I have absolutely no ideas, this is pure hypothesis/wishful thinking on my part.)

However, if we broaden out a little and consider the research that’s being done – the existing integration of facial recognition in social media and photo sharing sites, and how object detection/computer understanding of what it’s seeing is developing – then it’s obvious that computer recognized metadata will start to be a viable alternative.

Facial recognition

I mentioned Polar Rose, now integrated into Apple where I’d expect it will be used to improve the accuracy of facial recognition in iPhoto and Aperture, but also likely to be added to the facial detection framework now in iOS 5. As iOS and OS X are merging, I would expect the same frameworks to become available to OS X developers in due course. From GigOm - Apple’s iOS facial recognition could lead to Kinect-like interaction:

The unearthed APIs are described as “highly sophisticated,” and can determine where a user’s mouth, and left and right eyes are located, as well as process images taken by the iPhone for face detection. Aside from providing Apple an easy way to introduce Faces (which recognizes specific people in iPhoto) to both its own Photos app and any third-party apps that access that library, it should also open the door for much more advanced facial recognition applications.

Beyond Apple, facial recognition is becoming ubiquitous, even to the point where Slate’s Farhad Manjoo wrote in his “Smile, You’re on everyone’s camera“:

Soon, face recognition will be ubiquitous. While the police may promise to tread lightly, the technology is likely to become so good, so quickly that officers will find themselves reaching for their cameras in all kinds of situations. The police will still likely use traditional ID technologies like fingerprinting—or even iris scanning—as these are generally more accurate than face-scanning, but face-scanning has an obvious advantage over fingerprints: It works from far away. Bunch of guys loitering on the corner? Scantily clad woman hanging around that run-down motel? Two dudes who look like they’re smoking a funny-looking cigarette? Why not snap them all just to make sure they’re on the up-and-up?

This is absolutely a technology who’s time is coming. Further in the Slate article:

In 2006, Google acquired the biometric recognition company Neven Vision, and Hartmut Neven, one of the world’s experts in computer vision, is a respected engineer at the company. A Microsoft research team in Israel has built a fantastic app that uses face-recognition systems to search the Web for pictures of people who are in your photo album. And last year Facebook rolled out a tool that automatically suggests names of people to tag in your pictures.

The Google acquisition has now rolled out as facial recognition in Picassa.

Late 2011 the New York Times had a feature article Face Recognition Makes the Leap From Sci-Fi in their business section. In the article they list these examples:

• SceneTap, a new app for smart phones, uses cameras with facial detection software to scout bar scenes. Without identifying specific bar patrons, it posts information like the average age of a crowd and the ratio of men to women, helping bar-hoppers decide where to go. More than 50 bars in Chicago participate.
• Immersive Labs, a company in Manhattan, has developed software for digital billboards using cameras to gauge the age range, sex and attention level of a passer-by.
• Those endeavors pale next to the photo-tagging suggestion tool introduced by Facebook this year. When a person uploads photos to the site, the “Tag Suggestions” feature uses facial recognition to identify that user’s friends in those photos and automatically suggests name tags for them. And apparently you don’t need a person’s permission to tag them on Facebook.
• “Millions of people are using it to add hundreds of millions of tags,” says Simon Axten, a Facebook spokesman. Other well-known programs like Picasa, the photo editing software from Google, and third-party apps like PhotoTagger, from face.com, work similarly.

Powered by technology from Face.com (one of the key players in white label face detection) Facebook processed 400 million photographs in 30 days!

So my dream of having all the people in my source video recognized and grouped. Perhaps for manual naming (once) but more likely we’ll be able to use existing resources to match the face somewhere and associate the name with it. In a study reported in the Cloud-Powered Facial Recognition Is Terrifying article at The Atlantic:

Unlike Groucho Marx, unfortunately, the cloud never forgets. That’s the logic behind a new application developed by Carnegie Mellon University’s Heinz College that’s designed to take a photograph of a total stranger and, using the facial recognition software PittPatt, track down their real identity in a matter of minutes. Facial recognition isn’t that new — the rudimentary technology has been around since the late 1960s — but this system is faster, more efficient, and more thorough than any other system ever used. Why? Because it’s powered by the cloud.

….

With Carnegie Mellon’s cloud-centric new mobile app, the process of matching a casual snapshot with a person’s online identity takes less than a minute. Tools like PittPatt and other cloud-based facial recognition services rely on finding publicly available pictures of you online, whether it’s a profile image for social networks like Facebook and Google Plus or from something more official from a company website or a college athletic portrait. In their most recent round of facial recognition studies, researchers at Carnegie Mellon were able to not only match unidentified profile photos from a dating website (where the vast majority of users operate pseudonymously) with positively identified Facebook photos, but also match pedestrians on a North American college campus with their online identities.

At most you need one photograph of the individual, and good matching algorithms. CNET News has a background article that gives another take on the experiment identifying random people in public spaces.

It’s getting quite common:

• FTC Calls For Comments On Facial Recognition Technology
• Ubiquitous facial-recognition software is coming
• FBI rolling out nationwide face search and recognition system
• London Police Use Flickr to Identify Looters

Facial Recognition Technologies

Before moving on to other ways computers are “seeing” images, here’s a short summary of the primary technology providers.

• Google purchased PittPatt and are integrating the technology into Picassa
• Apple purchase Polar Rose and are integrating the technology widely through their OS and applications.
• Facebook relied on Face.com technology. In fact Face.com are one of the major providers of facial recognition technology with 7 Billion Photos in the last year in its Facebook apps, Face.com is now available for developers (such as Facebook, and well, even Assisted Editing) to use:

• Developers who are interested in building their own facial recognition apps can now get full access to the open Face.com API, free of charge. That basically means developers can tap into Face.com’s face detection and recognition technology and create brand new ways for friends to engage through photos at zero cost. Hard to beat that offer.
• It is also well funded: “Yandex, operator of Russia’s largest search engine, has invested in Tel-Aviv based facial recognition technology startup Face.com, marking its first investment in an Israeli company. In total, Face.com has raised$4.3 million in Series B funding in a round led by previous investor Rhodium.”

I want to draw special attention to Affectiva who use the tag line “Respectful emotional measurement and communication.” Yes, they read emotion from video.

The software “makes it possible to measure audience response with a scene-by-scene granularity that the current survey-and-questionnaire approach cannot,” Mr. Hamilton said. A director, he added, could find out, for example, that although audience members liked a movie over all, they did not like two or three scenes. Or he could learn that a particular character did not inspire the intended emotional response.

I fully expect facial recognition to come into the postproduction world rapidly and provide useful metadata. Automatically identifying and labeling people in video content will be very empowering. I do foresee an issue with *automatic* identification with actors and roles in narrative, but the software would surely have the ability to manually tag every instance of “this identified face” with a character name instead of the actor’s name sourced from IMDB!

If software can detect emotional responses to movies, it can detect emotional performances and – for documentary/reality/news – detect the emotion in the face to help drive editing.

Oh, if you really want to avoid facial detection? Social Beat has a few tips.

Metadata for asset management

Both DigitalSmiths and Asterpix have tools specifically intended to create visual metadata automatically for asset management and exploitation.

Asterpix

The actual process of machine-tagging involves pulling in imaging data from the video clip and matching it up to whatever text was included by the video’s creator. Nat Kausik, CEO of Asterpix tells me the process is a little similar to Google’s search algorithm in creating relevancy based on what bits of parts of the video get the most screen time. For example, in a video of a someone walking through a grocery store there would be a wealth of information about other products and people, but if you’re focused on that one person for the majority of the clip the engine will pick on it and react accordingly.

DigitalSmiths

Digitalsmiths is the technology leader in the rapidly growing segment of video search and recommendation. Digitalsmiths Seamless Discovery™ has revolutionized the accuracy and ease with which end-users find relevant, personalized entertainment across multiple channels and devices.

From powering tablet-enabled set-top-boxes, to live integration of sporting events, to first-run premium web-content, Digitalsmiths has deep experience and proven results for increasing engagement and viewership though our unique, holistic solutions. Our solutions serve customers that span across all media channels and devices and reach a combined audience of millions of consumers.

Generating other visual metadata

While we’re only just seeing facial detection and recognition rolling into useful applications, but already computers are being taught to read the text in images, and process the images themselves to recognize people, place and things.

Text recognition isn’t exactly new, but:

Google Goggles Can Now Read Print Ads. Oh, And Play Freaking Sudoku!

OCRKit 1.2 – The simplest Text Recognition for the Mac

Beyond Text

Computers That See You and Keep Watch Over You

A New York Times article outlining how computer vision is being used:

Perched above the prison yard, five cameras tracked the play-acting prisoners, and artificial-intelligence software analyzed the images to recognize faces, gestures and patterns of group behavior. When two groups of inmates moved toward each other, the experimental computer system sent an alert — a text message — to a corrections officer that warned of a potential incident and gave the location.

“Machines will definitely be able to observe us and understand us better,” said Hartmut Neven, a computer scientist and vision expert at Google. “Where that leads is uncertain.”

The applications are quite amazing. From observing prisoners, to reminding hospital staff to wash their hands if it’s detected they haven’t, to a host of other uses, smart computer software is observing us and making accurate observations.

Google Experiments With Next Generation Image Search

Notably, the new image search technology doesn’t just index text associated with an image in determining what’s in it. Google is now talking about using computers to analyze the stuff in photos, and using that to associate it in a ranked way with keyword queries. In effect, they’re talking about something similar to PageRank for images (but without the linking behavior).

Teaching Google To See Images

Nuno Vasconcelos, a professor of electrical engineering at the UCSD Jacobs School of Engineering, discusses the approach, called Supervised Multiclass Labeling (SML), in a recentnews release from the school (hat tip to Threadwatch for the pointer). Though SML sounds like a mouthful of jargon, what it really amounts to is systematically training a computer to recognize statistically similar objects, and teaching it to differentiate them from other objects that have similar characteristics.

Google Researchers Teach Computers Out How To Recognize Images Of Famous Landmarks

In the experiment, the researchers fed “an unnamed, untagged picture of a landmark” found on the Internet and the system would spit back the name and location of the landmark, such as the Acropolis in Greece. Each untagged photo was be compared to 40 million GPS-tagged images on Picasa and Panoramio (both owned by Google), as well as related photos found through Google Image Search. Using clustering and new image indexing techniques, the Google researchers were able to identify untagged photos of the same landmarks from different angles and under various lighting conditions.

The researchers report that their system can identify 50,000 landmarks with 80 percent accuracy. I’m not sure that’s quite good enough to even roll that out in a beta product, but if Google can get it to 90 percent or 95 percent that would start to be consumer-friendly.

Researchers from MIT’s CSAIL teach computers to recognize objects

The system uses a modified version of a so-called motion estimation algorithm, a type of algorithm common in video processing. Since consecutive frames of video usually change very little, data compression schemes often store the unchanging aspects of a scene once, updating only the positions of moving objects. The motion estimation algorithm determines which objects have moved from one frame to the next. In a video, that’s usually fairly easy to do: most objects don’t move very far in one-30th of a second. Nor does the algorithm need to know what the object is; it just has to recognize, say, corners and edges, and how their appearance typically changes under different perspectives.

The MIT researchers’ new system essentially treats unrelated images as if they were consecutive frames in a video sequence. When the modified motion estimation algorithm tries to determine which objects have “moved” between one image and the next, it usually picks out objects of the same type: it will guess, for instance, that the 2006 Infiniti in image two is the same object as the 1965 Chevy in image one.

SeeIT.com

 SeeIT.com is in beta while the company is scaling the index from millions to hundreds of millions of images. You can try it by clicking here, then entering the user name picture and password picture93AE (exclusive access for Search Engine Land readers). See thisinformation for new users for more information, including some of the limitations of the current beta release.

Riya

Riya started out focusing primarily on facial recognition, but now has a beta visual search that lets you find similar faces and objects on many images across the web and then refine your results, using color, shape and texture.

Riya also powers the visually oriented product search service Like.com that lets you find clothing and a few home furnishing items based on visual similarity. Like also has a “celebrity” search that lets you see what the stars are currently wearing and find similar accoutrements for your own adornment.

DARPA building search engine for video surveillance footage

According to a prospectus written in March but released only this month, the Video and Image Retrieval and Analysis Tool (VIRAT) will enable intel analysts to “rapidly find video content of interest from archives and provide alerts to the analyst of events of interest during live operations,” taking both conventional video and footage from infrared scanners as input. The VIRAT project is an effort to cope with a growing data glut that has taxed intelligence resources because of the need to have trained human personnel perform time- and labor-intensive review of recorded video.

Check out the “simple’ diagram accompanying the article. This stuff isn’t simple.

VideoSurf: New, Genuinely Radical Video Search

VideoSurf is a computer vision search engine that processes all of the kinds of information most video search services do, but then goes a step further, applying a proprietary process using “multigrid fast computation” and some heavy-duty computer processing power to analyze videos, identify people, and extract all kinds of additional information directly from the video itself. Until I saw the demo, I thought this type of technology was still years away.

Apple wins patent on 3D object-recognition technology

 The USPTO has awarded Apple a patent on 3D object-recognition technology that goes well beyond the current face recognition already included in apps such as iPhoto and the iOS 5 camera application, allowing a device to “build” a 3D face or object by analyzing the curves, contours and shadows of a 2D image. Such technology would give Kinect-like detection and recognition capabilities to cameras such as those found in iOS and Mac devices.

Another benefit of the Polar Rose acquisition, where the technology behind this patent was developed.

Developer APIs

Kooaba

The Swiss company aims to unlock its library of over 10 million images, ranging from album covers to books and movie posters, and provide access to all that precious data via the cloud.

Kooaba hopes that the launch of the API will trigger third-party developers to develop more mobile applications –iPhone and Android versions exist already – or tools that tap into social networking services like Facebook and Twitter, etcetera.

OpenCV

OpenCV (Open Source Computer Vision) is a library of programming functions for real time computer vision.  It has C++, C, Python and soon Java interfaces running on Windows, Linux, Android and Mac. The library has >2500 optimized algorithms.

Out at the Leading Edge of technology

New Image-Recognition Software Could Let Computers ‘See’ Like Humans Do

Using such small amounts of data per image makes it possible to search for similar pictures through millions of images in a database, using an ordinary PC, in less than a second, Torralba says. And unlike other methods that require first breaking down an image into sections containing different objects, this method uses the entire image, making it simple to apply to large datasets without human intervention.

Developing artificial intelligence systems that can interpret images

 Torralba is also attempting to develop systems that can scan a short video clip and predict what is likely to happen next, based on what people or objects are in the scene. To do this, the systems will need to understand what actions each object or person in the scene is capable of making, and what their limitations are. This will allow the systems to make predictions about what each of these entities is likely to do in the near future.

We already have facial detection in our software, and identifying the person is definitely coming. There are technologies to recognize a smile in a cheap iPhone app, or recognize human emotion currently exploited for focus group work. There are computers patrolling prison yards and making sure doctors and nurses wash their hands between patients. There is no doubt in my mind that pre-edit processing will give editors name, context and emotion metadata. And smart companies, like us, will exploit that as input for automating certain editing tasks.

Essentially Prelude is an ingest, logging and rough cut tool. It’s designed to make it easy to add log notes (a.k.a. metadata) quickly and easily, then perform a rough first string out to send to a craft (skilled) editor for the real work. Now obviously, any tool that encourages the entry of metadata is good by me!

So far, there’s not page for Prelude I can link to but from what I remember the key features are:

• Works directly off non-tape media
• Media can be previewed and selects trimmed before ingest to Prelude
• Icons can be “skimmed” (I think Adobe has a different term) FCP X style.
• Will copy media from the card to a specified location
• Will (optionally) do proxy (or really any specified format) generation in parallel with ingest
• Has great tools for subclipping and adding log notes easily
• Can build simple cuts-only timelines
• Exports direct to Premiere Pro or to Final Cut Pro 7 XML.

That’s about what’s known right now. Given that Al finished off by saying “See you at NAB” I can only guess we’ll learn more then (or perhaps before).

Adobe's Al Mooney previews Prelude. Shown here is the simple timeline assembly tool.

Many thanks to my friend Don Berube for pointing this out.  While the headline slightly overstates the case, it’s clear we’re heading for an era when computers in general will understand meaning and the content of images. 

Face Detection and Recognition is now prevalent in digital cameras which can detect many faces and even recognize those you tag, giving them preference as to focus and exposure. But Antonio Torralba is more concerned with what’s not the face in a photo: his Artificial Intelligence (A/I/) software tags all objects in an image and makes the image more easily searchable so as to be able to place the photo in an appropriate context. This ability will eventually mean that robots can recognize their surroundings, say in a house or office building, based on what furniture and objects they see around them.

Going one step further with this research, Torralba is also developing systems that can scan a short video clip and predict what is likely to happen next, based on the people and objects in the scene. This should eventually allow robots to anticipate how their actions will influence future events.

Seems like magic to me, but I seen enough of these type of articles – fully acknowledging they’re at the cutting edge of research and many years from practical application in an editing situation – to know that the future will be computer aided way beyond what we have now.

I’ve long said that the application of at least basic editing algorithms (such as in our First Cuts software) isn’t that difficult being largely a matter of modeling the behavior of human edits and what drives their decisions. The problem is having the source information – metadata if you will – to feed the algorithm. Speech transcription, facial detection, emotion detection, and now software to tag all objects in the image and predict what will happen next all point to a much more computer-assisted editing future than most expect.

My interest in this story is simply because I want to harness that power to speed the pre-post process or understanding what content we have, in order to better (and more quickly) use it. It also confirms my long-held belief that we are – at least for some kinds of work – be able to semi-automate first assemblies.

In this context:

My best guess is that this will be a growing part of the behind the scenes internet services industry. Google would be a natural contender, indexing as it does much of the data one would need to reach a reasonable judgment. But Google isn’t really in the judgment business. Sure, you’ve got their “best guess for Patrick Swayze age” if you search for it (59!), but evaluating natural-language claims, political or what have you, doesn’t seem like their business. They store and index data and surface what you’re looking for. I think it will be a startup, or someone in academia like Schultz, who provides the first germ of this and starts a movement, though his own contributions may in the end be minimal. The competition will, hopefully, be based on the accuracy of their evaluations, just as the search engines competed on speed and simplicity, or device makers on build and design.

Although let’s not forget what my friend Doug Luberts pointed to: Colossus: The Forbin Project

Forbin is the designer of an incredibly sophisticated computer that will run all of America’s nuclear defenses. Shortly after being turned on, it detects the existence of Guardian, the Soviet counterpart, previously unknown to US Planners. Both computers insist that they be linked, and after taking safeguards to preserve confidential material, each side agrees to allow it. As soon as the link is established the two become a new Super computer and threaten the world with the immediate launch of nuclear weapons if they are detached. Colossus begins to give it’s plans for the management of the world under it’s guidance. Forbin and the other scientists form a technological resistance to Colossus which must operate underground.

I’m mildly more positive.

Apparently one of the patents Apple acquired when they purchased Polar Rose last year.

The USPTO has awarded Apple a patent on 3D object-recognition technology that goes well beyond the current face recognition already included in apps such as iPhoto and the iOS 5 camera application, allowing a device to “build” a 3D face or object by analyzing the curves, contours and shadows of a 2D image. Such technology would give Kinect-like detection and recognition capabilities to cameras such as those found in iOS and Mac devices.

The technique could be used, for example, to create biometric logins that only unlocked the device when the owner was identified (though as with other such techniques, keeping the device able to distinguish the actual owner versus a picture of the owner would be the key to real security). It could also be used to automatically take and upload timed pictures of users who were not the owner, or lock out machines when the owner’s face was not detected. Apple mentions also being able to identify persons who are not aware that they are being recognized.

Those who know me will realize that I’m thinking of how we can apply this technology to production metadata. Facial identification is a powerful tool if it’s accurate, and the ability to recognize more objects will give us even more metadata to feed into identifying and editing algorithms.

Facial Recognition App Detects, Captures Smiles Technology intrudes more & more into “human” territory

Meet Swivl, The Motion Tracking iPhone Dock That Always Keeps You On Camera More and more automatics!

Now, it would be really cool if Swivl tracked you and kept you on camera using facial detection but it does not: instead it uses a hand held transmitter/controller to “know” where to point the camera.  Even with that it will make a great addition to a video blogger, web episode producer as the producer/talent can move and have the camera follow them as they do.

The SmileClick App is much more interesting:

SmileClick is a newly launched, unique application that uses proprietary facial recognition software to detect how big a photo subject is smiling. Once the app determines that the person to be photographed is sporting a large, genuine smile, it automatically snaps the picture at the right time, helping to create happy memories.

Waltham MA based  Affectiva takes it one step further:

…has facial recognition software that can accurately determine the difference between a happy smile, an embarrassed smile and a smirk!

As well as other emotions. Currently used only in focus groups and the like, wouldn’t it make a great source of metadata if, as well as the person’s face being (accurately) identified, there was another tag “smiling”, “embarrassed”, “annoyed”, et. al. I see it as inevitable but a few years off.

Naturally, any automation of “creative” processes interests me because I believe that some parts of the creative process of video postproduction can be automated. However, author Robbie Allen is right when he says:

Software isn’t a panacea, though. Not all content can be easily automated (yet). The type of content my company,Automated Insights, has automated is quantitatively oriented. That’s the trick. We’ve automated content by applying meaning to numbers, to data. Sports was the first category we tackled. Sports by their nature are very data heavy. By our internal estimates, 70% of all sports-related articles are analyzing numbers in one form or another.

Similarly there are many types of video that lend itself to automation. For example SundaySky create up to 1.4 million unique videos a month:

SundaySky, Tel Aviv/New York-based start-up with 50 employees, with no video camera or production staff, will produce 1.4 million video clips this month for a range of big retail and real estate corporate customers including Overstock.com and the History Channel. The company pulls customer data into a customized template which creates videos with movement, music, narration and graphics and video.

These are neither “cat videos” nor are they the sort of video that requires enormous human creativity. These are not for the big screen or for broadcast, but they are part of the online video trend (revolution?) to provide more video content at lower cost to produce, which seems to be everyone’s goals.

Our goals with Assisted Editing are to “take the boring out of post” as a tool for an editor or producer to filter through clips and find stories, or to simply automate dull parts of the process, and perhaps, build this type of simple video automatically.

Following on the heels of yesterday’s post about facial recognition in the cloud here’s information on how Apple are applying the technology they gained when they acquired Polar Rose last September, at least within iOS frameworks.

When coders dug through Apple’s beta versions of iOS5 they found what were deemed to be “highly sophisticated” API systems that let an iPhone automatically track eye positions and mouth positions (so the angle to the user, and possibly where their attention is being directed could be calculated) as well as passing key data on to a face recognition algorithm that would be accessible to all apps…not just Apple’s own.

Combine this with the Nuance-licensed voice recognition technology in Siri – also new with iOS 5 and iPhone 4S – and we have the foundation of a very powerful metadata generation system that would automate naming people in clips and form the basis of speech transcription and then keyword extraction.

In my dreams these are technologies that will come to Final Cut Pro X 10.2 or 10.3 in future years.

Back in the mid 1990′s my email sig line read (for a while) “Dynamic Media Evangelist’ because I was a serious advocate of interactive media of the lean forward, get involved kind. Well, disappointment after disappointment followed and I realized that, for most people, the act of “watching video” was a lean back, turn off act, not an active one.

So, my subsequent thinking was that the interactive technologies that appealed to me could be also used to create personalized media experiences. For example Arcade Fire teamed up with Google to create the Wilderness Downtown HTML 5 demonstration piece that picked up maps and images from the nominated location of your “downtown”.

Warner Bros and  ”Charlie’s Angels” director McG teamed up to bring together elements from the viewer’s Facebook profile:

“The show becomes personal,” McG told Reuters. “Music that the characters are listening to comes from your playlist, pictures on the walls, TV screens and picture frames inside the show are from your profile.”

“Aim High,” which will play at http://www.facebook.com/aimhighseries uses computer programs to access a user’s profile from which to draw material.

For instance, as Nick performs his spy duties in his high school’s hallway, he may pass a poster for a class president candidate and the picture on it is you. Or, the end credits might have photos of your friends when listing Nick’s spy accomplices.

This is another example of the way storytelling is moving out of its traditional structures. Of course these are experimental pieces and it will be a long time before a majority of entertainment is personalized, and some may be creeped out by it, but I like the idea.

At one level this is kind of scary – these were the folks who discovered a Social Security number way too often, from a casual photograph in the street – at the level of production automation it shows the direction we’re heading for automatically generating metadata for postproduction.

 In their most recent round of facial recognition studies, researchers at Carnegie Mellon were able to not only match unidentified profile photos from a dating website (where the vast majority of users operate pseudonymously) with positively identified Facebook photos, but also match pedestrians on a North American college campus with their online identities.

and

In our third experiment, as a proof-of-concept, we predicted the interests and Social Security numbers of some of the participants in the second experiment. We did so by combining face recognition with the algorithms we developed in 2009 to predict SSNs from public data. SSNs were nothing more than one example of what is possible to predict about a person: conceptually, the goal of Experiment 3 was to show that it is possible to start from an anonymous face in the street, and end up with very sensitive information about that person, in a process of data “accretion.” In the context of our experiment, it is this blending of online and offline data – made possible by the convergence of face recognition, social networks, data mining, and cloud computing – that we refer to as augmented reality.

The benefit for documentary/reality postproduction would be to have every person in a shot identified automatically and tagged with name (and perhaps other relevant details).

And then apply  Affectiva‘s emotion detecting algorithms…

This photo is titled "gradingbenchmark1.jpg"

Wait, what’s that logo on the display showing the picture output?  I didn’t know Apple made broadcast monitors?

But that’s probably just a mockup shot because the Apple monitors look so good.

Simply titled "Picture-2.png".

Or there’s this shot, which pretty much rounds out the product shots BMD-DaVinci use to promote Resolve.

Simply titled "resolve.jpg".

Now, I’m open for correction but these last two shots show the same monitor design/model used for computer display and image display of the image being worked on. I am unfamiliar with any broadcast monitor that also functions as a good computer monitor. And clearly in the first example that’s an Apple Cinema Display being used as the output video monitor on the system, and I think the other two shots also show a computer monitor as a video output monitor.

Good enough for Resolve but not Final Cut Pro X? What am I missing?

Until that point we’d only done black and white printing, and it’s easy to proof what you’re going to get on a B&W laser printer. Not so with color. Color output wasn’t as common then as it is now and we didn’t get the first Kinkos until very late in the 1990′s, so we really only had one choice for our runs of 2-3 covers for each job.

This became a serious problem when – while developing a food product for my parent’s company during the period I managed it (in addition to my own two companies) – we needed a very specific purple on mockup packaging we were presenting to food buyers at the national department store chains in Australia. Cadbury – Australia’s biggest chocolate company – have always used a specific purple in their packaging, and had just spent several million dollars on a campaign that heavily featured this purple. Since the new product was a chocolate variation on a traditional English Christmas Pudding, having the purple match was beyond important. And we got blue-purple, and red-purple: seemingly every color except the one we wanted.

The Chocolate "Blast of Mouthwatering Bliss" had to match Cadbury purple.

The problem was, what we saw on the screen and what they printed were two entirely different images. Just like that, the print industry’s color management problems were now my color management problems. There’s nothing like first hand knowledge of the problem to appreciate the cure!

By early 1998 we were full on into publishing, with the release in February 98 of the Media 100 Editor’s Companion. By this time we had  learnt – although that particular output bureau never did – the joys and benefits of ColorSync. While the interior was done with xerography (i.e. photocopying) the color covers were done in Sydney by a direct-to-plate digital offset and we never ever had color accuracy problems with an output bureau that used ColorSync.

Every printed cover looked the same, and the same as it did on the computer screen when it was desgined.

ColorSync was the solution the print industry needed to deal with color consistency and matching. The challenge is quite significant when you consider that each device treats color slightly differently: they have different “color profiles” in ColorSync Profiles. What ColorSync facilitates is a color accurate workflow, also referred to as a color managed workflow.

The challenge in print is to ensure that a scanned image reproduces with correct colors on an RGB display and prints consistently on a wide range of types of printers (and size of output) in CMYK color. On computers the images are additive (three light sources) while on paper they are subtractive.  If you worked hard you couldn’t create a less consistent workflow.

And yet, for the last 15 years or so, this has not been a problem. Colors in Photoshop, Illustrator, InDesign, Quark, etc. are accurately reproduced in the final output despite the different color models, device variations, display variations, differing types of paper and even ink variations. This has been possible because of ColorSync, largely developed by Adobe and Apple but now supported on every platform and by all manufacturers of scanning or printing hardware.

What is ColorSync?

This I knew when I wrote recently on how ColorSync might be used as part of Final Cut Pro X’s color management, but from the comments it quickly became obvious that I was very uncertain about:

• How does ColorSync work with video files, and specifically
• How does Final Cut Pro X use ColorSync to accurately monitor color.

So I took it upon myself to start researching the topic. I very quickly discovered that, while there is a lot of information regarding ColorSync and printing workflows, there was really nothing about ColorSync and video short of an OS 9 era article on ColorSync and QuickTime, noting that we can add ColorSync profiles to QuickTime movies using Terran Interactive’s Media Cleaner Pro! (That really dates it. Neither company nor product still exist.)

I took the opportunity to ask my contacts at Apple some specific questions about how ColorSync works in Final Cut Pro X and then synthesized that with the best information I could about ColorSync in general to research what I found. What was also interesting is that, while ColorSync has been mentioned in the PR materials and demos, there has been no explanation as to how that facilitates accurate color. This is my attempt to redress that lack.

ColorSync is built on ICC profiles and the two  terms are interchangeable: an ICC profile managed workflow is a ColorSync managed workflow. Each device – scanner, printer, camera, monitor – has at least one ICC profile. In an oversimplified explanation the ICC profile tells ColorSync (or equivalent engine in Adobe’s applications) how this device’s color representation differs from the standard. In this case it’s the XYZ Profile Connection Space: a colorspace with a very wide gamut.

So, for a given scanner the ICC profile is used to create an accurate representation of the scanner’s result into the Public Connection Space (PCS from here on) and the display’s ICC profile tells ColorSync how to accurately display that PCS on that specific monitor for most accurate fidelity to the original image content data.

In simple terms each device tells ColorSync how that device’s files relate to the PCS either to convert to the PCS or convert from the PCS to a display or printer. This way, each display or printed copy is color accurate to the original file, despite variations in printers or displays.

Source to PCS to destination. From Apple's 2003 ColorSync in Mac OS X document.

Workflows that correctly follow this practice are known as Color Managed workflows.  (Check and see if you’re reading this in a color managed browser.) From that page:

This is happening BECAUSE your color-managed browser is reading each file’s embedded profile and Converting or Mapping them to your monitor profile for a theoretical display of True Color.

In practice, while you may have to set up Photoshop or some other tools, for the most part, like the color management in Safari demonstrated from the link above, it just happens. You’ll find scanners and printers install ColorSync (ICC) profiles for all the functions – a scanner/printer will have a scan-to-PCS profile, and a PCS-to-printer profile – and OS X uses them.  Earlier versions of the OS required some setup but recent (10.5 onward at least) have implemented ColorSync so it “just happens”. As long as each device has a profile, and files are tagged with the appropriate source ICC profile, color accuracy along the entire process is assured.

Color consistency in the video world

Meanwhile, it was easy to monitor video: use a calibrated video monitor. Connect output of your edit system to said calibrated monitor. If you were very particular about your video signal you connected Waveform and Vectorscope to the output as well, so you could really see what’s going on in the signal.

The accuracy of the color entirely depended on how good a monitor you put on the end of the chain. Now in a perfect world this would be a broadcast level monitor, calibrated at least once a year, that could be relied on. I confess, I never had one, even though I delivered a decent number of TV Commercials for national TV broadcast, and other content that went to air. The majority of my editing career, however, has been for non-broadcast purposes.

Even today, I support systems that do work for major studios and none of them have a broadcast grade monitor, let alone calibrate it. One system is finishing HD footage to a 10+ year old SD TV set of no particular brand. Another restoring historic footage works primarily with a decent (but domestic) Plasma display that’s distinctly off axis from the editing position.

I’ve set up movie edit stations that have monitored on a Cinema Display via an AJA SDI to DVI adapter (with CLUT); or to an LCD TV. In fact that device is doing in hardware exactly what ColorSync is doing in software. Now these movies were not graded on this system, just for editing so that’s probably fair. Another client working on a TV series  for Cable Networks, has only a broadcast monitor (JVC 24U) in the grading room; the other four bays all work with consumer LCD or Plasma.

JVC-DT-V24L is a low priced broadcast spec monitor.

Color fidelity required that the NLE made no unintentional changes to the signal coming in and ultimately the accuracy of color depended entirely on how accurately the specific display represented the colorspace it was attempting to represent. In SD that would normally be Rec. 601 and for HD normally Rec 709.

It’s up to the NLE software to convert colorspace from the different source profiles it encounters: not only Rec 601 and 709 in video files but sRGB or any of another dozen possible profiles for still images.

This type of monitoring – a conventional video output – allowed monitoring in the destination colorspace, but appears to be not part of Final Cut Pro X’s design, at least according to Matrox. So, how does Final Cut Pro X ensure color consistency? Of course, ColorSync.

ColorSync was implemented in QuickTime. Badly. Final Cut Pro 7 did not support it, instead working with it own color conversion/matching technology.

Final Cut Pro X and ColorSync

As I’ve discussed the important considerations for ColorSync are:

• The colorspace, which can be considered to be the equivalent of an ICC profile, must be known and accurate for the file (taking the place of the source device profiles in print workflows);
• The rendering engine must use ColorSync to convert from source to destination colorspace/profile;  and
• The frames must be converted for the desired output, be it a Cinema Display, SD 601 or HD 709 result.

This is exactly what Final Cut Pro X does.

As files are ingested, Final Cut Pro X will read any ColorSync information if the files are tagged with information about their colorspace. If it’s a QuickTime file that contains an explicit tag for color information, that is respected by Final Cut Pro X, Motion 5 and Compressor 4.

If there’s not an explicit tag, then Final Cut Pro X infers the colorspace from known information. For example DV files automatically imply either an NTSC Rec 601 colorspace or a 1280 x 720 or 1920 x 1080 file will be tagged as being HD Rec 709 colorspace. These are quite reasonable assumptions and yet another use of Inferred metadata in the Final Cut Pro X world. Beyond those examples, Final Cut Pro X will simply make an intelligent guess based on whatever information is available. Fortunately, image size and aspect ratio are very strong clues to how the source color should be interpreted (the purpose of tagging it). 720 x 576 pixel files are almost certainly PAL SD files and so Rec 601 PAL is reasonably inferred.

Generally it’s older files that have no useful information and require inferring colorspace, but since there really are so few colorspace choices for video, compared with print where each device has its own characteristics, the source ColorSync information is likely to be highly accurate.

I’m sure someone will take me to task on the fact that Rec 601 doesn’t fully describe the color profile because there is no primaries defined, but Final Cut Pro X uses other information in the file to determine which of the well know primaries should be associated with the file.

So we have source color profile information associated with the files. Now it’s up to Final Cut Pro X to ensure that the rendering is accurate.

For Final Cut Pro X and Motion Apple created a shared render engine, touted as the “Linear Light Engine” at the NAB Supermeet Preview but that branding seems to have been dropped ahead of release. This shared render engine uses ColorSync to conform an image from one colorspace to another. ColorSync creates the steps that are required for a specific transform and then assembles them into a series of GPU operations into the GPU processing pipeline. Or in simpler terms, it manages the transforms so the source image content is accurately represented into the processing space.

There is no assumption about the input colorspace at render time. ColorSync profiles were derived or inferred on ingest or come from the image file. Most processing uses the Rec 709 primaries with a linear (1.0) gamma, ready for the final stage.

The third point above, is that the render engine has to accurately prepare that processing space for the appropriate output. If outputting to a Cinema Display, then the render engine conforms the processing space so that the visual appearance on the Cinema Display will look like the image should be viewed. (This works well with the recommended gamma and color temperature for the display: 2.2 and 65oo K respectively.) For the most accurate results you’ll want to create a ColorSync profile for the specific monitor rather than using the supplied “generic for this type of monitor” profile that ships. You can use the ColorSync Utility for that, or a tool that reads values off the screen to generate the profile.

If you’re working with an HD Rec 701 file but request an output for DVD, then the 701 frames (and other graphics from their colorspace) are converted and output so they are correct for the Rec 601 colorspace of a DVD.

So Final Cut Pro X determines the colorspace (ICC Profile) of the source on ingest; manages color using ColorSync through the rendering engine and then uses ColorSync to make sure that the representation on the display – whatever type of display – is also an accurate representation.

Meaning, there is no need for an external broadcast monitor in the classic sense. What you see on a monitor that has a ColorSync profile, is an accurate representation on that monitor of the source colors displayed on a Rec 709 monitor. And you don’t have to do a thing to get the goodness!

Not convinced? Well, coincidentally the Academy of Motion Picture Arts and Sciences’s “ACES or Academy Color Encoding Specification” initiative is planning something almost identical to ColorSync, but specific to the needs of motion picture post production, particularly with so much digital manipulation and inclusion of digital elements. (Look for the heading “ACES: Academy Color Encoding Specification”).

Beyond Rec 709

While the ACES initiative is largely about colorspaces used in digital cinema, it should be pointed out that a “broadcast video monitor” would be no help for colorspaces other than Rec 709 (for HD). So if you want to work with, DCI-P3, also SMPTE-431-2, for example in a traditional NLE you’d be stuck previewing in the wrong colorspace.

With a color managed/ColorSync workflow, if you have DCI-P3 media, and the appropriate source ColorSync/ICC Profile (or colorspace definition,) then ColorSync in Final Cut Pro X would present an accurate representation of that source on the computer monitor.

Mixing any of these colorspaces into any one, and then outputting the appropriate colorspace becomes no problem when using ColorSync.

Conclusion

While it’s a significant change from traditional workflows, the approach taken by Apple with Final Cut Pro X is a better fit for what happens in edit bays outside the big studios, where the problem with early Final Cut Pro users wasn’t that they weren’t monitoring on calibrated broadcast monitors but that they were monitoring only on the computer display. How many times did I, and the other Final Cut Pro 1 pioneers have to remind people that the computer display is not an accurate representation of the video!

Well, 2.2 gamma on OS X (since Snow Leopard) and ColorSync have made that advice not only null and void, but downright wrong. With a ColorSync workflow, the view on the computer monitor, particularly in full screen mode, will be more accurate than using a consumer television as a display, with the sole exception of interlaced video output. If you’re delivering for web, for LCD or plasma screen (i.e. all currently available displays) the display is progressive. Produce and edit accordingly.

I don’t think we have broadcast video output from Final Cut Pro X, because we have something better that accommodates todays colorspaces and will expand to accommodate whatever the future holds as well.

A disclaimer. I do not have the equipment available to test this: I don’t have a broadcast monitor nor a system that could output to it via SDI or even analog. (I could do DV via Firewire, but really….) I am confident that the solution that solved my inconsistent print color issues is robust enough, with the right software design, to manage my consistent video image needs.

The purpose of monitoring video on a broadcast monitor is to ensure that we have consistent color accuracy from a known reference. The value of a ColorSync workflow is that we ensure that we have consistent color accuracy wherever its shown. It’s a superior choice that avoids the limitations and expense of the previous “standard workflow.” Instead of having to build in a single colorspace on a monitor, Rec 601 or 709 style, with ColorSync all wide gamut monitors can be calibrated (ideally with a colorimeter and on that specific monitor) and used as reference monitors.

Smart Collections can be dragged from Event to Event.  You probably already knew that, but it was new to me.

This is really useful. If I’m working on an ongoing project and create, say, an Event for each day’s dailies.  In the course of preparing that footage, I apply Keywords and Range-based Keywords, and have a set of Smart Collections that organize the material appropriately.

When I get the next day’s footage and bring that into its own Event (or each Episode’s footage into an Event) I can drag the Smart Collections across to the new Event and they’ll instantly update to the content in the new Event.

This also applies to Keyword Collections, but since they’re created automatically when you apply the keyword, I don’t see that being so valuable other than to save a few moments setting them up if you prefer the workflow that creates a Keyword Collection first then drags Clips or ranges to the Keyword Collection (thus applying the Keyword to that Clip or Range).

Both book publishing and film/television are industries that were built on scarcity, that are being disrupted n an age of non-scarcity. That’s not to say that there isn’t success and money to be made in the traditional businesses, but book publishing is an interesting place to look for parallels to television (particularly).

In “olden” times book publishers approached writers, offered them an advance against royalties, to write a book. Somewhat more dramatically than the budgetary pressures on television production, the advances offered to writers in the video technology and techniques space have dropped to about 20% of what was being offered as an advance a decade ago.

Given that even those advances really didn’t cover the time it took to write a book, book writing was done for profile/career rather than from the publisher. In fact an author makes more from the Amazon affiliate commission (for sales in a State where Amazon still has affiliates) than from the publisher for the sale.

I went for “self publishing” a few years ago, starting with some PDFs of Final Cut Studio tips and on to some more serious book writing: most recently Conquering the metadata foundations of Final Cut Pro X. I’m selling both PDF versions and print versions (thanks to Amazon Createspace): pdfs directly and print via Amazon itself.

Now, I’ve written before on what you need to know and handle for yourself if you’re going to self publish, but for me it’s worked out well, with returns commensurate with the time invested in writing the book, so I get both a decent financial return and the profile/career boost as well.

The television industry, like print before it, has many levels between creative and customer – in fact it confuses the customer relationship because the broadcaster and cable companies’ customer is the advertiser, not the audience. These levels cost the consumer, but when the technology limits supply (of airwaves or cable channel real estate) it can be justified.

Until technology bypasses the control and allows print authors to go directly to customers, and still have access to the biggest book distribution channel in the world for physical books (and increasingly their digital equivalents). Although nowhere near as complete, we’re seeing a similar disruption to the television and film industry that will play out over the next decade as the print industry transforms ahead of it.

All the things that a publisher (substitute network mentally) provided for me as an author now have alternatives. As I wrote in the article referenced above, you will need to find an editor to peruse your work (unless you’re supremely confident) and someone to lay it out, but there are tools for that, or services at very reasonable prices. Even cover design can be done by template or service (through Createspace). The other things a publisher did for an author – printing, access to distribution and publicity – are things I can do myself or through Createspace and its competitors. Print happens on demand (and for very small number provides an interesting PR tool); by filling in some web forms the book appears in the Amazon catalog and publicity really fell to the author beyond the publisher sending out a press release at the time of publication and organize some interviews: things any competent author can do for themselves.

And without a large upfront investment required, funding the book is nowhere near the problem it was. Now, with film or TV production there’s generally a crew to fund as well, so the numbers will be bigger, but ultimately I think the same type of disruption will affect studiio film and television production.

Thanks to technology (and some work on my part) an independent has had the best selling Final Cut Pro X book in Amazon for June, July and August 2011. (Because there are no other print books being sold for Final Cut Pro X!)

Clearly reading brainwaves is getting a lot more advanced than I otherwise would have thought. This project reads “something” from the collective minds of the audience to influence story.

Take watching a horror film, for example. The more frightened a headsetted viewer becomes during a film, the scarier the ending will be. Though they can’t control each characters’ precise actions, with Mynd-Play, viewers can train their brains to better control a movie with each viewing.

Content is produced specifically for the system, with films up to 20 minutes long, although a feature-length zombie pic is in the works. Films are live-action, since, as MyndPlay CEO Tre Azam notes, “Live action is how you get the best emotional response out of someone.”

Fascinating. The article goes on to describe how content needs to be written for the format (no surprise) and how directors feel about it.

But it reminds me of a TED talk Tan Le gave in 2010: A headset that reads your brainwaves. How quickly will we get to being able to “control software” by imagining the outcome we want? Tan Lee’s research senses actions; the Mynd project seems to be sensing emotion. These two teams should meet up!

Now, seriously, if we get to the point of being able to control a 3D  program by imagining the result, then is that “dumbing down” the software?

Author Aleks Krotoski starts out with the importance of story.

Stories are memory aids, instruction manuals and moral compasses. When enlisted by charismatic leaders and turned into manifestos, dogmas and social policy, they’ve been the foundations for religions and political systems. When a storyteller has held an audience captive around a campfire, a cinema screen or on the page of a bestseller, they’ve reinforced local and universal norms about where we’ve been and where we’re going. And when they’ve been shared in the corner shop, at the pub or over dinner they’ve helped us define who we are and how we fit in.

But “… the process of telling a story doesn’t have to be unidirectional” and that’s given rise to “multi-media” or “transmedia” storytelling.

But the tools they use to tell tales are evolving, becoming more modular and tailored, more participatory and more engaging than just the printed word or the moving image. The new form of storytelling that’s coming from a digitally enabled cabal moves beyond reinterpreting a text for radio or screen.

….

Frank Rose, author of The Art of Immersion: How the Digital Generation is Remaking Hollywood, Madison Avenue and the Way We Tell Stories,believes this is exactly what people want from their story experience. “The kind of multi-way conversation that the web makes possible is what we’ve always wanted to do,” he says. “The technology finally enables it.”

Rose celebrates the way that the new kinds of storytelling brings audiences together to traverse plots, but recognises that there are challenges for consumers and for creators: “It’s very different when you have a medium that forces you to engage with other people,” he says, reflecting on the arc of a narrative that is necessarily more complex, multifaceted, and demands more flexibility. “You don’t know if you’re going to have to tell a story for one hour, two hours or 10 years.”

The article goes on to describe examples of extended storytelling. It has always been true that technologies enable new ways of telling or sharing stories: from verbal only, to scribes and the elite literate sharing via scrolls and the like; to the advent of the printing press and later widespread literacy; on to radio, TV, film and video right up to the Internet age of multiple venues for stories.

Paul Berman, writing for Wired, would prefer you didn’t call it “Transmedia” though!

This is a long, and not new, article that rambles through a fascinating story of how a lawyer, “Mr Pink”, “Mr Brown” and “Mr Bootstrap” collectively cracked the code for predicting the profitability of movies, TV shows and (separately) another team shows the likelihood of whether a song is going to be a hit.

The specifics of how they achieved both breakthroughs is interesting: have the computer software (usually some sort of neural net) analyze existing successes – music or movie.  It then analyzes new music or movie proposals to determine whether it is likley be be a hit (music) or how much money it will make at the box office (movie).

The head of hit-predictor Platinum Blue, Mike McCready claims:

Record executives have tended to be Humean: though they can tell you how they feel when they listen to a song, they don’t believe anyone can know with confidence whether a song is going to be a hit, and, historically, fewer than twenty per cent of the songs picked as hits by music executives have fulfilled those expectations. Platinum Blue thinks it can do better. It has a proprietary computer program that uses “spectral deconvolution software” to measure the mathematical relationships among all of a song’s structural components: melody, harmony, beat, tempo, rhythm, octave, pitch, chord progression, cadence, sonic brilliance, frequency, and so on. On the basis of that analysis, the firm believes it can predict whether a song is likely to become a hit with eighty-per-cent accuracy.

The emphasis is mine in case you didn’t pick that the software was four times more likely to predict a hit song than experienced record company executives – the people who make final decisions on these things! (Quite separately, how does anyone with a 20% success rate keep their job? Just asking.)

What is interesting is that it takes no “industry knowledge”:

McCready didn’t care about who the artist was, or the cleverness of the lyrics. He didn’t even have a way of feeding lyrics into his computer. He cared only about a song’s underlying mathematical structure.

Similarly the movie profitability prediction software was built without regard to “artistic” elements at all. It picks up on plot points, settings, and other aspects to determine – based entirely on what’s come before – how successful a movie will be.

Both technologies can be used to fine tune and re-predict the likelihood of a success. Now, I have all sorts of concerns about the application of the technology and how it might “force out” small, niche movies or music that wouldn’t make the hit status but would otherwise still be a satisfactory story and a profitable project.

In both instances it is metadata at the foundation of the input to the software: for music metadata about mathematic relationships within the music; for movies the details of plot points, setting, story, characters, etc. It’s another way that metadata makes the data more valuable by fine tuning the music or the plot/characters to make the project more profitable. Not exactly classic metadata!

By the way, the first half of the article I linked to above is the most salient – mostly in the middle. The end is a verbatim account of a meeting and doesn’t really go anywhere. I kept looking for the conclusion.

And don’t forget, if you’re interested in Metadata, I have a book – Conquering the Metadata Foundations of Final Cut Pro X – and an upcoming full day seminar at DV Expo – Using Metadata for Production and Asset Management coming up on September 21.

Back in 2010 Apple purchases Polar Rose and now it seems we’re seeing the first uses in a Framework for iOS users. Frameworks are code building blocks that allow developers to use advanced technology features without needing to understand how the technology works. Frameworks largely abstract the technology for the developer.

Now this article tends to confuse facial detection – there’s one or more faces in this picture – with facial recognition – I see Steven in this picture – but is a good primer on what can be done with facial detection and recognition in iOS applications:

• apps that track eye movement to work out where someone is looking (and then place interface or other elements there)
• emotional state of the user (but not without additional work by the developer, this isn’t part of the iOS Framework but it is technology that’s working)
• User identification – no password to log in, just show your face to automatically log into your User Space.

My interest is, not surprisingly, how this might be adapted to postproduction. Certainly the ability to quickly identify all shots with “Steve” in them would be valuable metadata. Shots where “Steve” was “happy” would be even more valuable metadata.

Since Apple have indicated that (eventually) iOS and OS X will be feature compatible, how long before we can start integrating this metadata into editing workflows?

This is pointed to more as an indicator of the distant future – one where direct brain control takes over from keyboard, touch pad or voice control. Still, it’s interesting what’s being done now as an indicator of what will be the future. Will it be control of software by thought, or will it go beyond that to imagining the edit or look and having the software work off our brain waves to results?

For the Prius Project bicycle, a team from Deep Local is working on a helmet with built-in neuron transmitters that allow the rider’s brain patterns to trigger the electronic shifters to move gears up or down. The system is said to take just 10 minutes to learn, after which the rider will be able to shift gears by just thinking about doing so.

In this context I’ll once again point to Tan Le: A headset that reads your brainwaves – Tan Le (2010) – a TED video – to consider in the context of this Prius Project. Direct brain control of software is much closer than I ever thought.

Facial Recognition – actually identifying the person – is more advanced than facial detection – simply determining how many faces are in a shot – and is going to become an important source of postproduction metadata.

Final Cut Pro X attempts to analyze shots (optionally) for facial detection, as does Premiere Pro CS5 and later. Final Cut Pro X also attempts to derive from the size of the face, the type of shot: Wide, M, MC, etc. Right now the technology is a little “hit and miss” or basically unreliable. For now.  These technologies will get better. Apple purchase a Swedish company last September to boost it’s efforts in Facial Recognition.

Meanwhile Google are also building up their portfolio of recognition technologies with the purchase of PittPatt.

When we get reliable Facial Detection, we’ll be able to find shots of individuals across our source media wherever they appear in a shot, and we’ll only have to apply a name once. When we get reliable facial detection, which is not yet, today.