Latest posts

The Trouble with Training Sets for Machine Learning

Although the title is Defining what’s ethical in artificial intelligence needs input from Africans is accurate, the author is more generally raising the issues of poor quality training data sets for Machine Learning leads to bias in the results.

More Flexible Deepfakes lead to Deeper Problems

My original plan for this post was to comment on how Disentanglement is the Next Deepfake Revolution using CGI to fill in the “blanks” where data for the target identity is missing, and to point to a DYI guide to Deepfakes. When the site became unavailable when I first wrote this post, I feared it had been censored, but it is now back.

Super Resolution isn’t enhancing, it’s creating what it thinks it should be!

As we gain better and better AI based visual enhancing tools, and they are amazing. Every published research paper builds on the work of its predecessors. Two Minutes Papers frequently compares a current paper being discussed, with the previous research. As you would expect there are often dramatic improvements happening in just months, as researchers leapfrog each others’ work.

As exciting as results like this on Super Resolution, it’s always worth remembering that the AI is creating what it thinks should be there, not revealing what should be there. That’s okay if the results are accurate enough, but there is always room for some error, such as with the interpolated glasses in the examples.

Becoming an Amplified Creative

Beyond specific NLE implementations, ML is making it’s way into almost every part of the productions process: storyboarding, production breakdowns, voice casting, digital sets, smart cameras, synthetic presenters, digital humans, voice cloning, music composition, voice overs, colorizing, image upscaling, frame rate upscaling, rotoscoping, background fill, intelligent reframing, Aging, de-aging and digital makeup, “created” images and action, Logging and organization, automatic editing (of sorts), temporization of production, analytics and personalization, storytelling, and directing.

That’s a lot, and it’s only the examples that I’ve kept records of! I’m sure there are many more I’ve missed here.

This is a very detailed article, ultimately running to 12 posts. If you would prefer a briefer version I wrote an overview at the Frame.io blog. This version includes a lot more examples and references.

Amplifying Metadata

If you know me, you’ll know I’m a metadata nut. I’ve never metadata I didn’t like! For me this is both an extremely exciting time and yet somewhat disappointing. 

We now have the tools for accurate transcription, extract text from moving images, identify objects in the video, extract keywords, concepts and even emotion, and that is all good. 

In fact, for all types of searching, whether it’s in a project, or any form of asset management system, this metadata is awesome. Tools like FCP Video Tag make it possible to find that perfect b-roll shot in your FCP Library, or like AxelAI in your asset library.

Metadata is information about the media files. It can be technical metadata describing the file format, codec, size, etc, or it can be information about the content – what we also call Logging! 

For discovery and organization we need the logging metadata to be concise and associated only with the range in the media file that it’s relevant to. One way to achieve that is to isolate subclip ranges and organize them in Bins associated with topics, or in Markers with duration.For me the perfect embodiment of Logging Metadata is Final Cut Pro’s Keyword Ranges, that self organize into collections. 

I’ll be focusing on Visual Search and Natural Language Processing, but there are many commercial and open source tools for extracting or embedding technical metadata including Synopsis.video, which will also allow semantic searching of movies using terms like “an interior, closeup shot outside, in a vineyard with shallow depth of field”, and CinemaNet (part of the Synopsis set of tools) will understand and match because it has been taught to understand those visual concepts.

Amplifying the Pixel

The magic of our industry has always been to create something that isn’t real, out of some real, and some created, elements. Well, imagine if those elements were just a bunch of pixels, then you have a grasp of where this is going: creating video out of a description alone.

Another necessity for Mike Pearl’s ‘Black Box’ it would be a great tool even for educational and corporate production. Instead of Alton Brown needing elaborate props and sets to illustrate the inner workings of culinary concoctions, simply describe the example you want, and your Black Box will create it for you. In full 8K HDR!

Okay, we are nowhere near that, yet, but the research is surprisingly advanced. Keeping in mind how much advancement we’ve seen over two years in other examples, like Jukebox between 2014 and 2016: from 80’s video game to senior living commercial backing track! 

These may be research projects now, but I expect them to be very exciting when I update this article in 2023!