Latest posts

Amplified Storytelling

Everything starts with a story, so the obvious question is, “can Machines tell compelling stories.” Could a machine write or direct a “movie”? When it comes to results so far, we should be asking if they can even tell coherent stories!

People have been working on these questions for years. There have been experimental projects where a machine has been trained on “science fiction scripts before writing their own.” The result was Sunspring, which is definitely a swing and a miss! Each of the projects I’ve mentioned will be discussed in detail in this section, with links.

Another experimental had an “AI” create a highly stereotypical car commercial. Speaking of stereotypical, the Eclipse project used AI to create a music video. While there have been lots of experiments, left to themselves, the machines create laughable, or incoherent results.

Where there has been success, it has been in machine/man synergy, or Amplified Creativity. On Impossible Things the machine suggested the storyline and some oddly specific requirements for the Trailer, but the script was written by humans!

All these experimental projects are interesting. None of them suggests that creative storytelling by humans is at all threatened.

West End Musical

While not a movie or television project, an excellent example is a collaborative effort between composer and machine: Beyond the Fence. It was billed as the first stage musical “written by computers.” Like most of these articles, that’s a little stretch, in that no one computer or algorithm wrote the musical. Benjamin Till, credited as the creator, worked with a series of ML models. 

“There is not a single song in this show, not a single moment, that wasn’t at some point inspired by or written by a computer,” said Till. Altogether, computers fully determined the premise and generated 25 percent of the music and lyrics. 

Till turned himself into an amplified creative who finished the first draft of Beyond the Fence in a few weeks, and the team finished the entire show in four months. By comparison, the “unamplified” creative spent over a year just researching and writing the first draft of his previous musicals. That’s roughly a 13x increase in creative outputs: a pretty serious amplification of Till’s creativity. 

Or course, it’s one thing to write a musical, but was it any good? One reviewer described it as “risibly stereotypical but pleasant as a milky drink.” Even an associated research paper described it as “a plausible and acceptable musical.” 

Before we dismiss it though, “stereotypically bland” describes a lot of television production. It’s not high art, so maybe machines do have a role in creating hundreds of hours of “stereotypically bland!”

Even Amplified Creatives should focus more on the creativity than the “amplified.” Don’t use Machine Learning just because you can. 

This is also a good time to re-iterate that very little is created by only one person. Filmmaking, stage production, television production, etc are all team sports, so it’s probably unreasonable to expect AI to do it alone.

Looking at some examples will give us a fuller idea of how various creatives have experimented with ML for storytelling. 

Car Commercial

As an experiment, filmmaker Kevin Macdonald directed a 60 second commercial using AI to generate the script: 

“The computer produced a script featuring a sentient-like Lexus ES that hits the open road, whizzing by stunning vistas of shoreline and forests before saving itself from a dramatic crash.

“I thought this was something amazing that had all these ambiguities in it and strangeness in it,” Macdonald said. “It’s only a matter of time where the formula of what makes up a great story, a great character can be learned by a computer.” 

The LA Times article teases with the “AI writes script” trope, but in fact re-iterates the marketing and script reviewing tools I discussed above. The example – a 60 second car commercial with sweeping vistas – hardly makes the case for AI originality.

Science Fiction

In a more experimental mode, Columbia University School of Arts’ Digital Storytelling Lab has been working on AI in more experimental filmmaking with projects like the aforementioned Sunspring and Frankenstein AI: A Monster Made by Many. While way out of the mainstream, these type of experimental projects are valuable for opening the field of thought, and perhaps seeding an idea further into the future.

At the current stage of development these projects end up being hilariously bad (Sunspring) or rely on humans to complete the project . 

On Impossible Thingsthe machine did the data analysis and suggested the story points that would resonate with an audience of horror movie fans. Apart from suggesting that the general story be about ghosts and family relationships, the machine made very specific instructions for the trailer (“needs a piano and a bathtub”). The actual writing was done by talented humans.

Music Video

Eclipse is a pop video featuring a French Band that had “AI as its director.” While what was done was interesting and more than a little amazing, it’s a stretch to claim the AI “directed.” There was no AI auteur involved, A series of Machine Leaning models were used for different processes along the way, starting with the music track. 

From there:

• The first machine gave script and a treatment, derived from the music
• Machine 2, took the treatment and directed cameras on drones for the shooting
• During editing another machine decided where cuts should be made according to the beat of the music and its emotional content.

Even the cast were selected by examining EEC brain scan data from actors and matched them to the emotions detected in the singer on the original track. Interestingly, the producers (independently) made the same casting choices. Perhaps it’s not only voice casting that could fall to the machines. 

The final comment in the BBC article sums it up:

Eclipse will not win any awards for creative film-making, and its director won’t be snapped up to make a Hollywood blockbuster.

But AI is advancing every day – and a decade from now, actors may find a computer sitting in the director’s chair shouting: “Cut!”

The Black Box

In a 2020 article, A.i.Can Make Music, Screenplays and Poetry. What About a Movie?, Mike Pearl conjectures a future where he describes a putative future “Black Box” where you select from options like genres, plots, types of character, locations etc, and even content keywords, and “Black Box” will create a fully “produced” unique movie that fits all the desired characteristics. He describes the theoretical process that could make this happen, and pretty much every quoted authority agrees it’s highly theoretical.

Duke University AI scientist Davit Carlson Ph.D. thinks that someone will eventually try to achieve the Black Box, but even he says:

“That’s a long way off. You know, years at the minimum.”

And ultimately the article ends up where everyone else has: it will be a collaboration, another form of amplified creativity.

Documentary Production 

Lance Weiler wrote a very good and wide ranging 2018 article Here comes the Automation, in which he describes Tommy Pallota’s More Human than Human  (original page unreachable) documentary where Mr Pollota documents his journey to try and replace himself as director with AI Tommy Pallota is responsible for the rotoscoping that brought Waking Life and A Scanner Darkly into existence, and is a multi Emmy award winning director himself.

Interwoven with his efforts to collaborate with (and ultimately be replaced by) an intelligent machine are an examination of AI’s rapid acceleration and its impact on humanity.

The project was elaborate, involving robotic arms, robots interviewing people and being part of the edit team. 

The robot could formulate questions, read emotions, detect body temperature, determine whether someone was lying as well as recognize a human’s age.

I will note, that I do not detect body temperature, reliably tell if someone is lying, or reliably recognize a human’s age, so the machine has that going for it!

Ultimately with the current stage of AI’s involvement in storytelling is best summed up by Mr Pallota: 

“I started to realize that telling the story was harder for me in a way than the technology part of it.”

---

Introduction, AI and ML, Amplified Creativity

• Introduction and a little history
• What is Artificial Intelligence and Machine Learning
• Amplified Creatives

Amplifying Storytelling

• Can machines tell stories? Some attempts
• Car Commercial
• Science Fiction
• Music Video
• The Black Box – the ultimate storyteller
• Documentary Production

Amplifying Pre-production

• Greenlighting
• Storyboarding
• Breakdowns and Budgeting
• Voice Casting

Amplifying Production

• Automated Action tracking and camera switching
• Smart Camera Mounts
• Autonomous Drones

Amplifying Audio Post Production

• Automated Mixing
• Synthetic Voiceovers
• Voice Cloning
• Music Composition

Amplifying Visual Processing and Effects

• Noise Reduction, Upscaling, Frame Rate Conversion and Colorization
• Intelligent Reframing
• Rotoscoping and Image Fill
• Aging, De-aging and Digital Makeup
• ML Color Grading

Amplifying Actors

• Multilingual Newsreaders
• Translating and Dubbing Lip Sync
• Deep Fakes
• Digital Humans and Beyond

Amplifying the Pixel

• Generating images from nothing
• Predicting and generating between first and last frames.
• Using AI to generate 3D holograms

Amplifying Metadata

• Visual Search
• Natural Language Processing
• Transcription
• Keyword Extraction
• Working with Transcripts

Amplified Editing

• We already have automated editing
• Magisto, Memories, Wibbitz, etc
• Templatorization
• Automated Sports Highlight Reels
• The obstacles to automating creative editing

Amplifying your own Creativity

• Change is hard – it is the most adaptable that survive
• Experience and practice new technologies
• Keep your brain young

As we get closer to “Computer Enhance” there’s a big problem

Artificial Intelligence driven image uprez has been improving dramatically over the last few years. There are already commercial products like Topaz’ Video Enhance AI that seem like they will inevitably lead to the “computer enhance” featured in so many science fictions shows, where it is used to pull some tiny detail out of an image, or a license plate, etc.

Ironically, that’s probably one thing it cannot be used for.

Google have showcased a new approach whose development had been sidestepped in 2015 to allow other approaches to uprezzing to be explored. It seems like the original idea developed to fruition can create amazing results. From a Cined.com article:

How does that work? Well, the team took a high-resolution image and added noise until only pure noise remained. They then trained a neural network to reverse the process in order to recover the initial image. Finally, this trained neural network is used on images at 64×64 pixels to create superscaled versions. Jonathan and Chitwan then used this process in a stack to further improve the process. By stacking a 64×64 ? 256×256 model with a 256×256 ? 1024×1024 model, they were able to achieve impressive upscaling results.

https://www.cined.com/new-google-ai-image-upscaling-makes-science-fiction-a-reality/

Making super resolution out of noise seems like alchemy to me. It will most certainly have many uses, particularly incorporating legacy material into more modern productions shot at higher resolutions.

As for the “computer enhance” problem. The “detail” is being invented, based on what the AI thinks it should look like. For realistic images this is excellent. For enhanced detail you could rely on for detective or legal purposes, almost certainly not. The detail is, quite literally, made up!

Thoughts on Frame.io and Adobe

Now I’ve gotten over the surprise of the sale price, it really doesn’t matter what I think. What it comes down to this:

To Adobe, Frame.io is worth $1.275 billion. Ultimately that’s the bottom line. Adobe invest in anything they think will add value to the company and pay what makes sense for them.

I’ll admit, the ascribed value is close to double what I would have expected for a Frame.io exit, and that was a bit of a shock. What I wasn’t taking into account is that something created and working, is vastly more valuable than the best idea ever. Frame.io took an idea and executed, creating an entirely new workflow in Camera2Cloud.

I’m sure Camera2Cloud is where Adobe see a lot of the value, and rightly so. Elegant review and approval is a neat achievement, adding C2C has truly created something new and elegant. As technology develops what is currently a rarefied workflow will become more and more accessible. I can see why Adobe would want to be leading there.

No matter how talented your engineering team is, there’s no guarantee that an Adobe team could have replicated what Frame.io have already completed. Going it yourself, has inherent time-to-market disadvantages, and an ongoing problem playing “catch up” with the pioneer.

I can’t pretend I understand how all those variables are ascribed a value to get to the final deal, but I don’t have to! The deal was done and both parties are happy.

So, unqualified congratulations to Emery Wells and the Frame.io team.

EditMentor sets a new standard in educating future editors

Up front I should state that I have been a fan of EditMentor’s founder Misha Tenenbaum’s work since he first founded EditStock.com. I was excited when he first hinted at what he intended with EditMentor, although I did not envy the challenge.

You see, I have a little insight. For a large part of my career I was an educational designer, and came up with the concept and content for our series of “Intelligent Assistants” – ultimately for Final Cut Pro (classic), Boris Red et al, Cleaner and Media 100. Our goal was simple: let people learn how to use a NLE or similar tool, while working in that app.

That took significant work, but by comparison with what Misha and CTO Gor Vardanyan have done it was nothing. Teaching functions in an NLE is easy. Teacher people why they should use those fi\unctions is not easy.

EditMentor is based around a functional NLE, that runs entirely in a browser (specifically Chrome). A NLE interface that is smart enough to report clip, marker and trims to the back-end for scoring! Add in a student management system, and a complete set of tools for building lessons, EditMentor is an amazing technological achievement.

Except, I think that’s not EditMentor’s main achievement. Working through the challenges never feels like learning. The challenges level up slowly, and feel “easy.” Well, they are easy when you read the Challenge and don’t jump ahead thinking you already know what to do! Who would do that? Oops.

EditMentor doesn’t care. Miss an answer, and there’s the explanation as to what is correct. You can go back and retake a challenge after learning the answer, because it’s all about learning. Still uncertain? Go back and review with all the answers displayed!

When learners have the freedom to fail without consequence, they want to learn. If I needed to go over a Challenge a couple of times to work it out in my head, I’d have all the freedom I need.

From an Educational Design perspective, the lessons are well set out, with each Challenged contextualized, before being presented with some “rules of thumb” to use in approaching the Challenge.

Challenges use the full range of NLE tools: navigating Bins, Editing to the Timeline, a Range of basic editing and trimming tools, and a functioning multitrack timeline. Wherever the student progresses, they have a good foundation to take forward to working in any NLE. The Lessons and Challenges are well designed.

EditMentor’s challenge is to going to be to fill out the curriculum, which they are acutely aware of and working hard on.

Although EditMentor is working with some of the top educational institutes there is a free option for individual students. Combined with the readily available phone cameras and editing tool, there’s no reason anyone with a desire to learn editing skills to not learn.

This “quick” review has taken me far too long. I keep feeling that I should add more to it, but each time, I find myself simply repeating myself, raving about how good EditMentor is. If you want to learn, or your role is helping other people learn how to edit video, then you must check out EditMentor.

Rotoscoping and Infill tedium gets a new Machine Learning Replacement

In the back of my mind I always thought, if I had it to do over, I’d go into Visual Effects. The combination of creativity and technology that goes into VFX fascinates me. Then the reality of the industry that I read about and its financial woes, makes me rather glad my life took a different path.

But what would VFX be like if the tedium – rotoscoping and background removal – were simply a couple of clicks away. The video below compares “traditional” (up to Rotobrush 2 in After Effects) rotoscoping tools, against RunwayML for isolating moving actors, and in filling. Stick around to the end for a preview of ML creating cartoon images based on text descriptions!

Machine Learning drive RunwayML is a web service, also used in this real time – 10 minute – demo of rotoscoping three separate clips and exporting the results.

These incredibly powerful tools are only the beginning of how Machine Learning is transforming almost everything visual: from production tools like RunwayML, to deep fakes like the recently distributed one of Tom Cruise.

Machine Learning is making huge strides in almost everything visual. It’s a pity it’s not advancing as fast at deriving useful keywords!

These tools are, however, the way I see Machine Learning (a.k.a. Artificial Intelligence) integrating with human creatives to devote more time to polishing and finishing the effect, to to process effects shots that would never be done otherwise. That last part is the exciting part for me.

Subscribe to The Corridor Crew for a bunch of interesting VFX videos.

Or for more of the applied techniques ActionVFX for the “Best Stock Footage for Visual FX.”

Reviewing Larry Jordan’s “Techniques of Visual Persuasion”

When Larry Jordan asked me if I knew anyone who might like to review his new book, Techniques of Visual Persuasion I immediately volunteered myself. After all who doesn’t want to “Create powerful images that motivate?” Plus, I always feel I can learn more about marketing and motivating people.

The first chapter of the first section – Persuasion Fundamentals – sparked off ideas as I was reading. It seemed like every new topic inspired and idea I could use to improve my marketing.

There is so much good information in the first chapter alone that I’m scheduling time to reread and apply the lessons.

The balance of the first section takes the principles from chapter one and demonstrates how it applies to create persuasive writing, images, fonts and colors. It’s a nice balance between the theoretical challenge and practical application.

The first section was the most valuable for me, because it was the newest material. The book is a nice balance between the techniques of persuasion and the techniques of creating persuasive writing, images and video. A marketer can rely on the practical production techniques, while the visual creator can learn from the techniques of persuasion.

In the areas of the book where I have some expertise, it was reassuring that none of the basics had changed! Larry carefully, and methodically, draws the connection between the theory and why (and how) it applies when you pull out the lights and cameras.

With such a visual subject, you would expect a well designed book, and New Riders delivered an all color, very handsome work. The illustrations are spot on and make understanding the “point” comfortable.

An unexpected, and welcome, inclusion is a chapter on Persuasive Presentations. Personal Presenting is a skill I continually strive to improve, so I devoured that chapter and will take a few new skills to my next presentation. (Perhaps Larry can write an extra chapter “Persuasive Zoom Presentations!)

The ‘Persuasive Practice’ for that is Pecha Kucha (or “little talk” in Japanese): a 20 slide, 20 seconds per slide presentation. Taking up the challenge I created a presentation of 20 Lessons from Learning to Sing. (I took up singing as a mature adult about six years ago.) It was an excellent challenge for me. Creating a precise 20 seconds from my verbose starting point was good practice.

The presentation has now become a 20 day series my vocal coach will be publishing later this month.

Techniques of Visual Persuasion is full of useful, practical information that both marketer and creative will find compelling, packaged in an entertaining and easy-to-read format. If you want to improve your persuasion techniques, and who doesn’t, then I thoroughly recommend Techniques of Visual Persuasion.

Disclosure: I asked to review Larry’s book and was provided with a review copy from the publisher. I received no other compensation.

Content Creating Entrepreneurs

I do not immediately think about satirist Randy Rainbow as a Content Creator, but with a body of parody music videos that have hundreds of millions of views combined, he is obviously a successful Content Creator.

As an aside, I am totally jealous of his ability as a lyricist, but that’s another story.

In a recent CNN interview Randy Rainbow talked through the process he takes for each new parody music video. It turns out he shoots and edits himself, on Final Cut Pro X. (FCP X is for those who create for themselves, etc.) He makes his income from producing videos, so is he a “professional?”

There are not many performers with millions of followers/views that have complete control over their career. Certainly not in traditional media, but for those who want to create content finding their own niche and audience isn’t as impossible as it was to get to the top in “Hollywood” and be a star that way.

Sam Mestman weighed in on what’s needed to be a modern content creator in a recent article at FCP.co.

The Top Ten earners on YouTube earn between $15-22 million a year as Content Creating Entrepreneurs. Musical artist Troye Sivan started building his audience on YouTube when he was 13 living in Perth, Western Australia (not the center of the entertainment world). He just had his best year ever with a World Tour and new album.

When you bring the audience you end up with a lot more negotiating power when it comes to distribution, merchandising and marketing deals.

Not just content creators but entrepreneurs as well. But are they “professional” editors? Definitely not. Professional Content Producers, definitely.

We have a 10 Year Old App!

While it’s not the first app we released, Sequence Clip Reporter has now been serving the Final Cut Pro, Premiere Pro and Resolve communities for 10 years! That’s an impressive achievement for a small Independent Software Vendor.

Greg and I are fortunate that we are able to make a decent income serving our community with tools that give them more time to do what they really want to do, instead of the boring parts of post. We’ve seen other ISVs move on to employment where they can get a more stable return for their skills. Being an ISV is a hard business.

Sequence Clip Reporter arose from a suggestion by our friend Les Perkins after I demonstrated some (now defunct) apps Greg had written. In those early days we had no idea what software helpers people needed, so we tended to follow up on most suggestions. Sequence Clip Reporter was an exception as most ideas turned out to fit a fairly narrow workflow.

Sequence Clip Reporter was created for FCP Classic, and we expected it to fade out over time, but with a little tweaking to accommodate Adobe’s custom version of FCP Classic XML, it’s now a valuable tool for the Premiere Pro community. It turns out Sequence Clip Reporter is the better of our reporting tools for Resolve as Resolve is a track based NLE.

Of course Producer’s Best Friend is our FCP X reporting tool that fits in where Sequence Clip Reporter does for Premiere Pro and Resolve. While superficially similar – they both produce all types of reports – the structure and the way the two apps work is remarkably different.

Happy Birthday Sequence Clip Reporter.

The Terence and Philip Show Episode 85: Is It Time to Retire?

The discussion in this episode started with a recent article in The Atlantic Your Work Peak Is Earlier Than You Think. Is it time to think of a career as an evolving series of “careers?”

After listening to the show, what you you think?