Hypercinema - Runway Models and Concepts

This week, we’re looking at RunwayML. Three models and three concepts:

Kandinsky Model

This model style-transfers from the artwork of Kandinsky onto visual medium (video and images). The parent ML model is a generic style transfer model, but this particular one was trained on Kandinsky’s artwork and brings forth his vibrant colors, broad brushstrokes, and mixture of shapes.

There’s something extremely interesting to me about this model, particularly the idea that you could “paint” in motion. While we already have the ability to draw cartoons in a particular style, I like the thought of taking real, in-world motion and color, and using that to create.

You could take the motion of a dancer or a group of dancers and turn it into a painting. Really translating the vibrancy of a room, maybe combining it with some masking to generate an external environment or only focus on the people. I’d love to see this used to represent a kind of emotion or atmosphere that is hard to recreate otherwise.

Ascinte_Seated

This ML model is trained to apply an aged film-like distortion. All based on a single photographer’s photos, it’s fun to imagine plucking the degradation of an ancient work as the style to focus on, not the artists’ aesthetic intentions. We aim for so much in the world of aesthetics, but to value distortion flips that all on its head.

This reminds me of the “retro” nostalgia for Polaroid film. Those shots aren’t strictly better than the ones you can take on a DSLR today, but they hold a sort of charm that can’t be captured digitally. My concept for this model is to take modern pieces and disintegrate them, turning them into objects portrayed in ways that feel incongruous to their era. You could also pipe them back and forth between another model that “heals” or “repairs” images, and see what you lose in the process.

PoseNet

While a pretty familiar model, it’s still incredibly powerful. PoseNet takes images and video and makes a best guess at the basic skeletal positioning of people in that image. While used time and time again, I don’t think it’s lost its charm or potential.

I imagine using PoseNet to recognize particular poses, and allow a scene or environment to respond to a user’s movement. This would make an excellent set for performance, or a way to augment previously recorded shows. I think it would also be a powerful tool if turned on the audience rather than the performer, empowering audience members to influence and participate in the performances that they are attending, making each show individually unique.