Algorithmic Rotoscope

I had come across Texture Networks: Feed-forward Synthesis of Textures and Stylized Image from Cornell University a while back in my regular monitoring of the API space, so I was pleased to see Algorithmia building on this work with their Deep Filter project. The artistic styles and textures you can apply to images is fun to play with, and even more fun when you apply at scale using their API. While this was fun, what really caught my attention was their post on their open source AWS AMI for training style transfer models--where you can develop your own image filter.

I fired up an AWS instance, and within 48 hours I had my first image filter. For some reason, I went to bed that night thinking about drone video footage and began wondering if I could apply Algorithmia's filters, as well as the custom filters I create to drone footage. The next day I started playing with my prototype, to see what was possible with applying this type of image filtering to videos.

My approach has three distinct features:

  • Separate Video Into Images - Using FFMPEG, I created a function that would take any video I gave it and separate it into an image for each second of the video, and write them all to a folder.
  • Apply Filters to Each Image - After uploading all the images for each video to using their API, I would select one of their 30+ image filters or any of the filters I created.
  • Reassemble Video From Images - Once I've applied one of the algorithmic filters to all of the images I reassemble them back into a video.

While the videos I've been able to produce so far are interesting enough, I am intrigued by the process of defining and applying the filters. There are a handful of things going on here for me:

  • Defining Of The Algorithms Behind Each Image Filter - I'm having a blast with the image filters that Algorithmia provide, but their work got me thinking about how I can train image filters specifically for being applied to video in this way. I'm learning a lot of about the training process--something I want to keep working on.
  • Applying Algorithmic Filters To Images At Scale (Videos) - The videos I am working with currently have been 500 and 2000 individual images after I separate the video. I am learning a lot about working with video at the individual slide level, and have a lot more work ahead of me.
  • Changing How I Take Videos Using My Drone - I have been applying this work to 4K video I either took myself or was present when it was taken. As I'm applying these filters I am hyper aware of where the sun was positioned, how the shadows of rocks, trees, and buildings play in, making think deeply about how I can design future drone shots.

Algorithmia has enough filters to keep me interested for a while, but this is where I think the real value will exist--assembling well trained filters. If you can craft exactly the right filter, I think it can be used to define entirely new virtual landscape derived from the physical world around us. Some of the filters offer a pretty extreme shift in the video landscape, but some of them provide just enough polish to make you think you are in an alternate reality.

I'm not sure where I'm going with this work. I'm finding it oddly soothing to define new filters, and apply them to my drone videos from this summer. For now, I'm just thinking of interesting filters to build and apply the filters to the videos I already have. It also has me thinking about how I can take my drone out to get more shots--from both natural and urban landscape.

For near-term future, I'm just getting to know what the filters produce, and how certain drone shot(lighting, color, terrain) respond to each filter. It has me thinking a lot about what would make a good filter, and what types of images I can use to train image filter algorithms that can be applied in this way. If you want to learn more about what I'm doing you can submit a commit via the Github issues for this project, or you can contact me directly.