What is Rotoscoping? Ultimate Guide to Rotoscoping in After Effects

Have you ever been in a situation where you want to remove or extract a moving object from a shot?

Each and every way you try to extract them seems to fail terribly right? Well, this is where Rotoscoping comes in to save the day! So what exactly is rotoscoping?

Table of Contents

What is Rotoscoping?

This is the animation of masks across a series of frames by separating an image into a multi-layered space.

This term comes from the good old days when artists used to manually trace over the frames of films one frame at a time in order to add visual effects to a shot.

They would then project each frame of the film onto a transparent surface where artists would do their work. The name of the device they were using to project the image is Rotoscope.

Now that there is a shift into the wonderful world of computers the term rotoscoping/ Roto (for short) remains the technique of tracing over a visual element frame by frame for the purpose of extracting or separating it from the footage.

A Guide to After Effects Rotoscoping

Since the digital switch, there has been the creation of so many visual effects compositing tools, one of them being Adobe After effects, and in a nutshell, these tools let you create masks that you can animate along with the movement of the object you want to roto. The question most of you are asking is so how do you roto in After Effects?

Well, there are two methods to actually do that:

Using the masking tool

Using the Roto Brush

Masking tool

This is where you use the pen tool in After Effects to outline the subject or a part of the frame you want to separate from the background/ the rest of the frame.

The masking process in AE can be very tiresome because you’ll be adjusting your mask frame by frame. Also, when since you’re using the pen tool, it can be really hard for you, (a beginner) to create bezier lines for your mask. To help you with that you can use this tool to help with that.

Roto Brush

If in the past you have had the chance to try masking before, you’ll bear me witness that it is quite time-consuming and tiresome. My favorite way of doing roto is by using the Roto brush in After Effects.

The cool feature about this is that you get a better-refined edge around the elements you’re rotoscoping compared to the masking technique.

Tips to better Rotoscoping in After Effects

Rotoscoping is not the most exciting part of VFX compositing as it can be very time-consuming and repetitive.

While there are tools that can help you reduce the amount of time you rotoscope, learning how to do it correctly can really come in handy.

Here are some tips to help you speed up the whole roto process.

Analyze your shot. Roto will take most of your time that’s for sure, so the first thing you’ll need to do is take a close look at your shot. Take note of things like the duration, do you need to roto everything on the plate? Do you even need to roto in the first place To answer these questions you need to do what I call VFX Blocking of your shot – This is where you add VFX elements on top of your clips.
Break up your shapes. Once you already know what to rotoscope, you’ll need to take a closer look at your silhouette and break it up into multiple shapes. Do not do one big mask for your entire shape, instead break it into multiple masks. This is because of the fact that if you have one shape you are going to have so many points in one mask hence making it difficult when adjusting it.
Use motion tracking. If your clip has motion in it, get as much tracking data from your plate as much as possible as it will always cut down on time. If you actually do this you’ll be like 40% through to your goal. For tracking, you can use tools like Mocha AE in After Effects.

Look for key movements. Since the first thing that you did was to analyze your shots, you’ll know how much movement your subject makes. Don’t go for straightforward rotoscoping instead look for key poses of your shot and cover those first.
Color code your masks. As mentioned above the best way to roto is by breaking up your shapes and once you do that you’ll have a lot of masks. To help in the process of identifying and distinguishing these masks, the best way to go about it is color coding your masks.
Dealing with motion blur. Rotoscoping motion blur is really quite a challenge due to the blurry details not having any opaque color. When dealing with this, the rule of thumb is to always roto the solid opaque part of the motion blur first and then proceed to roto the blurry part by adjusting the size of the mask using the feather feature in AE.

Best practices to speed up rotoscoping in After Effects

Assuming that you plan to do rotoscoping later in post-production and you don’t want to use a greenscreen, ideally, you’d want to keep in mind which visual elements you will later isolate or extract before you shoot your scene. If you can’t use a green screen, there are a number of things to make the process easier:

Use a higher shutter speed when shooting to reduce motion blur. Motion blur can be very difficult to deal with when rotoscoping plus you can always recreate motion blur in most editing software.
Use proper lighting techniques to visually separate elements. Ensure that your scene has proper lighting and that the objects that you’re planning to roto are clear and stand out distinctly against all the background elements.
Film necessary clean plates. Make sure that you shoot any clean plates that you may require to make it easier for you to separate the visual elements in post-production.

Conclusion

Even though rotoscoping can be a very cumbersome process, learning the whole process, understanding what rotoscoping is, how to do it in After Effects and actually implementing some of the pro tips on how to speed up the whole process can really help up your filmmaking game and add that unique wow factor to your project.