How to Rotoscope in After Effects CS5

Adobe After Effects is one of the best visual effects programs on the market, and it's latest version, CS5, offers users the ability to easily rotoscope. Rotoscoping is a word with two definitions in the entertainment industry. When we talk about rotoscoping in animation, we refer to animator literally tracing over live action film or video.

When we talk about rotoscoping and visual effects, we are referring to the technique of manually removing a subject from it's background in a live action video, which will then composited on a different background. Normally this is done with green screens and chroma key. If you wanted to remove a subject from the background on a normal live action clip, the process was tedious and had to be done frame by frame. But, thanks to a new tool called the 'Rotobrush', rotoscoping can be easily performed in After Effects.

Step 1: Find the Video You Wish to Work With

Open After Effects and make sure that the video clip you wish to work with is in the 'Timeline'. Click on the video in the 'Timeline Panel' to work with it in 'Composition Panel'. This way, the rotoscoping you perform will only affect this layer. Also, make sure that the marker is positioned at the first frame of the clip.

Step 2: Use the Rotobrush

The 'Rotobrush' is located on the top tool bar. Click on it to work with it. The 'Rotobrush' works very similar to the 'Quick Selection Brush' in Photoshop. Click on you mouse and move the 'Rotobrush' around your subject. You should notice a pink outline forming around the subject. That means that your image is automatically being grabbed. Make sure that you look over the frame very carefully to ensure that the entire subject is being grabbed. You don't want to be missing a piece of clothing, a shoulder or an ear. 

You also need to tell the 'Rotobrush' what you don't want to cut out of the frame. Hold down the option or alt key and paint across your background with the mouse to do this. 

Step 3: Apply Rotoscoping to the Entire Clip

Start displaying forward down the timeline, and the program will automatically adjust the edges to the movement. It makes its determinations from shape and color information. But, keep an eye out because the system is not perfect. You may need to pause and make the occasional adjustment here and there. Once this is done, you can switch to composition view to see how the effect has turned out.

Step 4: Make Adjustments

You might notice that the subject has a very harsh edge in certain areas. You can clean that up with a feature in the 'Rotobrush' called 'Refine Matte'. This will help add some blending to your image so it doesn't look so much like a cut out.