Tracker2Mask: new tracker-assisted rotoscoping tool

mmohl · **Joined:** Mon Nov 24, 2008 10:40 am **Posts:** 20

Hi,
I have written a script, tracker2mask.jsx, that infers the movement and change of the shape of a mask based on the movement of trackpoints. You don't need one trackpoint for each maskpoint.

The workflow is as follows:
1) track a few points (something between one and four points is reasonable, even for complex mask shapes) that represent the basic movement of the object which you want to draw a mask around
2) keyframe the mask by hand in the first and last frame
3) use the script to predict keyframes in between (the script uses the tracker information for this prediction which is hence much better than the simple interpolation done by after effects itself)

You find a video tutorial that explains the usage in detail here:
http://www.vimeo.com/2393127
On the same page, there is also a download link for the script.

Any feedback/suggestions/comments welcome.

UPDATE:
Here is a second tutorial that explains some new features of Tracker2Mask Version 2.0:
http://www.vimeo.com/2535683

lloydalvarez · **Posted:** Thu Dec 04, 2008 8:50 am

Hey Mathias,

This script is absolutely brilliant!! Congratulations.. This script along with TrackerViz have brought AE's tracking game back up to the premier league!

I also love that you took the time to explain the falloff and filter settings. Your explanation was very clear and made it very easy to understand something that is very complex under the hood..

Just from watching the tutorial I have a small UI design suggestion. I imagine that it will be easy for the user to forget that you have a selection frozen so what if the button turns red (or blue for ice maybe :wink:

) when there is a selection frozen.. you can't do this for the AE7 version but you can for the CS3/4 version using the ScriptUI graphics.. Let me know if you'd like an example of how to do this..

-Lloyd

mmohl · **Joined:** Mon Nov 24, 2008 10:40 am **Posts:** 20

Thank you for the feedback.
I would be interested in example code of using the ScriptUI graphics.

Maybe you (or anyone else) can help my also with some other problems that I have to solve for some new features I currently implement:

In the script I have to translate all maskpoints and trackpoints from layer-coordinates to comp-coordinates and back, to allow the points to come from different layers. This seems to be a standard problem that must occur in many scripts, but I have not found a solution. Paul Tuersley proposed to use expressions (see this thread), but this slows down the computation a lot and I have to compute many of the transformations for each generated keyframe. I implemented something that takes into account position, anchorpoint, scale and parenting of the layers but in principle I should also take care about rotation and I am not sure about pixel aspect ratio.

Does every layer has its own pixel aspect ratio?

Is there some technical difference between shapes of masks and shapes of shapelayers? I want to allow the modification of shapelayers in the same way as masks but somehow shapelayers behave different that masks on solids. I assume, something goes wrong while translating shapelayer points to comp-coordinates (maybe this is some pixel aspect ratio problem? Do shapelayers have by default another pixel aspect ratio than the comp or is there any other relevant difference to solids?).
Shapelayers as well as the entire coordinate system transformation stuff seem to be poorly documented in the scripting guide.

Mathias

lloydalvarez · **Posted:** Fri Dec 05, 2008 7:30 am

Here's some example ScriptUI code for working with color:

var winGfx = pal.graphics;
var darkColorBrush = winGfx.newPen(winGfx.BrushType.SOLID_COLOR, [0,0,0], 1);
var greenColorBrush = winGfx.newPen(winGfx.BrushType.SOLID_COLOR, [0,1,0], 1);
pal.grp.main.txt.foregroundColor = darkColorBrush; // this sets the font color to black
pal.grp.main.backgroundColor = greenColorBrush; // this sets the BG color to green

As for the world coord transforms, i'm afraid you might have to do them yourself.. Scripting is missing a few of the expression built in function and this is one of them..

Every layer does have it's own pixel aspect but you can read that easily:

app.project.item(index).pixelAspect

I am not very familiar with scripting masks/shapes but Jeff Almasol has a shapes to masks (and vs) scripts that might help answer some questions:

http://www.redefinery.com/ae/view.php?i ... ksToShapes
http://www.redefinery.com/ae/view.php?i ... pesToMasks

Hope that helps, let me know if you have any more questions...

-Lloyd

mmohl · **Joined:** Mon Nov 24, 2008 10:40 am **Posts:** 20

A new version of the script is online. It allows the local manipulation of parts of a mask which is explained in this new tutorial:
http://www.vimeo.com/2535683

I also tried to colourize the freeze-button, but it did not work since buttons do not seem to have a backgroundColor property and then I got tired trying.

I also took a look at Jeffs Mask to Shape scripts. Thank you for the hint! In his "known issues" Jeff mentions the same problems/bugs as I have. In principle the script now also works with shapes but it does not take any transformation of the shapes into account, i,.e. it does not translate the shape point correctly from the local coordinate system to the comp coordinate system.

lloydalvarez · **Posted:** Tue Dec 16, 2008 8:07 am

Hey Mathias,

Interesting about the button, hadn't tried it myself.. One idea would be to put the button in its own group and change the BG color of that group. Not sure what that would look like but maybe it would be like an outline..

-Lloyd

jaist · **Posted:** Wed Dec 17, 2008 8:05 am

Hi, that's a wonderful script!
Its interface could use some redesign, but more importantly, you should let it talk a lot more with trackerviz!
Some people love using trackers instead of trackpoints, some other just rely on positional data from little solids, i think you should give more options..

P.S. i found a misspelling error in one of your functions:

function roateCounterClockwise(degree){

I don't know if it causes errors since i've yet to run the script, let's hope for the best!

Redsandro · **Joined:** Wed Jun 25, 2008 4:55 pm **Posts:** 108

Sounds awesome.

A sidenote, not directly related to the script, I haven't tried it yet.

How come the second video is not downloadable? (The first one is.)
Didn't you upload .mp4?

I like portability, download now watch later on a device without internet.

mmohl · **Joined:** Mon Nov 24, 2008 10:40 am **Posts:** 20

I must have accidentally disabled the download option. It should be downloadable now.

@jaist: Thanks for the typo hint. I will fix it, but it should not affect the script: I just did a copy and paste of my entire implementation of transformation matrices (see this thread), but this function is actually never used (including the file would have been cleaner, but I wanted to have all in one file to keep installation as easy as possible).
It is interesting that you look at the code even before trying it out :-)

If people really look at the code, I should probably document it more consistent...

Interaction with Trackerviz is a good point. I think about extending the script such that it interprets the position of each selected layer that has no property (as a tracker or mask) on it selected, as a trackpoint. You could use TrackerViz-Layers then in the same way as trackpoints and even combine them. Furthermore, only moderate changes would be necessary. The only tricky thing would be to generate the position-expressions for the circles on the control-layer (these would depend then on whether the respective trackpoint is a real trackpoint or a layer position)

jaist · **Posted:** Thu Dec 18, 2008 5:19 am

Well, the thing that you should nail first is this: change the selection process using trackers, not trackerPoints.
Since there's no explicit way of adding another trackpoint ( you can only specify up to 4 linked trackpoints per tracker or duplicate them inside the timeline) and you can't do separate tracks (you can only hope to get all of them working at the same time) you should require to choose the container tracker instead.
That way you could also introduce the first layer of compatibility with how trackerviz handles things.
Then you only have to use your matrix lib to also include layers.

I haven't checked the new features of version 2. I saw only 2 minutes of the video you posted and they were quite impressive, but i can see how those can be tricky to implement.

Paul Tuersley · **Posted:** Thu Dec 18, 2008 6:08 am

jaist wrote:

Since there's no explicit way of adding another trackpoint ( you can only specify up to 4 linked trackpoints per tracker or duplicate them inside the timeline) and you can't do separate tracks (you can only hope to get all of them working at the same time) you should require to choose the container tracker instead.

I'm not entirely sure this is what you're talking about, but you can add extra track points by selecting "New Track Point" from the Tracker panel's wing menu (top right of panel). Also, if you want to deal with each track separately, you can turn off the track point eyeballs in the timeline to stop them from all being tracked at the same time.

mmohl · **Joined:** Mon Nov 24, 2008 10:40 am **Posts:** 20

Thank's for this hint! I new about the possibility to add new trackpoints but I was also not aware of the ability to track the trackpoints of a tracker separately.

Concerning the trackpoint selection of tracker2mask, I think the following behaviour would be best:

- for each selected layer on which no properties are explicitly selected consider the position as trackpoint
- for each tracker that is selected but in which no trackpoints are explicitly selected consider all its trackpoints as trackpoints
- consider each explicitly selected trackpoint as trackpoint

Then you are free to either select single trackpoints or entire trackers or layers (or any combination of those).

jaist · **Posted:** Thu Dec 18, 2008 10:02 am

@Paul: I knew the wing menu option (that's why i said not explicitly, it does the same thing as duplicating the trackpoint) but i really didn't know the eyeball exclude, you've made my day!
I always tracked one tracker at a time and then performed operations on nulls by hand, this is nice.
Thanks!

@mmohl: Seems a fair approach. The only thing you should consider is that when you chose the layer you should have the code to exclude the trackers if you want to get positional data later. But other than that seems logical enough.
What really stands out talking about this is that tracking and rotoing are two tasks that usually benefit from each other.. so Hype maybe we should think of a way to standardize workflows between the two.
Personally i don't really know if the explicit way of selecting (like trackerviz does, layers/trackers from the listbox) is better that the implicit way (just selecting the properties or the layers like Tracker2Mask is doing-plans to do)
But It's quite confusing having the two tools behave differently.

hype · **Posted:** Thu Dec 18, 2008 12:30 pm

Quote:

I think about extending the script such that it interprets the position of each selected layer that has no property (as a tracker or mask) on it selected, as a trackpoint. You could use TrackerViz-Layers then in the same way as trackpoints and even combine them. Furthermore, only moderate changes would be necessary.

Quote:

- for each selected layer on which no properties are explicitly selected consider the position as trackpoint
...
Then you are free to either select single trackpoints or entire trackers or layers (or any combination of those).

I love this idea, especially if "only moderate changes would be necessary."! The ability to use averaged and corrected tracks (represented as little solids with Position keyframes) with Tracker2Mask would be wonderful!

TrackerViz only lets you apply tracks directly to specific mask points. Tracker2Mask lets you use the influence of tracks to affect mask points. So the two tools kind of perfectly compliment each other, making it possible to have tracks help with roto in almost every way imaginable.

This is so exciting that these two tools are out there now for us AE users!

Simma · **Joined:** Tue Jun 08, 2010 2:57 pm **Posts:** 89

Just tried this and it's really good.

But I have some questions:

- I often track in Mocha and end up with a null in AE containing the tracking position. Can I use position keyframes instead of tracker points somehow? Or do I manually have to create a tracking point and copy the position data in there?

- Is there anyway to use rotation as well as position when tracking a mask? A lot of footage requires rotation as well as position, and it would save a LOT of time if there was a possibility to use rotation in the mask.

Or maybe all these things are possible and I haven't played around with it enough yet

AE ENHANCERS

Tracker2Mask: new tracker-assisted rotoscoping tool

Who is online