Fixing jagged edges when compositing:

Usually you work with two types images when compositing: premultiplied and not-premultiplied (a lot of compositing software also have a unpremultiply function, to save the day if renders were set up wrong)

Great! but what the h... does that mean?

It means wether just the colour channels (rgb) or both the colours and the alpha (the mask or matte) was antialiased.

ok I think we need an example, .... right?


First maya's default setting 'premultiplied' :
 
RGB
Alpha

as you can see both the colour and the alpha is antialiased against the background.

This can actually get you in trouble if you need to composite the later on.

Why?
because right now the colour is antialiased against black, but, were you to composite this with a very bright background, you might be getting artifacts.


Then 'not-premultiplied' :
to get this turn on the 'composite' checkbox in RenderGlobals->RenderOptions
and leave the 'Composite threshold'  on 0.
 
RGB
Alpha

as you can see only the alpha is antialiased against the background.

This is perfect for compositing, since the colour hasn't been antianliased yet, it will blend perfectly against 'any' background when using the alpha, as mask.


Finally, for gamers.

In games you often (still..) only have one masking bit, so either its there or its not. Maya can do this aswell.
to get this turn on the 'composite' checkbox in RenderGlobals->RenderOptions
and set the 'Composite threshold'  close to 1 (like 0.99), for some odd reason setting it to 1 has caused my version of maya to, render the image (right) and then make it black???. In theory setting it to 1 should give you the right result.
 
RGB
Alpha

no antialiasing anywhere!

This is perfect for gaming.