Author Topic: [TIP] Better AO bakes on objects with overlapping parts  (Read 4476 times)

If like me you've used substance designer to bake Ambient Occlusion on a model that isn't water-tight, but instead has overlapping parts, you may have noticed that the AO bake doesn't handle overlapping geometry very well. In areas with overlapping geo the outside of the overlap bakes fine, but inside the overlap, instead of being in dark shadow it suddenly transitions brighter. This tends to cause a bright halo around the edge of overlapping parts. Here's an example from the Pod object from the Pod tutorial.


You can fix this by always building your models water-tight, or lessen it by baking AO at a higher resolution and down-sampling, or by blurring the AO. If your object has a lot of high frequency details in that area like rust or dirt, it's probably not even noticeable, but if your art style is more clean and pristine, it can be a real issue.

I've stumble upon a trick however that fixes these overlap "misses" in the AO bake. I call it the double reverse normal trick. When baking AO, you check the "Invert Normals" check-box to invert the normal, but you also load a normal map that re-inverts the normals back again. I'm not sure why (maybe it forces the geo to be double sided during the bake), but this somehow fixes the issue.



For your inverted normal map, just create an image that's a solid 128,128,0 color. That's for a plain object AO bake, if you have an actual detail normal map you want to use, invert it's blue channel instead.



This will produce an AO bake where overlap areas are now in shadow, and the bright edge halo is replaced with a contact shadow instead.



It does result in a slightly lighter more subtle AO bake, but you can always add a levels node if you need a darker more intense AO. I've gotten so I use this trick on every AO bake I do now, I also haven't noticed any perceptible increase in AO baking time (but I haven't done any accurate benchmarking).

Interesting, thanks for the tip.

Indeed nice trick, the resulting AO looks a lot better.