Author Topic: Can we improve the normal baker?  (Read 1806 times)

Hi everyone, I would like to discuss something in regards to baking done in Substance Painter. If some guys from Allegorithmic would join us to speak from the perspective of their technical expertise — it would be really great, because I'm not sure that I understand everything correctly.

So, the thing is that recently I was trying to troubleshoot a problematic Substance Painter normal map bake on Polycount. After the source of the problem was identified, I thought that it was a pretty interesting one. The day after I have faced the same issue personally when I was trying to redo UV mapping and baking on the Sci-Fi Container from a Wes McDermott's course "Substance Painter Texturing for Beginners" (by the way, thank you, Wes!). I've decided to start this thread after it came to my understanding that such kind of issue could be not so rare in practice. Here's the link to the Polycount in case someone would like to read through the mentioned thread: http://polycount.com/discussion/167576/normal-map-baking-hard-edge-problem/p1

In short, some specific lowpoly/highpoly geometry configuration can produce non-obvious shading issues in baked normal map. Let me illustrate this with an example. See first image attached below: https://forum.allegorithmic.com/index.php?action=dlattach;topic=9519.0;attach=15757;image.

So there I have a simplified geometry setup that illustrates this possible scenario that can produce some shading issues. In this example I have a lowpoly that consists of two quad faces with a hard edge between them and a highpoly that has some concavity in its corner. Nothing special about baking setup — it is Substance Painter 2 that bakes without a cage with Average normals = ON.

The problem is that with configuration like this (and many more that is similar) some baking rays projected for the bottom lowpoly face around the corner of a mesh will hit that concave area of a highpoly, due to how baking rays are distributed when using averaged normals projection. To my understanding, because the normal map will be computed in regards to a flat lowpoly surface normals (remember, the hard edge is at the corner), it will record a pretty extreme value in the blue channel of a normal map (around RGB 113, actually). On a practical side this will result in that when such normal map will be applied to the surface, a rendering engine will treat this part of a polygon face like it is oriented "backwards" in relation to the actual geometry and its face normal. I have a more detailed explanation on this in the Polycount thread linked above (though it's more like a theory of mine). Short version: if value in the blue channel is above 128, we "can" see the surface and it is oriented towards us. The value of 128 is the last that actually meant to be seen on the object's surface and it means the surface is parallel to the camera. If value is 127 or lower — the surface is effectively facing away from us, so we shouldn't really see it.

When rendered inside the application, be it Substance Painter or some game engine like UE4, it looks like picture shown below. First is the actual geometry from an example above, second and third is the Sci-Fi crate I've talked about, with lightning being cast at different angles. Notice how shading works on a problematic area — it is dark when in the light and bright when in the shadow. Looks weird, like an artifact. See second image attached below: https://forum.allegorithmic.com/index.php?action=dlattach;topic=9519.0;attach=15759;image

It is important to note that shading issues discussed is unlikely to happen when using all-soft lowpoly mesh, because due to softer lowpoly surface normals it is less probable that blue channel of the normal map will have values lower than 128.

So, my question is — should we really allow the baker to record in the blue channel of normal map values below 128? As I wrote above, to my understanding such value means that we're basically looking at the polygon from inside, i.e. it is backface to us. And this normally shouldn't happen, no? Is there any practical scenario when we would want to have such "backfacing" normals on a normal map applied to the mesh? Maybe some other operations or features use this data in a way I am not aware of?

I mean, if we're to clamp all blue channel values below 128 to actually record as 128 or 129, this will prevent such shading issues to come up and will make the baker a bit easier to handle, especially for less experienced users. I totally understand that in examples shown there's clearly some problems with the shape of a highpoly (no concavity should be allowed in situations like that) and how close the highpoly and the lowpoly are aligned, but it's not the question of how to "fix" some particular bake setups. At the same time, I'm pretty sure that such configurations with lowpolys that use some hard edges to reduce gradients on normal map do pop-up here and there pretty often, and it can be pretty confusing to troubleshoot them without knowing where to look specifically (blue channel) if such shading issues are to come out.

Any thoughts on this will be much appreciated. Thanks!
Last Edit: April 06, 2016, 07:15:49 pm

I am not sure that clamping the normal values above the tangent plane in the baker is a good solution. It might actually make things worse because, as you wrote, the correct solution is to make the low and high poly geometry closer to one another. Clamping the values actually hides the problem and makes it more difficult to localize and identify, and eventually to solve. Worse, it may hide other and more common problems, like wrong intersections with different geometries when using incorrect min-max distances or too large cages.

Also, even if we don't see a compelling use case for tangent-space normals going under the tangent plane, that does not mean there isn't one, nor that there never will.
Last Edit: April 07, 2016, 03:01:20 pm

Makes sense. Thank you for your response.