Author Topic: Average and non-averaged normal blending  (Read 13353 times)

I am investigating if it is possible to use this technique in Substance Painter instead of designer:
https://www.youtube.com/watch?v=sGC2X6Qazvs

Furthermore, what is often forgotten (even by me) is that the normal map is not the only map suffering from skewing. It basically applies to all maps, most notably the material ID and AO map. These need to be fixed as well.

I suppose getting this blending to work in Substance Painter is done with a custom mask generator (or filter?), but you need to be able to bake a non-averaged bake and save it separately, so it won't get replaced by the subsequent averaged bake.

Any thoughts on how to make such an effect in SP?

Edit:
I figured it out. See the new thread here:
https://forum.allegorithmic.com/index.php?topic=12713
Last Edit: August 14, 2016, 08:32:36 am

I am investigating if it is possible to use this technique in Substance Painter instead of designer:
https://www.youtube.com/watch?v=sGC2X6Qazvs

Furthermore, what is often forgotten (even by me) is that the normal map is not the only map suffering from skewing. It basically applies to all maps, most notably the material ID and AO map. These need to be fixed as well.

I suppose getting this blending to work in Substance Painter is done with a custom mask generator (or filter?), but you need to be able to bake a non-averaged bake and save it separately, so it won't get replaced by the subsequent averaged bake.

Any thoughts on how to make such an effect in SP?

HI,

Can you post some screen shots of the skewing issues you are having? Are you using a synced tangent workflow between your baked maps and renderer? Skewing can also be affected by having straight edge geometry but the UVs edges are curved from an unwrapping algorithm.

Cheers,
Wes
Head of Substance Demo Art Team
the3dninja@adobe.com
Twitter: The3DNinja

Here is a screenshot (skewing.jpg). This is from the SP viewport, not an external render. Note the extreme skewing in the texture map. The hexagons are supposed to by symmetrical. The UV edges are not curved as you can see. If you bake without averaged normals, the skewing disappears, but black lines appear at the UV edges.

This is a common problem with all bakers out there and has been discussed at length at various forums. Not that it can't be solved at the baking stage, but it seems the will isn't there. So we are forced to work around it with time consuming workflows.

The most common way to fix this is to add extra support geometry. Unfortunately in my case I can't do that because all the models are from a CAD source. These models all have explicit (custom) normals and adding geometry will mess up these normals. Fixing this is extremely time consuming and is not really an option. One other way to solve this issue is to use a skew mesh, but that is only geared towards fixing the normal map, but people seem to forget that all maps are skewed, not just the normal map.

So in my case, the only way to fix it is by blending the averaged and non-averaged normal map. This can be done in SD as show in the video, but if I can make it work in SP, it will save a lot of time bouncing back and forth between different applications.

In theory, the UV to SVG bake can be replaced by a pixel processor looking at the normal map colors. The space between the UV shells is gray and that color doesn't exist anywhere else inside the UV shells. This can be used to detect where the UV borders are and generate a mask out of it. The additional non-averaged normal bake can be dropped in a generator slot. Next, I need to check if the Combine normal mixing is suitable to apply the additional normal map the regular way.

I managed to get the blending part to work using the guide here:
https://support.allegorithmic.com/documentation/display/SPDOC/Normal+Map+Painting
See "Painting over a baked normal map".

I made an averaged normal bake, exported the textures, and re-imported it into the shelf.
Then I made a non-averaged normal bake, and set the Normal Mixing setting to Replace.
The non-averaged normal map is added to a fill layer, and the averaged normal map to a fill layer on top of that.
The blending mode of the normal map layers is set to Norm.
On the top layer I added a black mask and hand painted white lines along the edges.
The result you can see in the attached screenshot. No skewing and no lines at the edges.

Next challenge will be to make a generator to auto-generate the mask and possibly writing a script to automate the setup process.

I managed to get the blending part to work using the guide here:
https://support.allegorithmic.com/documentation/display/SPDOC/Normal+Map+Painting
See "Painting over a baked normal map".

I made an averaged normal bake, exported the textures, and re-imported it into the shelf.
Then I made a non-averaged normal bake, and set the Normal Mixing setting to Replace.
The non-averaged normal map is added to a fill layer, and the averaged normal map to a fill layer on top of that.
The blending mode of the normal map layers is set to Norm.
On the top layer I added a black mask and hand painted white lines along the edges.
The result you can see in the attached screenshot. No skewing and no lines at the edges.

Next challenge will be to make a generator to auto-generate the mask and possibly writing a script to automate the setup process.

Very nice! Thanks for posting this workflow. The devs are looking into it as it's a known problem as you mentioned. At this time, we don't have an automatic solution. However, it might be easier to just add support loops on the borders to influence the averaged normal. This way, you can get a result without having to do all of the blending. However, this may not be an option if you are wanting to limit the vertex count.

Cheers,
wes

Last Edit: August 10, 2016, 04:02:59 pm
Head of Substance Demo Art Team
the3dninja@adobe.com
Twitter: The3DNinja

...it might be easier to just add support loops on the borders to influence the averaged normal. This way, you can get a result without having to do all of the blending.

Thanks for the suggestion, but unfortunately that won't work in my case, as I explained here:

The most common way to fix this is to add extra support geometry. Unfortunately in my case I can't do that because all the models are from a CAD source. These models all have explicit (custom) normals and adding geometry will mess up these normals. Fixing this is extremely time consuming and is not really an option.

But it is good to know the issue is being looked in to. The only other developers I know of having a lukewarm interest in solving this at the baker stage are from Knald and Handplane. Funny that together with SP, these are all relative newcomers. I suppose innovation isn't going to come from the establishment.

you can add divisions to your low poly mesh, bake a world space map and then convert that to a tangent base using your original, final low poly mesh. you'll get a perfect bake.  but I don't think this can be done in SP at the moment, it can be done in SD.

maybe Allegorithmic can implement this as an automated solution, would be awesome!
you load your low poly mesh, the baker will duplicate that, divide it, and use the divided mesh for all the bakes, then convert the normal map and load that into the additional maps slots like it's doing now.  I'm sure it can be done :]

It would be awesome if that can be automated, but I don't think I can currently use that method because the normal map is not the only map which is skewed. The ID, AO, and some other maps are skewed too, and I don't think you can use the skewmesh method to fix those.

I made a custom mask generator in Substance Designer which generates a mask from the UV edges. See the screenshot below. It is very simple and basically takes a custom normal map (baked in SP), converts the black color in the normal map to a mask, performs an edge detect, blurs it, and outputs the result to a mask.

To bake the custom normal map, you need to bake a normal map with the following settings:

Dilation width: 0 (to generate a texture printed tightly around the UV edges)
Average normals: true (to make sure no UV corners are cut)
Apply diffusion: false (to generate a black color between the UV shells)

A normal map baked with this settings has a black fill in between the UV islands. This black color is detected by the generator and it will use that to generate a mask along the UV island edges. A better approach would be a UV to SVG (or PNG) baker in SP, but unfortunately that is not supported. Please vote for that feature here:

https://allegorithmic.uservoice.com/forums/261284-substance-painter/suggestions/15625746-uv-to-svg-bitmap-baker

Next will be to explore the scripting API to automate as much as possible. Hopefully we will have an easy fix for a crappy baker soon.
Last Edit: August 11, 2016, 11:09:22 am

It would be awesome if that can be automated, but I don't think I can currently use that method because the normal map is not the only map which is skewed. The ID, AO, and some other maps are skewed too, and I don't think you can use the skewmesh method to fix those.
you'd use the divided mesh to run all bakes, but at the end of the process the world space normal map would be converted to tangent space map using the original low poly, to maintain correct normals.  the other bakes don't need this conversion process - they will give you accurate results on your original low poly as is.

Ok, I didn't realize this. I actually posted a feature request for this (which I noticed you saw) and having this build in SP would be of great help. Currently it requires SD as you mentioned, so for now I will stick with the generator for SP. Either way, Allegorithmic should fix this issue at the baker stage. It is good to see they show *some* interest in this. Most baker devs say "this is a problem which can't be fixed and you need to add support geometry" or something like that. Sigh.

I got the normal blending trick to work now, but in order to automate much of the workflow, most of it needs to be scripted. Unfortunately the current scripting implementation is extremely limited and it still requires a fair amount of manual work. If all UI functions are fully scriptable, the crappy normal map bake could be fixed with the press of a button. Let's hope Algorithmic takes scripting seriously and publishes a proper implementation.

You can read about the things I need here:
https://forum.allegorithmic.com/index.php?topic=12683.msg55289#msg55289

Here is a beta version for you to try. Download the smart material which includes the generator here:
https://drive.google.com/file/d/0Bwk4bDWv3jAcNzVNM1Z5dF9qWFU/view?usp=sharing

Instructions:

Three different normal maps, each with different settings need to be baked. Two of those normal maps need to be exported and re-imported into the shelf to prevent them from being overwritten by subsequent bakes.

-Bake a normal map with the following settings:
Dilation width: >= 1
Apply diffusion: true
Average normals: true

This normal map is used to get the correct shading at the UV edges.
-Export the normal map (CTRL + SHIFT + E)
-Open the export folder and rename the normal map to "Fix averaged normal".
-Drag and drop the normal map onto the Textures shelf.

-Bake a normal map with the following settings:
Dilation width: 0
Apply diffusion: false
Average normals: true

This normal map is used as an input for the mask generator to detect where the UV edges are.
-Export the normal map (CTRL + SHIFT + E)
-Open the export folder and rename the normal map to "Fix mask normal".
-Drag and drop the normal map onto the Textures shelf.

-Bake all maps, including a normal map with the following settings:
Dilation width: >= 1
Apply diffusion: true
Average normals: false

This normal map is used to get the correct shading at all areas except the UV edges.
There is no need to export and import this normal map into the shelf because it is placed on the shelf automatically. The name of this map is "Normal Map from Mesh TextureSetName".

Once all the normal maps are baked, they can be added to the Normal Fix smart material.

-Drag and drop the file Normal Fix.spsm to the smart materials shelf.
-Place the Normal Fix smart material on top of the layer stack.
-In the TextureSet Settings window, set Normal Mixing to Replace.
-Expand the Normal Fix smart material in the layer stack, so it displays all sub-materials.

-Select the "non-averaged normal" material. Drag and drop the "Normal Map from Mesh TextureSetName" texture from the shelf to the texture slot on the material properties window.

-Select the "averaged normal" material (not the mask). Drag and drop the "Fix averaged normal" texture from the shelf to the texture slot on the material properties window.

-Select the mask of the "averaged normal" material. Then select the "normal_bake_blend" generator below it. Now drag and drop the "Fix mask normal" texture from the shelf onto the texture slot on the generator properties window.

-On the mask on the averaged normal material press ALT + LMB to visualize the mask. Adjust the Edge Width parameter on the "normal_bake_blend" generator until the mask lines extend just beyond the chamfered edge. The model should now be shaded the correct way with no skewing and no black lines along the UV borders.

Note that if there is any high poly detail on the edge, the low poly version should have a chamfered edge instead of a hard edge.


Here is a beta version for you to try. Download the smart material which includes the generator here:
https://drive.google.com/file/d/0Bwk4bDWv3jAcNzVNM1Z5dF9qWFU/view?usp=sharing

Instructions:

Three different normal maps, each with different settings need to be baked. Two of those normal maps need to be exported and re-imported into the shelf to prevent them from being overwritten by subsequent bakes.

-Bake a normal map with the following settings:
Dilation width: >= 1
Apply diffusion: true
Average normals: true

This normal map is used to get the correct shading at the UV edges.
-Export the normal map (CTRL + SHIFT + E)
-Open the export folder and rename the normal map to "Fix averaged normal".
-Drag and drop the normal map onto the Textures shelf.

-Bake a normal map with the following settings:
Dilation width: 0
Apply diffusion: false
Average normals: true

This normal map is used as an input for the mask generator to detect where the UV edges are.
-Export the normal map (CTRL + SHIFT + E)
-Open the export folder and rename the normal map to "Fix mask normal".
-Drag and drop the normal map onto the Textures shelf.

-Bake all maps, including a normal map with the following settings:
Dilation width: >= 1
Apply diffusion: true
Average normals: false

This normal map is used to get the correct shading at all areas except the UV edges.
There is no need to export and import this normal map into the shelf because it is placed on the shelf automatically. The name of this map is "Normal Map from Mesh TextureSetName".

Once all the normal maps are baked, they can be added to the Normal Fix smart material.

-Drag and drop the file Normal Fix.spsm to the smart materials shelf.
-Place the Normal Fix smart material on top of the layer stack.
-In the TextureSet Settings window, set Normal Mixing to Replace.
-Expand the Normal Fix smart material in the layer stack, so it displays all sub-materials.

-Select the "non-averaged normal" material. Drag and drop the "Normal Map from Mesh TextureSetName" texture from the shelf to the texture slot on the material properties window.

-Select the "averaged normal" material (not the mask). Drag and drop the "Fix averaged normal" texture from the shelf to the texture slot on the material properties window.

-Select the mask of the "averaged normal" material. Then select the "normal_bake_blend" generator below it. Now drag and drop the "Fix mask normal" texture from the shelf onto the texture slot on the generator properties window.

-On the mask on the averaged normal material press ALT + LMB to visualize the mask. Adjust the Edge Width parameter on the "normal_bake_blend" generator until the mask lines extend just beyond the chamfered edge. The model should now be shaded the correct way with no skewing and no black lines along the UV borders.

Note that if there is any high poly detail on the edge, the low poly version should have a chamfered edge instead of a hard edge.

it didn't work

What exactly didn't work? There are a lot of steps in there. Make sure you follow each one carefully.

I will make a video some time later. Hopefully that will help.

When scripting is fully supported it should work at the click of a button, but unfortunately we are stuck with this convoluted workflow for now. But since asset creation is incredibly convoluted as it is, this "little" extra step shouldn't matter too much.
Last Edit: August 13, 2016, 12:56:23 pm