How to make clean normal maps via multiple smoothing groups with 3Ds Max?

[franman]

Based on this video guide:

From the 14:11 mark, the vid showed a 2nd alternative to baking clean normal maps that is by multiple smoothing groups.  However, this was demonstrated using Maya, how can I do something similar in 3ds Max?

I'll break this broad question down:

• From 14:22, he deleted his "history".  Does 3Ds Max have that feature as well? and if so, do I have to delete it?
• Starting @ 14:25, he harden the edges.  I'm not sure if what I did was correct but in 3Ds Max, I placed an edit polygon modifier on my mesh, went into edge mode, select certain edges on my mesh, opened up "edit edges" and clicked on the "Hard" button.  Does this translate to what he exactly did in Maya or does 3Ds Max yield a different result?
• @15:06, the angle of the vertex is suppose to be 0.  I cannot find the angle of the vertices in 3Ds Max, but in polygon faces I do.  If there's a way of setting the angle of the vertices to 0 in 3Ds Max please by all means enlighten me; otherwise, would changing the angles based on Polygon faces be an alternative?
• From 19:35 onward, he went into edge mode and set the smoothing constraint to hard; does 3Ds max have something like that? If so, where do I find that?
• Further in regards to question 4, there was mention of UV and smoothing splits with padding, let's say initially I have a box model and is UV unwrapped in a way that 5 of its 6 faces are stitched to together and nicely tucked within the UV boundary; on further note, the top and bottom faces of the box have the same smoothing group and the side faces adjacent to each other have different smoothing groups.  By setting the smoothing constraint on the edges to hard does that automatically split the UV geometry in Maya or do you have to manually break these faces that have different smoothing groups?

All these questions (with the exception of question 5) summarize how procedures done in Maya can be translated to 3Ds Max.

I can understand that these questions can be lengthy and complicated to a procedure that is already complicated, but fun to do; if I find the time, I'll try to simply a question or two.

[franman]

Thanx to user EfilOne on Discord who contributed these answers to my questions.

• First, there's no "history" to delete since max is "stack-based" (which refers to the modifiers that stacks on top of each other), else the usual history (through ctrl-z) must not be deleted, it's not as sensitive as it is in maya, so no need to worry about that.
• For your second question, yes you can define edge hardness through edge modes in max, note that hard edges must follow the way your UVs are cut (as in, edges must be hard only where an UV cut exists), and this due to the normal projection method used
• For your third question, you can visualize vertex normal angles in max using the "edit normals" modifier, which allows you to edit in vertex/edge/face mode at your liking. If normal anchors are too overkill you can also reduce their size from there
• Fourth one, everything happens here, in Edit Poly (you sort of answer yourself through your second question with this one.
  
• For your 5th question, no it doesn't "automatically" split in the UV map, unless you have a script that autodetects edge weighting and mark a cut there (which is also possible to do in 3ds max, by the way).

Also, another way to doing it all is through the edit normals modifier entirely :

• Selected edge is hard =>
  
• Selected edge is smooth, by clicking on "unify" after selecting it =>
  

Of course, this works with vertex and face selections as well, or normal by normal, at your liking.

[franman]

Following the above procedure, I exported my low and high poly.  After baking in the normal map, I notice some odd shading on the low poly mesh.


You can see the sharp edges appearing where the rounded edges are.


Also if you noticed, there's a sorta V shading on the corner of the frame.

Here's my low poly version


The high poly version


I exported these meshes in FBX for Unity using these settings.


This is my low poly UV unwrapped.


And here's the normal map result of the high poly (Just an update I've altered it in Photoshop)


What should I fix?

[Dennis.Porter.3d]

Francis,

After looking over your post you have a few things that are causing you trouble:

1) Your high poly actually isn't high enough.  You can see clearly how the light is bending around your surfaces.  You need more supporting geometry so the model smooths properly and flat surfaces are rendered as you would expect.  This is the main cause of the result you're seeing with your bake.

2) Your UV shells are very close together.  Make sure you give them a few pixels of space away from each other.

3) Really harsh angles between polygons (such as right angles) should probably have their smoothing groups and UVs split in that area.  Any gradients you see in your high or low will bake into the normal map which reduces the quality of the map.

4) Make sure each UV island is set to its own smoothing group.

[franman]

I've updated 1, 2 and 4 but I'm not fully following on 3.

3) Really harsh angles between polygons (such as right angles) should probably have their smoothing groups and UVs split in that area.  Any gradients you see in your high or low will bake into the normal map which reduces the quality of the map.

In my high poly, I have those brackets or frames that surround each face.  Where the corners are for each of the frame are you saying that they need to be in a different smoothing group?

I'll illustrate what I mean

So the inner edge of the frame between the red highlight and the other that isn't highlighted both should have different smoothing groups?

Same question but on the outer edge on the next image:


Also I've exported the high poly as an fbx with Smoothing Groups, Tangents & Binormals, TurboSmooth, and Triangulate enabled to Substance Painter but when trying to bake onto my low poly mesh it gave me these error messages:

[franman]

I've did an update to the mesh and here's what it looks like now in Painter.

Also on the error message from my previous post.  It seems to give that when I set the resolution only to 1k, but if I bump to 2k it no longer shows.

Any feedback just only on the normal baking part are welcomed.