Updated Report( As best I can understand how things function anyhow):

Display of Normal maps uses incorrect Vertex Normal Data.

If I take an asset, and squash it flat, or near flat, as best I can tell using tools that are not the Render Debug > Vertex Normal settings~ that object has it's vertex normals handled correctly with an inverse-scale relationship, and will render correctly in the presence of light, as long as it does not have a normal map applied to it's surface.  This is confirmed by reading the various bits of code Beq dug up for me in the viewer code, and testing with a single point light in the scene and an ambient dark windlight.  Angles of illumination seem to confirm that this largely behaves correctly.

However!! If I turn on Render Debug-Normals, it's clearly visible that the little drawn debug normal vector lines are being squashed with object scale, and not with inverse_scale, which lead to the rather confused testing / bug hunting loop that is in the preserved text below.  Furthermore ~ the issue still remains that when a normal map is assigned to any scaled object ( Almost ALL objects are scaled in SL, no one really makes unit cuboid meshes), the normal map renders as if it was using the vertex normal vectors that the (apparently) incorrectly drawn vertex normal debug settings illustrate.  It reflects light as such, and distorts the normal map in a manner that is consistent with the manner that SL renders the little debug normals lines.  That is to say, if I take a basic cylinder prim and slap a normal map on the round barrel part of it ( not the circular end caps ) and flatten the cylinder until the end caps are nearly thin lines and the cylinder itself is a near planar entity: it still renders the normal map on the side of the cylinder as if the cylinder was still in its original rounded form.  This further lead me to believe that the underlying mesh normals were being incorrectly stored, though I presently believe now, that this is not the case, though with debug normals being drawn incorrectly, it's difficult to prove.

At this point ~ my stab-in-the-dark guess is ~ ~ that somewhere, deep in the shader calcs, the original unit-scaled mesh form ~ ( before any transforms get applied ) is getting piped directly into the normal map shader calculation, or the object scale * vertex normal results that we can see with the debug normals are being piped in....  ( One of these two ?  ~or something?? ) and this data is being used to draw the lighting info ( on the correctly flattened surface, with proper correct inverse-scale calculated normals ) ~ making the flat surface look completely bent.

That being said, with anything that has a normal map on it, because of this error it looks like SL is using scale ~ rather than inverse scale in it's vertex normal calcs.  Good news, the mesh assets (probably) aren't all corrupted.  Bad news is they still all look corrupted if they have a normal map on them.

As an addendum : Rather unrelated to this bug, 3ds Max apparently uses scale * vertex normals in its' own internal functionality.  So that has further contributed to the elaborate amount of confusion about this bug.  Maya and Blender on the other hand both seem to use Inverse_Scale to calc their vertex normals.   However, 3ds Max corrects this bizarreness upon export or "Apply XForm" within the software.  Why it does this ~ I can only guess.  But it's interesting to know!  Or at least I thought so...

Original Bug Report:

I have been chasing the trail of this bug for the better part of the last 5 years.

When a mesh is uploaded to Second Life, the original DAE asset is compacted into a unit-cube. This squishification process processes all the vertex locations correctly. Unfortunately, during this process, the vertex normal data is pushed through an inverse-scale matrix calculation, instead of a scale matrix calculation ( like the rest of the mesh asset is ), which causes all the vertex normal data to be slightly bent by several orders of magnitude, dependent upon the original scale of the object. This bad data is then stored in the SLM file on the server ~ where it is then retrieved and rendered incorrectly for all eternity. This also completely borks any baked out normal maps that are subsequently uploaded and applied overtop of this bad mesh data, causing a visual discrepancy in lighting environments, that has been driving me bonkers for half a decade~ This is the case for any meshes that aren't uploaded with perfect 1:1:1 ratio bounding boxes. ( which is pretty much all of them ).

Just to use an example ~ if I upload a squished mesh sphere that's 0.25m x 1.0m x 1.0m large, the vertex normal calculation measures that the source mesh is 1/4 of a unit bounding box ( and that the mesh needs to be scaled up 4x on that axis ), but it recalculates each normal as if the mesh is a 4.0m x 1.0m x 1.0m object. ( a 16x difference )  In fact ~ in order to get my vertex normals back to the way they were in my 3D application, the mesh asset in Second Life needs to be scaled up to that size in order to get the vertex normals to "point" properly in the proper world-space coordinates, and get handled by the lighting calculations correctly. As you can imagine, the thinner the object ~ the more pronounced this defect is.

It was a forum post by ZedConroy : that tipped me off on where to look: https://community.secondlife.com/forums/topic/456171-vertex-normals-issue/

What were you doing when it happened?

I've included a test mesh object where I intentionally modified the vertex normals to be very specific angles. This is a squashed hemisphere has 3 specific zones to it.

1 : The flat portion of the hemisphere ~ that has all of its vertex normals canted at a 45 degree angle off of the plane ( instead of the usual 90 degree angle ) These vertex normals get calculated with the incorrect inverse scale matrix and come out nearly flat with the plane at default scale.
2: The top half of the sphere portion has vertex normals pointed straight up, as a kind of "control" these should not be ( and are not ) affected by the calculation.
3: The bottom portion of the sphere has vertex normals of a sphere that has not been squished. That is to say, that the verticies are angled in a manner of a sphere 3.5 times "fatter" than it is.

When uploaded with a broken viewer the area that's easiest to tell what's going wrong with is the flat back face.
In order to get vertex normals that have a 45 degree angle relative to world-space this test object needs to be scaled up to 7.0m x 1.0m x 1.0m. Which was my proof for the inverse scale matrix, though Beq Janus found exactly where an inverse scale is in the code, this test object was integral in proving we'd fixed the issue when she patched it.

What were you expecting to happen instead?

I'm expecting the vertex normals to function like they were in the original asset.

Other information

Beq Janus and ZedConroy helped out with this a lot. Beq as she always does, Zed, with his forum post and bounding box discovery.  Beq Janus made a modified viewer, where we tested uploading meshes with the proper scale calculation, and you can see the difference in how the lighting calculations deal with the data on your average clothing object here : https://gyazo.com/c66e087da14b3e99459e6f28b17c28f0

Here is a side by side comparison of a mesh uploaded with the current SL viewer, and Beq's modified viewer.  The red dress has incorrect surface normals, due to the scaling issue, the second dress copy in the video with perfect perpendicular surface normals was uploaded using a viewer with the fix Beq implemented : https://gyazo.com/983df72993027132af3a3458d4c6ad68

Note: This dress is only approximately 2:1 scaled off of a cube, it's very far from being the worst case scenario, but I included it as an example of how this affects your average mesh.

Appended Jira picture is of two instances of the test-hemisphere, one encased in a cube (named appropriately 'inaBox' ) to make sure there is no scaling operations performed upon it during the upload process, regardless of the viewer that's used,  and the other is not.  In the appended picture, you can see in order to get the "Flat faced" 45 degree vertex normals aligned between the two objects, the test object that is not "encased" needs to be scaled up to 7x the size of the 'boxed' mesh ( 49x it's original scale in that dimension ~ see the attached screenshot ), giving it an awkward topedo shape, which makes sense, since approximately 7 of these little wonky half-spheres can be packed into a unit cube at default scale.

One last thing: I made the test-meshes in Maya, and they exported in cm scale, so they're tiny.  When you upload them, hop over into the scale tab and give them a uniform 100x scale or they appear in SL super tiny, but this doesn't affect the test in any way.

Attachments

• meshNormals.patch 2020-06-19 17:46:41
• TestHemisphere_ExplicitNormals_squish_2.dae 2020-06-19 17:13:23
• TestHemisphere_ExplicitNormals_squish_2InaBox.dae 2020-06-19 17:22:49
• TestObjectsInMaya.png 2020-06-19 17:54:50
• vertexNormalComparison.png 2020-06-19 17:28:17

Links

Related

• [BUG-228823] Incorrect calculation of vertex normals for appearance sliders with rigged meshes #6815

{
  'Build Id': 'unset',
  'Business Unit': ['Platform'],
  'Date of First Response': '2020-06-19T16:45:55.703-0500',
  "Is there anything you'd like to add?": 'Beq Janus and ZedConroy helped out with this a lot.  Beq as she always does, Zed, with his forum post and bounding box discovery.',
  'ReOpened Count': 0.0,
  'Severity': 'Unset',
  'System': 'SL Viewer',
  'Target Viewer Version': 'viewer-development',
  'What just happened?': 'I have been chasing the trail of this bug for the better part of the last 5 years.\r\n\r\nWhen a mesh is uploaded to Second Life, the original DAE asset is compacted into a unit-cube.  This squishification process processes all the vertex locations, correctly.  But during this process, the vertex normal data is pushed through an inverse-scale matrix calculation, instead of a scale matrix calculation ( like the rest of the mesh asset is ), which causes all the vertex normal data to be slightly bent by several orders of magnitude, dependent upon the original scale of the object.   This bad data is then stored in the SLM file on the server  ~ where it is then retrieved and rendered incorrectly for all eternity.   This also completely borks any baked out normal maps that are subsequently uploaded and applied overtop of this bad mesh data.  This is the case for any meshes that aren\'t uploaded with perfect 1:1:1 ratio bounding boxes. ( which is pretty much all of them ).\r\n\r\nJust to use an example ~ if I upload a squished mesh sphere that\'s 0.25m x 1.0m x 1.0m large, the vertex normal calculation measures that the source mesh is 1/4 of a unit bounding box ( and that the mesh needs to be scaled up 4x on that axis ), but it recalculates each normal as if the mesh is a 16.0m x 1.0m x 1.0m object.  In fact ~ in order to get my vertex normals back to the way they were in my 3D application, the mesh asset in Second Life needs to be scaled up to be a 4.0m x 1.0m x 1.0m asset to get the vertex normals to "point" in the proper world-space coordinates, and get handled by the lighting calculations correctly.  As you can imagine, the thinner the object ~ the more pronounced this defect is.\r\n\r\nIt was a forum post by ZedConroy : that tipped me off on where to look: https://community.secondlife.com/forums/topic/456171-vertex-normals-issue/',
  'What were you doing when it happened?': 'I\'ve included a test mesh object where I intentionally modified the vertex normals to be very specific angles.   This is a squashed hemisphere has 3 specific zones to it.\r\n\r\n1 : The flat portion of the hemisphere ~ that has all of its vertex normals canted at a 45 degree angle off of the plane ( instead of the usual 90 degree angle )  These vertex normals get calculated with the incorrect inverse scale matrix and come out nearly flat with the plane at default scale.\r\n2: The top half of the sphere portion has vertex normals pointed straight up, as a kind of "control" these should not be ( and are not ) affected by the calculation.\r\n3: The bottom portion of the sphere has vertex normals of a sphere that has not been squished.  That is to say, that the verticies are angled in a manner of a sphere 3.5 times "fatter" than it is.\r\n\r\nWhen uploaded with a broken viewer the area that\'s easiest to tell what\'s going wrong with is the flat back face. \r\nIn order to get vertex normals that have a 45 degree angle relative to world-space this test object needs to be scaled up to 7.0m x 1.0m x 1.0m.  Which was my proof for the inverse scale matrix, though Beq Janus found exactly where an inverse scale is in the code, this test object was integral in proving we\'d fixed the issue when she patched it.',
  'What were you expecting to happen instead?': "I'm expecting the vertex normals to function like they were in the original asset.",
  'Where': 'Mesh Uploader',
}