Maestro Linden commented at 2022-12-22T19:58:06Z

When you notice objects failing to render, sometimes doing one of the following will force the object to render, but fairly often nothing works and you need to relog to fix it.

1. Right clicking where the missing object should be,
2. Zooming camera out and back
3. Toggling wireframe - see BUG-232625
4. Rule out if it's a case of BUG-9439 - technically not a bug.

• Case (1) would definitely be a viewer bug - being able to select the object implies that the viewer already knows about the object, but for some reason decided not to render it. Otherwise, right clicking would just select whatever object (if any) behind it

• Case (2) could be viewer, simulator, or packet loss between the two - it's hard to say

  • I think we should focus on this case for this jira

• Cases (3) and (4) are already covered by the other jiras you mention; both full squarely into viewer rendering

  Looking at the pasted conversation, those hints about area search not working points to a possible issue with the interest list - whether that's a matter of

• the simulator not sending the ObjectUpdate message to let the the viewer know the object exists

• the message being lost in transit (it's a UDP message)

• the viewer not processing the message properly

  The next step to investigate this issue is to get a solid repro case - how can one make this happen on demand?

  I visited the Catwa store with an empty objectcache and draw distance set to 512m, and looked at the missing area shown in the video, and I'm unable to reproduce the issue there - the building appears to be intact, and I don't see any obviously missing objects.

  I also set up a test case at secondlife://Aditi/secondlife/Mainland%203/40/179/30 - this setup has 22500 large unlinked boxes arranged to form a platform, such that any missing box will be easy to see. The boxes are static, so they should offer a pretty consistent test case. Are you able to repro this issue there?

animats commented at 2022-12-27T05:45:59Z

I've been able to reproduce this. It's intermittent, but the same objects disappear. See "hyperionfirestormbad06.png", attached. There are supposed to be solid panels in that building wall, which is the big GT building in Hyperion. I've had this happen several times in Firestorm for this specific object.

Notes:

• Sometimes getting closer makes the missing object reappear, but sometimes I was able to get really close without it coming back.

• That missing object is part of a very large linkset, over 200 prims. Changing the LOD of the linkset via Edit has no lasting effect; going to "Lowest" will blank it out, and it comes back broken at "High".

• Changing the draw distance may make the building disappear. Changing it back restores the broken version.

• Attempting to edit the object will not make it appear.

• Flying to another region out of draw distance and back made it appear.

• I've seen this happen with both a fully loaded viewer cache and a freshly cleared cache.

  I've had this building fail with Firestorm 6.6.3, Firestorm 6.5.6. All on Linux.

  Over at Cretopia, looking northeast from 0,0, there are also missing objects. There, loading progressed normally for a while, then stopped. After several minutes of no activity, a Yava street car went by in the distance, and loading started up again, very slowly, under 1mb/s. Gradually more distant objects appear.

  Those are the two regions where I see this happen frequently. I've seen it elsewhere, but in those places it occurs frequently.

  I've even seen this happen, {}on the same linkset{}, in my own experimental Rust viewer, which is a totally different viewer implementation and architecture. I can't say definitively whether this is an interest list error or a mesh load error. But it's definitely associated with specific linksets in world.

  Tried to visit Mainland 3 on Aditi for the test described above. Currently standing on Mainland 2 looking at open water where Mainland 3 is supposed to be. World map says "Offline". 2140 SLT 2022-12-26. 

animats commented at 2022-12-27T07:21:19Z

Some large linksets to test with at Mainland 3 would be appreciated.

I've been looking at the logging in my own system, and I see tens of "orphans", objects where the child first appeared before the parent or the parent is missing. I need to look at that more closely. It's legitimate for that to happen; we're not guaranteed in-order delivery.  Orphans have to be parented when the parent shows up. You don't know their location in world until the parent shows up, because child prims have coordinates relative to the parent. Has anything changed that would make out of order object updates more likely? Such as introducing more parallelism server side?

This may be related to this issue, or not. More later.

animats commented at 2022-12-30T19:24:49Z, updated at 2022-12-30T19:44:58Z

More info. I've been testing this further with my own experimental viewer. I've been spending too much time on this, trying to make sure it's not an error on my side. My experimental viewer is 100% safe Rust, has no common code with the LL-based viewers, has a rather different architecture, and has completely different bugs. So if we see the same problem in both systems, it's probably not viewer side.

• There are a few specific objects that tend to drop out. They're usually big linksets. The big Gentek Telecom building in Hyperion is the worst case. It has linksets with over 200 objects. I've seen this specific building with big missing linksets in both Firestorm and my experimental viewer.

• I tested the "orphan" hypothesis by checking my experimental viewer's un-parented orphan list after scene loading was complete. While there are  thousands of child objects without parents during loading, eventually, all the needed parents showed up, the orphaned children were re-parented, and zero children were left un-parented.  The ObjectUpdate and ObjectUpdateCompressed messages described a consistent parent-child tree. So I don't think that's it.

• I am logging zero packet loss in my own experimental viewer. That's not the problem.

• My test setup is to request the entire region's objects. This is done by sending RegionHandshakeReply with SENDALLOBJECTS | SENDALLCHCEABLEOBJECTS | SUPPORTSSELFAPPEARANCE set, viewpoint at <0,0,0>, draw distance 400m, and look at direction <1,1,0>. This does get all the region's objects, except for a few omissions.  I don't move the viewpoint as sent to the server, and look at what showed up. Always an initial login, not a teleport.

• For test purposes, I draw all objects at Highest LOD. So this is not a LOD problem.

• Which objects are missing is rather repeatable. The objects which are missing are always the same ones. Sometimes they're present, but it's not that some random object disappears. It's the Gentek Telecom building in Hyperion, a clock in the Nox building in Port Babbage, and a few other very specific objects. Usually, but not always, big linksets. Big objects missing are very visible. Sometimes it's possible to get close to the object and still have it missing. Example:  Those openings in the walls are supposed to be solid recesses. Logging in with Firestorm 8.8.3 at Hyperion/20/20/20 produced this. The missing item here is link [BUG-100683] At login and more seriously after teleporting, the scene is loading much more slowly than usual #213 out of 244 links. That one tends to disappear a lot.

• In Firestorm, missing objects may eventually appear if the viewpoint moves. I suspect this is a problem which most users simply see as "lag", or as bad levels of detail.

  I think this eliminates but case (2) above, and focuses attention on the interest list.

animats commented at 2022-12-30T20:29:12Z

More info: See bad version and good version, below, with the same prim count. Object updates are not being lost, but some of them may be bad.


Bad version above. Good version below. Note 244 prims in the linkset for both.


animats commented at 2023-01-02T21:10:13Z

More notes:

Far from the viewpoint, like 250m away, but within a large draw distance, the interest list seem to omit small objects. (1m diameter omitted, 50m included.) That's reasonable. It also tends to omit objects that have small scripted rotational movement, such as clocks. Maybe even if in a child prim. That's wrong.

This may be a bug in my viewer. More later. But I'd suggest looking at code that decides some objects don't need to be in the interest list.

Mainland 3 is still offline. When it comes back up, I'd suggest having some big linksets and some simple objects with slow LLSetRot rotation. Preferably with copies in the corner far from the origin.

animats commented at 2023-01-10T19:23:25Z


animats commented at 2023-01-10T19:24:46Z

Mainland 3 (Aditi), the test area for this bug, is still offline.

Erik Mondrian commented at 2023-01-11T13:23:33Z

I've been encountering this same issue with some objects failing to render and requiring a relog, most recently when I was visiting the New Kadath Lighthouse Art Gallery yesterday (New Kadath/34/40/23). I noticed that some of the walls were bare, missing maps or collections thereof that I knew had previously been there. The "Early Grid" object was one of those that did not show up, even when I was standing right next to it (and even after right-clicking and moving camera out and back). It's a linkset with 27 objects. Elsewhere in the gallery, another one that did not show up until after a relog was "The Virtual University of Edinburgh" exhibit, a linkset with only 17 objects. This is, unfortunately, not just happening with linksets that are 200+.

I've included the Inspect Objects info in the snapshots, in case that helps. And I'm on Arch Linux, running Firestorm 6.6.3 (67470) Aug 27 2022 16:07:25 (64bit / SSE2) (Firestorm-Releasex64).

rick.daylight commented at 2023-01-13T20:46:22Z

I've seen this issue a lot too; far more missing objects that I'm used to seeing happen, and very often the objects will not reappear with any of the known 'tricks' like zooming out and back in, or even TP'ing out of the region and back.

This is the latest: I was testing a build with an alt, and the alt could not see the doors in the wall. My main account, also logged in, could see them.

The alt could interact with objects through the (closed) doors as if the doors did not exist, and could not click the doors to open or inspect them. However the alt could not walk through the doors until they were opened. Nothing, not even TPing out and back, made the doors appear for the alt except relogging.

Alt's view of the (missing) doors:

What was really there, view from main account:

animats commented at 2023-01-13T23:38:21Z, updated at 2023-01-13T23:39:47Z

Here's a fail which is a bit more informative. This is one of my NPCs at Bruissac. The NPC seemed to have disappeared, but the hovertext on the root prim was still showing. So I could find it and open Edit. This NPC is an animesh, with an invisible but solid root prim that's a simple cube as the collision model. The animesh mesh and other links (clothing) are gone, but the root prim remains. The root prim is still selectable. Edit says it's 22LI, which is the full total with all links, but Edit only shows the root cube. As far as Edit is concerned, the other links are just not there. Trying to advance through the links shows this.

Bruissac seems to get hit by this a lot. I've seen half the buildings missing.

Open image in a new browser tab to see it clearly.

animats commented at 2023-01-25T03:51:26Z, updated at 2023-01-25T05:46:33Z

Some progress:

I've found a workaround for my own experimental viewer. I delay sending RegionHandshakeReply for 2 seconds at startup, so that AgentUpdate arrives first.  For any viewer which sends AgentUpdate unsolicited but waits for RegionHandshake to send {}RegionHandshakeReply{}, that's a race.

Without the delay, my test at Hyperion for large objects not appearing failed 4 out of 10 tries.

With the delay, it failed 0 out of 25 tries.

The sim server doesn't start sending object updates until both RegionHandshakeReply and AgentUpdate have been received. That's correct behavior, because it needs info from both of those messages to decide what to send. But the order in which they are sent does seem to make a difference. It shouldn't.

Since the SL UDP network protocol does not guarantee in-order delivery, it's the receiver's job to cope with out of order messages. So this would appear to be a sim-side  bug. Working around it from the viewer side is possible, but to do it right, the viewer would have to insure that the AgentUpdate message was acknowledged, not just sent, before sending {}RegionHandshakeReply{}. This requires message delivery tracking, which the network level does not currently have.

So, it looks like an out of order receive condition is being incorrectly resolved by the sim servers.

Henri Beauchamp commented at 2023-01-25T21:25:59Z, updated at 2023-01-25T21:45:58Z

@Animats

The problem is that, when I tried to thread the LLVOCache file reads in my viewer (to prevent the FPS hiccups seen when these reads happen), I also delayed the RegionHandshakeReply reply until the read was completed (so that the cache is primed and the viewer can set the proper flags in that reply, to tell the server whether it actually got a cache for that region or not, which determines whether the server will send cache probes or just anything in the interest list).
I then observed a weird second (and sometimes a third) RegionHandshake coming in from the server, even before the first RegionHandshakeReply was sent by the viewer. Also, it looks like the server does not even wait for the first RegionHandshakeReply before sending object data (this caused totally empty neighbour regions in some occasions, in my case, probably because the data for them was sent even before my threaded read would complete, meaning on completion the received data was wiped out).

So, your workaround may work for your Rust viewer since you are queuing messages in it (which is not the case for other viewers, that process them in real time, as they come in), but it will not work for other viewers (the delay in the RegionHandshakeReply seems to confuse the server; maybe it makes the assumption that its first RegionHandshake message got lost and is resending one ?).

Note also that AgentUpdate is a message sent by the viewer in both process_agent_movement_complete() (which is a callback ran as a reply to the AgentMovementComplete message from the server, the latter arriving after a completed TP or login), as a "reliable message", and every second (or when something about the agent changed) from the idle loop of the viewer, as a normal message; on login and far TPs, the AgentUpdate reliable message might contain a very approximate FOV (camera axis) info, which might explain the bogus interest list with "missing" objects in it...

animats commented at 2023-01-25T22:21:51Z

 (the delay in the RegionHandshakeReply seems to confuse the server; maybe it makes the assumption that its first RegionHandshake message got lost and is resending one ?).

So that confirms that the sim server's behavior can be made to change depending on the timing of those messages. That's what I see, too.

Viewer side workarounds for this are just for testing. This has to be fixed sim-server side, so that behavior doesn't change just because one message came in early or late. Otherwise, random network delays will continue to cause intermittent failures.

Beq Janus commented at 2023-01-26T01:09:07Z

@henri

I think I concur with the FOV/culling having some role in this.

I went to a location suggested by @Animats today to see if I could reproduce the missing items, and while I could certainly see missing parts, it seemed to me not so much a race condition type effect but poor frustum culling, as you suggested. In particular, a tall building that, on first login/arrival, towers above your vertical FOV loses wall panels; these panels are large prims, possibly mega, but certainly large enough that the centre of the object is offscreen. This led me to wonder whether there was some overly aggressive culling going on. As soon as I cam up a little to get a "clear view" of the missing items, they appear.

animats commented at 2023-01-26T02:31:17Z

I'm getting exactly what Beq describes today. Note the missing wall panels of the large building.

It's not a LOD problem; I can change the LOD factor and it has no effect.

As Beq says, it's as if the center of those large objects has to be within the viewing frustum.  This particular bug is quite consistent.

But, as you can see from the pictures above, there are other fails.  There are cases where you can approach the building closely and still have pieces missing. Tonight, I can't reproduce those.

We may be looking at two different bugs here.


animats commented at 2023-01-26T02:54:55Z

Now, here's the fail that does NOT seem to be related to the viewing frustum. Here, I can cam around and get the entire area of interest in the viewing frustum and still not get some objects loaded. Those specific building panels are one of the objects that disappear reasonably often.

All these tests are right after a login.


Henri Beauchamp commented at 2023-01-26T09:00:41Z

@BeQ

I went to a location suggested by @Animats today to see if I could
reproduce the missing items, and while I could certainly see missing
parts, .../... As soon as I cam up a little to get a "clear view" of the missing items,
they appear.

Beware, you might be seeing "the other bug" here with missing objects being due to a race condition between incoming interest list data from the server and the pipeline objects rebuilding in the viewer; that other bug is worked around, viewer-side, by any of the following actions: camming around the missing object, right-clicking in the "empty place" where they are located, switching wireframe on/off, or using the TPV built-in "Refresh object visibility" feature (initially implemented by Marine Kelley in her RLV viewer, and that I extended/improved when backporting it to my viewer, by adding an auto-trigger feature that kicks in a couple seconds after each login, far TP or sim crossing).
If you can see the objects reappear when camming around (not far), then this is not the bug dealt with by this JIRA issue.

The only workaround for the bug in this JIRA issue (short of relogging or TPing far away and back in), is to zoom out far, far away (beyond draw distance), wait a few seconds, and reset the camera to its normal FOV.

This led me to wonder whether there was some overly aggressive culling going on.The culling is done exclusively viewer side, based on the info sent by the server via the interest list. The server does not do any culling of its own for the objects in the latter. So, no, it's not a culling issue, but just that the object is not in the viewer object list, but yes, their absence might be due to the fact their center was not in the approximated FOV initially sent by the viewer in the first AgentUpdate message...

@Animats

As Beq says, it's as if the center of those large objects has to be
within the viewing frustum.  This particular bug is quite consistent.

But, as you can see from the pictures above, there are other fails. 
There are cases where you can approach the building closely and still
have pieces missing. Tonight, I can't reproduce those.

We may be looking at two different bugs here.

Yes, there are several bugs that appear to have the same effect for the end user(missing objects), and even the "new" (introduced early in 2022) bug may have a couple causes (race condition and/or FOV issue)...