Skip to content
This repository has been archived by the owner on Feb 28, 2024. It is now read-only.

[BUG-231582] Newly rezzed objects are invisible after relog under certain circumstances. #9034

Closed
6 tasks
sl-service-account opened this issue Dec 18, 2021 · 57 comments

Comments

@sl-service-account
Copy link

sl-service-account commented Dec 18, 2021

Update after filing

This bug is really annoying me now so I'm going to stop poking it to get a better repro for now.
The tl/dr though is:
Just login, rez cube, relog, rez cube, relog, rez cube, relog, rez cube & repeat. Eventually (could be after a couple of relogs or 20) all or some of the cubes will not render after relog and as soon as it repros, every new cube you rez will be invisible after relog after that with 100% certainty
Till you purge cache. Then it's fine again - for a while but it will reproduce again if you repeat the rez/relog cycle.

Thoughts from Chaser: Honestly sounds like it is buffering the cache, that being, every write is to memory and every so often it flushes that buffer onto disk. If relogging too quickly it never calls the flush, but has an object cache in place with whatever checksum thing LL uses to say it has those objects in cache.


This bug is a bit of a faff to reproduce, so apologies.
There may be simpler reproduction steps which I have not found yet, but if you follow the steps below it should reproduce 100% of the time.

Steps To Reproduce

  • The trigger for this bug appears to be having cache created on a pre-simplified cache viewer & then updating to a simplified cache viewer.

  • Install both the Maint G+H build (not using simplified cache) from https://releasenotes.secondlife.com/viewer/6.5.0.565607.html
    And the Simplified cache + 360 build from https://releasenotes.secondlife.com/viewer/6.5.1.566335.html
    Install each build to a different folder.

  • You will need to change the channel on the Maint G+H build to stop the auto-update.

  • Delete the SecondLife viewer cache folder at C:\Users\USERNAMEr\AppData\Local\SecondLife - this is just to start off with a clean slate because you may alreadyhave the cache corruption,

  • Login on the Maint G+H build.

  • Rez a default cube from the build tools.

  • Relog on Maint G+H.

  • Observe you see the cube you rezzed.

  • Rez another default cube.

  • Relog on Maint G+H.

  • Observe you see the 2 cubes you rezzed.

  • Rez a 3rd default cube.

  • Relog on Maint G+H.

  • Observe you see the 3 cubes.

  • Rez a 4th cube & relog on Maint G+H then a 5th cube & relog on Maint G+H, observing you can see all 5 cubes.

  • Rez a 6th cube and relog on the simplified cache viewer.

    Observerved Behaviour

  • After relogging on the Simplified cache viewer, the 6 cubes you rezzed will be invisible.

  • Rez another default cube {[BUG-10008] Viewer Managed Marketplace issue #7)

  • Relog on the simplified cache viewer.

  • Observe all 7 cubes are invisible.

  • The invisible cubes cannot be selected anddo not show in wireframe.

  • Once the cache is corrupted in this way, you can keep rezzed a new cube & relogging on the Simplified cache viewer and none of the cubes will render.

  • Log out.

  • Delete the objectcache folder in C:\Users\USERNAMEr\AppData\Local\SecondLife

  • Relog

  • Observe all the cubes you rezzed are now visible.

  • Rez another default cube ([BUG-10009] I tried to migrate my unique items to VMM and quite a few items disappeared the next day #8) & relog.

  • Observe the bug reproduces again & the 8th cube is invisible.

  • Log out

  • Delete the full cache folder, Secondlife from C:\Users\USERNAME\AppData\Local

  • Login & observe all the cubes are visible.

  • After deleting thefull cache folder, the bug no longer seems to reproduce & all newly rezzed cubes will be visible after relog.

    Other Information

  • I can reproduce this bug on both my Windows 10 laptop & Windows 7 desktop.

  • I'm not 100% happy with this repro yet so may change it.

  • I have a suspicion it might not be anything to do with the changes for Simplified cache because only one time I did reproduce this when I'd started from a clean cache & had only used the Maint G+H viewer.

  • The steps above should hopefully let you reproduce it though. I'll poke at this more once I know others can repro it.

  • And edit to add, I just reproduced this when only using the Simplified cache viewer after a cache clear.

  • I have a feeling that the trick to reproduce this is timing how quickly you wait after rezzing the cube before relogging - how log after you rez an object does it get cached???

Attachments

Links

Related

Duplicates

Original Jira Fields
Field Value
Issue BUG-231582
Summary Newly rezzed objects are invisible after relog under certain circumstances.
Type Bug
Priority Unset
Status Closed
Resolution Cannot Reproduce
Labels fail_to_render
Reporter Whirly Fizzle (whirly.fizzle)
Created at 2021-12-18T20:43:45Z
Updated at 2022-05-31T21:24:40Z
{
  'Build Id': 'unset',
  'Business Unit': ['Platform'],
  'Date of First Response': '2022-01-05T15:59:04.825-0600',
  'ReOpened Count': 0.0,
  'Severity': 'Unset',
  'System': 'SL Simulator',
  'Target Viewer Version': 'viewer-development',
  'What just happened?': 'Filling in...',
  'What were you doing when it happened?': '...',
  'What were you expecting to happen instead?': '...',
  'Where': 'Reproduces on any region.\r\nBut I did most of my testing here: http://maps.secondlife.com/secondlife/Main%20Channel%20Sandbox%20A/153/140/23',
}
@sl-service-account
Copy link
Author

Dan Linden commented at 2022-01-05T21:59:05Z, updated at 2022-01-05T22:06:16Z

Thank you for the report, Whirly!
I am unable to reproduce this bug. I'm importing it for investigation anyway because there are several reports of this.

If anyone has a recipe to repro this bug, please add it in a comment.

Thank you,
Dan

@sl-service-account
Copy link
Author

NiranV Dean commented at 2022-01-19T18:14:17Z, updated at 2022-01-19T18:14:42Z

As far as i can tell this is not an issue with the latest Viewer code nor the simplified cache. Black Dragon (currently based around 6.4.18 and thus far behind the simplified cache) does have it too, i get frequent reports of this happening since December last year.

@sl-service-account
Copy link
Author

AndreyK ProductEngine commented at 2022-01-20T19:44:09Z, updated at 2022-01-21T18:07:03Z

If possible, please provide logs. Preferably whole folder from first instance of the issue reproducing.

Broken cache will be very usefull to have as well along with location where this cache was taken from, likely the cause is in \AppData\Local\SecondLife\objectcache.

@sl-service-account
Copy link
Author

Henri Beauchamp commented at 2022-01-23T09:47:20Z, updated at 2022-01-23T09:52:00Z

Indeed not a simplified cache issue.

I have had a couple of reports too for the Cool VL Viewer, starting from the very end of December (while I backported the new asset cache back in March 2021 and did not have a single issue with it, once all the original bugs and quirks fixed, a month or so after my initial backport).

The first reporter mentioned that clearing the objects cache (LLVOCache) solved the issue for them. I then have been suspecting an issue with changed interest list code/algorithm on the server side (perhaps to improve the 360° snapshot reliability ?) and a recent commit to LL's code fixes a bug in caching of objects updated via the OUT_FULL UDP message; I backported this commit in my last release of my viewer and so far did not hear any more about the initial reporter...

I since had a second report about disappearing mesh objects by another user, however they were rebuilding the viewer themselves and were using libcurl v7.64 (this is an option in my build system) instead of v7.47, which is the version I am using for official builds because all newer versions experience HTTP pipeline trashing and corruptions (affecting textures and meshes data). I asked them to use v7.47 and did not hear from them since either...

Not experiencing the issue myself I have to recourse to wild guesses, but I see two possible explanations:

  • The wider usage of OUT_FULL UDP messages since December, which would hit the bug that just got fixed in LL's git, and cause objects cache corruptions (this would also explain why a simple TP away and back solves the issue, since the interest list is resent on the come back). This could also be an interest list code change server side that is not directly related with this bug...

  • A change in AKAMAI's CDN servers configuration (perhaps in some Points of Presences and not others, which would explain why not all users are impacted), that would cause more trashing of the HTTP pipeline for viewers using curl versions newer than v7.47 (i.e. all viewers but mine...). The resulting corrupted mesh data would then get cached in the assets cache (or VFS for older viewers), prompting a clearing of that cache (or simply a change of mesh LOD on TP back) to solve the issue.

     

@sl-service-account
Copy link
Author

Chorazin Allen commented at 2022-01-26T11:46:28Z

I've also been chasing this following reports with the Kokua viewer (currently based on 6.5.2). I have a fairly reliable repo. I'm located in the UK.

Log into the region Bigger Bang Park within the RHI Asylum building at the NNW edge of the region (note: it's an adult region). Log out from there and back in several times (typically the bug will hit within 10 times, generally within 2 or 3). Note: the region has a landing point set. If necessary, choose RHI Asylum from the teleporter inside the RF store by the landing spot.

Certain parts of the building, most usually some of the mesh objects that form a door frame plus two sliding doors unit, will not appear. As above, a cache flush or teleport away and back will solve the issue (although other items have been noted to disappear instead, again typically other sliding doors).

My investigations with Kokua are pointing at the object being present in the viewer's cache but the viewer doesn't receive a cache probe from the server for the affected object(s)

@sl-service-account
Copy link
Author

Henri Beauchamp commented at 2022-01-26T13:21:46Z, updated at 2022-01-26T13:23:34Z

@Chorazin

Went there, at position 81,229,92, i.e. inside the building at the entry hall, looking North, at the sliding door. Let us know if there is a "better" position to reproduce that bug.

I logged out/back in over 15 times with the Cool VL Viewer v1.28.2.56, and just as many times with Kokua v6.5.2 [*], both under Linux. I could not reproduce the issue with either of these two viewers.

Hypothesis: it might be a network issue (HTTP pipelining bug, when connected to given point of presences of the Akamai's CDNs ?).

 

[*] By the way, I noticed that anti-aliasing is totally non-functional with Kokua under Linux (whatever the AA setting chosen): the last working version (I got) of Kokua in this respect is v6.2.3. v6.4.16 was broken as well.

@sl-service-account
Copy link
Author

Chorazin Allen commented at 2022-01-26T16:52:46Z

@henri Beauchamp

Somehow I'm not that surprised it didn't repro for you. The spot I've been told is standing in the gallery ring above the entry level looking towards the dormitory half of the lower level, however I don't think the position is too critical.

I'll get in touch directly about anti-aliasing - I know what's going on there.

@sl-service-account
Copy link
Author

Whirly Fizzle commented at 2022-01-29T00:43:45Z

I removed the "Simplified Cache" tag from the issue title as it appears this bug is nothing to do with the Simplified cache code.

@sl-service-account
Copy link
Author

Whirly Fizzle commented at 2022-01-29T01:02:51Z, updated at 2022-01-29T01:05:37Z

Hiya AndreyK,

Sorry for the slow reply, I'm currently away from home.
So, this bug still reproduces for me on a different computer & network 200 miles from home :D (99.8 Mbps download, 23.6 Mbps upload).

See zipped file: Whirly_repro_1 attached.
File contains:
Session_1.log - logged in & rezzed default cube then logged out.
Session_2.log - logged into last location, cube not rendering. Rezzed a 2nd cube, logged out.
Session_3.log - logged into last location, both cubes not rendering, logged out.
Test location: http://maps.secondlife.com/secondlife/Main%20Channel%20Sandbox%20A/128/128/23
Note that Session_1 was run on a clean cache too & the bug still reproduced.

System Info:

Second Life Release 6.5.2.567427 (64bit)
Release Notes

You are at 128.0, 128.0, 22.9 in Main Channel Sandbox A located at simhost-01339a1fb48f3143f.agni
SLURL: http://maps.secondlife.com/secondlife/Main%20Channel%20Sandbox%20A/128/128/23
(global coordinates 336,768.0, 303,488.0, 22.9)
Second Life Server 2022-01-06.567269
Release Notes

CPU: Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz (2808 MHz)
Memory: 16273 MB
OS Version: Microsoft Windows 10 64-bit (Build 19044.1466)
Graphics Card Vendor: NVIDIA Corporation
Graphics Card: NVIDIA GeForce GTX 1050 Ti/PCIe/SSE2

Windows Graphics Driver Version: 30.0.15.1123
OpenGL Version: 4.6.0 NVIDIA 511.23

Window size: 1920x1027
Font Size Adjustment: 96pt
UI Scaling: 1
Draw distance: 128m
Bandwidth: 3000kbit/s
LOD factor: 1.125
Render quality: 5
Advanced Lighting Model: Enabled
Texture memory: 512MB
Disk cache: Max size 204.0 MB (20.2% used)

J2C Decoder Version: KDU v7.10.4
Audio Driver Version: FMOD Studio 2.01.07
Dullahan: 1.12.3.202111032221
  CEF: 91.1.21+g9dd45fe+chromium-91.0.4472.114
  Chromium: 91.0.4472.114
LibVLC Version: 3.0.16
Voice Server Version: Vivox 4.10.0000.32327

Packets Lost: 3/4,335 (0.1%)
January 28 2022 17:03:55

@sl-service-account
Copy link
Author

Whirly Fizzle commented at 2022-01-29T01:12:50Z

Also added my cache folder from the sessions in the above comment.
Whirly_Repro_1_cache.zip
I removed the inventory cache, name cache, texture cache etc so the file would upload to JIRA. Object cache is untouched.

@sl-service-account
Copy link
Author

Whirly Fizzle commented at 2022-01-29T01:17:05Z

Would it be possible to have a region on Aditi with the serverside interest list changes that were made for the 360 snapshot viewer removed?
I suspect Henri might be correct that this is an interest list bug caused by the changes that were introduced for the 360 snapshot viewer.
Seeing as I can reproduce this bug so easily (lucky me! - though seriously it's driving me mad lol) I can easily test if it's indeed fixed with the interest list changes rolled back.

@sl-service-account
Copy link
Author

Whirly Fizzle commented at 2022-01-29T01:45:43Z, updated at 2022-01-29T01:47:55Z

Henri, I can repro this bug consistently on the Cool VL Viewer too.
Note that I've never had Cool VL Viewer installed on this system so settings & cache were clean.

Cool VL Viewer v1.28.2.56, 64 bits, Jan 15 2022 10:58:48
Release notes

You are at 336768.0, 303488.0, 22.9  in Main Channel Sandbox A located at
simhost-01339a1fb48f3143f.agni.secondlife.io (50.112.73.73:13027)
Alias: ec2-50-112-73-73.us-west-2.compute.amazonaws.com
Second Life Server 2022-01-06.567269
Release notes

CPU: Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz (2808 MHz)
Memory: 16273MB
OS version: Microsoft Windows 10 64 bits v10.0 (build 10586.1466)
Memory manager: OS native
Graphics card vendor: NVIDIA Corporation
Graphics card: NVIDIA GeForce GTX 1050 Ti/PCIe/SSE2
Windows graphics driver version: 27.20.0100.8681
OpenGL version: 4.6.0 NVIDIA 511.23
Detected VRAM: 4096MB
J2C decoder: OpenJPEG: 1.4.0.635f
Audio driver: FMOD Studio v2.02.05
Networking backend: libcurl 7.47.0/OpenSSL 1.0.2u/zlib 1.2.11.zlib-ng
Embedded browser: Dullahan 1.12.3/CEF 91.1.21/Chromium 91.0.4472.114
Packets lost: 0/1408 (0.0%)

Built with: MSVC v1916
Compiler-generated maths: SSE2.

Compile flags used for this build:
/O2 /Oi /DNDEBUG /D_SECURE_SCL=0 /D_HAS_ITERATOR_DEBUGGING=0 /DWIN32 /D_WINDOWS /W3 /GR /EHsc /std:c++14 /EHs /fp:fast /MP /TP /W2 /c /nologo /GS- /Zc:threadSafeInit- /DLL_WINDOWS=1 /DUNICODE /D_UNICODE /DWINVER=0x0601 /D_WIN32_WINNT=0x0601 /DXML_STATIC /DLL_PHMAP=1 /DLL_IOURING=1 /DLL_NETBIOS=1 /DBOOST_ALL_NO_LIB /DLL_FMOD=1 /DAPR_DECLARE_STATIC /DAPU_DECLARE_STATIC /DCURL_STATICLIB=1

@sl-service-account
Copy link
Author

Henri Beauchamp commented at 2022-01-29T11:21:28Z

@whirly:

Yes, I just had the initial reporter for that bug posting that they are still seeing it. I first inferred from their silence that it got fixed with LL's OUT_FULL fix, but they were simply away from SL and did not use the "fixed" viewer until yesterday...

What is driving me mad is that I cannot reproduce it myself (which would allow me to instrument the viewer to understand what's the Hell is going wrong) !

@sl-service-account
Copy link
Author

Whirly Fizzle commented at 2022-01-29T12:35:06Z, updated at 2022-01-29T12:35:24Z

If you want me to post logs or do any other debugging on your viewer Hemri, just ask.

@sl-service-account
Copy link
Author

Henri Beauchamp commented at 2022-01-29T13:17:11Z, updated at 2022-01-29T13:18:34Z

@whirly:

"As is" (with default logging settings), the logs would likely be totally useless. If I were able to reproduce the issue myself, I would add more LL_DEBUGS messages, with a specific tag in parts of the code corresponding to what I could infer as being related to the bug, from what I am observing (such as the type of the "missing" objects, or even by allowing to track them via log messages by their UUID).

If you wish, you could yet try and reproduce the bug with the following LL_DEBUGS tags enabled (you can enable them before login (if the issue only happens on login) via the "Advanced" menu, "Debug tags" enrty/floater, or after login in "Advanced" -> "Consoles" -> "Debug tags", or add those tags to app_settings/logcontrol.xml):

"ObjectCache", "UpdateType", "ViewerObject", "Mesh", "MeshQueue" (for the latter two, if any of the "missing objects" is a mesh).

The log might then become useful...

@sl-service-account
Copy link
Author

Whirly Fizzle commented at 2022-01-30T02:06:42Z

@henri

  • "ObjectCache", "UpdateType", "ViewerObject" debug tags enabled for each session.

  • Please see files in Whirly_Cool_1.zip attached
    This contains...

  • Session_1.log - logged in at http://maps.secondlife.com/secondlife/Main%20Channel%20Sandbox1/99/152/23
    Rezzed a default cube from build tools, instance UUID of cube: b9532a4a-1d15-265d-3508-35342b90e784
    Logged out.

  • Session_2.log - Relogged to last location.
    Cube did not render.
    Rezzed a 2nd default cube via build tools,, instance UUID of 2nd cube: e0d65b4b-4218-5f56-cd5b-b0c90715c4ba
    Logged out.

  • Session_3.log - Relogged to last location
    Both cubes failed to render.
    Logged out.

  • Also included the files from the folder C:\Users\Owner\AppData\Local\CoolVLViewer\objectcache

@sl-service-account
Copy link
Author

Whirly Fizzle commented at 2022-01-30T02:16:23Z, updated at 2022-01-30T02:32:01Z

Additional info - I feel this is important to find the cause of the bug

  • When the bug reproduces & there are objects that fail to render, surround selecting the area the objects are in does not work to force them to render. You are also unable to see the invisible objects in wire frame mode. Changing active group tag does not work to render the objects either.

  • BUT if another avatar right clicks the objects you cannot see, the objects will immediately render for you!

    @henri
    Session_4.log attached - this is session after Session_3 above on Cool VL Viewer with the debug tags enabled.
    Whirly logged into last location again and the 2 cubes b9532a4a-1d15-265d-3508-35342b90e784 and e0d65b4b-4218-5f56-cd5b-b0c90715c4ba still failed to render.
    My Alt Kippy Adored then right clicked on the 2 cubes and they instantly rendered for Whirly.
    Kippy was logged in on Firestorm to keep the Cool VL logs clean.

@sl-service-account
Copy link
Author

Chorazin Allen commented at 2022-01-30T10:02:43Z, updated at 2022-01-30T10:41:48Z

Further to Whirly's last message - from my testing, a change in the object (eg someone else opening one of the sliding doors mentioned in my repro) also causes it to get drawn properly.

@Henri/Whirly - I suggest heading for the area of cache probes and local object IDs (rather than UUIDs) coming from the server - I found these weren't happening for affected objects in my testing

@sl-service-account
Copy link
Author

Chorazin Allen commented at 2022-01-30T10:40:26Z

I've just been doing some more testing following a report of this from another Kokua user. I can confirm that it's very particular about whether it will repro at all - it won't repro for me at the second report's location.

There seem to be two distinct states - is never going to repro OR will repro fairly quickly and consistently on some of the same set of objects.

@sl-service-account
Copy link
Author

Henri Beauchamp commented at 2022-01-30T10:45:11Z, updated at 2022-01-30T10:45:54Z

Thank you so much, Whirly !

At last, some relevant info.

Even though I am still puzzled, because I reproduced your procedure (went at the exact same spot, rezzed a cube, logged off, logged back on), but did not see the bug happening. I am posting my logs (MissingObjectsNotReproducing.zip): as you can see and unlike what happens for you, on relog, I do get the FULL_OUT message, and so the cube appears...

There is definitely something fishy happening server-side, but the reason why it happens for some people and not for others is a total mystery for me !  Did you try with an alt avatar of yours (on the same computer, of course) ?

Note that I could likely work around this bug (by forcing an object cache update/write on OUT_FULL messages), but this won't fix the root cause for the appearance of this bug since December 2021...

MissingObjectsNotReproducing.zip

@sl-service-account
Copy link
Author

Whirly Fizzle commented at 2022-01-30T13:33:42Z

@henri

  • I can reproduce this bug on my alts too.  I tested with 5 different accounts, each with a different character at the start of their UUID, so presumably they use different inventory servers - just to cover all bases  :D
  •  So far I have been able to repro this bug very easily on my Windows 7 desktop computer at home, my Windows 10 laptop at home, both using Wifi, location Edinburgh Scotland, ISP Talk-talk.
    I have also been able to repro at my parents on my Win 10 laptop and on my Dads desktop (wired connection). Location Hull Uk. ISP KCOM.
     

@sl-service-account
Copy link
Author

Whirly Fizzle commented at 2022-01-30T13:49:18Z

A couple of thoughts...

  • Could this bug be caused by antivirus software not playing nice with serverside or interest list changes?
    All my computers are using Avira antivirus & so is my Dads.
    I will uninstall Avira later & see if I can still repro with no antivirus on my laptop.
    All systems I have tested on are using the default Windows firewall.
    Is everyone who has reported this bug been using Windows? Does it repro on Mac/Linux?

  • Back in 2013 (I think) I remember a bug caused by changes to the interest list that caused lots of objects to fail to render for certain people only. At that time I wasn't affected.
    I can't find the JIRA issue for it but I remember it being discussed & debugged on the forum, however I can't find those posts because it was on the old forum software.
    I remember Andrew Linden fixed the bug just before he left LL.
    As far as I can remember the bug only affected people with a certain "feature" on their connection, but I can't remember what that was. I think it was something outof the users control though.
    Can anyone remember what that was?
    Could this bug be caused by a similar thing?
    I will login over my cellular connection later & see if I can still repro.

@sl-service-account
Copy link
Author

Chorazin Allen commented at 2022-01-30T13:58:51Z, updated at 2022-01-30T14:28:16Z

@whirly - Windows 10 or 11 (10 initially, now 11). Norton AV. UK based with Openreach-hosted ADSL broadband, All my reports are on Windows, however that's most of the Kokua base so I don't think that can be considered statistically significant for Kokua. I'm currently trying to get another stable repro (the first one was someone testing beta builds for me). I'm going to give your cube-based repro a try and see if that triggers for me.

Update: Reproduced on Mac with LL 6.5.2, I also have what looks like a potential stable repro with LL 6.5.2 for Windows or Mac. More on that once I've done more testing and grabbed screenshots etc

@sl-service-account
Copy link
Author

Chorazin Allen commented at 2022-01-30T15:43:06Z

Summary: Reproduced easily with LL 6.5.2 on Mac and Windows. On Windows this is with and without Norton AV active. Test repro based on Whirly's

Test location: Thirdspace/128/128/21 (opened for testing of this bug - 60 min autoreturn)

Tip: should the floor prim fail to rez (seen this on one test cycle) say anything on channel 0 to make a script in the floor prim perform a colour change that should bring it back into view.

Mac test sequence

  1. Start LL 6.5.2, trigger a cache clear, quit viewer
  2. Log in at Thirdspace 128/128/21, ensure avatar is within the blue square and sees six black squares as rez targets for testing. Rez a standard box in one of the black squares. Relog
  3. The cube from step 2 is absent. The rest of the scene is correct. Rez another cube in a different black square. Relog
  4. Both cubes are now absent. Rez a third one in a different black square. Relog
  5. All three cubes are absent.

Windows test sequence
As above, with similar results, however the floor prim also vanished from view on one test cycle. Same results on two different laptops.

It seems that the most recently placed objects can be more vulnerable. In a test run where I placed cubes 1-3 on one Windows laptop then changed to another laptop cubes 1-3 remained present whilst newly placed cubes 4,5,6 each disappeared on the subsequent login.

A note about the test location - Thirdspace is a very sparsely populated Homestead - only 24 LI before any test cubes get rezzed. So, ideas I had before about this happening with very visually busy regions are one more factor that's actually not relevant.

Also reproduced it with Cool VL 1.28.2.57 using same repro. More on that in the next comment (from a different computer)

@sl-service-account
Copy link
Author

Chorazin Allen commented at 2022-01-30T15:48:54Z, updated at 2022-01-30T15:52:24Z

Tested with Cool VL 1.28.2.57 on one of the Windows laptops used above. Same test sequence as before at Thirdspace 128/128/21. Each time the cube rezzed on the previous login and previous cubes failed to appear.

Log files (with ObjectCache, UpdateType and ViewerObject) attached in a zip file as follows:-
001 - post cache flush, wait for inventory to reload, place cube 1
002 - no cube 1, place cube 2
003 - no cubes 1 or 2, place cube 3
004 - no cubes 1-3

cube 1 uuid was 250bb219-3cd3-1c17-0de4-5fc2e359c309
cube 2 uuid was 3f14a8e8-1b1e-c500-7583-ed1bf34d13d0
cube 3 uuid was 8482835d-be21-65ef-5c05-9939de974b07

However, none of these uuids appear in the four log files

Attaching file: CoolVLViewer_CA.zip

@sl-service-account
Copy link
Author

Henri Beauchamp commented at 2022-01-30T16:10:38Z, updated at 2022-01-30T16:21:44Z

The initial reporter of this bug for the Cool VL Viewer is using Linux, so it is OS-agnostic and cannot be an anti-virus issue (we do not use, neither need to use such a thing under Linux).

-I doubt very much that the issue is related with anything network-specific, unless you see a high rate of UDP packets loss (but even then, why just OUT_FULL messages ?).- [*]

To me, it furiously looks like a server-side bug, likely in the interest list code. Why it affects some people and not others yet stays a mystery (AFAIK, CDNs are not even involved in UDP messaging with the sim, and even though sim servers are AWS-hosted now, everyone in the same sim is connecting to the same AWS server, be them in USA, Japan, Europe or even at the North pole ! :D).

[*] EDIT: I just had an idea: what if the OUT_FULL UDP message got too large to fit a single UDP packet ?... Normally, an UDP packet is 508 bytes at most, but can be accompanied with a payload of up to 67Kb split in several (larger: up to 1400 bytes or so, IIRC, to which you must add your network link protocol overhead) UDP packets. The MTU may then get in the way. What is your network I/F MTU ?

@sl-service-account
Copy link
Author

Chorazin Allen commented at 2022-01-30T17:31:02Z

@henri MTU=1500 is the default 1500 on the Mac and both Windows systems

@sl-service-account
Copy link
Author

Henri Beauchamp commented at 2022-01-30T17:55:53Z

Well, yes... But it is in fact a bad question of mine, since your MTU determines the fragmentation of the packets you emit, not the ones you receive... However, if any router at your ISP is dropping UDP packets from LL servers because the packet is too large to fit the standard MTU they use, it could totally explain the issue.

Since LL changed their server "tooling", it is possible they did not notice they are using too big a MTU on their end with the new configuration.

AFAIK, some broadband providers limit the MTU down to 1454 bytes or so (depending on the way you are connected: cable, ADSL, FTTH, and with which encapsulating protocol: PPPoE, PPPoA, ATM, etc). It means that on LL's side, their MTU must be smaller or equal to the smallest MTU used by ISPs, else large UDP packets will get lost...

We will need for an answer from Lindens...

@sl-service-account
Copy link
Author

Monty Linden commented at 2022-01-30T18:39:17Z, updated at 2022-01-30T20:02:20Z

The SL UDP protocol is not ideal.  Both ends assume an MTU of 1500 and segment filling attempts to keep a segment within that limit, usually with a margin.  These turn into packets that leave the endpoint with Do Not Fragment flags set.  So, they are subject to being dropped in transit.  This is relevant because there are still bugs in segment filling as well as cases where other MTU values are used.  E.g. 1200 shows up in places, and 8192 is used for special purposes and produces fragmented packets (they may or may not be further fragmentable).

So a transport issue is possible.  UDP being UDP, internet infrastructure cannot send icmps back to us about this.  But you'll get hints on the viewer side from breaks in the packet numbering of viewer-bound messages.  I just tried Thirdspace and couldn't reproduce the problem but the build may be simple enough to show breaks easily.  (Wireshark might also do.)

Still smells like Interest List to me but that's the transport story.

@sl-service-account
Copy link
Author

Whirly Fizzle commented at 2022-01-30T18:52:26Z, updated at 2022-01-30T19:00:07Z

@henri

Ha! That's the 2013 bug with the interest list I was trying to remember above! You mentioning MTU allowed me to find it.
That old interest list bug that caused many objects to fail to render needed a fix to keep ObjectUpdate packets within an ethernet MTU of 1500
I believe this is the server with that fix: http://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_Magnum/13#13.01.17.269162#
I'm unsure if any of this is helpful though :D

Edit to add: Andrew Linden explains the cause of that old MTU issue here: https://community.secondlife.com/forums/topic/141117-deploys-for-the-week-of-2013-01-21/?do=findComment&comment=896518

@sl-service-account
Copy link
Author

Chorazin Allen commented at 2022-01-30T19:48:21Z

@henri - I agree completely, but if you want to go after the UDP reception angle, the offer stands

@sl-service-account
Copy link
Author

Monty Linden commented at 2022-01-30T21:55:00Z

I have a negative test that doesn't tell us anything, unfortunately.  Did a UDP packet capture while repeating the Threespace test.  Doesn't reproduce for me.  No UDP packet with data over 1250 in length.  No fragments in either direction.  All outbound with DF.  (Inbound all have DF clear - looking at AWS).  Might want to repeat this test with someone who can reproduce it but will want to coordinate that.

@sl-service-account
Copy link
Author

Whirly Fizzle commented at 2022-01-30T22:16:11Z

@Monty
I'm pretty sure I can repro on any empty region rezzing just default cubesY relogging.
I can test if you tell mewhat to capture.

@sl-service-account
Copy link
Author

Chorazin Allen commented at 2022-01-30T22:22:50Z

@Monty
Same here for a second sample - get in touch via chorazinallen@gmail.com

@sl-service-account
Copy link
Author

Monty Linden commented at 2022-01-30T22:31:45Z

[~whirly.fizzle] [~chorazin.allen] Okay, let's do some fast captures.  I'll coordinate via email.

@sl-service-account
Copy link
Author

Monty Linden commented at 2022-01-30T23:52:04Z, updated at 2022-01-31T00:31:42Z

Many thanks for the testing. Early results:

[~chorazin.allen] had 3 out of 3 failures:

  • No segment larger than 1385 bytes in either direction

  • No fragments in either direction

    [~whirly.fizzle] had 6 out of 6 failures:

  • No segment larger than 1383 bytes in either direction

  • No fragments in either direction

    Going to look at this more closely later. And also might be interested in doing a src vs dest packet comparison because unreliable transmission is still a possibility here.

@sl-service-account
Copy link
Author

Henri Beauchamp commented at 2022-01-30T23:55:04Z

Sounds like MTU is out of the picture, then... This lets us with the interest list updates, but I do not see how it could explain only certain persons are affected and not others...

@sl-service-account
Copy link
Author

Chorazin Allen commented at 2022-01-31T01:02:18Z, updated at 2022-01-31T01:04:37Z

I did one more test pass using Thirdspace and the LL 6.5.2 viewer.... with Chorazin Allen I can still make it happen reliably. With an alt on the same computer, doing the exact same test, it works fine.

What's different? Their avatar uuid of course, plus Chorazin is wearing more scripted attachments, more complex clothing and has a much larger inventory - ie more data to be transferred at login.

@sl-service-account
Copy link
Author

Whirly Fizzle commented at 2022-01-31T01:51:25Z

@Monty, I'll not be logghing in for about 12 hours so if you want to hijack Whirly to see if you can reproduce with my account, please go ahead.
I don't think worn attachments matter. I put whirly in the default female avatar & the bug still reprod.
As for inventory - Whirly's is horribly large & quite flat so when cache isn't clear, logins are a bit slow.

@sl-service-account
Copy link
Author

Chorazin Allen commented at 2022-01-31T08:49:00Z

Checking one more test vector: same computer as used previously, connecting this time via mobile 4G wifi; still occurs with Chorazin

@sl-service-account
Copy link
Author

Henri Beauchamp commented at 2022-01-31T09:39:34Z, updated at 2022-01-31T10:32:31Z

@Chorazin

AFAIK, the avatar UUID is discriminated for the inventory servers (IIRC, based on the MSB of the UUID).

But it might also play a role with the login server(s) (won't be too surprising seeing how logins are sometimes failing with your main avatar and not with an alt, or vice versa). In which case there could possibly be some communications issue between servers (between sim and a specific login server, for example; a race condition or something in that vein, that would cause an incomplete interest list to be sent on login for groups of avatars ?).

I could sadly not test my alts in the "Main Channel Sandbox A"; they get denied TPing there: is this region reserved to a selected few, or to premium accounts ? So I did it in my home region (Hunburgh), but it is running the RC channel [*]... All 5 avatars (Henri + 4 alts) are "immune" to this bug.

[*] Which raises another question: can anyone repro this bug on another sim server channel, or is it specific to the channel running in this region ?

@sl-service-account
Copy link
Author

Chorazin Allen commented at 2022-01-31T10:04:49Z

@henri

There are three regions where I've reproduced the issue (Thirdspace being the most reliable / most free of other factors). All are running Second Life Server 2022-01-06.567269. I don't have any regions where I can't reproduce it at present, however that's mostly because I've been focusing on those where I can.

If Hunburgh's on a different version and I can have access, I'll do a test run there

@sl-service-account
Copy link
Author

Henri Beauchamp commented at 2022-01-31T10:25:18Z, updated at 2022-01-31T10:29:28Z

@Chorazin

You may rez prims on my land plot (Cool Shop, for example just outside of the shop, here: https://maps.secondlife.com/secondlife/Hunburgh/60/134/54 ). There is a 5 minutes auto-return.

I tried Thirdspace with my 5 avatars. Cannot repro the bug...

@sl-service-account
Copy link
Author

Chorazin Allen commented at 2022-01-31T12:46:47Z

@henri - Thanks!

Tested at Hunburgh. Reproduces for me there too.

@sl-service-account
Copy link
Author

Whirly Fizzle commented at 2022-02-10T19:45:39Z, updated at 2022-02-10T19:52:47Z

I can still reliably repro this bug on Second Life Release 6.5.2.567427 (64bit)

  • I can also repro this bug on Aditi.
    Tested at secondlife://Aditi/secondlife/Testylvania%20Sandbox/144/145/22
    Second Life Server 2022-02-10.568376

  • A relog isn't necessary to repro this bug either.
    Rez a box on Region A.
    Immediately TP to Region B.
    Immediately TP back to Region A.
    Box does not render.
    Now this is weird....
    If I then rez another box in the location of the 1st invisible box, the invisible box will then render too.

@sl-service-account
Copy link
Author

Chorazin Allen commented at 2022-02-15T23:16:34Z

Retested at Thirdspace - still occurring

@sl-service-account
Copy link
Author

saglie commented at 2022-04-05T03:45:38Z

I'm among those who reported this issue (to Chorazin). This is an FYI that the issue appears unresolved in that latest update. (Kokua 6.5.3.51729 for me.) 

@sl-service-account
Copy link
Author

Chorazin Allen commented at 2022-04-05T06:47:51Z

I can still reproduce it reliably on the main grid at Thirdspace and other test locations using LL 6.5.3

@sl-service-account
Copy link
Author

Chorazin Allen commented at 2022-04-10T19:07:41Z

Retested on beta grid at 127.7, 128.5, 3.1 in Arnthrud located at simhost-0e546300051cb1e42.aditi
SLURL: secondlife://Aditi/secondlife/Arnthrud/128/129/3
(global coordinates 290,176.0, 275,329.0, 3.1)
Second Life Server 2022-04-01.570311

Still happens

@sl-service-account
Copy link
Author

Marine Kelley commented at 2022-05-12T11:38:39Z

I'd like to chime in and say that it happens to me every other login in my home sim on a viewer based on SL Viewer v6.5.3, my mesh house frequently disappears, and/or some furniture, until I clear my cache or TP out and back (but with that second option, it happens again at the next login).

I was told this probably has something to do with the size of the user's inventory. My inventory is over 115K items (being a creator tends to do that), if that helps.

@sl-service-account
Copy link
Author

Chorazin Allen commented at 2022-05-25T18:28:14Z

Retested at Thirdspace running server 571998 which does indeed resolve the issue according to my tests. Well done!

@sl-service-account
Copy link
Author

Dan Linden commented at 2022-05-26T18:08:38Z

This may be fixed by change in RC server 2022-05-20.571998. If your region happens to be on that version, let us know if this bug is fixed.

https://community.secondlife.com/forums/topic/487186-deploys-for-the-week-of-2022-05-23/#comment-2458378

@sl-service-account
Copy link
Author

Henri Beauchamp commented at 2022-05-28T13:08:55Z, updated at 2022-05-28T13:41:02Z

Sadly, I am still seeing the issue happening in sims running 2022-05-20.571998.

I just saw it now (2022-05-28 12:57 UTC / 05:57 PDT) on arrival in sim "Into The Woods": a mesh wall (not a newly rezzed one) refused to rez (including with the "Refresh visibility of objects" feature of my viewer, which basically triggers a full rebuild of all objects in the objects list at the render pipeline level). As usual, "camming out" (ALT + left click to focus, then ALT + move mouse backwards) beyond draw distance, staying 30s or more like that and then resetting camera fixed the issue (likely re-triggering an interest list full update).

Equally, in my home sim (Hunburgh, running the same RC server), I sometimes (rarely, but just as often as before) see my (non mesh) HUD missing all but its root prim after login (the prims are there, and can be edited, but refreshing the child prim textures or anything else won't make them appear: I must relog or detach and reattach the HUD to see it rez fully). Same symptom as before the putative fix...

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

1 participant