|
|
|
I have to wonder if these two may be connected....if some of the leaking memory may be broken texture cache.
I can confirm this bad behavior under a Linux based system too.
The problem occurs with the 1.20.4 RC and as well with the 1.20.17 Release. I've tried both clients on two different machines and got the same result on each test...always reloading textures. Greetings, Johann Second Life 1.20.17 (98669) Oct 5 2008 10:13:01 (Second Life Release) ... CPU: Intel(R) Core(TM)2 Duo CPU T7300 @ 2.00GHz I think this might be related to this issue:
There seems to be no or maybe a wrong prioritization in the order textures are loaded. Entering a location textures in the distance are loaded first and not those close-by. This is especialy annoying while shopping: I see the shop and the trees outside, but the vendor pictures appear at last. The same issue with map window: when I open the map, one should think that I want to see the map asap and NOT the location I'm in! It's the same in 1.21.6 release viewer.
I think this relates to VWR-8503. At least it's part of my problem.
I agree with Joshua...even with my draw distance set to 64m and not moving at all, on the mainland my textures console is a continuous shower of purple lines; it never settles down.
I believe the ordering problems to be strongly related to the heavy cache scrubbing that occurs in the new viewers. Textures loaded by priority appear first and then they get scrubbed out. Perhaps there is an ordering bug, but this scrubbing one needs to be fixed first because it is perturbing many of the other texture loading optimizations. See http://jira.secondlife.com/browse/VWR-8503
 .
I really do think the texture loading / cache logic is fundamentally flawed. I've been standing in a sim for some time now (40 minutes+) ... so far as i can see the whole sim is loaded ok, but the texture console still shows lots of activity. Worse, I changed my T shirt and after 25 MINUTES (yes, minutes) it still hasnt rezzed.
I strongly suspect that a dropped packet during a texture download causes the texture load to abort and/or wedge, cause if I TP to another sim, the T shirt loads at once. But of course, this is really another symptom of using UDP where TCP should be used. The entire network stack in SL really should be flushed down the toilet. I've seen that "wedged texture" problem too, Carl. One hopes LL dev management realizes that this problem (and others like it), scaled across their entire user base, do go directly to their corporate bottom line as unnecessary infrastructure costs.
As of SL 1.22.2 (104576) Dec 2 2008 12:10:34 (Second Life Release Candidate) this is as bad as ever, if not worse.
After 2 hours at the same location (a club) textures are still being loaded like crazy. Huh? A few ppl have some and gone, but mostly things are stable. What is going on? I leave a place, and return at once. All textures are reloaded from scratch. I just tried to display a texture .... after waiting 55 minutes I gave up. 55 minutes??? SL is showing 24,000 people logged. There is no way i can do the math that makes this performance level make sense. Somebody's gonna have to show up at office hours and plead for this to actually be assigned. Something is very, very broken in texture land.
OK, as of 1.22.4 this problem is as evident as ever. I'm currently dancing in a party at a sim with about 20 other AVs. The sim doesnt appear overly loaded, yet a change of clothing i made 45 minutes ago (yes, 3/4 of an hour) still has not shown up. Numerous AVs are permanently gray.
I can't really make the math make sense. There are about 1000 textures in this location according to the console. Most textures are about 16K in size, giving a total of about 16Mbytes of textures to download. After waiting 45 minutes this means i'm getting an average download speed of about 5Kbytes/second, which is pathetic (and it hasnt finished yes, so its actually worse). There are 20 AVs here, meaning a sim apparently can only manage 100Kbytes/second download bandwidth??? My CELLPHONE can run faster than that. GUYS; THIS IS BROKEN! Why should I pay for a game that after 18 months you still cannot make work properly? Reverted title to focus on one bug. Was "Texture Cache Appears Ineffectve – distant textures loaded first?"
If you find further information about distant textures loading first, please add it to SVC-3630. I've tested the cache behaviour over some hours of SL Usage.
System: Intel Core 2 Duo 6600, 3 GB RAM, XP-Pro SP3, 7600GT. I've set the Cachesize to 750 MB and used a 1GB-RAM-Disk to avoid any sideeffects of acces and settling times of my harddisk. For testing purposes i used Mark Russinovich's Process Monitor 2.0 (Sysinternal Tools at Microsoft). This tool gives a protocoll of The RAM-Disk was newly started to get a really clear cache, then SL 1.22.5 RC was started. I started at my home position and got a lot of "Disk Write" access - quite normal, every texture and every inventory item is loaded now. Now the Process Monitor Log is interesting: This "Create File" behaviour may be the reason for the inefectiveness of the texture caching if i'm not misunderstanding the whole "cache" thing. Sincerely I love the SysInternals tools. They are indispensable on Windows.
Hi Raban,
A while back, I did another test to see how effective the texture-cache actually is. I set my cache to minimum-size (10 MB) using the hard-disk (I have made a 4GB partition for the SL cache). It was clear that textures loaded a lot slower than I was used to with a large cache. ======= The cache doesn't store a complete sim, it just stores large chunks of data, like textures. ======= The texture-console (ctrl-shift-3) is another thing to look at. Arriving at a new place you see the blue progress indicators grow with various speeds. ======= I'm not saying the cache is perfect, but it certainly lowers the load on the network. @ Raban
What is your draw distance set to? What location(A) are you standing at in your home? What location(B) are you teleporting to? How many seconds are you away from your home location? I believe objects that are no longer in view are removed from memory after 60 seconds. I'm wondering if the texture loading you see after returning to home(A) is actually textures requested from location B. Dan:
> I believe objects that are no longer in view are removed from memory after 60 seconds For what reason? If the cache is full, okay, then these are the candidates to get deleted from cache, but without pain? What left the view may return easily after just a few minutes. And even if they are deleted from memory, they should be availiable fast from harddisk. @ Daniel: Objects are removed from memory (not the cache) to keep the viewer's memory footprint low.
Can anyone provide a specific repro for this bug? I just checked my original repro on the latest (Mac) RC ... it still applies.
Clear your cache. Logon to a known location (e.g. Home). Wait and watch texture console till all textures are loaded and the system is essentially quiescent. Exit the SL viewer and note the size of the cache (so you know it is below the max allowed and has not overflowed). Restart SL at the same location. If the cache is doing its job then you should see the view load very fast as all the textures will be in the disk cache. But thats NOT what I see ... I see lots of texture loading from the network (in texture console) and an appreciable delay before the view is updated. As a further comment (putting on my software archtiects hat here) ... any cache is only as effective as its hit rate, which can be significantly degraded by cache thrashing. Some informal tests lead me to believe that most "rich" sim locations need about 250MB of cache each, which means a maxed out cache can only hold about 4 sims worth of data.
It seems to me that a typical AV will visit several places before heading home, and if they visit more than 4 (typical), then when they return home then it will already have been evicted ... not good. I strongly suspect that increasing the maximum cache size from the current 1GB to (say) 2GB or 3GB (not a lot of disk space these days) would dramatically increase the long-term hit rate on the cache, with the commensurate decrease on network and server load. @ Dan Linden, Carld Wilder
i made a new testing with 1.22.8 RC, this time with logging out instead of teleporting away to avoid flooding the cache as Carl described well with his software architects hat on. Again i used a RAM disk for cachce, restarte´d the RAM Disk to get the cache totaly empty and used the SysInternals Process Monitor to track the cache read and write access. Going to my home point and waiting for inventory/cache are ready - logged out and logged in again with started Process Monitor. It's interesting to see, that on the same position (my home position) and view í got a lot of write access to the cache but very low read access... I repeated this with distance Settings from 96 to about 250 m without significant changes. Why does the cache get data written it should have already stored instead of reading them from the disk? The network load goes high, this points to loading much things that should be in the cache, writing them into the cache again and almost never get them out of the cache? As Moon Metty posted, there seems to be room for improvement on the cache. I don't think this is a question of memory size footprint: You may remove items from the memory - put them into a disc cache! In either case a disk will be faster than the download and keeps down network load and the ressources on Linden Lab's data centers. The cache manager should be re-engineered or rewritten to more efficiency, even keeping in mind a later enlargement of the maximum cache size . This takes some load to the client computer, but it takes a lot load from the network and the servers. Greeetings Raban ... what exactly are you monitoring with Process Monitor? Remember that aside from SecondLife, Windows itself also has a disk cache and if you have a lot of RAM (e.g. 4GB) that its possible that the entire SL texture cache has been cached by Windows itself. If you monitor physical disk activity you would not see the Windows cache hit reads (only the writes, which you are seeing).
Carl, remember that whereever cache there is or ramdisk... SysInternals will monitor disk reads and writes at the OS level. It is pretty obvious that SLviewer is writing it's cache and barely using it.
@Carl
Process Monitor 2.0 shows every access to a file or a device (and more) separately thus generating a huge list of accesses within short time. I've made the cache to a RAM-Disk, there is nothing beside the SL cache on the RAM disk.. (normaly, SL cache runs on a separate partition here - i've checked, makes no difference). You are able to see "file opened" - "read"-"write" - "file closed" events in the output, so this should be access to data, not a physikal disk activities. I filtered for displaying only RAM-Disk Access while testing on the Process Monitor output, so there is no windows traffic shown - Windows itself (and other programms) are tending to make disk access now and then. It should be possible to see the network traffic, too. But there will be possibly other traffic because it's the same device so this is possibly a bad idea. I took a look on the clients network activiy display on the clients statistics display. Search the Microsoft pages for "sysinternals" - the Process Monitor and the other tools from Mark Russinovich are a sine qua non - and they are free! There are detailed descriptions of this programs on the page, too. Greetngs Raban Here's a cache-test with the script from this link:
Cycle through all XY10 textures -Go to a place where there's no XY10-text around. The object is now ready to change cached textures. -Click the root-prim, keeping your mouse over it while the texture is loading. Observe: First there is a period 0.3 to 1 seconds of unloaded grey. After that the new texture goes through the discardlevels very fast. -Click the child-prim, keeping your mouse over it while the texture is loading on the root-prim. Observe: Again, first there is a period 0.3 to 1 seconds of unloaded grey. After that the texture is blurred for a comparable amount of time. The rest of the discardlevels are loaded fast. ======= So, when the mouse is not hovering over the texture, loading from cache gets stuck in the first discardlevel temporarily. ======= And why do we see grey in the first place? ======= Raban, can you try this test using your Process Monitor? ======= Second Life 1.22.8 (109366) Feb 1 2009 13:13:47 (Second Life Release Candidate) CPU: Intel Core 2 Series Processor (2401 MHz) libcurl Version: libcurl/7.16.4 OpenSSL/0.9.7c zlib/1.2.3 Yeah I'm very familiar with Process Monitor .. in fact was discussing it with Mark two weeks ago
And I agree, as long as u are monitoring at the file read/write level you will seeing logical texture cache access activity. I'm going to repro your test setup here and see if i get the same results. I strongly suspect the cache hit logic in SL is broken and returning fails when it should be returning hits. If so, then the impact on SL might be enormous ... the extra bandwidth to the servers alone would be huge. As for conformation from the server .. this really should not be necessary ... textures are (as I understand it) immutable under a specific UUID so there is no concept of "stale" data in the texture cache (and even if there were the client could still use the texture while checking with the server). I think it'd be much interest on trying under Snowglobe 1.1.2 or Snowglobe 2680 or more.
OK, I'm commenting to keep this issue alive. I'm standing at my home as I type, which I left not 5 minutes ago. And EVERY SINGLE texture is being fetched from the servers.
THE CACHE IS TOO SMALL. THE CACHE IS TOO SMALL. THE CACHE IS TOO SMALL. Can you hear that? It's the SINGLE MOST CRITICAL ISSUE WITH SL. Think about it ... i've done the math. A cache 8x larger, 8GB, peanuts by todays standards, would make the ht rate DRAMATICALLY increase. I estimate that it would reduce server load for textures by a factor of 3x to 10x. Think about that. Texture traffic to servers 3x less AT LEAST. Clients that load after TP far far faster. Vastly reduced network load for SL. And yet years (literally) go by and we are stuck at 1GB. And NO-ONE seems to be addressing this issue. But hey! We're gunna get lip-sync right? FAR more important!!! Absolutely agree. After a brief apparent improvement, overall texture performance on the main grid is at least as bad as it ever was, and much worse most of the time. This is killing in-world commerce; operating in a shop full of vendors is a study in grey boxes. Particle systems all begin as a spray of grey squares; many never recover.
Yes, amazing. Cache is almost useless... Since this jIRA has been brought up I have a computer 10 times faster, an internet link 3 times faster, and a video adapter 8 times faster... and you know what? doesn't help. The sea is beautiful... but my basic transport poofer is still gray everyday.
Where's the HTTP promisse? I would love to host my own 500GB proxy cache to FLY Secondlife. Absolutely agree, too.
As i mentioned above i'm rechecking this issue regularly, with the same measurement procedure. Linden do you ever read this: Nothing has changed! This should be set to an urgent "showstopper"! Shopping in a mall is far, very far away from fun - sometimes, iv you are patient, it works - sometimes it is completely impossible. Technical Side: I've added a network monitoring to my testdrive here... You can see what happens. Go to a new place: TP to the other side of the SIM and back This is a not the behaviour of a cache This cache is useless, a gigabyte desert. i agree - the cache logic seems screwed up. @ Carl:
i just made a simple Check. I've logged out, and cleared Cache - with setting my cache in a separate partition, this is simply done with a quick format and leaves a really clean cache. Then i logged in to my home position (which is mostly sea and land texture and a volcano, not really much to load.) After everything has loaded, i logged out and checked my cache: It counts about 500 MB, far away from 1 GB limit. Rebooted to clear every possible cacheing in RAM. Logging in again this small amout of textures does not load from cache but are comming from the server. Process monitor shows only Write Access (again!) - nothing is read. Acess is done vie Network. I'd really like to have a bigger cache, the effects you described are correct. If there is a way to test i'll be happy to do testing. But it is a senseless discussion at all if nothing is done by Linden or any of the alternative viewers. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
|
Bug
Open
Critical
Texture Cache Appears Ineffectve
Second Life 1.20.17 (98669) Oct 6 2008 12:24:29 (Second Life Release)
You are at 223182.7, 306352.0, 25.6 in Atmosphere located at sim4948.agni.lindenlab.com (216.82.27.159:13001)
Second Life Server 1.24.9.98659
CPU: Dual PowerPC 970 (2300 MHz)
Memory: 2560 MB
OS Version: Darwin 8.6.0 Darwin Kernel Version 8.6.0: Tue Mar 7 16:58:48 PST 2006; root:xnu-792.6.70.obj~1/RELEASE_PPC Power Macintosh
Graphics Card Vendor: NVIDIA Corporation
Graphics Card: NVIDIA GeForce 6600 OpenGL Engine
OpenGL Version: 1.5 NVIDIA-1.4.18
LLMozLib Version: [LLMediaImplLLMozLib] - 2.01.18637 (Mozilla GRE version 1.8.1.13_0000000000)
Packets Lost: 0/564 (0.0%)