|
|
|
[
Permlink
| « Hide
]
kwakkelde kwak added a comment - 08/Feb/09 06:57 AM
I am looking at times over 2000ms now in spikes, Net time seems to spike aswell, I am not sure this is because of the same problem though.
I've been seeing this in Pak and other sims too, it's something that needs to be addressed A.S.A.P. before it turns into one of those widespread and universally-felt "Why did this take so long to fix, and alienate so many residents?" issues!
Please see see JIRA issue posting guidelines
 For Region-specific help please post a Ticket with Support
Please also see this article in Knowledge Base Someone in Second Life is doing something that makes the whole region run much more slowly than it should. Thank you : ) @Ella
I don't see how this post is in violation with the guidelines. Next to that this affects more than one region, as Nemesis already stated. This is bigger than one region and needs to be addressed. I'm not a premium member so filing a ticket is rather difficult ( no not impossible). Plus this way people can vote so it will become clear how people think of this issue, a ticket just won't do it here. Also, if one avatar causes this with big scripts (or more people with big scripts) the times would drop when they are gone. This is not the case unfortunately. Times go up overtime until we can't walk or talk or run a single script properly. Vendors and updaters malfunction with all the misery of undelivered products or bugged, halffinished updates. i don't understand why this is tagged resolved, nothing has changed...
two other sims i can think off offhand that get this excessive sim time (other) are Stonehaven Island (where the sim owner reboots it, sometimes every 24-48 hours) and Maemi (on mainland like Pak, it just gets superlaggy til the sim crashes).
I've always assumed that Sim Time (Other) in the server stats was a measure of serverside memory leaks P.S. if someone wants to mark this as [RESOLVED] because it's in the wrong place, please let us know where the RIGHT place is in the JIRA for it! At the moment (sunday feb 8, 8:13 AM PST) the situation in Pak is very bad again.
total frametime varies between 50 and over 400 ms, caused by excessive high "simulation Time). I have seen this in other sims too, eg in Dolores, but this sim is ok now. I hope someone can fix this problem. @Ellla
< rant >It's more helpful to read and think about what's being posted, and try to think of why it may be occuring, than it is to rubber-stamp a verbal pat on the head and a rather noticeable "Please don't sit with the grownups" air of condescension. None of your advice was of any use in this matter, please save that sort of reply for noobs who post "NOBODY WANTS TO HAVE SEX WITH ME"-level things. < /rant > -Having said that, Kwakkelde's reporting of this is timely and relevant to SL as a whole, not just a small slice of it. This issue will become more and more of an annoyance as more regions become affected, nipping it in the bud is the smartest course. Just because one person doesn't experience it or notice it enough to want to bother about it doesn't make it irrelevant to the larger population. More and more, I see places I go to becoming hellish pits of uberslowdown, making the experience of being there somewhat akin to trying to breathe molasses. The dinosaurs in the LaBrea tar pits have kindred spirits in here, let's smooth out and speed up SL as a whole, wouldn't that be nice? I have requested 25 to 30 resets of this sim over the past 3 months. It appears that memory is being exhausted and disk swap is commencing when the times begin to increase. As far as the load we run several islands where the load is comparable and these rarely require a reset. Pak, as long as I can remember has had this problem and it is now approaching 3 years. This might be a good place for you to observe and analyze a classic problem that affects second life.
Stonehaven has the additional issue that it has lots of heavily scripted objects written by different persons, some of them with little experience how to write little-lag scripts.
Pak and Stonehaven have, however, the same target group. A target group which wears the same attachments. I don't want to point at the creator of those attachments. And even if my little theory below is right it is not her fault. Having said that, here is my little theorie: The attachments worn on the "r forearm" create lots of llLinkMessage (about twice per second) and llWhisper/llSay (about once or twice per second). There is a HUD which listens to those chat events and is known to drop events on laggy sims. Sometimes it does take much more than one second to process them and so the queue fills up. There is similar spam created on channel -9119 by some of those attachments. And the exhibits in the store next door may be listening on that channel. With the same problem that they cannot keep up processing the events in time. Is it possible that dropped chat events or link_message events fill up something? Sim Time (Other) - The amount of time that frame spent running other simulation (agent movement, weather simulation, etc.)
The problem here doesn't seem to be caused by scripts in the first place, but by the high number of avatars. So either it's something in another piece of LL software specific for this sim OR the entire grid could suffer from this under certain conditions. Some Lindenhelp here would be appreciated very much, in the first place some better explanation what the simtime (other) means, the description they give above is vague to say the least. Can we please get sims/regions this is happening in?
personally i only know of Pak, which is fine right now btw, i suspect it's been reset, frametimes really go into the 1000s of ms sometimes
Stonehaven Island ..are also mentioned above After a reset of Pak yesterday simulation time was .3 to .5 ms today it has risen to 5.9 to 7 ms today's current load on the Pak is 18 avatars. Comparing our home Island of Ashford Park with a load of 14 avatars and a simulation time of 1.3 to 2.1 ms. As people leave AP the simulation time drops, on Pak it remains and/or increases
If a restart of the region helps, I suggest not restarting the region until the problem becomes apparent, so it can be investigated.
I've been checking these four sims regularly since sunday. I attached screenshot "Stonehaven Island stats", from the statistics window (ctrl-shift-1). Not requesting a reset means the at least 5 stores on the sim can't run properly, that's not an option for us. Also a steady 40 ms is dreadful, since lag starts at a total of 22.3. 10 ms of scripts is a bogus number when the total time exceeds 22.3. It's the time the sim spends on scripts, not the time the scripts need, this means on bad days script time sometimes drops to less than 3 ms. 40 ms indicates a problem since SL tries to keep it at the 22.3.
In the picture I see 13000 running scripts for 14 + 0 agents on Stonehaven, Pak is no where near that with 7800 scripts for 17 + 7 agents right now. Hi Kwakkelde Kwak,
The trick is to catch the problem at its worst. I wasn't suggesting letting the sim run into the ground .. I know script-time and script-speed are two very different things. Here's another screenshot from Pak: "Dilation with spare time".
As you can see there are only 4 agents on the sim. Time dilation is around 0.87 and frame-time is made up of mostly simulation- and script-time. With more agents on the sim the spare time disappeared. This looks like a new mystery to me. I would say when Time Dilation is under .99, there is no spare time. Also when the total frame time is above 22.3 (which it is on all screenshots, both by me and Moon) how can there be spare time?
Anyway, we'll keep the numbers building up some more, it seems LL has fixed something since it's been about a week now and we can still function, although I did witness a corrupted product update on the sim yesterday (Pak). Times are still way too high, total frametime above 22.3 constantly and sim time (other) close to 9. My script performance meter doesn't come close to 50% btw, when quiet on the sim I get the 30% or so Moon posted earlier at best. Yes it didn't take long, I added a new screenshot. Script performance about 1-2% of an empty sim and times that speak for themselves, this is an average time, not a spike. The sim is completely useless again.
edit Hi Kwakkelde Kwak,
I was just on my way to Pak when I saw your comment. Here's yet another screenshot: "Tortured sim" (pun intended). ======= Second Life 1.22.9 (110075) Feb 10 2009 12:43:24 (Second Life Release Candidate) CPU: Intel Core 2 Series Processor (2401 MHz) libcurl Version: libcurl/7.16.4 OpenSSL/0.9.7c zlib/1.2.3 If the screenshots from yesterday weren't enough, another one, now on a near empty sim. Times are higher again. Can't walk can't run a script..... Some help please !
Oh and I decided this is enough to call the issue showstopper instead of major (although in the past we have seen much much higher times) And now, two seconds after we filed a ticket the times have dropped...weird, still too high though with 12ms sim time (other) and only 4 agents on the sim, but the sim runs again.... for a few days
Yay I'm way too optimistic...the better times were a hickup, worse than ever...as shown on the screenshot
Same thing is happening on our sims - Huge lag spikes, but overall lower sim performance, scripts lagging etc. Its more noticable on our class 4 sims, but its happening everywhere. As its most noticable on class 4 sims, the only support help we can get is "upgrade sims" - these sims all ran well a few months ago :/
Region: Snark - Second Life Server 1.25.5.109327
Snark found running with a high and persistent Simulation time, increased frame time, reduced FPS and time dilation Time Dilation: 0.72, Sim FPS: 31, Physics FPS 31.6 Total FRAME time: 31.9, Net time: 0.3, Physics Time: 0.1 The statistics read as if the region was running with a target frame rate of 35fps, Moon Metty's benchmarks show reduced script performance (430 vs 650) and the usual time dilation variations but around a reduced baseline. Snark is a class 5 sim and normally runs with spare time and little or no time dilation. This condition persisted for at least 4 hours. I'm aware of the problem described in this JIRA, Stonehaven Island is adjacent to Snark. The conditions observed on Snark represent an mild expression of the problem on an empty sim that normally runs with little or no time dilation and Simulation time <1ms. There were no unusual objects of script in the region. Disabling scripts/physics had no effect on the Simulation/Frame times. As this appeared to be a clear example of the problem in it's early stage on a 'clean' sim I contacted concierge and asked that any availabe logs and debug information be saved (Ticket 4051-326362). The region as crashed at 2009-02-28T04:05 to produce a dump. After the restart the region performance returned to normal. I contacted concierge about this issue taking place at Stonehaven Island back in November. TJ Linden took a look at that time. I'll paste in some of our conversation:
[2008/11/13 11:08] TJ Linden: It looks like the issue is with a group of animations... we can;t tell what is using them, but it pretty much either has to be an Animation Overrider on an avatar or possibly an Animation vendor. edit 2009/03/09: Just my 2 cents. Sim Statistics, before and after a restart. This is a fairly mild example. Sim Time (Other) has been known to climb to 100ms+. Typically it gets into the 20's before it becomes intolerable, I become aware of it and perform a restart of the region.
I've uploaded 2009-03-06_Stonehaven Island-24h.png which shows fps (black), time dilation (red) and agent count (blue bars) for Stonehaven Island 2009-03-06 14:00 to 2009-03-07 14:00. The two region restarts can be clearly seen with the slow drop of frames/sec to <10 and time dilation to <0.2.
Hmmm it appears that the Lab has no idea how to fix this problem. Pak continues with a 20 hour cycle after reset going from a perfect sim to one that has 250 ms of frame time all created by the Simulation time other. This is a direct quote from the Wiki on what simulation time encompasses.
Since there is no weather on Pak one might make the assumption that the few people who use the sim create this problem. This sim's traffic is not out of the ordinary so there must be a problem with the under lying software. It certainly would be nice if they could solve this problem and not send back a response that says there are many scripts running, none that have high times. Since simulation time other is not a by product of scripts it seems there is still confusion of the technical side by the helpers. At the moment performance of Pak is very bad again. a Timedillitation of about 0.1 and a framerate of less than 5 frames per second.
Leaving Pak is completely impossible. TP's to out of Pak fail, and walking out of Pak gives an "expired region handoff". The only way to leave Pak is to logoff and logon in another region. I posted a support ticket, i hope they will restart Pak so it will be bearable again for some time. I gathered some statistics of Pak over the last 24 hours, i will add these in an attachment. I hope someone can find the cause of this problem and solve it Performance statistics of Pak from 13 march 12:45 (SL time) till 14 march 12:27 (slTime)
Note: times in graph are UTC+1. Digeut seems to be having this problem, too - simulation time seems to move around some but mostly hovers at 20ms/frame. I looked at the regions mentioned above and they seem to be a lot happier than this, even though their avatar/script counts are higher.
Simulation time on digeut was hovering near 40ms tonight.. Asked concierge for a restart and now it's around ~3.5ms, 15 minutes after the restart. Still seems a lot higher than it should be with 25 people here but it's a whole lot better than 40ms..
There are still issues with 1.26, as discussed at Prospero's OH today.
2009-04-28 Stonehaven Island sim3139.agni.lindenlab.com Server 1.26.2.117266 The Simulation time on Stonehaven Island increased from ~10ms at 13:30 to ~100ms by 19:00 with Time dilation dropping to ~0.20 before the regions was restarted. Screenshot attached showing the Sim stats and a 24h history of FPS (black), TD (red) and Agent count (blue). it seems now that im experiencing a similiar problem on the isle of prodigious , its a busy sim but always i have mantained good stats and time dilations of 0.95 and above..since approx 2 weeks ago this has slowly degraded over a few days unless i reset the sim. at which point everything goes good for a day or so before slowly becoming worse again..
the culprit seems to be something effecting the simulation time..after a reset ..sim time is 1-2ms alll good but over time this increases..at one point 8-10ms before i reset the sim due to much lag / time dilation.im sure this would just get worse and worse but i cant afford to watch people walk in a pea soup. script times seem ok..i can't spot anything thats jumping at me for causing this from the information i can get from the stats bar. sorry for the lack of plots..it was more of a ..theres someone else outt here with a similiar problem..right now i raised a ticket and await lindens views. hopefully they can identify the rogue script, object.. As a point of interest is there anything out there that allows me to monitor scripts/ objects in more detail , and see if i can spot it? |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
|
Bug
Reopened
Showstopper
Ever rising "sim time (other)" up to 100s of ms
