History | Log In     View a printable version of the current page.  
  • All submissions to this site are governed by Second Life Project Contribution Agreement. By submitting patches and other information using this site, you acknowledge that you've read, understood, and agreed to those terms.
Issue Details (XML | Word | Printable)

Key: SVC-385
Type: Bug Bug
Status: Closed Closed
Resolution: Fixed
Priority: Showstopper Showstopper
Assignee: Tofu Linden
Reporter: Ruud Lathrop
Votes: 286
Watchers: 0
Operations

If you were logged in you would be able to see more operations.
2. Second Life Service - SVC

Huge decrease in simulator performance after 2007-06-28 rolling restart

Created: 02/Jul/07 02:15 AM   Updated: 19/Mar/08 05:13 AM
Component/s: Performance, Scripts, Simulation
Affects Version/s: 1.17.0
Fix Version/s: None

Environment: This issue is unrelated to any client environment.
Issue Links:
Duplicate
 
Relates
 

Linden Lab Issue ID: SL-47081


 Description  « Hide
Earlier info on the blog: http://blog.secondlife.com/2007/06/28/rolling-restart-starting-1045-pst/


Several people have notice that after the rolling restart of Thursday, June 28th, 2007 that the performance of the simulator is decrease by 50% or more.

Thomas Conover has done some testing and described its as follows: (Copyed with his permission)
Somehow Linden Labs has managed to decrease all SIM's ability to execute scripts - that seems to have led to about 50% performance-decrease or more, globally.

This means, all products in-world will run about half-speed or less, in all SIM's in secondlife - and it also means SIM's can only handle about 50% of the amount of total scripts as they did previous to this patch. I have now done deep tests with alot of different scripts and advanced objects in many SIM's, and it seems after this patch from Linden Labs, the SIM's can only handle about 50% of what they used to be able to handle. Specially scripts that handles and stores data thorugh "lists" will suffer greatly. And this seems to be a permanent and persistant effect, affecting all residents - nomatter if its mainland or private islands. I find these results abit disturbing myself.

Please vote for this issue, so Linden will pick this up fast and fix this.

 All   Comments   Change History      Sort Order: Ascending order - Click to sort in descending order
Barney Boomslang - 02/Jul/07 04:42 AM
I have observed drastic performance drops with just very few avatars in the sim. I run the sim Lummerland and we have regular music events there. Before the rolling restart, we have been fine up to 50 avatars without any problems, going up to 70 with just slight sim FPS drops when ppl TP in and out. Now we can have as few as 10 avatars in the sim and observe regular sim FPS drops to below 10 FPS. I observed it with many other sims, too, it's not something connected to my sim (and the code and builds on that sim didn't change significant anyway).

A good way to notice it is to use the MystiTools sim FPS monitor and just wander around - there are often sim FPS drops on sims, even on very new ones (and so supposedly class 5 machines) with only few avatars in the sim.

Torley Linden - 02/Jul/07 07:40 AM
Tateru Nino pinged me about this. Related: http://www.secondlifeinsider.com/2007/07/02/residents-complaining-of-widespread-performance-drop

I'll ask our devs about this...

Lex Neva - 02/Jul/07 09:16 AM
I haven't noticed any performance drop in my sim, Suffugium...

mabb dilweg - 02/Jul/07 09:17 AM
In addition, search is very slow at times, and so is inventory search. I even experienced a slight resurgence of the old problem where textures couldn't be found from the texture-select dialogue tonight. I am experiencing this in both the standard client and the First Look client.

Scripts and gestures are taking longer to start, and Group IMs are also borked again. I'm continually getting the chat failure message.

Nobody Fugazi - 02/Jul/07 09:31 AM
Man I hate JIRA. adding a comment is unintuitive.

that said, despite all the meat puppets who might come in here with an axe to grind after reading the SecondLife Insider article, I have not noticed a change in sim performance. The problem I have seen is related to the rezzing of objects, which has been fixed.

I find it difficult to support that there has been a 50% performance decrease. If there was such a decrease, Second Life would be unusable for many residents who would be up in arms... I don't see that. Granted, I could be a minority but my connection to SL is actually slower than most people's connections.

However, the observations of degraded sim performance should not be discounted. All I am saying is that I have not seen it. I would suggest specific sims use their estate management tools and check which scripts are the ones using the most processor time. That would be a great indicator. All sims that I own/manage are within normal limits. The one time Sentience had an issue, it was because someone had placed 150 scripted jellyfish around the sim. It adds up.

If it is not that, then this may be a more focused issue.

What I have noted is that the asset server hamster needs to be fed and given some aspirin. But that is a known issue.


Dimitrio Lewis - 02/Jul/07 09:40 AM
The patch has caused other complications with scripts. Since Thursday I've received numerous reports of scripts crashing in products I sell, compared to zero reports prior to the patch. The replacements I've issued appear to run ok, however. I hadn't noticed the lag effect until now but it does look like performance is pretty bad.

Inigo Chamerberlin - 02/Jul/07 10:16 AM
The week before the rolling update was spent doing a 'cleanup' after encountering sim performance issues in Lyonesse. I managed to change performance measured by Thomas Conroes Advanced Border & Performance-Scanner from 'average' (around 900) to 'great' (around 1900).

Immediately after the 'update' I'm back to 'average' at best, sometimes dropping to 'bad' (3-400)! There is definitely an issue arising from that last rolling restart.

Ashira Legien - 02/Jul/07 10:23 AM
I live in a residential sim and use the Conroes monitor. Before the update this sim ran consistantly above 3500 awesome. Now it averages about 1500 good. People would not see the difference here if they haven't been watching the performance. I was at the new Australia sim for the big opening party. It was painful to be there. Performance was down to 20 and it was almost impossible to move. In all the sim I normally visit I have noticed a major drop in performance.

Torley Linden - 02/Jul/07 11:09 AM
Thanks for the info so far, our devs have been notified and there appears to be a large increase in simulator agent time usage since after the rolling restart (an observation some of you have emailed me about). We're investigating further.

Aimee Congrejo - 02/Jul/07 11:46 AM
Linked to VWR-864.

I've noticed the same problems in all sims dedicated to C:SI combat since the rolling restart...Samurai Island, Samurai Edo, Shrouded Isle, Endless Dreams, and others. My screen FPS is 30 to 50, sim ping is 80ms to 100ms, sim FPS is in the upper 40s, and time dilation is normal. But everything happens in soupy slow motion when fighting.

Luciftias Neurocam - 02/Jul/07 12:16 PM
Adminning three sims with a massive amount of concurrent ecosystem scripts running on them (Terminus, Pseudozoa, and Second Nature 3), I can't say that I've seen any evidence of a consistent script related drop in performance on any of them.

The several thousand plants on my sim tick once every few minutes, but the animals in them run with clock ticks every few seconds. It's conceivable I wouldn't notice any slowdown in the plants operation do their already dinosaurish pace, but I'd expect to see a performance drop in the animal behavior, which I don't.

Perhaps the problem is not quite grid wide, but still large scale enough that it seems as such.

Bridie Linden - 02/Jul/07 12:31 PM
Mani's investigating!

Funk Schnook - 02/Jul/07 12:53 PM
Ive noticed a major drop in sim performance on my own sim too (FNKY Cake). Agent time and Net time both spike at the same time (>700ms) causing sim fps to drop to 0 for a second or 2. Sim fps doesn't always drop to 0 but Im getting dips every few seconds. It seems a bit random because there will be times where 3-4 minutes go buy without any dips in sim fps. My script time and performance all seem normal though. I hope you guys can figure it out because this is starting to become really painful

chilly charlton - 02/Jul/07 12:54 PM
I've had a ticket in on this since soon after the rolling updates. Although I didn't specifically note script execution I have some stats scripts I regularily run which are in fact taking up to TWICE the normal time for them to complete. Cycles for script execution are definitely being stolen with this last rolling update.

kelley shortbread - 02/Jul/07 12:58 PM
I am the operations manager and estate mgr of Heartland sim, I have seen a huge difference in our sim since the roll back,,,, the contest boards do not work correctly now.. av's have more trouble than before... the lag is bad even with very few on sim.... I also own Lost Paradise sim and I NEVER have lag there and since the update it is very bad

chilly charlton - 02/Jul/07 01:01 PM
Although script execution is definitely hurting here the WORST symptoms since the update are time dilation dropping to 0.01 several times an hour. Maybe this is only on script heavy sims ... I don't know, but scripted objects seem to hobble along and get done what they need done, time dilation dropping to 0.01 a couple times an hour is NOT acceptable however. I don't know if script execution is the culprit or simply another symptom of what is wrong. The freezing sims concern me more then the script execution.

This seems to happen worse when people are tp'ing in. Sim doesn't need to be full either it will happen with 4 in a sim as well as with 70 in a sim. Dilation just drops.

enktan gully - 02/Jul/07 01:19 PM
I'm experiencing a serious drop in performance at my sim Orbiuna. Before thursday I was able to have 60 ppl on the sim without any problem at all. Now 30 agents is enough to have the total frame rate exceeding 22,5ms wich never happened before, even with 60 agents on the sim. Whenever someone arrives at the sim, frame time peaks, up to 2000ms at times. I run almost no scripts on my sim.
Also someth is wrong with scan range, radars that are set to 96m are scanning ppl that are 400m away now.

coldfusion cheeky - 02/Jul/07 01:44 PM
I am co owner of a private island region. We were very careful in choosing what scripted objects we set out and monitor them on a constant basis via the estate tools in an effort to keep script time very low. Prior to the rolling restart our time dilation was locked at 1.00 and 'total frame time' averaged around 10ms. Since the rolling restart total frame time is now averaging 18ms and time dilation frequently drops below 1.00 presumably because the sim is now throttling.

I also work for a gaming island region (Parrothead Cove) where they host a game called Devil May Care (DmC) This game has always been resource intensive but prior to rolling restart we were able to host the game with 25-30 simultaneous players. Since the rolling restart the game has become virtually unplayable even with only 15 simultaneous players. It runs very slowly and freezes (stops playing) regularly while the match is in progress. We have had to close it down indefinitely as a result. I have heard that many other regions that also run this game have been forced to close it as well for the same reason.

A region manager of Parrothead Cove who regularly monitors the scripts via top scripts under estate tools observed that average script time from all scripts prior to rolling restart was around 18ms and jumped to 47ms after the rolling restart.

will geiger - 02/Jul/07 02:05 PM
I heartily agree that this is a huge problem. It doesn't seem to effect all sims all the time. Rather, a sim can be working smoothly, then go into a bout of performance degredation with no warning, then snap out of it an hour or two later. I have witnessed this on the sim I live on when I am there with only one other person, and I have witnessed it on numerous other sims with as few as 1 person and as many as 30.

Yngwie Krogstad - 02/Jul/07 02:06 PM
Use control-shift-1 to pull up the FPS info and all that, and click at the bottom to expand the data about Time. Check the Total Frame Time field. I'm seeing it usually averaging in the 22 to 30 range where I usually go. However, I have on multiple occasions seen the TFT when merely standing in one place in Mumun, jump up to in excess of 2300!!!!!!!!!!!!

cammi hudson - 02/Jul/07 03:29 PM
I am a host at Parrothead Cove. now the DMC game is very resource intensive and does produce lag but we have never had much problems running dmc and the other games at the same time, now you cant rum dmc and anything else or the performance comes to a grinding halt. Time dialation goes to .035 sometimes Sim FPS goes to 20 or lower and it frequently freezes and stops altogether causing us to reset the game and hope it gets through a game. It takes 30 minuts or more to geta game done. I've been to other sims and universally every place I goto with DMC running goes to 9 or so FPS and sim ping goes from 200 to 2000 ms varying. Even when 1.17 came and everyone screamed about that at least the sim performance wasnt impacted that greatly. Where I live on a private island I have seen the FPS degrade by up to 50%. My home sim usually has 30 to 35 FPs and .99 Time dialation now its lucky to get 20 FPS. I have a shop on Katrina sim where the 777 casino is..and turning around is adventure. Turning 180 degrees is a major tester of your patience.. I see FPS go to 1,9 FPS to just turn around and thats even with my setting which are tuned to not cause much lag or performance degredation.

valerie viking - 02/Jul/07 04:27 PM
In SL I am a pilot so I try to keep aware of SIM Performance levels. I use a couple of Sim Performance Analyzers to do this. Immediately after last Thursdays update SIM performance in the areas with which I am familiar significantly degraded. These poor SIM Performance ratings were corroborated by Time Dilution and SIM FPS stats as well as real in world attempts to fly aircraft. This degredation of SIM Performance remains wide-spread throughout the grid.

I am a Kim Seifert fan. On Sunday she had three concerts scheduled. Two of them were ruined by SIM crashes. These were severe crashes and it took almost an hour in each instance for the crashed SIMs to come back online..

t0xic scissor - 02/Jul/07 05:05 PM
I have also experienced severe reduction in script performance on my sim since the rolling restart as well. the difference is like night and day.

Ashley Neva - 02/Jul/07 05:12 PM
I own and operate the sim Misfits Cove. And have noticed the aforementioned complaints. Since the last update our server has taken a nosedive in performance, and I would wager that it has decreased in performance and stability by more than 50%. Landing on the sim, causes a resident to shoot up into the air and hang around while flailing their arms. If you are ever lucky enough to find pixel manifestations of ground, you are left doing an levitated - moonwalk, that would shame even Michael Jackson.

Incidentally however, when I had filed bug reports and had gone through other mechanisms of support, no Linden ever acknowledged that such an issue even existed. It was not until I called in for customer support that I was informed of this JIRA.

Lastly, those of you that complain you have not noticed such an issue - are counter productive. I dont understand why you would feel it necessary to negate other peoples concerns. If the issue isnt effecting your gameplay - you need to release your overwhelming sense of worth, and allow those of us who's game it is effecting to comment on the JIRA, instead of blogging it down with our self righteous, "no not I" comments that flood the forums and dont help anyone.

Ashley Neva - 02/Jul/07 05:18 PM
What I forgot to mention is. Our sim has over 8000 available prims. And we have restarted daily, sometimes 2-3 times even. None of which seems to clear up the issues, there is a brief window of "satisification" which fades quickly.

I would think with 8000 prims available, if there is a spawn of jellyfish, it wouldnt cause such severe lagging and inoperation.

athena sterling - 02/Jul/07 08:12 PM
My main sim Hantal has been effected by this as nasty bug since the rolling restart on the 28th. Also I am and Estate Manager for Surreal and have noticed most of the sims viewed are also effected by this as well.

From what I've noticed, Agent Time has creased by almost 10 times min with avg of 6~8ms avgs for about 10 avatars on the sim. And when you get more than 25, script time is limited to about 2ms or so while agent time peaks around 2000ms quite often.

Blinky Lane - 02/Jul/07 09:38 PM
I spend a lot of time in gaming places and have noticed a large reductuion in speed.

I had been playing about an hour or 2 prior to the rolling restart and went back as soon as the reset had moved south and immediately noticed this drop in preformance and allerted Athena to the issue thinking it was a local glitch. Since then me and a friend have visited many other places also suffering the same slow down. In some places certain games are vitually unplayable, to the extent that a high score that used to take 20-30mins playing time, now takes up to an hour to achieve.

This needs to be found and fixed swiftly, as businesses will suffer, though I am sure no ones as allarmed by this as Linden are and I bet they are frantically searching for the problem.



Vickie Maidstone - 02/Jul/07 09:54 PM
I have not used the optional update !.17 yet and have noticed horrendous slow downs in my home (Pandorus) and my Medieval village in Laila. Since the rolling updates this past week, even on log in, it takes as much as ten to fifthteen minutes for everything to rez in before I can even do anything. I didn't know that even without the optional update what the problem was (is) until now. I cannot even finish getting my village in (Laila) done and readied in time for opening in August be cause even TPing there everything is borked and taking 3xs longer to rez in. I know I'm not the only one with this problem and it needs to be addressed as soon as possible.

Smith Fizz - 03/Jul/07 12:08 AM
I agree with Thomas's findings, I have noticed a huge decrease in my SIMs preformance all around, rezzings slower, more ghosting, More Lag, Slower TPs, My few month old sim ran awsome until that last update

Zimzim Yongbo - 03/Jul/07 02:45 AM
Tech Expo has observer these issues almost immediately. The islands scrpt execution times went from around 8s to around 15s. When I signed up to pay for a server at $295/mo it was on the understanding it would perform at the speed of a new class 5 as was available at the time. I now have around half that speed - so I'm willing to offer maybe $150/mo. We all know that LL aren't going to accept such an offer, but it illustrates very well what they've just done to me with the new patch -- they've effectively taken my Class 5 and handicapped it in such a way that it's now a Class 2.5!

How much do YOU think I should be paying for that?

Duke McDonnagh - 03/Jul/07 06:31 AM
I own 8 Sims and they all have dropped by 50% in preformance since this happened. This appears to be script related and does not manifest itself by looking at just the status screen data. Scripts are defiinately MUCH slower. This is impacting the quality of the environment for all residents on these sims. Some Scripts are failing to execute totally and are timing out or showing very ertatic behavior.

luxa budan - 03/Jul/07 07:41 AM
Own 5 sims would have been 10 by now but decided to slow down purchases due to the increasing unreliability in SL like this. Yes, I have noticed the same in my sims. I did notice that the problem runs cyclical from varying degrees of bad to horrible. Yes a credit would go a long way here since my traffic is directly impacted by this problem and my traffic is one of the ways the lindens get paid. We are paying a premium for class 5s and not getting the results. Oh well, may have to go to an even lower gear on sim purchases until things start to stabilize and support to actually catch up with demand.

Lex Neva - 03/Jul/07 09:49 AM
I haven't seen one single comment that lists enough information to make an objective claim about sim performance. We need more information from all of you sim owners.

What's important is not whether the sim has suddenly decreased in performance, but whether it has done so _while load has remained the same_. If you had an influx of visitors after the rolling update, then I would expect performance to decrease.

Coldfusion Cheeky's comment above was somewhat helpful. It's interesting to know that the "total frame time" went from 10ms to 18ms. The question I have is: was that with the same number of avatars as before? What about the number of active scripts? I'd expect to see a jump from 10 to 18ms if avatars showed up with a few thousand scripts on them (as often happens).

So what we need to know is whether sim load (number of avatars, number of active scripts) remained the same as before the rolling update while statistics like total frame time, agent time, script time, etc increased. This would point to a systemic worsening of performance.

As it is now, we just have a lot of people who seem to think their sim is performing worse. While it's pretty impressive that there are 180 votes on this issue, it's still within reason to think that maybe this is just due to transient increases in sim load caused by large gatherings of users or increased numbers of scripts or other content in the sim.

Drew Dwi - 03/Jul/07 10:05 AM
midian city agent time has been the source of high lag when before it would spike but return to normal numbers quickly. This number is higher and remains higher than normal usage in the past. No new scripts or objects from estate tools are adding the lag. Even disabling scripts at the estate level this performance degregation remained constant. I think this issue may also be tied into the underlying issue discussed in SVC-206.

chilly charlton - 03/Jul/07 10:24 AM
To address Lex Neva, yes ALL THINGS BEING THE SAME performance has degraded. It almost goes without saying for those who run some of the places which are commenting in here. On my sims I run the same events day in and day out. My previous comments speak to this being a problem with 4 avatars or 70+ avatars on a sim. This problem CYCLES and happens a couple times an hour. I have scripts I regularily run which take up to 18 hours to complete that now take 36. They are on an EMPTY sim. This was observed DIRECTLY AFTER the rolling update and has persisted. All of these symptoms have. My sims have the SAME OBJECT THE SAME SCRIPTS AND THE SAME ATTENDANCE they have had for over a year.

I'm getting tired of the nay sayers here. I for one do not "THINK" sim performance has been crippled I KNOW it has.

I shut down events maybe three times a year for different reasons. This is one of those times, my sims are shut down for all events until this is fixed, because it DON'T WORK NO MORE.