• All submissions to this site are governed by Second Life Project Contribution Agreement. By submitting patches and other information using this site, you acknowledge that you have read, understood, and agreed to those terms.
Issue Details (XML | Word | Printable)

Key: SVC-3819
Type: Bug Bug
Status: Fix Pending Fix Pending
Priority: Normal Normal
Assignee: Simon Linden
Reporter: NeoBokrug Elytis
Votes: 24
Watchers: 12
Operations

If you were logged in you would be able to see more operations.
2. Second Life Service - SVC

"Top Scripts" in estate tools is inaccurate

Created: 10/Feb/09 01:24 PM   Updated: 22/Oct/09 03:13 PM
Return to search
Issue 217 of 236 issue(s)
<< Previous | SVC-3819 | Next >>
Component/s: Performance, Scripts, Simulation
Affects Version/s: 1.25 Server
Fix Version/s: None

File Attachments: None
Image Attachments:

1. after1.21.6 .jpg
(36 kB)

2. JIRA_3819_001.jpg
(535 kB)
Environment:
Second Life 1.22.8 (109366) Feb 1 2009 13:13:47 (Second Life Release Candidate)

You are at 201174.2, 265896.4, 4001.0 in Malady Bog located at sim2253.agni.lindenlab.com (216.82.17.4:13001)
Second Life Server 1.25.5.109327
Issue Links:
Duplicate
 
Relates

Last Triaged: 24/Feb/09 08:01 PM
Linden Lab Issue ID: DEV-27949


 Description  « Hide
Investigation reveals that the script time located with CTRL-SHIFT-1 is on average double what is reported via the Top Scripts window with the estate tools.

These tests were performed on sims I have estate privileges on, with an Av with no scripted attachments, and me being the only av in the sim.

This error seems to be affecting the performance of the sims I've tested on, skewing Time Dialation and Sim FPS.

edit: Further testing shows that if there are physical items with a high top colliders, that time seems to be subtracted from script time, and added to physics time. Resolving the physic collider issue, makes script time jump to double what is normally reported.



 All   Comments   Change History      Sort Order: Ascending order - Click to sort in descending order
Jahar Aabye added a comment - 10/Feb/09 02:40 PM
Does the EPS (script performance, Events Per Second) change when the script time does?

NeoBokrug Elytis added a comment - 10/Feb/09 06:18 PM
Unfortunately, I wasn't paying attention enough in the past to look at the EPS prior to the most recent server version.

RIght now Region Top scripts is showing me 5.3ms
Time (ms) > Script Time is showing me 10.4ms
Active scripts are around 2.7k
EPS is 300-400.


Maggie Darwin added a comment - 11/Feb/09 10:14 AM
I wonder what fraction of the running scripts are compiled to Mono?

SVC-3352 possibly related?


NeoBokrug Elytis added a comment - 12/Feb/09 09:04 PM
Maggie: I don't think this is related. My estate in general has always been very light script usage in the past. I only noticed the bug because my script time seemed highly inflated, on every sim.

Alexa Linden added a comment - 17/Feb/09 01:51 PM
Can you provide us with more details and number you are seeing and what you've experienced in the past or are expecting?

NeoBokrug Elytis added a comment - 19/Feb/09 01:41 AM
@Alexa: In the past the numbers were the same. This affects pretty much every viewer I have come across, so I assume this is a simulator issue. I am not sure when the bug came up. Attached is an image.

CTRL-SHIFT-1 says: Script time of 7.9ms Which is incorrect.
Top Scripts says 434 scripts taking a total of 2.1ms, which is more correct.

When script time of CTRL-SHIFT-1 spikes it affects the time dilation of the simulator, and I don't believe it should because the numbers are inflated or incorrect. Vagabonds (the sim in the pic) should be running at 1.00 time dilation, but is frequently at 0.99 or less.


Catten Carter added a comment - 19/Feb/09 09:04 AM
All of the 8 sims I have estate privilages on, shows difference in script time and top scropts. It should be easy to reproduce imo.

It would be nice to know which is correct, and what causes the difference?


Moon Metty added a comment - 19/Feb/09 02:39 PM - edited
Thanks to Psi Merlin, who made me estate-owner, I found this test.

-Rez a box, call it "%%%" so it can be found easily, and add this Mono-script:

default
{
    state_entry()
    {
        while (TRUE) llSleep(0.001);
    }
}

-Open the top-scripts window.

Observe: The script uses around 0.08 ms according to the top-scripts.

-Take the box in inventory and rez it again.

Observe: The script does not show up in top-scripts

-Recompile the script.

Observe: It's back to 0.08 ms again.

=======

I did this test on a 1.26 aditi server. Can someone try it on agni?


Moon Metty added a comment - 19/Feb/09 04:46 PM
Amazing, with llSleep(10) I get exactly the same results ...

telexa gabardini added a comment - 03/Mar/09 08:38 PM
I have been posting my problems about this on http://jira.secondlife.com/browse/SVC-3867 but seeing as this is my actual problem I'll post it here. Daily this is getting worse! At this point seeing my homestead sims with a Dilation over .80 and FPS over 40 im getting excited!!! 25.0, 30.0, 40.0ms + script time is now become the norm on sims that never ran higher then 4.0ms prior to Feburary 19th. I am now loosing people off the sims! This is costing us money and we run on a very tight budget with the sims. Here is an idea, what ever you did on the 19th, how about going back to the way it was on the 18th dumping this POS you deployed then and get back to when it actually worekd till you can figure out what happened, this has become completely unacceptable for the tier fees i pay. As im writing this im on a sim monitoring it and im seeing Dilations of the mid 70's Fps in the 30's and script times in the high 20.0's to mid 30.0's ms. This is a sim with 235 scripts not a single script on it that ever ran above .100ms. Now they are pulling 2.0+ms for the top 10 of them. This is one of 12 homsteads i have doing this. How come this is only effecting homesteads? This is what we get for the extra $20 a month on them? Another thing im noticing on these sims is Child agents above 20 when im the only person on the sim. Whats that about? Both this and SVC-3867 are serious issues and need to be addressed above all other issues pending!

Catten Carter added a comment - 04/Mar/09 04:42 AM
I know this isn't related to this Jira, but just to answer, this doesn't affect only homesteads. Class 4 sims show this too, and while it might not be as easy to see, our class 5 sim is showing increased script time as well.

Last year we had a mardi gras event on a class 4 sim. 40 people all wearing floats and approximatly the same amounts of scripts running on the sim. There was lag for sure, but it was doable and we got through it. Since then we upgraded the sim to a class 5 sim, and this year, with the same amount of scripts, the event lags even more with 40 people. Scripts are slowing almost to a hault, objects not updating position or registering touches etc. One would expect better performance on our class 5 sim, but its the same, perhaps even a bit worse.


Amber Barnard added a comment - 07/Mar/09 09:06 PM
I just want to add that my homestead is suffering from this same issue. There seems to be not pattern to which scripts will be top scripts on each refresh. And script times are 17-25 ms where they used to be about 1.25 ms. Dilation is usually about .80 and sim fps about 33. This all started about two weeks ago for me.

telexa gabardini added a comment - 09/Mar/09 10:13 AM - edited
Ok so im guessing nothing was done about this in the 1.25.6 server deployment and yet i see its still not assigned to anyone. Take a look at my stat bar pick up there. This is insane!! This was taken from the sim Kinu Inu about 10 minutes after the server was deployed WOW a 0.47 Dilation, 21 FPS, Im the only person on the sim yet it shows 6 agents, and look at that rocking script time there 23.6!!! Come on now this sim never seen higher then 3.0ms on the script time and never droped below 44.0 on the FPS Dilation was always at 97 or better. Can we PLEASE get this imported and assigned!!!

Al Supercharge added a comment - 09/Mar/09 11:14 AM - edited
This was a while ago and I cant remember what RC version was running during Burning Life 2008.

But here is the problem then ( and I bet it still exists):

I was told my mono script was taking up too much resources. But when I looked at it - it was mostly a WHILE loop with a good few seconds llSleep in the middle of it !

All I did was remove the WHILE and replace it with a self looping llLinkedMessage and I was told that was considerably better!

The logic remained the same and if anything a llLinkedMessage should have ADDED slightly to performance rather than improved it.

Please investigate and fix.


Darien Caldwell added a comment - 09/Mar/09 01:19 PM
That is likely due to them adding more reporting statistics, and your viewer doesn't yet know how to interpret them. I was already seeing this issue using viewer 1.21 with server 1.25. It is likely they have added even more. In doing so the statistics don't line up with what the viewer expects anymore.

What the viewer is showing as Script Time is actually a new statistic called "Total Frame Time" or something to that effect. I could be wrong, but I suspect that is what you are seeing. If you use the RC client 1.22, or are fortunate enough to have a copy of 1.23 viewer, things display correctly.


Al Supercharge added a comment - 09/Mar/09 11:51 PM
Darien,
If your comment is referring to mine?
Then I am not talking about anything on the viewer. I'm reflecting what I was told by the SIM Admin.
She was giving me feedback on what she was seeing from SIM Stats - not Viewer Stats.

Viewer Stats had SCRIPT TIME of > 15ms throughout all this and showed no noticeable difference before or after.

PS Lindens - Please make the STATS window persistent over sessions - I'm getting tired of re-opening it EVERY login. And please also persist STATS settings - I'm getting tired of deleting gauges and expanding TIME )
Thanks in advance.....


telexa gabardini added a comment - 21/Mar/09 12:55 PM
Whats the status here with this and SVC-3867? No coments since the 9th.

still getting ghost scripts, still have script times off the damn chart, still have tenants leaving due to lag and constant restarting of the sims, even after they read this and SVC-3867, still cant understand why this and SVC-3867 are not fixed yet.


Harleen Gretzky added a comment - 22/Apr/09 11:14 AM
A possible cause of this is SVC-4152, lots of good data and pictures there.

ee oh added a comment - 26/Apr/09 11:32 AM
Assignee: Unassigned

that s crazy..

i had before the 1.25 server version sim times from 6ms.. now it s up to 50ms..

and nobody notice?? or ask to reproduce..

all estate owners i know see the same at their sims.. and it s not only ghosts scripts trouble..
sim times 10 times higher on the get top script window.. it s not all lag but a mix of it..
and for scripter i d like to know how to control and fine tune scripts with such bug.

so what?

is there another jira may be speaking about that.. and here is the wrong place..
if yes please give me the link..


Moon Metty added a comment - 30/Apr/09 09:08 PM
Today I found that loops throttled by built-in delays (like in llSetPos) show the same increase in topscript-time.

Simon Linden added a comment - 01/May/09 04:14 PM - edited
Did some testing on this and it does look like the 'sleep' time for various LSL commands causes the difference.

A region with 100 scripts doing simple counting loops doesn't show the problem - the two sets of numbers match. If the scripts contain LSL commands that sleep (such llSetPos()) then the difference happens.


Al Supercharge added a comment - 01/May/09 07:16 PM
Simon,

If llSay has an implicit SLEEP then can u please update the WIKIs.

http://wiki.secondlife.com/wiki/LlSay
and
http://lslwiki.net/lslwiki/wakka.php?wakka=llSay

(I know that llSetPos has an implicit 0.2 secs SLEEP but I didn't read anywhere about llSay having one.)

Thanks


Moon Metty added a comment - 03/May/09 09:59 PM
llSay doesn't have a built-in delay. A tight loop with llSay is really a topscript.

=======

Here's something interesting:

-Create a box, call it "%%%", so it can be found easily.
-Add this script:

default
{
    touch_start(integer total_number)
    {
        llSleep(30);
    }
}

-Open the topscripts-window, sort by name.

Observe: The object uses very little time.

-Click the box.
-Refresh topscripts a number of times.

Observe: Topscripts reports a high time, identical each refresh. When the 30 seconds sleep is over, time goes down to almost zero.

-Click the box again.
-Refresh topscripts a number of times.

Observe: Topscript-time does not go high anymore. After a reset the bug manifests only once, in this setup.

=======

Second Life 1.23.0 (118316) Apr 23 2009 21:20:14 (Second Life Public Nightly)
Built with MSVC version 1400

You are at 180358.8, 254087.6, 40.5 in Snark located at sim3012.aditi.lindenlab.com (216.82.49.224:13001)
Second Life Beta Server 1.26.3.118650


Simon Linden added a comment - 04/May/09 11:16 AM
First, yes, llSay() doesn't have an implicit sleep. It's throttled in other ways – sorry for the confusion

Some more info – there are two different accounting mechanisms involved here: the script time shown in the ctrl-shift-1 window is the block of time spent in the frame running scripts. It's a running average of the last second. It's basically a stop-watch that covers when it starts running scripts until it ends for each frame.

This includes the overhead of looping through all the objects with scripts, making sure they are on parcels that allow it, aren't about to be deleted, etc. In my test (100 objects moving randomly) the overhead was significant but varied widely each frame - it could be almost zero, or close to 40%.

The individual script times are an average time over the last few frames in which the script ran, only counting the time executing script byte code. The overhead mentioned above is not included, so that can contribute to some of the observed difference between the two values.

It also looks like a sleeping script suspends that running average calculation, while it should be considered a zero-time duration to execute for that frame. That may be why it's sticking around in the top scripts as Moon found. I'll try a fix there.

I'll poke a bit more at this but it looks like it's lower priority (as compared to a crash, for example). I think the 'top scripts' is a decent indication of the relative usage but with the significant overhead for each script, its may be difficult to get them to match perfectly.


Simon Linden added a comment - 06/May/09 10:36 AM
Dropping this to "Normal" since it doesn't cause content loss or significant problems.

Simon Linden added a comment - 24/Sep/09 03:40 PM
1.30 has a partial fix here. Scripts that sleep, have zero script energy, or an error will no longer remain in the Top Scripts list, so that ranking is more accurate.

The problem I mention above, where the overhead of switching between all scripts, scanning, etc causes a difference between the sum of all the individual script times and the total time in scripts still remains. We could essentially add that small amount to every script, I guess, but that would need another loop through all the scripts, which will take up even more time.


Moon Metty added a comment - 26/Sep/09 07:31 AM
Hi Simon,

Sorry I wasn't available when this was in beta. The problems with sleeping scripts have not been solved yet, as this test shows:

-Create a box, name it "recompiled", add this script:

default
{
    state_entry()
    {
        while (TRUE) llSleep(10);
    }
}

-Drag a copy sideways, name it "reset".
-Drag another copy sideways, name it "cloned".
-Select "recompiled" and "reset", drag a number of copies upwards. (10 is fine)
-Select all boxes, recompile the selection in Mono.
-Select the "cloned" box, drag a number of copies upwards.
-Append a unique number to the name of all "cloned" boxes.
-Select all boxes in the "reset" and "cloned" column, reset scripts in selection.
-Check topscripts.

Observe: All boxes "recompiled" show up high in topscripts.

-Restart the region.
-Check topscripts again.

Observe: The "recompiled" boxes still show high. The "reset" boxes show high too now. One of the "cloned" boxes shows high.

=======

I agree with what you say about the overhead-times. It's better to try to reduce them than to try to calculate and correct, causing even more overhead. It's a systematic error, and we can probably best interpret when we don't fiddle with it.

=======

About the priority: When an estate manager is looking for causes of lag, the highest topscripts are always first suspect.
A good example is my dilation graph, a powerful, low-lag tool. Typically it shows up about 100 times too high in topscripts.

Now avatars show up in topscripts, which is great. But what if your shoes have some harmless, sleeping scripts?
I think topscripts should give a reasonable indication of script-time, maybe not too accurate, but it should work for every script (even mine ).

=======

You are at 258432.1, 290944.2, 21.6 in Mersenne Prime Too located at sim2.aditi.lindenlab.com (216.82.15.195:12035)
Second Life Beta Server 1.30.1.134345


Moon Metty added a comment - 20/Oct/09 10:02 AM
Hi Simon,

Good news, your fix made it into 1.32.
There are no release notes yet, but topscripts looks convincing now.

=======

Second Life 1.23.5 (136262) Oct 14 2009 12:08:26 (Second Life Release)
Built with MSVC version 1400

You are at 258440.7, 290951.3, 21.0 in Mersenne Prime Too located at sim1.aditi.lindenlab.com (216.82.15.194:13000)
Second Life Beta Server 1.32.0.136438

CPU: Intel Core 2 Series Processor (2401 MHz)
Memory: 2048 MB
OS Version: Microsoft Windows XP Service Pack 3 (Build 2600)
Graphics Card Vendor: NVIDIA Corporation
Graphics Card: GeForce GTS 250/PCI/SSE2
Windows Graphics Driver Version: 6.14.0011.8206
OpenGL Version: 3.0.0

libcurl Version: libcurl/7.18.1 OpenSSL/0.9.8j zlib/1.2.3
J2C Decoder Version: KDU
Audio Driver Version: FMOD version 3.740000
LLMozLib Version: [LLMediaImplLLMozLib] - 2.01.27714 (Mozilla GRE version 1.8.1.21_0000000000)
Packets Lost: 0/6584 (0.0%)