• All submissions to this site are governed by Second Life Project Contribution Agreement. By submitting patches and other information using this site, you acknowledge that you have read, understood, and agreed to those terms.
Issue Details (XML | Word | Printable)

Key: SVC-2511
Type: Bug Bug
Status: Resolved Resolved
Resolution: Fixed
Priority: Critical Critical
Assignee: Andrew Linden
Reporter: Eata Kitty
Votes: 283
Watchers: 44
Operations

If you were logged in you would be able to see more operations.
2. Second Life Service - SVC

Serious Object-Avatar collision problems in 1.22.2 and above

Created: 08/Jun/08 01:44 PM   Updated: 23/Jan/09 05:44 PM
Component/s: Physics
Affects Version/s: 1.23.4 Server
Fix Version/s: None

Issue Links:
Duplicate
 
Relates

Linden Lab Issue ID: DEV-16593
Linden Lab Internal Branch: havok4/havok4-8


 Description  « Hide
Physical objects in 1.22.2 easily and frequently pass through avatars. This is noticable with the standard Linden popgun, although the behaviour is much less frequent.
  • Fire the library popgun on an avatar, frequent balls (Around 1 in 5) will pass directly through without colliding.

To fully demonstrate the problem:
1) Drag bullet object from popgun inventory onto floor.
2) Change the bullets prim type to a cube.
3) Set rotation to 0,0,0.
4) Change X length to 2.0 meters.
5) Pick up new larger bullet.
5) Delete bullet from popgun and replace with new lengthed bullet.
6) Increase SPEED variable in popguns script to 75.

  • The vast majority of bullets (9/10) will now pass through the avatar with no physical collision or scripted collision event.

Expected behaviour:
Bullets as small as minimum size with two meters in length would reliably collide with avatars at velocities of 100+. Even much larger bullets no longer collide accurately with avatars.

Impacts with prims are still physically reliable but suffer from greatly increased ricochet, almost as if triggering of collision events is delayed. I would file an issue on this but it is hard to provide example behaviour. Land collisions appear to be behaving normally.



 All   Comments   Change History      Sort Order: Ascending order - Click to sort in descending order
wildefire walcott added a comment - 08/Jun/08 10:39 PM
There are other symptoms of recent physics weirdness: You can see this in action on the steps of my house in Bay City: Docklands. Last night my partner and a friend and I were all standing on the front steps talking, when both of them aksed why the house was moving. I noticed what they were talking about soon enough; the picture was jittering around as if a distant herd of elephants was running past. At first I wondered if there were some errant rez faux scripts in the house, but I checked all the nearby linksets to find them empty. I then found that if I watched closely, the house was not moving relative to the sidewalk or adjacent buildings- it was I who was moving, and the camera was just following me, giving me the impression that everything else was moving. It felt like collision jitter, and this confused me because they were just regular-ass prim-per-stair steps. No sculpties, phantom or otherwise. SL was just having a hard time making our avatars stand motionless on the staircase.

Catten Carter added a comment - 09/Jun/08 01:30 AM - edited
I've noticed the same thing. It seems that collision is triggered with a sometimes huge delay. A surf track we created is fubar since the 1.22.2 rollout. Checkpoints sometimes does not register a collision, and ramps that adds a small speed boost, most of the time doens't activate, and when they do, it might give you a boost seconds after your actually over the ramp.

Please fix this sooner rather than later


Hg Beeks added a comment - 09/Jun/08 09:47 AM
I think the simplest thing I've found as an example is making a script that just applies an impulse to a basic <.5,.5,.5> sphere on collision. When you let the sphere bounce, it will usually go once or twice before simply falling and rolling off.

Jessicka Graves added a comment - 09/Jun/08 12:18 PM
It's been like this for a while, now it's to the point of unbearable of course, but there has been a problem with the collision system for months. =/

Jahar Aabye added a comment - 09/Jun/08 12:59 PM
I've noticed the rocochet effect from prims as well recently, but wasn't sure what was causing it. Bullets scripted with

collision_start(integer collide)
{

llDie();

}

should die on impact. Over the past few days I have seen them not only bounce back, but continue going for a distance of 10m or more. Ironically, I have had these bullets hit me and trigger collsion events on scripted items I was wearing. I have also seen them go through prims without either bouncing back or dying, giving me repeated messages that they had entered restricted parcels despite the fact that I was firing at a half-meter thick target sitting in front of another half-meter thick wall. These were 2.5m long spherical bullets travelling 115 m/s, which have never behaved like this under normal conditions.


Catten Carter added a comment - 09/Jun/08 01:11 PM - edited
Edited to 1.22.2 - It stated that it was unreleased, but I guess the Jira hasn' been updated

ScriptScavenger Lei added a comment - 09/Jun/08 01:11 PM
I think this has something to do with the Havoc 4 engine was aware there were going to be problems with the changeover. Won't be surprised if this is a side effect of trying to reduce griefing by making AV's harder to target.

Increased ricochet may be an isolated issue if I recall there's a way to adjust the ricochet in the physics engine.

Should be noted that "move to target" of a physical object still reliably produces collisions with AV's. Volume detect is also functioning fine.

Appreciate the heads up .Will be conducting some further tests myself shortly as I'm not sure if it's affecting "collision", "collision start" and "collision end" equally,scripting system may be registering things physics engine isn't processing properly . Not sure if ricochet is affecting all material types equally will have to look into this as well.


Catten Carter added a comment - 09/Jun/08 01:14 PM
What I'm seeing is a big delay before collision_start is triggered on object (vehicle) to object collision with volume detect. I still belive it's related since the same delay would cause bullets not to die on impact, and things passing through objects, because the engine doesn't react to a collison in time

Jahar Aabye added a comment - 09/Jun/08 07:58 PM
Just to clarify, these bullets behaved perfectly fine in Havok4 up until now, so I don't believe that it's a general H4 issue, but specific to this server release.

Catten Carter added a comment - 12/Jun/08 06:52 AM
Agree. I was using our track just before the restart, and when I tp'ed back after the restart, everything was broken

Daemona Razor added a comment - 12/Jun/08 01:41 PM - edited
A Related issue needs your vote!!!

There is a similar, if not identical, report open at

http://jira.secondlife.com/browse/VWR-7724

Though voting here is > 60 as 6/12/2008, the related/identical one has just 2. Vote there as well. It doubles the likelihood of them assigning this fun wrecker.


Andromeda Recreant added a comment - 12/Jun/08 01:43 PM - edited
This happens with all guns ingame. Every single gun i've fired at an avatar since 1.22.2 has had this problem. It is now present in 1.22.3. Needless to say, SL is acting up like a troubled child on crack

Suzanna Soyinka added a comment - 12/Jun/08 03:19 PM
Just getting in behind my peoples here, this is definitely an issue with all collision based weaponry in SL post 1.22.2.

Catten Carter added a comment - 12/Jun/08 03:33 PM
Yay - linden assigned

eianna cale added a comment - 12/Jun/08 10:32 PM
/me does the B-Boy stance behind Suz who is behind all the people that beat her to this issue. It is teh sux.

MarkusChristopher Llewellyn added a comment - 14/Jun/08 01:19 PM
We need resolve this as Suzanna has, as usual, clearly alerted us to this problem then it is a issue requiring some serious attention and priority.

Darkstar Heliosense added a comment - 18/Jun/08 06:07 AM
i have noticeds that its not just cubes. i use cylinders for my bullet objects. no matter what size i use or what speed i fire them at they all seem to pass through the object. however i have found a way around this but it its still a horrid pain to work with.

from what i have gathered there are 2 spots that you can infact trigger a collision with the agents.

just left of agent such at an elbow in the default stand animation where the hand is on the hip and just top-right of agent.

but yes i do agree we need to have this fixed. this surely breaks all projectile based products in SL.


Jahar Aabye added a comment - 18/Jun/08 02:08 PM
Yeah, it shouldn't just be those points that return collisions, there should (ie expected and observed behavior seen thus far in H1 and H4 up until now) be a rectangular "hitbox" roughly half a meter wide and as tall as the avatar's shape. This holds true (under past normal behavior) even if the avatar is in an animation. For instance, if an avatar were lying in a "prone" animation, a bullet will still result in a collision if you fire slightly below their nametag, which you will notice is still located where it would normally be.

Julia Faulkland added a comment - 22/Jun/08 08:07 AM - edited
The consistent delay on execution of collision events is just as serious as the fact that they pass right through objects. I don't know if these two issues are related but both are causing major issues for many products, especially weapons and shields.

An interesting thing to note is that bullets will hit avatars more consistently from some angles than from others, and hits are also more frequent if the avatars are moving.

Brief testing on the preview grid indicates that this issue is still present in 1.22.4.90215 as well.


Maggie Darwin added a comment - 26/Jun/08 03:03 PM
I can confirm the "aim to the left of the avi's right elbow and hit them" report.

Julia Faulkland added a comment - 03/Jul/08 06:02 PM
Still broken in server version 1.23 on the preview grid.

Maggie Darwin added a comment - 07/Jul/08 05:10 AM
It sure would be nice to know where this month-old critical pervasive high-impact physics bug stands without having to camp out in somebody's office hours. We do weapons QA, testing and evaluation, and it's nearly impossible to do any productive work until this is resolved.

Has the defect been identified?


Andrew Linden added a comment - 08/Jul/08 11:00 AM
I'm working on this bug this week (2008.07.07). I've already reverted a few things that I thought might be related and failed to find the problem. So now I'm regressing into older versions until I can narrow in on the culprit changes.

Maggie Darwin added a comment - 08/Jul/08 11:14 AM
Thanks, Andrew, I appreciate the update, and recognize that working inside Havok is non-trivial. It's good to know this is not languishing.

Kenn Nilsson added a comment - 08/Jul/08 12:35 PM
Not sure if it helps, but I've noticed that bullets with 'blood splatter triggers' on collision will often register a blood splatter 50-60 meters behind an avatar without having hit ground or another prim...in other words, the 'bullet prim' SEEMS to be recognizing that the collision-event happens, but very, very late. The avatar, however, still fails to recognize that a collision even occurred.

My testing in this regard is by no means scientific (was a result of 'try shooting me and seeing how bad things are still broken while on a completely empty sim), but I figured I'd throw in the info anyway.


Jahar Aabye added a comment - 09/Jul/08 05:15 AM - edited
hmmmm, that would be consistent with other behavior such as bullets bouncing off of objects or going through multiple objects without dying (despite being scripted solely to die on impact), in that it seems as though there is some sort of problem or delay in the triggering of the collision_start() event.

Is it possible that the reason that the collision is not being passed on to the avatar is due to the avatar's key not being passed in the physics engine's calculations (ie it belatedly sees that it hit something, but doesn't register a key for the collision)? Or could it even be the reverse, that the collision detection is just fine, but gets delayed because the database is taking too long to return the key of the collided object? I mention the second possibility only because it offers another possible place for the bug to be lurking that hasn't yet been explored.

I might try scripting a bullet to llOwnerSay() the name of the object it hits before dying, to see what comes up.


Julia Faulkland added a comment - 09/Jul/08 04:27 PM
Not sure if this is helpful, but it may be related. When running llGetAgentInfo, there is a significant delay on returning the CORRECT value for AGENT_IN_AIR. In other words, it returns FALSE shortly after jumping, being pushed upward, or otherwise leaving the ground, as if it hasn't yet realized you're in the air. Additionally, the "ground friction" seems to apply during this time, even while in the air. This is happening in a script which worked correctly some time before this bug appeared. If it is indeed related, it suggests this is deeper than just the collision event code and is a delay in other physics/script interaction too.

Maggie Darwin added a comment - 10/Jul/08 06:53 AM
Julia-- I wonder if that's related to the significant delay we're seeing in flying recognizing a change in command state...the reason that lately one tends to overshoot a desired altitude; for example you stop commanding "fly down" but continue to descend for somewhat less than a second, where before you would halt immediately.

I wish I knew more about the internals of this stuff, even at a at broad-brush level. It would really enhance my contribution on the "given enough eyeballs, all bugs are shallow" effort.


Andrew Linden added a comment - 10/Jul/08 01:09 PM
I have a fix for this bug in my code sandbox. Turns out there were two problems...

(1) Indeed some LSL collision events are not handled in a timely manner. I've tightened up the handling rate, particularly for the first collision event after a period of none.

(2) Some anti-avatar-entrapment code is preventing the bullets from colliding sometimes, especially when there is a somewhat rapid shot rate.

The flying/stop response time problems, and llGetAgentInfo delays are unrelated to the above bugs, but for all I know the llGetAgentInfo() could be similar to (1).


Catten Carter added a comment - 10/Jul/08 03:20 PM
Great news Andrew - Will be great when it hit the grid!

Julia Faulkland added a comment - 10/Jul/08 06:52 PM
Excellent news and thank you for narrowing this down so quickly.
From the sound of it these were changes made to fix much more minor issues with physics, and in the process killed physics entirely. Will your new fixes scrub all traces of this and put us back at the proper, expected behavior from before those changes went live? My only concern is that, like the original sound throttling code, we may see some unexpected behavior once this goes into heavy real-world use which would put us right back where we are now, if those older changes aren't fully undone. Well intentioned as these features are, I would just prefer not to see any experimental tinkering while we're having shotgun server code updates with < 5 day testing windows on the preview grid.

Andrew Linden added a comment - 11/Jul/08 09:56 AM
Julia, I think "killed physics entirely" is not an accurate assessment of the situation. I'm confused by your questions so I offer the following alternate questions/answers in an effort to provide some of the information that I think you seek:

(1) Q: Are you fixing bugs or adding features?
A: Fixing bugs. Changes will be made to pre-existing "features". In particular, when the LSL collision_start() and similar events are handled, and also how some anti-avatar-entrapment features kick in.

(2) Q: Will these changes be deployed too fast?
A: No. These will go into the normal testing and deploy pipeline. I wouldn't expect these changes to be deployed for about two weeks or more.

(3) Q: Are you worried that these changes are in the middle of fragile code and will introduce new unforseen and horrible bugs?
A: No. The collision_start() changes are very safe and correct. The other stuff is definitely an improvement and if anything will prove to not have been enough – I've fixed the popgun recipe above, however it may be that larger bursts of bullets may still have problems (that is, a gun that shoots more bullets per second for longer periods may still eventually stop colliding – the rate/duration would have to be accurate hits against the avatar on the order of 12 bullets/sec for 1.5 seconds).


Andrew Linden added a comment - 11/Jul/08 09:56 AM
I'm marking this as "Fix Pending".

Maggie Darwin added a comment - 11/Jul/08 01:36 PM
Congratulations on the fix, Andrew, and thanks. We'll beat it up when it gets to the beta grid.

steady enzo added a comment - 16/Jul/08 04:10 PM
I have experienced same issue with some monsters that use collision detection to accumalate damage.
I cAn stand there and get no damage from attacks? unless its a violant or hard collision. It makes my parcel less effective with damage all screwed up

Phantom Ninetails added a comment - 26/Jul/08 04:36 PM
As this issue is technically not about the script collision event but about collisions with avatars, I've created and linked an issue about the lag in the script collision event.

See SVC-2698


Kenn Nilsson added a comment - 05/Aug/08 10:34 AM
Do we have any update on when the fix for this issue will be applied? I know that the difficulties with implementing 1.23 have slowed down things a bit, but I have a long-list people begging me for an update of when to expect the fix...

Kristin Parness added a comment - 05/Aug/08 01:12 PM - edited
^^ I'm with Kenn on wondering when an update for the fix will be applied. Is this on the agenda for the 1.24 server release? I know it may not be on the top of the priority list due to the 1.24 server release utilizing the Mono VM, but it would be great if this is rolled out with the server upgrade if it has been thru the normal testing process without any noticeable issues. If the 1.24 server release is still slated for August 8th-10th, this would make a great addition! Any feedback would be highly appreciated Andrew. Thanks for addressing this promptly for all of us.

Andrew Linden added a comment - 05/Aug/08 09:13 PM
I'm hoping to get the fixes into the 1.25 release (the MONO project will be 1.24) however the relevant collection of fixes has failed the first QA pass (not this one, but two of the other bugs). I need to fix them and resubmit.

Andromeda Recreant added a comment - 05/Aug/08 10:35 PM
Ya know Andrew, this is a huge problem. This messes up combat in all combat sims, including safezone ones. All guns made to play in these types of RP situations are dead. There are hundreds of people missing out due to this.

Maggie Darwin added a comment - 06/Aug/08 05:35 AM
The in-world press is beginning to cover this issue.

http://foo.secondlifeherald.com/slh/2008/07/zomg-military-w.html

...is coverage in the style we've come to expect from SLHerald.

A more precise and helpful treatment is on page 100 of the August issue of SL Monthly Combat Report.

(free copies of all issues of SLMCR are available from a vendor at:

http://slurl.com/secondlife/Harrington/22/214/381


Andromeda Recreant added a comment - 06/Aug/08 07:16 AM - edited
^^^^^^^^^^^^^^^^^^^^^^^^^

You see, thats exactly what i hate about the lindens update process. They'll put something like "glowing objects" in immediately, OH WOW glowing objects! But they wont fix a major bug that negates all SL combat. You see, its not just passing through prims thats broken, its hitting the avatars too. Bullets pass right through avatars without triggering a collision, and even though this bug is marked CRITICAL, its going to take to 1.25 to go in? This is a fix, not a new feature, it should precede MONO!

As long as major fixes to the system are not given proper priority and put at the bottom of the list of things to do, more bugs will occur. I do not doubt that with releasing MONO, many more bugs will be created and this bug will be pushed back to the bottom of the pile again.


Maggie Darwin added a comment - 06/Aug/08 07:42 AM
I'm as passionate as anybody in wanting to see this issue resolved.

But I'm also sympathetic to the issues involved in managing a code base of this size and complexity. Mono should deliver huge benefits gridwide, and it's a much more impactful change. The patch for SVC-2511 is held back because it's in a group of fixes that just aren't ready to ship yet.

When software is this complex, it's necessary to work on new code in parallel, (whether you view it as implementing a "fix" or a "feature"). But while the coding work goes in parallel, you must resist the temptation to implement too much change at once, or when something breaks you won't know why. It turns out that at least part of SVC-2511 is due to changes made to fix avatar entrapment problems with the released version of H4. So it's not a result of H4, it's a result of a result of H4.

The automated QA/testing art for SL is still very, very far from where it needs to be. The testing that does get done is mostly "depending on the kindness of strangers"...people willing to donate their in-world time to getting on the beta grid and bug-hunt.

I have to agree, even though I'd really, really like to see SVC-2511 fixed, that if Mono is ready to go, it should go first. And you really, really don't want anything else to go at the same time Mono does. I suspect we're going to see two, maybe three point releases of the server implemented with rolling restarts before Mono settles down enough to start fixing physics again.


Kayla Stonecutter added a comment - 06/Aug/08 09:02 AM
You have to understand how development and bug fixing works in a large company like Linden Labs. They don't have just one or two people working on the code, they have perhaps dozens. Not every programmer knows how to work on everything inside SL. The people working on physics stuff may know little to nothing about MONO, while the people working to implement MONO may not know anything about physics. So you are going to get different projects working in parallel like Maggie mentioned. And thus some things may get done before others.

As for the glowing objects, that was implemented into First Look before there was a physics glitch. MONO implementation will most likely have a few issues and require a few patches to smooth out, considering the scope of the project. However like I said not everyone will be working on fixing that since not everyone will know how. Plus you get the "too many cooks in the kitchen" problem of making things worse if you have too many people trying to work on the same part of the code. MONO probably won't push back the physics fix much if at all, considering the problem seems to be rooted very deep in the physics system from what Andrew has said.

I run a safe zone combat sim with a system I made myself, believe me I want to see the physics glitch fixed as much as anyone. I won't however bitch and moan about how long it takes because I'm a programmer myself. I know what it's like to bug fix something, and can only imagine how much of a pain it is to work on code as complex as SL. Andrew is working on the problem and will get it out as soon as he can. As he's said it's not one but multiple bugs and that makes it just that much harder.


ScriptScavenger Lei added a comment - 06/Aug/08 10:06 AM - edited
I keep seeing the word "combat" and it's starting to get on my nerves. I have been making weapons since mid 2007 and not all combat systems use collisions. Furthermore this is not a combat system issue it's a worldwide malfunction of the physics caused by a modification made by linden labs to fix another problem." It turns out that at least part of SVC-2511 is due to changes made to fix avatar entrapment problems with the released version of H4. So it's not a result of H4, it's a result of a result of H4." I'm not sure how vital fixing avatar entrapment was. In fact I'm not even sure what avatar entrapment is. But I'm fairly certain it doesn't occur as often as collision events. This is a System Wide Problem and I hope it's being treated as such. I think by making comments related to combat we are minimizing the perceived impact.

I would make the suggestion that anyone running a sim ask their combat system provider to provide a temporary sensor and chat based system (ie. Safezone emulator).

There are many systems that use such a system for extended damage but are keeping the workings of the extended damage system proprietary to reduce "god mod" weapons from third parties. All of my extended damage weapons work fine even when the base impact damage fails.

I personally would like collision based systems to go away rezzing high volumes of even temporary prims creates a higher system load than chat and single sweep scanners.

I have been scripting for quite some time and have found there are many ways around "interface inconsistencies". We don't have to be patient and wait for a fix. If I waited for fixes I would never get anything done.


Kayla Stonecutter added a comment - 06/Aug/08 11:04 AM
ScriptScavenger Lei said:
"I keep seeing the word "combat" and it's starting to get on my nerves. I have been making weapons since mid 2007 and not all combat systems use collisions. Furthermore this is not a combat system issue it's a worldwide malfunction of the physics caused by a modification made by linden labs to fix another problem."
--------
Yes we know (at least I hope the others know) that it's not just combat that's affected. However combat is one that's affected hard by collisions not working right.

--------
ScriptScavenger Lei said:
"I would make the suggestion that anyone running a sim ask their combat system provider to provide a temporary sensor and chat based system (ie. Safezone emulator)."
--------
Sensor-based combat systems have plenty of their own issues for combat:

First off they have a 96m max range, since that's the farthest a sensor can reach.

Second there's the problem of the sensor arc. Too narrow an arc and you can miss avatars nearby since sensors must pick up the middle of the avatar. Widen the arc for closer range and then you have a very wide arc at a distance. Easy to pick up multiple avatars and have to work out which one should be hit, if any.

Third, you have the issue of things in the way. Sensors cannot determine things like path cut, hollow, taper, etc. So if you scan for objects in the way an open window built with a hollowed cube would stop the 'bullet'. Similar issue with linked sets, sensors can only determine the size of the entire set, not where individual prims are. And again the sensor has to pick up the middle of the object so you'd need a wide arc to find things nearby, and then you have the problem of the 16 object limit. Pick up lots of things nearby and you'll not detect things at a distance. Basically means cover is useless unlike with physical bullets.

Sensor-based combat systems have their place, but not as a replacement to standard combat using physical bullets, IMO.

In any case, I think we've hijacked this JIRA entry enough, should let Andrew get back to fixing the problem.


Un4given Spoonhammer added a comment - 15/Aug/08 05:50 AM
I noticed the same thing, I'm the creator of the Zombie Spawning Pool and Drift or Die, my zombies are suppose to die when hit by bullets or cars. But since the server update they hardly or don't die at all, this bug is a reall show stopper. I really hope it will be fixed soon.

Eata Kitty added a comment - 22/Aug/08 03:55 AM
I am impressed that Andrew Linden managed to get it fixed within a month and I think he did a good job there, thank you Andrew.

I am just disappointed that a bug I think a lot of people would consider to be fairly major for physics has not been deployed yet nearly three months after the issue was filed.


Catten Carter added a comment - 22/Aug/08 05:34 AM
I second that - It's cool to know it has been fixed, but 1.22 broke a lot of physics, and therefore a lot of products/games on SL. I would have thought it would take priority over Mono, which is a new feature, and not a content breaker like this bug

Un4given Spoonhammer added a comment - 22/Aug/08 05:42 AM
I agree with the above, most of my products aren't working properly because of this bug. I'm getting a bit tired of new features being introduced while bugs like these still exist.

Andrew Linden added a comment - 22/Aug/08 09:13 AM
I agree with the above. In fact, there has been much gnashing of teeth inside LL over the clogged pipeline that is our QA and deploy process. It isn't clear to me whether the process will remain slow or speed up – the current extraordinary backlog is affected by MONO which is a 2.5 year project and is a major overhaul of a lot of functionality and they are trying to deliver it with less pain and suffering than the Havok4 project. On the other hand, the whole update process is slowing down to try to reduce the introduction of new bugs so all changes must suffer more scrutiny and proceed at a slower pace than the past.

Current status of the bug fix is: passed QA, now in queue for merge back into the main codebase. Turns out there is a rather long queue because MONO has been blocking everything else.

@ Un4given, it is unclear to me that your zombies are actually affected by the fixing of this bug. I don't know how their death algorithm is implemented but this bug actually has two parts, one of which is very avatar-specific and will not apply to the zombies (assuming they are scripted objects):

(1) There is some special code that will disable avatar collisions when suffering repeated penetrations. The logic for this uses timing/history information to determine if the avatar is stuck in penetration over a period of time, and was not smart enough to distinguish a long stream of interpenetrations with new objects, and repeated penetrations with one. I fixed this, but it will not apply to normal objects.

(2) collision_start() events were not being called in a timely manner – also fixed. If the zombie deaths are affected by the proximity of the bullets when their collision_start() events are processed then the fix may actually repair your zombies.


Maggie Darwin added a comment - 22/Aug/08 04:37 PM
@Catten--it remains to be seen that Mono is not a content-breaker.

Catten Carter added a comment - 22/Aug/08 04:53 PM
Hehe - Agreed. It's not easy to see whats happening under the hood, but from my understanding, no existing content should break unless you recompile your old scripts to mono, and it turns out mono doens't react just as LSL did.

Maggie Darwin added a comment - 22/Aug/08 07:59 PM

Catten Carter added a comment - 23/Aug/08 02:26 AM
Odd - Guess there are some stuff that I don't understand about the LSL / Mono architure

Andromeda Recreant added a comment - 05/Sep/08 09:56 AM
Ok, Mono got released so its not blocking anything anymore. Why has this not been fixed yet? And WHY was llEmail fixed before it when its problem occured AFTER the problem described here? That is just rediculous. If theres a queue, everything should be done in order.

Kristin Parness added a comment - 05/Sep/08 11:07 AM - edited
I'm not tying to step on toes here Andro, but it's important to understand the pipeline. Mono has indeed released, but it is important to note that they are still investigating any and all bugs related to the 1.24 server update, hence why we are on 1.24.4 so far. The development team for the Mono rollout is most likely still investigating any issues as they arise. If you check the Status page on the SecondLife Grid, there is a tentative date of September 9-11 for a 1.24.5 server rollout. In addition, Prospero has also stated on the forums that they placed those tentative dates in case they choose to do an additional server update.

Please see http://wiki.secondlife.com/wiki/Release_Notes/Second_Life_Beta_Server/1.24 as it outlines the fixes in 1.24.4. The llEmail fix was "bumped to the front of the line" due to it being related to mono and necessary in a bugfix version of 1.24, hence it being included in 1.24.4. With a rollout on this scale, there will always be issues that need to be fixed ahead of any previous bugs. It also doesn't help matters that the QA and deploy process is rather clogged, per Andrew Linden.

It may seem ridiculous to those that don't understand how the pipeline actually works.

"Current status of the bug fix is: passed QA, now in queue for merge back into the main codebase. Turns out there is a rather long queue because MONO has been blocking everything else. " - Andrew Linden

The best thing to do right now is be patient. I know it's tough since the collision bug has needed to be fixed for some time now. Passing QA and being placed in queue is a good indication that we can expect it to be included in a server update sometime soon. I know it's probably not the answer you wanted to hear, but it's just the nature of the game of product development. Thanks and see you in SL!


Maggie Darwin added a comment - 05/Sep/08 11:55 AM
Mono has been "released', but not stabilized.
Of course, fair minds could say Havok 4 wasn't stabilized before Mono.

@catten: turns out SVC-2903 was bogus...the beta grid isn't the same as the live grid hardware wise, and this affects timing-sensitive testing.

Like physics engines, that sort of thing.

Of course, SVC-3001 is a hoot too...


Aquarius Paravane added a comment - 14/Sep/08 06:02 AM
I'm wondering if the following is related to this issue.

I have a script for a doormat; when an avatar steps on the doormat, it uses collision_start to open a set of doors using linked messages. It works fine when the linkset is no more than the doormat and the doors.

Now I link several sets of (doormat + doors) into a building and use linked message parameters to ensure the doormat only controls its own set of doors. (Thanking havok4 for how easy the linking is to do nowadays). However the doormat generally detects the first collision with the avatar and then not subsequent collisions; the collision_start event doesn't fire at all. I can stand on it indefinitely and nothing happens. If I open the doors by touch or leave them to time out and close, the doormat may randomly start working again. It's almost like firing other linked messages into the system gets the collision_start to wake up again. Replacing collision_start with collision makes no difference. Collision_end is fired immediately after collision_start while the avatar is still standing on the doormat.

It seems the collision_start is not working properly in a linkset and the "special code that will disable avatar collisions when suffering repeated penetrations" Andrew mentioned is not well tuned for this application. If it's relevant to the discussion, the root prim of the linkset is a huge prim, the floor of the build, but has no scripts.


Andrew Linden added a comment - 15/Sep/08 08:41 AM
@Aquarius. A standing avatar at rest may provide no collision events.

As an optimization feature the physics engine will "deactivate" an object that comes to rest and does not have other dynamic objects hitting it. "Deactivate" really means the physics engine puts the object onto an internal "inactive" list, whose objects are NOT processed during the integration step, and therefore do not change position or receive collision events callbacks (such as collision_start()). Various events will cause an inactive object to be put back onto the active list: hit by a dynamic object, pushed by an internal force, explicitly activated via some Havok API call, or any other object in its "simulation island" is activated (the entire pile of objects will activate at once).

When an avatar stands on a platform, both avatar and platform are in the same "simulation island". When the linked door changes position the platform will be activated, which activates the entire simulation island, including the avatar, which may cause a collision_event() to happen. If nothing else moves the whole pile will be detected to be at rest, and will be deactivated – this can happen within about 16 frames of the physics engine which typically steps at 45 frames a second.

The "special code that will disable avatar collisions when suffering repeated penetrations" I was talking about earlier is different:

(1) it only happens if the avatar is overlapping another object
(2) is applied on a pairwise basis (avatar vs penetrating object)
(3) is applied only when the avatar "active" (since otherwise it wouldn't be getting any collision events anyway)
(4) kicks in after about 1 second of overlap
(5) the pairwise collision disable is removed as soon as the avatar ceases to overlap with the object


Aquarius Paravane added a comment - 15/Sep/08 04:05 PM
@Andrew - thanks for the comments. For clarification when I said "detects the first collision with the avatar and then not subsequent collisions ... I can stand on it indefinitely..." I can see now that was ambiguous:
  • step on mat first time - collision_start then immediate collision_end then nothing
  • step off mat
  • step back on mat - no collision_start; waiting on the mat now makes no difference
  • touch a door and step off mat
  • step back on mat, now collision_start may or may not be detected again.

I tried this a bit more tonight, and the major factor is the thickness of the mat and whether it's linked with the floor.

  • If the mat thickness is an appreciable step up for the avatar (0.1m), collision_start is always detected when you step onto the mat, whether or not it's linked.
  • If it's thin (0.01m), or sunk into the floor to be nearly flush with the floor, AND linked to the floor, collision_start isn't detected when you step on .
  • If it's equally thin but NOT linked to the floor, then collision_start is always detected when you step on.

I would expect the collision detection to be equally reliable whether or not the prim is linked. This seems a bit different from the other reports under this issue, since no physical objects are involved. I can put it into a new issue if people prefer.

I also tried putting the script in the floor (the root prim), and interestingly the floor prim often reported multiple collision_start and collision_end with prims with various different link numbers other than 1, when I stood still on the floor, and didn't move. The linkset is over 180 prims.

Detecting collisions with the floor would be an alternative if the new llDetectedTouch* functions were replicated for detecting the position of a collision on the prim (e.g. llDetectedCollision*) - is that likely to happen?


Andrew Linden added a comment - 15/Sep/08 06:47 PM
Hrm... sounds like a bug, however I'm not too surprised there would be a bug here. The collision events are "filtered" such that per pair of objects only about 10 collisions per second are submitted for special handling for events (such as collision_start(), sound triggers, and such). A linked set is a single object, and it may be that the relevant sub-prim involved in the collision is not measured and determined to have interest in collision events until the colliion handle event itself – I'll have to double check the code tomorrow.

Meanwhile, the Havok physics engine has another optimization where it keeps "contact point" information over several frames (as opposed to deleting all contacts at the end of each frame and building a new set at the beginning of the next). Contact points are also maintained, even if two objects are very close but not quite touching, such that no collision forces are exchanged – the two objects have to separate at some threshold before the contact point is reaped. It may be that when the mat is very thin and close to the base level that the legacy contact point between the avatar and the base of the floor is somehow preventing the new contact point between the avatar and the mat from falling into the event handler.

In any case, I would guess that the bug is distinct from this one, and would deserve its own jira item. A simplified copy of the script for the mat would be a helpful comment to add to the new jira item.


Anthony Beatty added a comment - 23/Sep/08 04:53 PM
bump

wat is the status on a fix ? is this fix out on BETA grid yet?

just wondering its been over 4 months now thats all..... and imagine its going to be more months to


Maggie Darwin added a comment - 23/Sep/08 06:46 PM
I'd love to see both this and a fix for SVC-2723 get out there as soon as possble. But they're still rolling Mono fixes last I saw.

A status update on both would be nice. Pjira would be a better tool if the information flowed two ways.


Lucian Collas added a comment - 29/Sep/08 04:05 PM
Hi, this issue has been fucking up my company now for the past 4 months. When are you lot going to get your fingers out your asses an stop inventing the things (new problems) and focus on the issues that are causing GRID wide disruptions.

Jahar Aabye added a comment - 29/Sep/08 07:30 PM
Hey man, we're all frustrated about the physics issues and the time it's taking to fix them, and you're not the only one who's getting hurt by this.

Trust me, though, when I say that we will all be better off in the long run if Linden Labs takes the time to make sure that they fix this correctly. It is far more important to get this fixed correctly than to have it fixed quickly.

And please also use more appropriate language. This is a public issue tracker, not some internet bulletin board.


Maggie Darwin added a comment - 29/Sep/08 09:10 PM
Actually a fix has been available for months; but the implementation of the fix has been delayed behind fixing stuff Mono broke.

Myyst Jewell added a comment - 29/Sep/08 09:16 PM
sighs and counts the number of players leaving and complaining. If there is a fix it should be released, if they can't fix it or release it tell us. I have players screaming at me daily and my sim owners are losing money. I certainly understand about getting it done in one, but months on end is not acceptable, when most here are paying a huge amount of money, I am not going to pretend to understand the tech end of it, but the user end of this is really really bad business and bad customer service.

ScriptScavenger Lei added a comment - 01/Oct/08 08:19 AM - edited
theres a few rp sims where they actualy rp the fights...
but if you realy want to beat people down with huds and weapons
Theres this old opensource code called safezone.
it requires no collision and since its open source u can mod it and make your own hud.
You can probably also find somthing free like battlezone v1 or v2 still lying around somwhere but i wouldnt expect much in the way of support.

much like everything else in second life if u cant fix it just work around it.


Maggie Darwin added a comment - 01/Oct/08 08:31 AM
Object-avatar collisions need to work correctly. So does push in push-enabled land (see SVC-2723).

Havok 4 broke some things...and the fixes for the breakage broke some other things. But that doesn't mean they're not fixable...only that the fixes have been delayed so Mono could be implemented (...and fixed....and refixed).

Andrew has promised us eventual fixes for at least this problem. Just has to make it though QA...


Andrew Linden added a comment - 01/Oct/08 01:28 PM
This fix for this bug has passed QA, and has been merged back into the main code base, however what we're now waiting for is the 1.25-Server project to actually branch off. I spoke with some of the release team yesterday, and it turns out that 1.25-Server is now waiting for a list of other development projects that are done, but not yet passed QA or merged back. They hope to get 1.25-Server out before the end of this month (October 2008).

Maggie Darwin added a comment - 01/Oct/08 01:43 PM
Thank you for the update, Andrew!

We really do appreciate your efforts to keep us informed of your progress!


Andrew Linden added a comment - 28/Oct/08 02:30 PM
The theoretical fix for these bugs in in 1.25-Server which is currently deployed to the preview grid. Please use the preview grid to test your content to see if it really is fixed or not.

http://wiki.secondlife.com/wiki/Preview_Grid


Jahar Aabye added a comment - 28/Oct/08 03:53 PM
I'm standing in Sandbox - Weapons Testing with an M4A2 assault rifle and my much abused alt as a test subject, and things do appear to be working as they should now. Bullets die on impact with a 0.5m thick plywood wall, and bullets correctly collide with the avatar all over the hitbox, including the center of the av where they had previously been going straight through...and the particle spray confirms that the collision_start() messages are being triggered at the right times. The DCS combat meter also correctly scores the hits.

Obviously a sample size of n=1 is probably not sufficient to truly confirm that this fix is successful, but it looks good so far. Excellent job.


Maggie Darwin added a comment - 28/Oct/08 04:01 PM
Testing with several different guns on beta grid looks very, very good.

Siobhan McCallen added a comment - 28/Oct/08 04:06 PM
I also tested with both melee and bullets and found that RCS works fine with collisions to center-of-mass, and direct melee strikes now work properly. Before, only slashing attacks were certain to hit, a forward-arrow strike was unlikely to cause a proper collision.

I also tested in a 1.25 OpenSpace region (Kapor) on Beta Grid at Prospero's suggestion, and found that while there is still some hesitation from some weapons when firing, which I feel is probably a sign of the low script power in an OpenSpace Sim, the collisions worked fine.


Jahar Aabye added a comment - 28/Oct/08 04:10 PM
Yes, I've found that combat in openspace sims on the main grid (I know, I know, we're not supposed to do that), rate of fire is sometimes slower and there's sometimes a hesitation from some weapons when firing, so yes, that sounds like just normal openspace issues. I didn't think to test on an OpenSpace sim, good call.

I'm glad to see a whole range of weapons developers, who normally would be competitors, working together to solve a common problem. Like a big, happy, homicidal gun-crazed family.


Andrew Linden added a comment - 28/Oct/08 04:32 PM
Thanks for testing everyone, much appreciated.

Julia Faulkland added a comment - 01/Nov/08 04:03 PM
Results look good and combat may finally be stable again. Thanks Andrew.

As indicated earlier, avatars do seem to become intangible to bullets when taking consistent fully automatic fire for about a second, and this effect quickly wears off. I can't really predict if this is going to happen enough in practice to be an issue, but it seems to be working as intended. We'll see what people say when they get back to murdering each other.

Apart from weapons, all other collision-dependent products I tested (grapple hooks, shields, vehicles, etc.) seem much improved by this as well, and I don't want to jinx it, but I didn't fall through a basic 10 x 10 x 1 floor even once, which happens every 10-20 minutes on the main grid. So I would say huge success, and hopefully everything else in 1.25 works this well so we can see it on the main grid soon.


Andromeda Recreant added a comment - 11/Nov/08 09:48 AM - edited
So, whens this finally gonna be fixed? Its been 5 months and we had to WAIT for mono which was stupid.

Maggie Darwin added a comment - 11/Nov/08 12:28 PM
I'd venture to say that when we get server 1.25 this will be pretty much fixed. It looks fixed on the 1.25 preview.

And 1.25 is still scheduled to roll out next week...and you may be able to find pilot-rolled regions to test in on the main grid today.

I agree it was a bitch and a half to get stuck behind Mono. But honest, you couldn't do both at once.


Navar Harbinger added a comment - 02/Dec/08 06:27 AM
So i guess this means half a year fix pending now? I am not sure why people are so left alone with the issue for so long... the collision system used to be just fine. Now all that works fine are sim ground collisions. The phantom bug happens in almost all cases now, of course depending on the "projectile" speed and length of the object...however additionally it affects prim to prim collisions as well showing as collision event being triggered way to late. Maybe fix coming as christmas gift?

Maggie Darwin added a comment - 02/Dec/08 07:34 AM
Having tested it myself, I'm confident it's fixed in 1.25. But there's a lot of changes in 1.25 that were held up while Mono was rolled out and nailed down.

Honest, neither 1.25.0 not 1.25.1 could have been left in place; they had absolute showstopper bugs. But there's another 1.25 roll on the schedule, and if this one is a keeper I think your Christmas guns will work fine.

I'm looking forward to it too.


Oren Itano added a comment - 02/Dec/08 08:53 AM - edited
Whatever the excuses, whatever the promises, whatever other more pressing issues came up, the length of time it has taken LL to fix something they broke is freaking ridiculous. Come on, 6 months for something that used to work just fine? This is just another example about how LL doesn't care about the player's experience in SL, just about how many L$ are being purchased. I bet if we all got organized, those of us who play in sims where avatar collision is an integral part of the experience, and boycotted SL, or even just buying L$, they'd have this fix rolled out in less than a week. One of the integral problems with SL is the lack of accountability to there customers. I think they have made it obvious time and time again that they don't care what we think-- look at how they tried to mark this issue as fixed not too long ago and we had to raise cane to get them to understand it wasn't fixed for example. Come on LL, fix what you broke and give us a positive SL experience or just admit that you don't care what you customers want so we can have some closure and find some other online experience to fill the gap leaving you will create.

Maggie Darwin added a comment - 02/Dec/08 09:35 AM
Rushing out fixes causes breakage.

(In fact, if I'm not mistaken, this particular problem was caused by a somewhat too-aggressive fix to another bug, one introduced by Havok 4.)

I've followed the evolution of this bug, and the LL response. I've been frustrated by how long it's been broken, and annoyed by my perception that Mono was greenlighted before Havok 4 had truly stabilized.

But given the actual tactical situation from a software point of view, I can't say I'm overall dissatisfied with what Andrew's done for us on this one. Screaming and yelling about it at this point isn't going to cause 1.25 to roll out any faster.


Maggie Darwin added a comment - 02/Dec/08 09:42 AM
Furthermore, this issue was never marked as fixed, that's just not true:

Change by Catten Carter - 09/Jun/08 01:10 PM
Affects Version/s 1.22.2 Server [ 10330 ] Affects Version/s 1.22.1 Server [ 10320 ]
Change by Alexa Linden - 12/Jun/08 02:00 PM
Assignee: WorkingOnIt Linden [ WorkingOnIt Linden ]
Linden Lab Issue ID DEV-16593
Change by Andrew Linden - 12/Jun/08 04:55 PM
Assignee: WorkingOnIt Linden [ WorkingOnIt Linden ] Andrew Linden [ Andrew Linden ]
Change by Andrew Linden - 12/Jun/08 04:58 PM
Status Open [ 1 ] In Progress [ 3 ]
Change by Maggie Darwin - 26/Jun/08 03:03 PM
Summary Serious Object-Avatar collision problems in 1.22.2
Serious Object-Avatar collision problems in 1.22.2 and above
Affects Version/s 1.22.3 Server [ 10331 ]
Affects Version/s 1.22.2 Server [ 10330 ]
Change by Andrew Linden - 11/Jul/08 09:56 AM
Status In Progress [ 3 ] Fix Pending [ 10001 ]
Linden Lab Internal Branch havok4/havok4-8
Change by Maggie Darwin - 05/Aug/08 11:36 AM
Affects Version/s 1.23.4 Server [ 10343 ] Affects Version/s 1.22.3 Server [ 10331 ]


Oren Itano added a comment - 02/Dec/08 09:47 AM
I didn't scream or yell, I presented my opinion Maggie. And I think 6 months is hardly rushing out anything. SL was launched in June of 2003, so let's put this is perspective, this issue has been ongoing for 10% of SL's lifespan. I find this unacceptable and feel entitled to express my opinion on it.

Jahar Aabye added a comment - 02/Dec/08 09:51 AM
In addition to testing 1.25 on the beta grid, I'm an admin at a combat sim that was part of the pilot roll for 1.25.1 (we had specifically requested to take part in the pilot roll because our system utilizes physical bullets and HUD-worn meters that use collision_start(), so we're a prime testing ground for this bug).

During the few hours that 1.25.1 was on the grid, the fix for this particular bug worked flawlessly. It was WONDERFUL. Our meters registered hits perfectly, it no longer took 5 million shots to "kill" someone (granted, it was also a lot easier to be "killed"). I spoke with Andrew and Simon about the rollback and from what they've said about the bugs that forced it to be rolled back to 1.24, they were definintely showstoppers on the level of "oh dear lord, we need to roll this back as soon as possible." As much as I really want to see 1.25 out there, since it directly affects my business, I fully understand why it was rolled back and I'd have made the same decision if I were in their shoes. Fortunately, the bugs that necessitated the rollback were unrelated to this particular issue, and so once they're straightened out, 1.25.2 or 1.25.3 or whatever iteration is finalized ought to fix this.

As for people who think they can do a better job....well hell, send in your resume to Linden Labs, I'm sure they'd be happy to hire you if you're really as good as you seem to think.


Oren Itano added a comment - 02/Dec/08 10:10 AM - edited
"As for people who think they can do a better job....well hell, send in your resume to Linden Labs, I'm sure they'd be happy to hire you if you're really as good as you seem to think."

I never said I could do it better, I lodged a complaint, much like I would if my mechanic had not been able to fix my car for 6 months. I can't change a transmission, but I can certainly be unhappy if my mechanic can't get it right. I can't fix SL, but LL provides a service that we pay for and I have every right to state my opinion, whether i personally have the ability to fix the problem or not. Isn't this what our money is supposed to go to? Paying the salaries of the people at LL who are supposed to keep SL in working order?


Maggie Darwin added a comment - 02/Dec/08 10:29 AM
SVC-2511 went from reported to fix-pending in about a month.

The rest of the time was spent stabilizing Mono. (I would have preferred to see it Havok 4 fixed first, but it wasn't my call.) Changing the scripting engine while tinkering with the physics engine is a recipe for disaster, and that's what I meant by "rushing out a fix". This fix would probably be on them main grid already but for another bug that would have allowed anybody to have sex with anyone they chose in-world without permission.

Clearly A Bad Thing.

Oren, if you have no technical contribution to make and wish only to complain, I'm sure you can find a place to do so on the Forums.


Andrew Linden added a comment - 02/Dec/08 10:58 AM
This is an ok place to complain – go ahead. If the complaints are on-topic and remotely constructive/informative then I think they are valid. In this particular case I think the complaints meet those requirements.

Your money is not for paying LL salaries – your money is for the service, and LL must spend its profits to help it provide the service customers are willing to pay for. Some complaints help guide LL as it struggles to walk the correct path to success. When the service fails your requirements you are left with a few modes of feedback. Some that come to mind are: discussion, complaint, and change of service.

For the record: yes this bug was fixed much faster than it has been deployed (er... not yet deployed). Mono blocked this bug fix (and many others) for good reasons. There is risk that combining changes B with changes A... if B carries blocker bugs then A will unnecessarily be delayed. This is why the Mono project didn't want to pick up any unknowns from the various existing maintenance fixes and feature branches that were piling up behind it. The pile that became server-1.25 is suffering from this scenario. There are a lot of necessary infrastructure changes in 1.25, most of it related to 64bit server support and changes necessary for scaling the system – a few of the changes introduced blocker bugs, the rest of the changes are blocked by the problems. Unfortunately there are tradeoffs between deploying small groups of changes frequently and large groups of changes less often – LL is trying to balance those tradeoffs.


Oren Itano added a comment - 02/Dec/08 12:30 PM
Thank you Andrew for taking the time to break that down for someone less than technical like me. I will sit on my hands and attempt to not type anymore rants as I eagerly await deployment. I would not be so passionate in my complaint if I did not enjoy SL so much. At the risk of yet another bad metaphor-- while I like pie, I like a fresh pie made with quality ingredients from a good recipe, not dog feces scooped into a store bought pie crust. I feel like that is what we have right now, but your comments make it obvious to me you all have obtained a good recipe and ingredients and have a new pie in the oven for us all. I'll try to be patient. Can I get it a la mode?

Jahar Aabye added a comment - 02/Dec/08 12:58 PM
hehehe, in this case he's just making sure it's cooked thoroughly so we don't all get a really bad case of food poisoning.

/me remembers the 1.23 rollout kerfluffle and shudders.


Escort DeFarge added a comment - 30/Dec/08 01:17 AM
Thanks for the attention to this issue – but please note that this has really been going on since Feb. (SVC-1705) when the first easy repro was provided. I think another real issue with this is that everything keeps changing owing to all the side-effects. So a fix for one in-world effect surfaces as a breakage of another. Have Havok been asked for input on this?

Acadia Miami added a comment - 03/Jan/09 07:48 AM
k so this was working nicley on the beta grid, you did a few pilot regions on the main grid which introduced a new bug

so wat now? is it going to get pushed back in the pipeline for another 6 months?

An update on the progress of this would be apretiated thanks

/me looks at andrew


Maggie Darwin added a comment - 03/Jan/09 09:04 AM
The fix is in 1.25.

Tentative pilot roll for 1.25.3 is 1/12, with full roll slated for 1/19-20.


Acadia Miami added a comment - 03/Jan/09 10:24 AM
thanks for the heads up

but im not gona get my hopes up lol ,just hope it goes well.


Andrew Linden added a comment - 23/Jan/09 05:41 PM
Fixed in server-1.25 which was finally completely deployed today.

Maggie Darwin added a comment - 23/Jan/09 05:44 PM
Hooray, and congratulations and thank you Andrew and all the other folks involved in getting this resolved!