is there:
1) corruption to any sort of 'object hook' tables in the servers?
2) overactive/buggy capping?

previous fixes like copying the object or taking/rerezzing it, do not seem to be working this time. sim resets may be the only fix.

\\\\\\\\\\\\\\\\\\\\\

UPDATE 080228

i can confirm that 'slave'/out-of-sim messages are not reaching home sim at least (1st stage not reached, remote-to-home)... ALSO can confirm that 2nd stage does not occur on further random occassion (home reads incoming message and deploys a responding reply; outgoing email does not make it to the remote vendor). ..... can objects mute objects?? (and if so, is this a hidden part of the cause??) .....

there is also extreme lag from sim-to-outofworld email, confirmed (i am a comcast/insight subscriber; a purchase made at 2:02pm only received messages first out-of-world contact at 2:25pm!! a look at long-headers confirms that the message managed only to be sent out-of-world at the later time. lindens can call for a copy request...)

okay, let's get the ducks in a row...

1) inworld/insim coms (i have 'slave' vendors within the same sim) - ALWAYS okay and on-time
2) sim slave to out-of-sim home/master: BAD (via inworld ownersays)
3) sim home/master to out-of-sim slave: BAD (via inworld ownersays)
4) sim to out-of-world: BAD (via vendor-to-email confirmations)

since all out-of-sim comms sim to the be the sticking point, i'd recommend looking there...

i have before twice fixed this problem by setting up new home-boxes and re-keying the rest of the network (prior to the past week's events; i am sure this would be an appropriate fix, however that is a long haul for something that should be bug-free, considering a previously normally-working network box now cannot reconnect... )

this seems to happen in a haphazard manner on either end, in both 'inworld networking' formats, xml or email, which makes me think there is a) either something wrong with caching/storing ongoing/established out-of-sim connections (in-sim though has worked completely and reliably though), b) overactive/buggy email/xml capping, or c) can objects 'mute' other objects, like avis?? is this a bug there??

'ANY CASE'... it's 'OUT-OF-SIM', somewhere, in the comms, and does not matter whether the server- or slave-side. BETWEEN-SIM/OUT-OF-SIM COMMS possibly corrupted... anything in message-preprocessing (affecting contents of the actual message)?? anything in caching (affecting the addressing of object comms)? anything in object-to-comm tables?? anything??

i have reprogrammed my own machines to now de/re-activate according to network conditions, and am rolling them out now...

Interesting enough it received mails originating from objects (requesters) inside the sim (but not from other sims)
After a couple of tests we figured it out and tried several things to fix it:

Basic checks:
UUID in requester was correct
Mails were not capped

Complete reset and recompile of the update server script ==> did NOT fix the problem
Changes to the script with test output ==> confirmed the problem
Initiated a sim restart ==> did NOT fix the problem
We made a copy of our update server ==> fixed the problem (after updating the UUID in the update requester to the new object UUID)

This kind of leaves the impression that the problem is related to the object (task).

however i still have one problem slavebox in a sim called 'Lag Free' which is being super-resistant - i have even laid out a new box, ran it with one of the other running vend's keys, then plugged in the reluctant network to again find it would not connect... to either the old or replacement key... this may clear up with another sim reset perhaps. (it's strange, it's like the sim has a grudge against running this one network - now it doesn't want to run either the new or old key - but must 'obviously' be something susceptible to the tabling of these things...)

well i'd only just finished my first round cleaning up all the 'spare network' stuff from this past week i'm gonna try one more home server replacement for this last box and see if it pulls it up this time. i'll check back about it...

– UPDATE just a few mins later... i return to Lag Free to find that the vendor box had finally started up with the new key! (so no need for the 3rd try maybe)... i've copied the box and am seeing if the old key will eventually punch thru too.

– UPDATE 2... well i don't think the old key is going to work, although 10-15 other locations have no prob with it. :\ i wouldn't call it 'case closed' as that could still be a lingering problem, but 'for now', it's more back to normal and the new key works... whew!

related or beyond the issue??: i'm usually assured that an email reaches me offworld if a vendor shuts down. if perhaps the slave vendor is not able to reach home via its initial key, and i do not receive a shutdown notification from the slave, then it perhaps is unable to email at all, seeming to have lost its hook to the sim's email handlers.

this has also happened to server/home machines too though; while - strangely - a master key can 'go bad' for a slave location (the only fix for a remote slave being a new master box) - that's the 'strangest' situation. otherwise if it's a completely 'local' breakdown of comms, that makes more sense...

UPDATE: thinking of which, THERE ARE CONFIRMABLE 'DEEP LAG BUBBLES' all the way down to MONEY 'TIME OUTS' STILL... i had a couple yesterday with two customers mid-afternoon (10/11 am sl time, and am having in excess of three minutes programmed time-out on my vendors today. (and i would not rely upon email 'shutdown' notifications of inworld objects at that point)... whether the lags are primarily responsible, or whether there is corruption in any sort of 'object comm tables', is difficult to say now. still... :\

<striketrhu>however, in one case, a drag-copy of the same vend created a startup. i am also seeing much 'wait-and-try-again' behavior with my vends today. (email-network based)</strikthru> RESOLVED... i was not getting a proper prefence-save of the notecard at the time for the new key! there are still underlying issues (re: one bad vendor earlier needing a copy to replace properly), and keeping an eye out...

Mail from a certain eMail address is polled every 5 seconds via llGetNextEmail("user@whatever.dom", ""); for new mails.
Scripts have been reset, are running and finally the object has been re-rezzed several times (UUID has of course been adapted to the new one). For verification 2 different eMail providers have been used to verify.

http://jira.secondlife.com/browse/SVC-23

":0 i'm having some outage probs too again with this morning's report; i am having to drag-copy remote vendors and starting the fresh ones to reach home again. so you may need to take/re-rez no-copy vends to get them operating again." -ny