And no this issue is not resolved. xmlrpc still has a 30 second to over a minute round trip time.

The issues does not appear to be occasional, its actually quite frequent.

Date: 9/9/07

I don't have hard numbers, but it looks like the error rate across the grid is optimistically somewhere around 5%.

In most cases our servers return "unable to connect to xmlrpc.secondlife.com:80 (Connection timed out)". The error happens when attempting to open a connection before any data is sent.

About 60% of my XML-RPC calls are failing in the end, although only 10 to 20% of this is due to the "Unable to connect to xmlrpc.secondlife.com:80 (Connection timed out)" error.

With this kind of success rate, XML-RPC is practically unusable.

expat reports error code 4
description: Empty document
line: 1
column: 1
byte index: 0
total bytes: 0

data beginning 0 before byte index: /1.1 200 OK

Please either correct these problems or give us a reliable alternative. As it stands right now, XML-RPC is for all intents and purposes a failed service.

Please Fix it

Regard

This is probably the highest failure rate of any system in Second Life, and some of your bigger business and most interactive, innovative products rely on XML-RPC.

I'm not suggesting necessarily that XMLRPC is the ultimate answer, but we really need to have some reliable and scalable mechanism of communicating from out-of-world to in-world. And since another alternative doesn't yet exist, XMLRPC is it.

Well I haven't tested it, so can't tell how well it works, but how about maintaining a kind of job queue for the object.
The object will be triggered by email for a job. And then it starts processing the jobs. Well the mail could be empty, just the mail needs to be sent to the box. A possible use case:

• Buy on website started
• website sends mail to the box containing the item which should be purchased
• box receives an email and requests the job queue from the website
• box delivers and tells the website that the job has been processed
• website removes the job from the queue
• box requests further jobs if there are

Additionally the box says hello to the website once a day and checks for updates, jobs etc

Regards,
Virtouse

http://wiki.secondlife.com/wiki/LlHTTPServer
https://wiki.secondlife.com/wiki/User:Zero_Linden/Office_Hours/Discussion#Incoming_HTTP
http://jira.secondlife.com/browse/SVC-913

The one thing I'd suggest differently, for security concerns, is to use a channel like XML-RPC rather than the object's UUID for the URL. llOpenHTTPServer() could get the object ready to receive, with an event http_server with types HTTP_CHANNEL and HTTP_INCOMING_DATA (or something to that effect) to identify whether this is the channel UUID or an incoming data transmission.

Regards,

-Flip

Melnik Balogh

Since we wasn't abled to rely on it in the past there I am still reserved about that topic. But I hope we can rely on it.

Regards,
Virtouse

Melnik

melnik

1:15
0:20
0:25
0:25
1:20
> 5:00 (failed or timed out)
0:15
0:25
> 3:00 (failed or timed out)
1:20

I can deal with under 30 seconds, but it very often takes over a minute or just never works at all.

Testing Region: Hawthorne

Test 1: 47 seconds
Test 2: 20 seconds
Test 3: 21 seconds
Test 4: FAIL (after 67 seconds; "failure to open socket to xmlrpc.secondlife.com:80" message reported in web browser)
Test 5: 19 seconds

Regards,
Virtouse

Anyway, my whole project, which with my luck started just as XMLRPC decided to s**t the bed, is hosed. I can't get more than one in 20 calls to my obects to not get timed out or just drift off towards the Planet Bork. I've asked around and people are getting wildly varying results as reported here in other posts. I had some scripted vendors that had to be taken offline, no biggie because I only had two which were sort of 'beta' anyway so that was not major - but when you are trying to test a program you are writing to acces objects in world (c++ in this case) when you are off-world its a maddening experience becuase you dont knkow if your code is crap if you can't test it properly.

Please fix this or tell us its not fixable so I can stop coding something that wont work anyway.

maybe this will help.... i hope....

Mel..

http://jira.secondlife.com/browse/SVC-913

It appears that the possible solution is to abandon the XML-RPC approach and provide a method where any prim can have a random URL assigned to it so the HTTP process could be made bi-directional.

To be clear these issues appear to be entirely a case of the server just being over loaded, I can't find any other bug especially no newly introduced ones. We attempted increasing the maximum connections (since it always sits at just under the max) and that just resulted in the rest of the server trying to melt, so we reset it back to the original limit. The second server should alleviate some pain but I have no doubts that it may not be long until we reach capacity on that server as well.

It, however goes against the grain having every object poll once every 6 seconds, 24/7 just to see if there is data to be retrieved. So far my host has not complained about the rate of MySQL queries but traffic will build with the project and I suspect I will run to trouble.

I'm no expert on the inner workings of these but would it be possible to have a function that establishes a sub domain URL for any prim? The root DNS would be a second life port that has a look up by UUID to send an HTTP request to the prim. I don't know if this makes sense but it would solve the problem without the XML-RPC headaches.

These kind of things is making me, and much others, to lose clients and money and it is, simply, not acceptable.

The solution is not to round or jump the wall (the "http" solutions posted before). Is as simple as if Linden pretends to maintain its business, it needs residents making business on the grid and this is not the way, having us waiting for a miracle (Torley, I know you're trying to help, but maybe someone needs a slap on the face to wake up and not only words).

In the last days I have done a couple of tests, not with my product, with a simple web page sending only 4 or 5 chars and locally, from my machine.

What I have detected is:

• From web pages hosted in USA the calls seems to work, slowly to connect, but working. The times to answer varies from 20 seconds to 1 minute.
• From web pages hosted in Europe (Spain, were I live) the time increases from 45 seconds to 2 minutes.
• Locally at home, from a test program, 3MB DSL connection... impossible, timeout always (I set timeouts from 5 to 10 minutes).

Greetingzzz

My business is being decimated with lots of customers screaming at me. I may suspend sales soon, with a corresponding reduction in tier. What seemed an intermittent problem is now a flood.

Also note that polling http is not merely a brutal abuse of bandwidth furthering lag, it is not even possible for certain types of communication.

Please please please solve this problem.

Fanciful talk of HTTP to a prim etc etc is lovely, but to take a system thats worked well and is current BBB (borked beyond belief) and tell us 'the servers overloaded, we'll add another, and it will be just as bad in a few weeks' is pathetic!!! I didnt think for a second that xmlrpc was bugged, it was just timing out repeatedly or sooo slloooww as to make it unusable. This indicates a huge overload or bottleneck.

Has anyone looked to see where all these calls are originating that are clogging the server all of a sudden? Is someone flooding them, to grief? How is it that in the space of two weeks the server gets THAT overloaded? If you're not sure adding another server will help, that means there will be a bigger flood of calls to the script. Can't you add two or three? Or even shudder four?!?!?

I'd be happy to build you a few servers if you need them. No really, nice ones too. Of course if there's ever a problem with them, you'll just have to post on my blog, or put in a "ticket" on my site, and wait for a reply. After all we have more important things to do at my company, like new wallpaper, installing the cappucino machine, and making sure the sky looks nice outside the window, but we'll get to it. Eventually. Maybe next week, at the earliest. If the plumbers don't come, that is.

Kelly, Torley, please... PLEASE.... do something!!!!!!

Previously I was using llHTTPRequest polling every 5 seconds to a queue on my webserver. If there was anything in the queue, it would get executed.

This ate a lot of unnecessary bandwidth on my webserver.

My new solution:

Inworld-Server-Object listens for email:

state_entry() { gServerKey = llGetKey(); llSetObjectDesc(gServerKey); llSetText(llGetObjectName() + "\n Key: "+ gServerKey,<0,1,0>,1.0); llSetTimerEvent(2.5); // not a good idea to drop below 2.5 seconds, it's laggy enough as it is, and I'll already process the entire queue each event. }
timer() { llGetNextEmail("", ""); } // check for email with any subject/sender
email(string time, string address, string subj, string message, integer num_left) {
checkHTTPQueue();
if (num_left > 0) { llGetNextEmail("", ""); } // Continue processing queue if any remain.
}
link_message(integer from, integer num, string str, key id) {
if (str == "TIMER.HTTPQUEUE") { checkHTTPQueue(); }
}

Plus a second script:
default { state_entry() {llSetTimerEvent(1800);} timer() {llMessageLinked(LINK_SET, 0, "TIMER.HTTPQUEUE", NULL_KEY);}}

When it receives an email (Any email), it will go and retrieve & execute the queue over HTTP.
The second script forces a poll every 30 minutes just as a backup.
Now the load is only on the timer/getnextemail on the simulator.

When I have larger server farms, I use the second script, replacing llMessageLinked with llRegionSay/llShout depending on the farm's size, and adding a listen event to the servers so only a single 30m timer script is needed for the whole farm.

Currently, I'm using internal-only object-to-object emails which never actually hit postfix, and are normally "instant".

For external email servers sending mail in-world, can be subject to delays on ll's postfix server getting it's queues filled up, but whenever that happens, I just poke LL on the sldev mailing list to empty the queues.

Harold Brown had this to say the last time it came up on sldev: "For a nice history of the reliability of external -> internal E-Mail from May 2 -> current date, you can view my test log at this URL: http://www.rpgstats.com/SL/maillog.txt "

As of this comment posting (Nov-17-07 - 7:45PM PST), the average delay of external -> internal emails is roughly 4-8 seconds.

I should also note, Sean Linden proposed adding HTTP server support to LSL – PLEASE Vote for it here: https://jira.secondlife.com/browse/SVC-913 "Allow LSL scripts to act as HTTP servers in order to replace XMLRPC with something scalable"

The operation was succesful, but the patient is dead.

Polling with llHTTPRequest() is an impossible solution, as I'm using PHP which is incapable of proper asynchronous. Trying to offload asynchronous behaviour to vendors is a nightmare solution.

We NEED that new machine soon, and need to know an alternative is in development that will prevent this just happening again when the new machine hits capacity as well.

Create multiple communication objects in multiple sims which are polling a notification queue on the webserver. (Multiple Objects in multiple regions to increase that the objects are polling the webserver at all.

If the notification queue has an entry the communication object sends an email to the object which needs to be notified. Including a sequence id so that the entry can be removed at the webservers queue.
When the object receives an email it checks if it the notification id is the same as the last one if not it will contact the server
and asks for a job and it passes it sequence id, if it was triggered by an email so the server knows which job can be removed from the notification queue.

NOTE: The jobs are not passed via email. The jobs are passed from the webserver directly to avoid
that someone is sending you faked emails.

Well however this architekture should work pretty stable for the moment and is at least more reliable than XMLRPC.
Plus the server is not bombed with requests from thousands objects.

Just a few communication objects are polling the webserver like every 20 seconds for the next notification 10 entries.

Regards,
Virtouse

"(Multiple Objects in multiple regions to increase that the objects are polling the webserver at all."

=>

"(Multiple Objects in multiple regions to increase the redundancy so that the objects are polling the webserver at all.)"

Let's not distract from what is important: fixing the bug. I hope one server is enough, add two or three or more if needed, please.

CONCRETE USEFUL SUGGESTION: Although there should be no time limiter (keep it RPC), perhaps LL can investigate if parties are abusing the system with spamming objects, bots etc. If one object is sending hundreds of these per hour, maybe cut that object off!

• Polling! If you have multiple objects per region polling your servers then your poor server has to constantly accept and close new connections, not to mention process them, probably with some database look-up. This is incredibly costly and would increase the work the server has to do by many orders of magnitude. Instead of simply receive maybe 100 sales requests a day, and routing these through to delivery boxes, it would have to do the 100 requests a day, plus process THOUSANDS of polling requests just looking for something to do.
• Asynchronous. In my case I'm using PHP as a back-end, what if my script requires some data in the XML-RPC reply? PHP doesn't do proper asynchronous behaviour, meaning I would have to offload all of that to the vendors, meaning THEY would have to poll for data as well, whenever they're waiting for a reply!

A working XML-RPC, or an incoming HTTP service is NECESSARY to making these applications behave correctly, and with a reasonable degree of performance.

I began doing some one-off data cleaning which would run over a couple of months, but have turned that off temporarily after reading of the problems that real businesses are already experiencing with XML-RPC, so as not to impact them (I hate it when my SLX purchases arent delivered - it will need to be turned back on eventually as my application gets closer to completion. The application itself will also rely on this, with objects scattered all over the world, once its completed.

How permanent is this solution? Is it just an increase in capacity that will gradually fill again and degrade performance in the future? And if so, do you plan to add capacity as necessary to maintain performance?

I had to reprogram my application to HTTP polling and it is working reasonably well. Switching back to XML-RPC is a big task so I would like some indication of reliability before commencing. Application performance and bandwidth load on the web site would be considerably improved but I only want to do it once.

I will be looking into the spikes, and we have some plans to further reduce load in the hopes that we can extend the time until this becomes a problem again. We will also be working on monitoring tools for this service.

I am closing this as fixed.

If you are still having problems let me know - kelly@lindenlab.com - before re-opening this. I am going to guess that any problems still happening are a different issue and not this same umbrella issue. We can discuss whether to make a new jira etc.

If you are reading this down the line and think you should re-open it because things have gone to hell again please don't! Instead please create a NEW issue. I ask this because it will be much easier to track and diagnose a new issue (or even a recurrence of this one) if it isn't tied to the large history of votes, watchers and comments that this issue now has.

The primary goal here is to give us enough head room to implement, test, deploy and iron out kinks of a replacement lsl feature for xmlrpc, plus a reasonable phase out period. At which point we would really love to officially phase out xmlrpc. Current theory for such a replacement is a super light weight http server inside lsl (SVC-913). I have a more thorough design for this feature that I hope to share this week, the general idea of which is in my comments in that jira.

Email has latency issues sometimes, and can be difficult to program a transport layer onto. Well, at the least it isn't a usual choice.

If you are communicating between objects llEmail is THE way to go. It isn't really email on the back end, and is much more reliable than any other option.

One fear I have is that residents will decide xml-rpc is 'fixed' and slam it, and break it again. Even with the round robin in place now I do not think we stand a chance of keeping up with the potential load of people believing it is a reliable stable system to use. If at all possible I still recommend llHTTPRequest at the top (the only option with no central server on our end!!) then llEmail second (more lightweight, scaling and routing email is a more known and solved problem) and lastly xmlrpc (a very hacky system with a round robin that I honestly don't know how high can scale, does incredibly bad and hacky things including DB writes on the central database with every connection etc).

Does this make any sense and is this what is intended for the HTTP solution?

I have tried xml-rpc again today, everything looked great..!! and fast (at my end anyway) so if it stay's this way im a happy camper..

thanks

I hope it stays this way. Thanks for your efforts!