Kelly, does the fact that this is listed as critical mean that it's a fairly high priority on LL's plate to get it implemented ASAP? and if so when might we see it at least in beta form?

I could see this being difficult because of the need to provision many SSL certificates, one for each region or sim. Perhaps LL could solve this by creating their own CA, and posting its CA certificate on the SL website accessible by HTTPS so we can be sure that we're getting an authentic certificate.

Then again, perhaps that's all a little too complicated

o urls will be lost if the object changes regions or the region restarts

this will create some problems as objects are not always aware of a region restart.

Lex: That is a good point. I actually think the cap server only listens on https, but I will look into that closer. My guess right now is it will be either HTTPS only or both.

2 objects communicating via LSL http requests (assuming this is permitted: see http://wiki.secondlife.com/wiki/LSL_http_server).

rolling restart occurs, both objects now have new url's having to revert back to llemail or xmlrpc to communicate the new url's once generated?

the issue is ability to keep track of the url of the object without having to fall back on another communication method is point i'm trying to raise if that makes sense.

You are right, this design has the draw back that any sort of presence for objects has to be maintained by the LSL scripter. I am aware of that. I think it is a necessary evil to create the most scalable feature possible and I have tried to add the features to make it as easy as possible. This aspect is actually mirrored by XMLRPC which requires you to track a key that may change (although less frequently than http_server will).

We need a way to get the GET variables. Thanks for this.

The wiki page has some more details I think. But basically I don't think that will be possible at first. The caps system just doesn't allow for anything past the end of the cap URL (no /<uuid>/Foo). You will be able to do POST or PUT to pass in a body with the message (or even with a GET I think?), which the LSL can look at and send back different data for. Sure it isn't the easiest, but I think it is workable.

If in the future we are able to extend the cap system to allow tacking data to the end of urls then the LSL event will be able to handle that as extra information in the META variable of the http_server event.

The cap server is essentially an url redirect service. The data from the request at the cap url is sent to the internal url - including any data in POST or PUT (and I think even bodies of GET). Or is that more what you mean by "actually proxied"?

Redirects don't forward the POST body. In fact, if your browser POSTs a form and gets back a redirect, it GETs the url specified in the "Location:" header. This is probably a good idea so that POST bodies don't get spewed all over the place. It's fairly common (especially in forums and the like) to use a POST -> Redirect -> GET system for submitting forms. The user fills out the form (ie me placing this comment), the browser POSTS the form body, the server acts on the POST, and then the server redirects the client to use a GET request to see the results of their action. This means that the user ends up, for example, just looking at a JIRA entry, and if they hit "reload", their comment won't re-POST. Generally a good thing.

From the HTTP RFC:

10.3.3 302 Found

The requested resource resides temporarily under a different URI.
Since the redirection might be altered on occasion, the client SHOULD
continue to use the Request-URI for future requests. This response
is only cacheable if indicated by a Cache-Control or Expires header
field.

The temporary URI SHOULD be given by the Location field in the
response. Unless the request method was HEAD, the entity of the
response SHOULD contain a short hypertext note with a hyperlink to
the new URI(s).

If the 302 status code is received in response to a request other
than GET or HEAD, the user agent MUST NOT automatically redirect the
request unless it can be confirmed by the user, since this might
change the conditions under which the request was issued.

Note: RFC 1945 and RFC 2068 specify that the client is not allowed
to change the method on the redirected request. However, most
existing user agent implementations treat 302 as if it were a 303
response, performing a GET on the Location field-value regardless
of the original request method. The status codes 303 and 307 have
been added for servers that wish to make unambiguously clear which
kind of reaction is expected of the client.

Interestingly enough, the common convention of automatically following a redirect resulting from a POST violates the spec... but it's generally considered a useful way to go about things, and EVERY browser does it. "303 See Other" basically does exactly what I described above, and the spec specifically says that the target URL SHOULD be retrieved with a GET.

All in all, HTTP redirects simply don't carry a POST body onward, so any data would need to be copied by the server from the original request's query string into the redirect-target's query string. So that means that something like I described above wouldn't be possible without GET argument processing. Not the end of the world, but something to consider.

I'd go as far as saying that's THE core idea of REST.

Unless I'm misunderstanding this situation badly, this whole thing goes strongly against the grain of REST the way it's designed now, forcing actual resource information into the body of the request.

I'm not saying this wouldn't be extremely useful even if we don't have a GET sort of functionality, but there might be a deeper issue here that needs to be resolved.

Also, an ETA, if you would be so kind?

• Cory

No ETA right now. This is not in active development.

http://wiki.secondlife.com/wiki/Http_server

• Feature Request function set needs to be updated to reflect any new changes/info.

At the time of conception that was the set we (Sean & others...) had come up with initially. Nothing about them is set in stone & totally theoretical. The delay is just there for "shits n' giggles" basically.

Please feel free to update/delete/rename them, etc... A lot has changed since they were made originally.

I need to be finished <.<

https://sim123.agni.lindenlab.com/cap/f23b4b94-012d-44f2-bd0c-16c328321221?var1=true&var2=false&var3=foo&var4=bar

What method would I use to get varX in the LSL script from a GET request to the above?

Evens - Param name
Odds - Param value

[param1, value1, param2, value2, . . . paramN, valueN]

http://wiki.secondlife.com/wiki/Http_request

And yes, that is I think the biggest down side to the proposed design here.

https://wiki.secondlife.com/wiki/LSL_http_server

There is a teaser image of a hello world server available also:
http://wiki.secondlife.com/wiki/Image:Lsl_http_server.JPG

I don't currently have a time frame other than Soon(tm).

Feedback is welcome, but please keep comments on the discussion page of the wiki, the way I am updating the design page (to reflect a changing internal design page) means edits there are likely to be lost. Final word of warning: while this is in active development, the design is being adjusted regularly to reflect actual implementation when needed.

> * system_changed(integer change) (possibly a subset of existing changed event)

Maybe this should be a separate event. Having a changed() event means that scripts get events for ANY change and have to filter. That could annoyingly clog an event queue when all you want to do is keep your http server open. It could increase simulator load.

The other concerns I have around the issue are:

• Events are a relatively scarce resource while trying to maintain LSL as a language. This should indicate we should use changed() if possible.
• Some (IMHO poorly written) scripts only ever expect specific changes and do not verify the changed event is for the event they expect. I've seen problems similar to this when adding a flag to the llGetAgentInfo results that caused some AO scripts trouble. For example because there were so few returned flags you rarely got more than 1 and a straight == comparison (usually) worked, when more were added that was no longer necessarily true. This is what makes it tricky and what really needs to be thought about and investigated - and may mean a new event is needed.

I wonder what other CHANGE_ flags the SYSTEM_CAPS_RESTART and SYSTEM_REGION_RESTART flags would be likely to occur in combination with?

Q: How are DoS attacks against sims handled?
While an attacker can't guess the public URL since it contains a random UUID part (the cap key), wouldn't they still be able to bring a sim to is knees by just flooding HTTP requests with random (non-cap) requests? I realize the requests will be rejected by the server as invalid, but it still have to respond to them all and that'll cause legitimate incoming requests to be crowded out, no?
It there any sort of source-IP address based throttling of the incoming requests handled upstream from the sim in the network path? That might not stop attacks that use bot nets or IP spoofing, but it'll stop the naive attacker who aims to bring the sim of someone they don't like down using a quick shell script or whatever (which it likely to be the most common case in my estimation).
-Cenji.

Things I Like about the current design:

• 'x-forwarded-for' and 'x-forwarded-for-port': for the ip/port of the requester.
• 'x-trusted-path': A string specified by the user at url request
• 'x-untrusted-argument': A header appended by the cap server containing any path and/or arguments appended to the cap

I'm glad you took my request for arguments into consideration, and I heartily approve the header name – "x-untrusted-argument" is a perfect description of what I was looking for.
The trusted path is also good, as I'm assuming if one of my scripts obtains two public URLs, that will be the metadata I will have to check to differentiate between the two public URLs.

• This is a first use for a more general Limited Script Resource system that should eventually also handle script memory and cpu cycles.
• Not all requests for an url will succeed, the scripter is expected to handle the failure case.
• The number of available urls will be based on the amount of land owned in the region
• integer llGetFreePublicURLs() returns how many public URLs are available.

I like the fact that you guys are really thinking about the limits to put in place to make this a useful AND safe thing – otherwise we'll end up with "the bling effect" of some jerk putting ~200 particle bling scripts in his 3 prim sculptie necklace. Very good forethought there.

• Define response codes - no script/object found, throttled.

Yet more forethought into the response codes, I would propose that response code 410 "Gone" be used for "No Script/Object found".

410 Gone

The requested resource is no longer available at the server and no forwarding address is known. This condition is expected to be considered permanent. Clients with link editing capabilities SHOULD delete references to the Request-URI after user approval. If the server does not know, or has no facility to determine, whether or not the condition is permanent, the status code 404 (Not Found) SHOULD be used instead. This response is cacheable unless indicated otherwise.

The 410 response is primarily intended to assist the task of web maintenance by notifying the recipient that the resource is intentionally unavailable and that the server owners desire that remote links to that resource be removed. Such an event is common for limited-time, promotional services and for resources belonging to individuals no longer working at the server's site. It is not necessary to mark all permanently unavailable resources as "gone" or to keep the mark for any length of time – that is left to the discretion of the server owner.

301 "Moved Permanently" should be used for an object that has recently changed regions, if the regions are still in communication with eachother (assuming they are adjacent) a 301 should be returned to the new CAPS URL if known.

Throttled requests should result in a 503 "Service Unavailable" with a proper Retry-After header.

503 Service Unavailable

The server is currently unable to handle the request due to a temporary overloading or maintenance of the server. The implication is that this is a temporary condition which will be alleviated after some delay. If known, the length of the delay MAY be indicated in a Retry-After header. If no Retry-After is given, the client SHOULD handle the response as it would for a 500 response.

Note: The existence of the 503 status code does not imply that a
server must use it when becoming overloaded. Some servers may wish
to simply refuse the connection.

• Define mime-handling

I believe that we should, like the llHTTPRequest() call, be very clear in our handling of bodies and mime-types.
In particular, accept only text/* mime types, and be sure to do proper charset handling and conversion into the
UTF-8 that LSL strings use. (The code should all be cullable from the llHTTPRequest() implementation.)
Zero Linden 14:37, 16 November 2007 (PST)

I support this, but we should have a mechanism to define RESPONSE types other than text/* – I regularly use ruby on rails and have ported a simple YAML implementation to LSL, and I would be quite happy to be able to respond with application/x-yaml instead of text/x-yaml.
Perhaps I might want to write a postscript generator to print from second life – I would need application/postscript to do that. Or perhaps to serve a link to a chat service by using application/x-chat or application/x-conference... Perhaps I might have an Eggdrop bot on IRC and wish to return it dynamic TCL scripts to execute with application/x-tcl, or return the current parcel music stream playlist as a PLS or M3U with audio/x-scpls or audio/playlist. Maybe even return javascript with application/x-javascript, that would certainly be useful.

• There will be a max number of 'in process' requests allowed per script (requests that have been made but not responded to or timed out yet), requests when past this limit will receive an appropriate http status code. This will be roughly on the order of maximum number of queued events in a script, give or take a factor of 2.
• We may wish to throttle at the CAP server level as well, if possible.

If my memory serves me correctly, the queue depth for script events is 63, is it not? Therefore we theoretically would be able to have up to 32 or 128 HTTP requests in progress? Certainly hope you guys are writing this is a multithreaded manner!

I need to update the design again, but I also need to update the internal one a bit (not as much as the external). The design is a little different, trusted path and untrusted path are gone (including from the llRequestPublicURL() call) replaced with 3 other headers. The full header list is now:

"x-remote-port", "x-remote-ip" as described above.
"user-agent" - as supplied by the requester, truncated to 255 bytes.
"x-public-url" - The base cap url, something like: http://sim1234.agni.lindenlab.com:12043/<uuid>
"x-path-info" - Any part of the path after the UUID in the url requested. Truncated to 255 bytes
"x-query-string" - Any query args supplied by the requester ( ?foo=bar ). Truncated to 255 bytes

I will have a look at those response codes. 301 will not be possible - we simply don't know where the object has gone and have no intention of tracking that.

LSL can only produce UTF-8 text. For now that is what we will be returning, and the content type will be fixed to "text/plain; charset=utf-8". At a later date it may be possible to implement setting headers including content type on responses and maybe even returning different response types.

For now we are going with the queue depth, 64, for number of pending http requests. However there is a relatively short time out as well - 25 seconds.

I'll try and get the wiki pages updated this week, but no guarantees.

Not sure I understand what possible use an LSL HTTP server would be if it can only return content MIME type text/plain, not even text/html?? The "H" in "http_server" stands for Hypertext after all. So if it can only return text/plain and not hypertext, why not just call the event "text_server" instead? Am I missing something?

I'd prefer if the script can set the content type to anything - there is no negative security implication of that I can think of, is there?
The first think I was hoping to implement was XML-RPC and JSON-RPC, which our scripts use for llHTTPRequest to our APIs already, but for incoming calls to objects. Unfortunately, that won't be possible without being able to set the response content-type to text/xml and application/json respectively. The restriction to UTF-8 seems reasonable (perhaps someone else has other ideas though).

If this feature is really going to be so crippled by only being able to return text/plain, I'd rather see work spent on something more worthwhile (I'll remove my vote, though I realize that'll make no difference).

I hope I've misunderstood or you'll reconsider. If so, it could be a very useful addition to the grid.

Thanks for keeping this issue up-to-date.

The greatest utility for this function is getting data into SL / LSL and out of SL / LSL when polling via llHTTPRequest would otherwise be required but impractical. Essentially when data needs to be transferred on the schedule of the external service as opposed to the LSL based service.

Being able to host user friendly or pretty web pages and interface seamlessly with various protocol systems are distant runners up in priority. Creating intermediate interfaces for these features are not that difficult, well within the realm of doable. Web sites at persistent URLs can retrieve data from objects and format it for viewing and interaction. In fact this is a much better scenario because it keeps the bulk of the load on the high performance and less restricted server and relies only on the low performance LSL server for the SL related data it needs. You may be surprised at how quickly you burn up 2k bytes when you start making pretty html pages. Redirecting services are going to be needed and popular anyway given the temporary nature of the urls.

Similarly it should be possible to create simple services that retrieve the text based data from LSL servers and format it as required to match desired protocols. There are probably business opportunities in providing these services, both for protocols and html, including the tracking of these very dynamic urls.

Our goal with this phase is to make the greatest amount of functionality available as quickly as possible. The more features we add the longer development and testing takes. No, the solutions above are not necessarily ideal and they do require extra work and extra servers and introduce latency.

There is also a social hacking concern around the ability to create pretty web pages on dynamic often changing urls. For example phishing systems could be created that linked to objects in world with the pretty mock bank site and disappeared entirely after used. This simply isn't possible if objects can only return text. I'm not saying this may not be addressable, only using to illustrate that there are security implications in these decisions.

That saddens me because "getting data into SL / LSL and out of SL / LSL" and "when data needs to be transferred on the schedule of the external service as opposed to the LSL based service" is exactly what we'd hoped to use this facility for (specifically, text) - just not with plain text, but "text/xml" and "application/json" (which is also text but uses "application" for historical reasons). It seems we'll be unable to do that using standard tools that expect those content types. If we can use it at all, it be via 'hacking' client libraries to deal with the non-standard response content-types.

If you're worried about the social engineering problem of people using it to serve HTML pages to standard web-browsers, why not just limit the response content-type to text/plain if the user-agent indicates a request from the top 3-4 most popular browsers? (e.g. IE/Mozilla-based/Webkit/Opera) - or even require a specific substring to be present in the user-agent client header (e.g. "SLUserClient"), or a special x-grid-user-agent header and just reject requests that don't contain it outright. Since, most HTTP-based protocol libraries allow setting of the user-agent and additional headers, but don't have any mechanism for dealing with incorrect/non-standard response content types.
That approach would still stymie any kind of phishing schemes.

Currently we used a mixture of llHTTPRequest polling (from tens of thousands of objects all over the grid), XMLRPC (since it does still work occasionally), email both ways (both inter-sim and external), IMs to bots and two-way region chat to between 'relay' devices which also use llHTTPRequest polling to our external service.

How difficult would it be to at least have some way to set text/xml instead of text/plain in the initial implementation? At least that would allow people implementing XMLRPC. It would even allow developers to re-use the code already developed for the ill-fated LL XMLRPC service. I can't think of any phishing schemes that would benefit from being able to serve text/xml.

Even if we can only serve text/plain, you can still pretty easily set up a system that will let you use your JSON clients or any other kind of client. It'd look something like this:

[LSL HTTP server] <===> [translation server]<===>JSON clients

The translation server would be a relatively thin server that would have these tasks:

1. Communicate with LSL scripts to keep track of their HTTP server URLs.
2. Accept requests from JSON clients, and in turn make text/plain requests to the LSL scripts. Translate the results into JSON (or whatever).

The overhead for such a translator will be minimal. You'll have to have something doing #1 anyway. The advantages are:

• Kelly says it's easier to only allow for limited responses.
• It's better to have something that we can work with now rather than wait for kelly to redesign and think through the implications of allowing anything but text/plain.
• 2kb isn't much room for XML anyway, so you might find yourself having to do this kind of thing regardless.
• This solution is far more flexible and allows you to support protocols that LSL wouldn't be capable of supporting (e.g. non-HTTP transports, things needing responses >2kb, etc)
• Your translation server can gracefully handle the case where the LSL script disappears. It can even do round-robin load balancing and whatnot.

Kelly,

I would like to point out one thing with respect to phishing. Mozilla browsers are good about paying attention to the Content-Type and trusting it regardless, but Microsoft IE sometimes tries to be "helpful". I know it was true in the past that if you pointed it to a url like "http://foo.com/bar.html", even if the server returned a Content-Type of "text/plain", IE would go ahead and render the HTML anyway. I think this was an ill-conceived attempt to save server admins trouble... but in practice it just made it so that server admins who only tested in IE could find that people using Mozilla browsers would get a page full of HTML. You might want to test recent versions of IE with this kind of thing, since your design allows scripts to pay attention to path info, which would allow them to produce URLs that "look" like they should be static HTML pages. Perhaps a "Content-Disposition: attachment" header might convince IE not to be "helpful"; it will cause firefox to always save the file rather than displaying it.

On the other hand, adding an additional 'translation' server tier means:

• greater complexity
• greater cost
• additional protocols & code to develop and maintain for the LSL<->translation-server interaction
• decreased reliability

Our using XMLRPC and JSONRPC calls from the grid implies XML and JSON in both directions and we are living within the current llHTTPRequest/LSL constrains and can live within the 2kb limit.
In our case, a translation service wouldn't perform 1 above as that is already done by our existing clients.

I'm not disagreeing with you Lex. I'm just voicing my wish to Kelly that in our particular case we'd greatly benefit from being able to serve responses with at least text/xml and if possible application/json content-types. I don't think our case is especially unique.

I've implemented a fully threaded HTTP server myself in the past. I can't speak for Kelly's implementation, but I find it difficult to believe that it would be such a big task to provide a mechanism to set the response content-type - even if it is 'filtered' by the implementation to be in the set ["text/plain", "text/xml"] (for example). Then the mechanism wouldn't need to be changed if a future implementation expands the allowed types.

So, it is a suggestion in the hope that when Kelly weights it with all his other considerations he decides the effort is worth it

I'll add the capability to the opensim code myself if necessary, since having the restriction only from the SL grid is better than having it from other grids as well.

If it is implemented with only text/plain, of course we'll utilize the facility. I'll probably just stop using the facilities of our framework for XMLRPC and JSONRPC and find an open-source implementation of them, then hack it to accept the non-standard response content-type and then have the maintenance headache of extra build dependencies and of re-applying my patches if I ever want to update the code.

Either way, it'll be interesting to see to what uses people but the facility, even if it is initially crippled.

LSL isn't so good at speaking XML, and it's going to be pretty painful to write a good, reliable, robust XML parsing routine. You've already done it, I know, but I bet it was a pain, wasn't it? LSL's so limited and difficult to maintain that I prefer to move as much code out of the LSL script and onto an external server as I can get away with. LSL really would rather just deal with simple text communication. On the other hand, your applications would really love to just do abstract XMLRPC, passing objects around or whatever the model is in your applications. To me this is just asking for a robust, reliable layer to act as a go-between, handling all the nitty-gritty communication with LSL and leaving you free to write nice-looking, easily-maintained RPC code. Furthermore, as Kelly points out, that translation layer need not be written by you; it could be an external service you subscribe to.

You're suggesting you'd go grab an XMLRPC/JSONRPC implementation and hack it to accept text/plain content. You said yourself: this is a maintainence headache waiting to happen. You're going to be subverting the code so that it's not necessarily working how the writers intended. You might introduce unexpected problems, affecting reliability. Upgradeability is out the window. Complexity goes up. This strikes me as a far less robust solution. Why do this when you can write a small, reliable, clean, simple little piece of code to act as a go-between?

Oh, and about the feasibility of adding other content types to the LSL HTTP server system: Of course if you just look at the technical aspects, it'd be a very simple thing to add different content types to an HTTP response. Just a few lines of code. But Kelly's not dealing with an HTTP server in isolation here. It's part of a big production system with lots of interconnecting parts, a QA department, design and implementation phases, etc. Making a change might well mean going back and modifying the design, figuring out the 5 or 6 places that the code needs to be changed, wondering if the changes will break the CAPS server and bring SL down, more QA cycles, more test cases, more thoughts about security, etc. I'm going to take Kelly's word for it, because I know that these things are much more complicated than they seem from the outside.

Each script can request as many urls as it needs. There is a limited amount of total urls available, but it isn't limited by prim or script. Also urls point to specific scripts - other scripts in the same prim will not receive the events.

So you could have 10 scripts in a single prim each with their own url. Or you could have 1 script with 10 urls.

https://wiki.secondlife.com/wiki/LSL_http_server

• "x-script-url": The base url, as originally recieved from llRequestPublicURL
• "x-path-info": Any trailing path information from the requested url
• "x-query-string": Any query arguments, the text past a ? in the url
• "x-forwarded-for": The host that made the request
• "x-forwarded-for-port": The port from the requester
• "user-agent": The user-agent header as reported by the requester
  And:
• llRequestPublicURL no longer takes any arguments.
• The changed_server event was changed to just a new flag CHANGED_REGION_RESTART which will be used with the existing changed function.

Also there is some discussion on the Talk page that might clear up (or cause more confusion) the issue of public urls as resources and resource pools: https://wiki.secondlife.com/wiki/Talk:LSL_http_server

https://wiki.secondlife.com/wiki/LSL_http_server/examples

I am talking about things like vending systems, where a vendor sends a HTTP request to a warehouse server object to deliver the product. This is a much neater method using using llEmail, which is currently the only way for two in-world objects to communicate if they are in different regions. Similar considerations apply to any external application which needs to know the URL of the in-world server

Is it possible to have a URL that uses the Region name, Object Key, and some user-defined given name, as follows:

http://myregion.regions.secondlife.com/object-key/myname

For static objects, this would persist as long as the object remains rezzed. Yes, maybe it involves an extra database lookup for each request, but surely this information could be cached somehow, because regions don't change sims THAT often.

My twopence from the UK

As documented on the wiki page (http://wiki.secondlife.com/wiki/LSL_http_server) all the ways that an url might be lost are detectable by a script, and while it may not be trivial to implement is surely is not that difficult to build a service that proxies requests to a known static name on to the dynamic script. This kind of system is used already for 'dynamic dns services' to match domain names to non-static IPs.

Yes it is a bit of a pain, but doing it this way makes this feature scalable (which our implementation of email is not). I also expect dynamic name services to pop up as a viable business opportunity, so after a short time it seems very unlikely that you would need to build your own custom solution unless you had some really specific needs.

This function is to replace inbound XML-RPC with something scalable. It does just that beautifully. It's designed to complement an external service, which you can easily poll with the latest http_server url through httprequest or email.

To provide persistent URLs would invite abuses of simulator resources.

I've already got plans to implement a web based dynamic name service for this anyways...either for my own use or possibly as a generally available service. I'm betting I'm not the only one.

The other advantage of the approach taken here is that we get a new CHANGED event when a region restarts - useful in many other ways!

What changes on region restart is the host and port number for the region, which are kind of essential for actually communicating with the object.

"Kelly Linden -

Yes, you are missing something.

Each script can request as many urls as it needs. There is a limited amount of total urls available, but it isn't limited by prim or script. Also urls point to specific scripts - other scripts in the same prim will not receive the events.

So you could have 10 scripts in a single prim each with their own url. Or you could have 1 script with 10 urls."

I've already apologized for entering the fray late in the day here, but it's often good to have outsiders come in and poke holes in something you have been working on for a long time. Sometimes you get so focused on something that you can't see the wood for the trees. I know - I've done it myself enough times. Please reconsider this, because I don't think I can find much use for this as it is, and I suspect a lot of other SL developers will feel the same.

1) You move to a different region, on a different host and port number.
2) The region restarts, and comes up on a different host with a different port number.
3) Your script gets reset. Since the URL request has to be performed again, the same URL can't be guaranteed.

for all of these reasons, and probably more, persistent URLs aren't possible. It's not that LL doesn't want to do it, more like the overhead to even try to attempt to have the urls remain persistent would be outrageous. I'd rather have something simple that works with a high amout of reliability (and so far so good.)

It's pretty trivial to handle non-persistent URLs. Get the new URL, phone home to your external server (who's address never changes), and tell it the new URL. If your application doesn't involve an external server, all the same tricks that worked for llEmail are still applicable.

And as Kelly correctly points out, Dynamic DNS is completely workable for HTTP-IN, there are numerous free services that provide this already, and I'm sure some which are SL specific may spring up.

Overall I find HTTP-IN to be fantastic and It is going to revolutionize many aspects of communication within SL and between SL and outside destinations. Many, many thanks for implementing this.

I'm glad people think this will revolutionize things in SL, but it seems to have an Achille's Heel. If we have to fall back on the things we're trying to avoid (e-mail and our own http servers) what's the point other than a cute, but incompletely thought out exercise?

As mentioned, 3rd-parties will provide such look-up services for sure. Of course, the less external dependencies one has the better - so I would love to see LL provide such a resolution service on top of it in time. LL obviously has the facilities as they've had to solve similar problems internally themselves.
Perhaps LL could provide a general service that could be used for following changing objects in general (and apply it to the current in-bound email system if that isn't currently scalable).

This functions EXCELLENTLY as a replacement to XML-RPC. It is both useful and trivial to get around the temporal nature of the URL. (especially with the functions provided). Programmers (and specifically scripters of LSL) have been working around this problem for years now. Frankly, if this isn't useful to you, then you aren't the type of developer this was intended for.

Input is of course appreciated - but attempting to derail the train in the eleventh hour will only succeed in delaying the feature for a lengthy period (you can't just bang out code, a major change like you and others are suggesting would most likely require a retooling of the entire feature, and then it would have to go through QA ALLL over again) Already this feature has been in development for (most likely) over a year. Worst case scenario they decide it's too much trouble and we get SQUAT.

So your opinions have been noted. After this feature has been released you can submit a ticket for a change or a new service.

I have plans to create a DNS type service and (as others have signalled) so do many others. Once several good DNS services are available, this will be something you never really have to think about.

Lamorna: Unfortunately scalability is a huge concern for us. We have a huge number of servers, a huge number of active scripts and a huge number of active users. I completely and fully understand the desire for persistent urls. However implementing persistent urls on our end requires pretty much the same thing we are asking users of this: a registration service. That registration service would require new hardware and additional cost. Also, if done by us at this stage (every url being persistent) it would have to be a registration of every url ever used, which increases the scope dramatically.

Doing it the way we are has many advantages:

• We get the feature out there quicker so our awesome content creators can start using it.
• We open a potential market for registration services, which can be simpler and quicker by only registering those urls that need to be registered and in specific cases can specifically match the needs of the content being created.
• At some later date there is still the possibility for LL to offer a persistent url feature, perhaps at some nominal L$ cost to offset the real world costs of maintaining that service. There is no delay or invalidation of work already done for this, it would just be a next step that requires everything already done.

It is also important to realize that this feature is not meant for user visible web pages, it is meant as a glue between webservices on the greater Internet and the scripts in Second Life. For a web service it is not that difficult to update the url it communicates with when needed.

Persistent urls are not something we just forgot or overlooked. We thought really hard and long about them and what is required to offer them at the scale we have now and the scale we expect to reach. We discussed the options with content creators at various venues and even sought out some of the highest users of xml-rpc and llemail for their input. The conclusion is that it is not free to offer persistent urls (but is essentially free to offer non-persistent ones), that the service required for that registration does not need to be built by us, that the service could in fact be built better by our residents for some uses and that we could always offer the service as a later feature if needed.

Your input is of course highly valued, as is everyone elses here. I agree the feature is less usable without persistent urls, but it is far from useless.

The present system has been working (nearly) flawlessly for a year so reliability would not increase but my website traffic volume and SL outgoing HTTP traffic would go down by just under 100%.

Not having kept up with the developments, I assume a log-in like that could easily keep the URL current?

http://wiki.secondlife.com/wiki/LSL_http_server/examples

I'll bury this deep in here, but I have re-written the landing page (http://wiki.secondlife.com/wiki/LSL_http_server) recently, and moved the design over to /design.

In the use case I had in mind, I have an in-world HTTP server and multiple clients. Either I use an external server to resolve where my in-world server is, or I use...duh... llEmail... to communicate with an intermediate in-world server whose address persists. But the latter is what I'm already doing. Therefore, I can't see the point of rewriting my application to use HTTP.

I'll bow out of this debate now, and I hope that those of you who are keen to have this feature find good uses for it. Personally, I find it a big disappointment as it is, and I hope that Linden Lab implement some kind of URL registration service in the future. This doesn't have to grow and grow – we could be told that URL's will time out if not renewed after some period of time, and it's a simple matter for a running script to have a timer to renew its URL(s) occasionally.

However, there's one small thing that I'd like to add, which is the limitation of data to 2kb. Kelly states that «You may be surprised at how quickly you burn up 2k bytes when you start making pretty html pages». Well - heck, yes!
It will be the same as with the current HTTPRequest. 2kb are just not enough.

I don't know about technical details, but I'm pretty sure that making subsequent HTTP-calls, just to get all the data we need, contributes more load to the server than being able to get all the data at once. Plus, I think it'll make some stuff more complicated than needed.

I have a service running right now, which pulls data from a webservice to track airplanes. The data contains 8 variables, including plane-type, carrier, scheduled start/landing type, origin/destination airport and some more. REALLY small pieces of data. But depending on the length of the carrier and the airport names, the data is being capped. So I have to split these variables into two and make multiple requests to the server, just to get all the data.

I'm pretty sure this will happen with HTTP-Server as well. So please, please, please - make the amount of data just a tiny little bit bigger (4k would be fine, 8k would be great!). And while you're at it, raise it on the HTTPRequest side as well

and (this being the idea i got now, that for some reason I thought would be similar tot he previously mentioned one) for pages hosted by scripts that the creator intends to be bigger than the maximum ammount that can be send at once, add one or two commands to create a page by appending a few 2KB chunks before sending to the client that requested the page

Haruki: The size limit is not quite etched in stone. We do need some limit, and we chose to match the limit of llHTTPRequest. I think this is a case where if we need to we can increase the allowed length later, but we would never be able to reduce it if needed. As such I'd prefer to err on the side of a smaller limit for now. Also, you can't make pretty html pages with this, or at least they don't display as pretty in most browsers.

The common expectation for what a restart does is that there is a short period of time between the stopping and the starting, but the way I'm reading this you intend to use the flag is to indicate when a region comes online. Additionally, the name of CHANGED_REGION_RESTART infers that it is warning of an impending restart and not the result of a restart.

I've made a separate feature suggestion for this that goes into greater depth: SVC-3254

(Basically work the design specification page into our normal style of documentation).

Essentually what I'm asking, is have the keywords been finalized?

I'm not trying to sound like a curmudgeon, but almost all of these types of questions are answered in the wiki. Please read there first before posting a comment as myself and all others with a watch on the issue get emailed every time you post a comment

...does vehicles eat up from the same 38 urls meant for attachments when they are using avatar resources?

and somthign I hadn't seen before, wouldn't it be better to inform the script it lost the url instead of returning the object?

• X-SecondLife-Shard
• X-SecondLife-Object-Name
• X-SecondLife-Object-Key
• X-SecondLife-Region
• X-SecondLife-Local-Position
• X-SecondLife-Local-Rotation
• X-SecondLife-Local-Velocity
• X-SecondLife-Owner-Name
• X-SecondLife-Owner-Key

(though if the full list wasn't to be provided, at least shard, object name and key, region, local position, and owner key would be very nice).

HTTP-In should hit Second Life with server version 1.27. We are currently on 1.25, and 1.26 became feature frozen before the final pieces of HTTP-In were in place. History we hope is a poor indicator of the actual time frame this implies. We are working to shorten the time between major versions, which is one of the reasons that 1.26 froze so early. Trust me I am as eager or more than anyone here to see this out the door.

We will keep the HTTP-In regions on aditi / beta up until 1.27 beta servers are deployed there.

Hewee: I can't remember if I made another jira for including the second life specific headers, but they will be available and should be on beta now.

To go around the text/plain limitation, simply write a firefox plugin that looks beyond text/plain for another header and use the second header instead if it is found - but only for second life type links.

Perhaps make a series of plugins, one per file type, so that users that trust *.BMP files from inworld servers aren't surprised by a scary *.EXE

Problem #2: dynamic URLs

If the object itself says its URL and asks the user to click it, it is good enough for inworld-export work like outputting a file for the user to download. I'd use it to export sculptie maps with my SculptCrafter tool, for example.

If the server is meant for offworld offline people, then have some bot login as an alt avatar. Have that bot stay just long enough to notice what the media URL is. Whenever the region restarts, your inworld server sets the media URL to its own URL so the bot has the info it needs always updated.

Neither solution is very pretty, but they don't need intermediate servers at all!

I think a better solution is for http_server to support more mimetypes =)

Regarding upcoming changes to the LSL scheduler and obj->obj comms.

I currently use llEmail extensively for Object to Object communications, and it works great for that purpose.
The trouble I have is the 20s delay for llEmail. I currently get around it (as I'm sure many do) with llMessageLinked to a set of scripts (32) which round robin the actual llEmail call.

I'm concerned that the large number of scripts (and related memory usage) will cause problems with the proposed scheduler changes and memory limits.

llRequestUrl et al looks promising, but there's the headache of managing presence. Also, I suspect this method may actually incur more overhead than llEmail for obj->obj.

What I'd REALLY like to see is just dropping the delay for llEmail in the case of obj->obj comms. If that's not possible, a specific set of zero/low delay functions for obj->obj comms would be nice.

Is such a thing possible? Is it on the roadmap?

--Doran Zemlja

http://sim3014.aditi.lindenlab.com:12046/cap/b180ede2-4850-c9d0-6d47-62f1e94fbefc

I need to compress these URLs.

Is it safe to assume they will always be
http://<regionhostname>:<port>/cap/<uuid> ?

Is it safe to assume the port will always be 12046?

Is it safe to assume the regionhostname will always end in .aditi.lindenlab.com or .agni.lindenlab.com ?

Doran Q#2:
Really, no. These work best if no assumptions are made on the format. They may change without notice. If you make such assumptions we may not feel sorry when they break. :/ We make absolutely no guarantees on the format of these urls.

The concept of a capability is that the url is a magic key, it is a whole entity, they are atomic in that the component parts don't have meaning or use. We could change the port or the host, the type of key used, add extra paths etc.

Essentially if you make your scripts work such that they just treat the url given as the base url they will be portable and not break. If you dissect it and assume some parts then it may break in the future and you aren't allowed to get upset at us when they do. Or if you do get upset I think we are allowed to say "told you so". Something like that. Sorry.

That said, please bear in mind that we're very conscious of memory limits, especially considering Babbage's proposed scheduler changes. Shorter URLs, better. Regular, predictable URLs, better.

Sure it is a contrived situation maybe, but perhaps common for some usage scenarios.

I am also discussing the idea of url-safe base64 encoded cap keys with an internal dev. This would in theory reduce the cap from 36 characters to 8. This would be essentially a presentation only thing: the same 128 bits that made the long cap could be encoded to the shorter base64 version to point at the same resource. I don't know if we will go anywhere with that, but it is an interesting idea.

http://wiki.secondlife.com/wiki/Key_Compression

Kelly,
Base64 would bring caps down to 24 or 22 chars. (22 will induce a small but manageable error) It's mathematically 16 bytes as an integer and any encoding is gonna inflate it about a third. Also, the standard base64 charset includes a slash which will not play well inside the base url. You would need a custom charset, perhaps hyphen and plus along with the 62 case-alpha-numeric. Even going from 36 to 22 would be nice but might be too much trouble?

Also, I've fought the +/ issue before, and settled on -_ ; both can url encode properly without escaping. (although the stock ll function escapes them anyway sigh )

You are right on the actual savings, dunno what math I was thinking. It would take 22 characters, really only saving 10. (I don't think 22 introduces any error: 128bits / 6bits-per-char = 21.33 characters) If we were caring that much we'd drop the dashes for a total savings of 14 characters. This does not seem worth the effort to me. Unfortunately I can't think of any more dense encoding that would be url safe.

If you provide us a LSL function for it we can either construct our own external function (or you can give us the pseudocode) and you wouldn't need to do any fancy foot work on the server side. The benefit of this is all the other applications we could use for key compression. Keys are the life blood of everything in LSL. I'd say 4/5 major scripting projects involve storing a fair number of keys (whether texture keys, av keys, object keys, etc)

Doing this in LSL is not very pretty or fast, here is a quick (rough) implementation that is not optimized for lsl:

string base64_set = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_";
string hex_set = "0123456789abcdef";

string hex_to_base64(key id)
{
    // Do this in 12 bits = 2 base64 and 3 hex
    string sid = llGetSubString((string)id,0,7) 
        + llGetSubString((string)id,9,12) 
        + llGetSubString((string)id,14,17) 
        + llGetSubString((string)id,19,22) 
        + llGetSubString((string)id,24,-1);

    integer i;
    string out;
    while (i < 32)
    {
        integer h1 = llSubStringIndex(hex_set,llGetSubString(sid,i,i));
        integer h2 = llSubStringIndex(hex_set,llGetSubString(sid,i+1,i+1));
        integer h3 = llSubStringIndex(hex_set,llGetSubString(sid,i+2,i+2));
        
        integer b1 = (h1 << 2) | ( (h2 & 12) >> 2);
        integer b2 = ( (h2 & 3) << 4) | h3;
        out += llGetSubString(base64_set,b1,b1) + llGetSubString(base64_set,b2,b2);
        i += 3;
    }
    
    return out;
}

key base64_to_hex(string sid)
{
    integer i;
    string out;
    while (i < 22)
    {
        integer b1 = llSubStringIndex(base64_set,llGetSubString(sid,i,i));
        integer b2 = llSubStringIndex(base64_set,llGetSubString(sid,i+1,i+1));
        
        integer h1 = ( (b1 & 60) >> 2);
        integer h3 = (b2 & 15);
        integer h2 = ((b1 & 3) << 2) | (b2 >> 4);
        
        out += llGetSubString(hex_set,h1,h1) + llGetSubString(hex_set,h2,h2) + llGetSubString(hex_set,h3,h3);
        
        i += 2;
    }
    
    // Convert to canonical UUID format.
    return (key)(llGetSubString(out,0,7) 
        + "-" + llGetSubString(out,8,11) 
        + "-" +llGetSubString(out,12,15) 
        + "-" + llGetSubString(out,16,19) 
        + "-" + llGetSubString(out,20,31));
}

default
{
    state_entry()
    {
        key id = llGetOwner();
        string b64 = hex_to_base64(id);
        key h = base64_to_hex(b64);
        llOwnerSay((string)id + " : " + b64 + " : " + (string)h);
        if (id == h) llOwnerSay("Success.");
        else llOwnerSay("Fail. " + (string)id + " != " + (string)h);
    }
}

I get:

[11:25]  Object: 0e346d8b-4433-4d66-a6b0-fd37083abc4c : DjRti0QzTWamsP03CDq8TA : 0e346d8b-4433-4d66-a6b0-fd37083abc4c
[11:25]  Object: Success.

ETA: Speaking of caffeine. Thanx for the multi-digit hex, alot nicer than shifting in one at a time (face-palm). Though I do still recommend working with 24bits at a time. It packs better, 22 vs 24 chars and is more "correct" if compatibility with another b64 routine ever comes into play.

string keyToBase64(key k){return
llDumpList2String(llParseString2List(llDumpList2String(llParseString2List(
llGetSubString(llIntegerToBase64((integer)("0x"+llGetSubString(k,0,5))<<8),0,3)+
llGetSubString(llIntegerToBase64((integer)("0x"+llGetSubString(k,6,7))<<24|(integer)("0x"+llGetSubString(k,9,12))<<8),0,3)+
llGetSubString(llIntegerToBase64((integer)("0x"+llGetSubString(k,14,17))<<16|(integer)("0x"+llGetSubString(k,19,20))<<8),0,3)+
llGetSubString(llIntegerToBase64((integer)("0x"+llGetSubString(k,21,22))<<24|(integer)("0x"+llGetSubString(k,24,27))<<8),0,3)+
llGetSubString(llIntegerToBase64((integer)("0x"+llGetSubString(k,28,33))<<8),0,3)+
llGetSubString(llIntegerToBase64((integer)("0x"+llGetSubString(k,34,35))<<24),0,1)
,["/"],[]),"-"),["+"],[]),"_");//DumpParse
}//keyToBase64

key base64ToKey(string b){
string h="0123456789abcdef";
integer i;
b=(b="")+llDumpList2String(llParseString2List(llDumpList2String(llParseString2List(b,["-"],[]),"/"),["_"],[]),"+");
i=llBase64ToInteger(llGetSubString(b,0,3)+"AA==");
string k=llGetSubString(h,i>>28,i>>28)+llGetSubString(h,(i>>24)&0xF,(i>>24)&0xF)+llGetSubString(h,(i>>20)&0xF,(i>>20)&0xF)
+llGetSubString(h,(i>>16)&0xF,(i>>16)&0xF)+llGetSubString(h,(i>>12)&0xF,(i>>12)&0xF)+llGetSubString(h,(i>>8)&0xF,(i>>8)&0xF);
i=llBase64ToInteger(llGetSubString(b,4,7)+"AA==");
k+=llGetSubString(h,i>>28,i>>28)+llGetSubString(h,(i>>24)&0xF,(i>>24)&0xF)+"-"+llGetSubString(h,(i>>20)&0xF,(i>>20)&0xF)
+llGetSubString(h,(i>>16)&0xF,(i>>16)&0xF)+llGetSubString(h,(i>>12)&0xF,(i>>12)&0xF)+llGetSubString(h,(i>>8)&0xF,(i>>8)&0xF)+"-";
i=llBase64ToInteger(llGetSubString(b,8,11)+"AA==");
k+=llGetSubString(h,i>>28,i>>28)+llGetSubString(h,(i>>24)&0xF,(i>>24)&0xF)+llGetSubString(h,(i>>20)&0xF,(i>>20)&0xF)
+llGetSubString(h,(i>>16)&0xF,(i>>16)&0xF)+"-"+llGetSubString(h,(i>>12)&0xF,(i>>12)&0xF)+llGetSubString(h,(i>>8)&0xF,(i>>8)&0xF);
i=llBase64ToInteger(llGetSubString(b,12,15)+"AA==");
k+=llGetSubString(h,i>>28,i>>28)+llGetSubString(h,(i>>24)&0xF,(i>>24)&0xF)+"-"+llGetSubString(h,(i>>20)&0xF,(i>>20)&0xF)
+llGetSubString(h,(i>>16)&0xF,(i>>16)&0xF)+llGetSubString(h,(i>>12)&0xF,(i>>12)&0xF)+llGetSubString(h,(i>>8)&0xF,(i>>8)&0xF);
i=llBase64ToInteger(llGetSubString(b,16,19)+"AA==");
k+=llGetSubString(h,i>>28,i>>28)+llGetSubString(h,(i>>24)&0xF,(i>>24)&0xF)+llGetSubString(h,(i>>20)&0xF,(i>>20)&0xF)
+llGetSubString(h,(i>>16)&0xF,(i>>16)&0xF)+llGetSubString(h,(i>>12)&0xF,(i>>12)&0xF)+llGetSubString(h,(i>>8)&0xF,(i>>8)&0xF);
i=llBase64ToInteger(llGetSubString(b,20,21)+"AAAA==");
k+=llGetSubString(h,i>>28,i>>28)+llGetSubString(h,(i>>24)&0xF,(i>>24)&0xF);
return (key)k;
}//base64ToKey

I was referring to a built-in function, though, like string llCompressKey(key input) and key llDecompressKey(string input), which would undoubtedly be super speedy(er). I will try using some of these out though, as their ms time isn't too shabby. =] Thanks again.

(sorry minor syntax error in the nonexistent function prototypes =P )

I've produced a base 32768 compression scheme that will get a key down to 9 characters. Rather than pollute this issue further with compression schemes, I've updated https://wiki.secondlife.com/wiki/Key_Compression to include both the Base64 and Base32768 schemes.

Here is a free DNS service.

http://www.everydns.com/aboutus.php

There are more free dynamic DNS services out there, by the way.

Problem #2: text/plain

The firefox plug-in "modify headers" will work.
https://addons.mozilla.org/en-US/firefox/addon/967

Someone could make that plugin explicitely support some second-life content type changes by default (so SL costumers of some file-outputing tool can "just install it and it works"). Just be weary of not letting content type being switched to anything else than what is strictly necessary for the tool in question so we don't make virus writers, spammers, or griefers work too easy.

Most of the work arounds mentioned all seem reasonable to me. If you fake the user-agent or install a mod to view some text/plain pages as text/html you are moving yourself out of the group that would inadvertently hit a spoofed page - you've made some very explicit exceptions and taken deliberate steps. By nature it seems unlikely for any of these methods to create a group large enough that phishing/spoofing/spamming them would be worth the effort.

If we built the functionality in to allow embedded browsers in SL to view these web servers as text/html (when specified by the script of course) we can add extra information and warnings to the viewer since we have some control of the browser in this case. There are many possible ways we could use to indicate to the user that the page they are viewing is hosted by an object in SL and give them access to some of the extra header information about object owner and location to help combat abuse.

What I haven't seen, though, is a service that lets the target URL be set dynamically from an URI-driven API the same way a dynamically-changing IP address is for the dynamic-IP DNS service. The services I've looked at all require a manual webform update to change the target URL (so: (tough and bulky to script and near impossible to maintain).

Does anyone know of an API-driven WebHop/WebForward service? Or do we need to start beating on the various vendors to provide this extension to the existing redirection services?

Trouble is, even with 3rd-party service offerings like this, we're going to end up with scripts embedded in objects that will go belly-up when/if the 3rd party service does – perhaps permanently. Persistence of service seems like a good reason for LL to host a URL redirection service and specify an API for it.... someday. I can appreciate the scaling issues, though!

I really need to know more about how people would intend to use it. If they would use it for a few central servers, fine, but if they intend to have 3000-10,000 items out in the world with persistent URL's, thats not something I could see sustaining, at least not for free.

I've been viewing this as just a method to send data into an object (from another object or an outside server). If the address isn't static, you can always just have the object register upon reset, or whenever the URL_Lost event triggers using http_request or llEmail. Considering that the URL appears to only really be lost when the region resets or region changes, it shouldn't happen excessively often.

"If the address isn't static..." — that's not an "if"; it's a given. The external URI will be variable and essentially random as far as the outside world is concerned – see the Linden explanations on the mechanism.

"... you can always just have the object register upon reset,..." – register WHERE, though? That's the point of the discussions on a sort of "dynamic DNS" scheme. In order to let the outside world know its current URI, the scripted object is going to have to 'register' that URI with some address-mapping entity on the external 'net – one that can be reliably located BY THE SCRIPT arbitrarily far into the future. The day someone's home-made address-mapping service goes down is the day the associated scripts in-world stop working (assuming no-mod scripts and lack of instant in-world creator support). Relying on a well-founded external address mapping service alleviates these concerns going forward. (Which is why, ideally, the Lindens would host that remapping service – but they said they will not.)

"... it shouldn't happen excessively often." – one time is enough to make the end-user of your objects very unhappy. Where you can go in and tinker with your own scripts in your own objects... that's one thing. Expecting an end-user to go through some elaborate (and long-winded) manual procedure every time this happens... that's another matter.

So I really think these concerns are quite valid even for uses well short of running miniature web pages from in-world objects (which the Lindens are discouraging; see above.)

that's not an "if"; it's a given

To clarify, I was speaking more along the lines of 'if ... then ... else', but if you want to say 'when ... then' thats fine,

register WHERE

2 places come to mind, the server of the script creator, or a central object inworld.
any product containing scripts that talk to the outside world will have a problem. Any dynamic DNS website can go away as well. Personally, I would point them all to a website I controlled and maintain the database myself.

What I meant about happening excessively often is that it won't happen often enough to cause excessive pings to a website. If the URL changes, it should be transparent to an end-user that the script updates that information where necessary. If the URL changed to often, it would be a concern that lots of scripts would suddenly update, which could bring an external server down.

I think all of these concerns are valid, but can be handled reasonably by careful script and external site design.

I also agree with Stephen that there is no way we can solve everything before it it gets out, so getting it out for the masses to use would help flush out any other really critical flaws or needed features.

At one time (which many of you no doubt remember) the resolution of names to keys was a huge technical and ethical issue that was debated back and forth constantly. Now we have quite a few ways to resolve names to keys (many of them reliable) and it is a non-issue.

Frankly, the issue was never really a technical issue - it was relatively easy to collect the keys, and the whole subject is still a huge ethical issue. Its just that the sheer amount of key collection databases have rendered the issue impossible to handle, silencing most people.

Either way, collecting URLs for llHTTPServer isnt really an ethical or a technical issue - more of an annoyance issue.

I'm just surprised that LL cant afford supporting a DNS server. User supported services tend to vanish after awhile while scripted items tend to stay in the world for as long as the grid is going to stay alive. I'd really prefer having the domain resolution handled by LL where the DNS is assured to stay alive as long as the grid is.

If LL is worried about the DNS server being hammered for requests, here's what i'd consider doing:

Rather then allowing every single object to register - i'd limit the DNS to supporting a limited number of domain names per avatar. Say, 3 or 5. You can then assign these names to objects that can be used as sub-domain hosts that will store currently updated URLs of all of your personal objects.

Its sort of the best of both worlds - take the pressure off LL, while still having the security of atleast one step of the DNS system always available and working.

My line of thinking is that even if the URL was static, or was made static by some DNS service, it wouldn't matter because if the management object or external server went away, those objects would lose that functionality anyways.

What use cases are you considering where the DNS or a static URL would keep a product 'alive' even if the other side was gone?

That might help frame what I'm misunderstanding.

• A single central mechanism for tracking and routing all http requests to any/every object in SL is not feasible.
• What is feasible is an optional 'registration' that could create more permanent urls for those cases it is really needed.
• The implementation that is complete is essentially free from a resource perspective, no new hardware etc.
• The "registration" service is not free, it requires hardware and infrastructure which has overhead cost in terms of money and maintenance.
• There isn't a whole lot of real benefit to users from a LL hosted and implemented registration service. There is some in terms of trust primarily, and a very slight nudge on ease of use.
• There is nothing that was done that we would have done differently or could have done differently if we had decided to build that registration service from the beginning, as part of the first phase. The implementation would look just as it does today with an additional LSL method to register urls. And it wouldn't be complete yet.

Http-in and Http-out are the magic bullet for communications into and out of SL. XMLRPC is definitely not scalable, and llEmail only slightly better. If your system can switch, please do when the time comes.

However, there's still a way for LL to allow domain -> capurl resolution without investing in a big resolution server. Allow me to elaborate:

• At this time, every avatar in SL has a profile. One of the features of this profile is the ability to write personal "notes" about people in their profile. Its really just a plain text entry field.

• What I am suggesting is an additional, similar text entry area with limited text - say, enough room for up to 3 or 5 entries. NOT necessarily editable via the client: LSL only.

• These fields will be poll-able via llRequestAgentData() as well as writeable by scripts with the proper ownership permissions.

• This would mean that every avatar will have a limited capacity to store a tiny index of keys. Say:

llRequestAgentData(my_key, DATA_HTTP_INDEX);

will return the following dataserver Data field:

<cap_url>,<cap_url>,<cap_url>

This is useful in two ways:

• The stupid simple way:
  Say you only have 3 or 5 objects that use capability URLs and you dont really care much about having more - just choose to use any of these fields to store a reference to that URL.

• The smart expandable way:
  Say that you want to store CAP URLs for over 50 items or more: Use LSL to set up your OWN "domain server". Then use one of the free profile fields to set this DNS object's capability URL.

The result: Even if the DNS object gets returned or loses its URL, you can still restore its usability by updating the profile field to the current HTTP cap url - hopefully, automatically and via script.

Coders will not hammer the dataserver more then they do already, since the scripts would (or should) be designed to query for the capurl list only if they lose contact with whatever object they were talking to, and have no other capurl to fall back on.

What do you think? Is this feasible?

And Crystal, That sounds good. If there's one thing that Scripters have been begging for, it's a way to store persistent data, even if it's the tiniest bit. We just need a toehold in-world, an external server is not the answer for everyone or every situation. I'm currently having to do something potentially risky just to store a persistent integer in-world and I don't like it one bit. But I have no other choice.

• There are no plans to remove script email. Outbound (llEmail) definitely has uses far beyond obj->obj communications. Inbound email, while not ideal at least rarely hits our central db. We are committed to continuing to offer these for the foreseeable future.
• XMLRPC is a more difficult issue. The implementation for it truly is horrible. We really want to get rid of it. We don't have any finite plans to do so at this point. What I imagine will happen is ~6 mo after http-in is live on agni we will survey XMLRPC use to see how much it is still being used, why it is still being used etc. I believe http-in to be far superior and that everyone should switch, but I could imagine reasons for it to be difficult (the DNS-like issues for example, of course). Based on information we gather then we will make a decision on the fate of xmlrpc, whether we can continue to support it or will need to do something else, or whether we need to wait and do another investigative round later.

I think Kelly's idea for an optional registration is a good solution for those who want something LL controlled (so it is more likely to be around for the grid lifetime).

However, I still think that considering http-in as a potential webserver (which is basically what I see would be needed for any use case that wouldn't have an external server) brings on a whole load of issues. I mean, it would be like running apache in every instance that would use this, which opens up many huge cans of worms. That's why I see it as a communications layer, instead of acting as an actual portal into SL for browsers. None of the current communications layers can do this (object UUID's change upon region reset, and XMLRPC does as well, and can actually be lost if it isn't used often enough).

In any case, any of the ideas mentioned so far could be done using an external server, and none would even require http-in. I'm looking for use cases (of the non-webserver type) where having a static name is necessary without being dependent on some creator item (inworld object or external server).

@Anthony, I think it would be a narrow view to think those who want to have something persistent only want to use it as a 'web server'. HTTP-IN has a lot of potential to be used for object to object communications, as a faster, more reliable form of llEmail. However, given that both the sender and reciever has the potential to 'change keys' suddenly, it can be problematic. It's a shame to say someone must buy/rent an external server just to manage addresses of in-world objects. It can be done, and likely will be done, but it's just kind of sad. We've had to deal with the same issues with llEmail for years and really there's a lot of things I built and never released, as I felt they simply would never be reliable enough given the transitory nature of keys/addresses in SL. It may be LL banks on that sort of thing, to keep the amount of communication low. But I can't say I favor that approach, if it's indeed the case.

For Crystal's proposal, I think you should read again, She's talking about additional hidden fields that would be added to a profile, accessed by script only. So yes, there would need to be a new function to write to these fields. To prevent random scripts from writing random junk to the fields, I think it would be necessary to implement some sort of access restriction. If my application uses slot A, I don't want some other application overwriting it. Of course this starts adding more complications and I can see where it can begin to get out of hand. If I thought there was a chance it would be implemented I might sit down and really work something out but, I'm never a fan of wasted efforts.

The thing that quite a few folks here seem to have missed, is that unlike XML-RPC and similar, part of the HTTP-IN implementation gives us an event that we can use to tell when the URL has changed. This means in the change handler we can use some code to tell any other objects that need to know our URL the new details. That way object / object comms can be self maintaining - each time an object knows its URL has changed, it updates its peers...and so on. The cases where the URL change are actually relatively rare, so it's not going to add a lot of work - unlike the current situation with XML-RPC. I have a timer set to check the current XML-RPC for my object every 1o minutes, and if it's changed, it tells my webserver - a total waste of resources that HTTP-IN will remove.

Yes it's not ideal - but it's a vast improvement on what we've got, so let's get it implemented quickish!

So two people in two sims both have an attachment on, that talk to each other. Somehow, mysteriously you have got them talking, each knows the other's key. The two people take them off about the same time. Then person A puts his back on. on_rez() fires,and the item says "oh, my key has changed", and tries to tell it's new key to the other attachment on person B. But it's not being worn, and once it is put back on, it's key has too also changed. now neither can talk to the other, and there's no way, save both of them talking to some 3rd party who's address doesn't change, to re-establish that communication.

This has been the problem with llEmail and XML-RPC for years and sadly, also remains the issue with HTTP-IN. I'm probably preaching to the choir but It's one of my favorite sermons.

Given that the email() event has been unusably broken since around 2005 (SVC-23) and XMLRPC is (unofficially) deprecated, I'd rather email was fixed BEFORE another new method is introduced. Actually, I don't know why LL is spending resources on this new feature when there are a huge and growing number of grid platform bugs that need fixing anyway.

Perhaps getting the Mono script engine in shape so that it can at least run the scripts that the old LSL engine can run would be a good idea first. (We had to switch thousands of users back from Mono to LSL scripts to patiently wait for Mono to become a bit more stable - but it is starting to look like LL is going to leave it broken while they implement more features. We have a business to run and users don't care about explanations like "LL is working to improve the stability of the new Mono script engine" - they just want their products and services to work, which for us means not using Mono due to the various script halting exceptions it causes - SVC-2908)
I've developed for Mono for several years and I know the quality of the Mono code base is more like a perpetual Alpha product - so by extension, LL shouldn't claim a Mono-based script engine can be release ready until the Mono IL runtime itself improves significantly.

Having said all that, we (apez.biz) will be announcing a free scalable public service to track server URLs using a DNS solution. I realize people have concerns about investing in 3rd-party services that might disappear if a company goes bankrupt, but honestly, LL could go the same way. Our other public APIs have been around for years already. It is a mash-up world. Take it or leave it.

I'm looking forward to HTTP-IN if it is robust and we're already developing for it, but honestly I hope it is delayed to receive MUCH much more testing that LL typically puts into features before deeming them 'release' ready.

Currently I have objects polling a data base every 10 seconds to check if there is an HTTP-IN message waiting for them. It is clumsy but it works.

With HTTP-IN, could I not have each object report its URL to a new slot in the database whenever a situation arises that changes the URL? And if necessary report once an hour just in case? That would be a reduction in traffic of 99.7%.

If there was a transition situation where a new URL report came in, outgoing messages for the last 10? seconds could be repeated as duplicates. The objects automatically log in when rezzed or worn now, so it would be an elementary change.

For a case like that, they must register somewhere. Whether it is an external server (DNS or URL tracking), a 3rd inworld object 'server', or LL run registration service, you couldn't avoid that. I am guessing here, but having keys/URLs themselves be persistent isn't feasible likely either due to database growth or a complex management of how items are populated within a simulator (when a new item is added how would you know it was the same one and not a copy)

One other possible thought is that both objects could fallback to sending an email to the owner's address (from a notecard, etc.) that comms have failed and how to restore it by sending information in one email to an address in another which would restore the communication. (Could also be done via IM I suppose). That way the scripts wouldn't have to rely on any 3rd party at all (except the owner) which is what some have mentioned as being problematic.

distributed-mesh-like-DNS:

it would be somthing like, people volunter one address of their quota to store a certain number of records, plus the address of other DNS servers, if the requested name isn't found locally, it relays the request to some of the other servers randomlly, the server that finds the result sends the data directly to the address that requested it. Scripts can request that an record be created for their address and an URL, if the server that received the request already got too many records, it relays the request to other DNS servers it knows about picked randomly, the servers that create a record will report the succses of the request and the server's own address so the script can if desired, store that DNS server address for future use. Details like time to live, pings and other things inspired by regular networks should be discussed by people that knows mroe about such things to see how it would be better. If enough people volunteer to host a DNS server I believe a system like this might become quite resilient.

It is possible to do this sort of thing without a central server by using a distributed hash table. The Open Stargate Network works this way. see http://ma8p.com/~opengate

See previous comment.

whether they were external webservers or inworld 'object servers' wouldn't matter, though I imagine the external web servers would be more scalable and efficient.

By limiting the number of entries you can save within your profile, it will force you to use your own, LSL made DNS server. Its not intended to store all of your capability URLs on the dataserver, and not intended to be refreshed constantly. Thats why its design will discourage such use and instead try to push people to use their own DNS servers with the profile fields only used as a pointer to that DNS server in case of a prim failure (auto return, delete, etc)

@anthony reisman:
You're right about the security issues, and frankly this makes me realize that restricting editing of this profile field to manual editing over the client would be a better solution rather then doing this over a script.

The reason for manual editing being superior is that it will force people to use this field only to point at their DNS service prim and avoid using it for anything else that may hammer the dataserver with updates - because their LSL based domain name server will be easier to update automatically compared with the profile that must be edited manually.

http://wiki.secondlife.com/wiki/LSL_http_server

I ask because I'd like to use Google App Engine to create services that interface with LSL scripts through http-in, but App Engine's urlfetch service is (deliberately) limited to ports 80 and 443. See http://code.google.com/appengine/docs/python/urlfetch/overview.html#Requests.

Is there any reason to use non-standard ports in http-in?

http://sim3015.aditi.lindenlab.com:12046/cap/3ff4f3f2-ea08-76c1-cef6-a22b4a573a7c

https://sim3015.aditi.lindenlab.com:12043/cap/a7717681-2c04-e4ac-35e3-1f01c9861322

These urls are atomic, and the port is part of that. It is possible the port will vary between url requests, just as the host might (for example if the region restarted and the simulator came up on a new host). I'm not sure on google app engine's reasoning for limiting request ports but it is unfortunate in this case since it will not be compatible with this feature.

For your last question: We run a fair number of services on each simulator host. The cap server is far from the first, and does not get the standard ports.

If I were to re-write my XML-RPC communication protocol ... how long would it last until HTTP-IN was released? If I release a new product heavily reliant on XML-RPC communication (as I am planning to do in the next 2-3 days), how long would it be valid before I needed to essentially force all buyers to update?

• I don't know why the wiki pages are marked as active. Perhaps because of the regions on aditi?
• I do not know when 1.27 will be live on the main grid / agni. It depends on how many issues are found in "beta", and how long it takes to resolve those. If I were to make a guess that guess would involve the term "weeks". 4? 10? 20? I dunno.
• XMLRPC will continue to function when HTTP-In is released. We would like to deprecate it, but that will be a long and loud process that will involve announcements, talking to content creators, tracking down who has not converted and why etc. etc. etc.
• Be careful of the load your "new product heavily reliant on xml-rpc" causes. Part of the reason for HTTP-In is the fragility of XML-RPC. Please try to minimize the number of XML-RPC messages as much as possible.

My personal wish would be for you to build out and test your system on aditi (where there are regions with HTTP-In now, and will be regions with 1.27 w/ http-in soon) and then wait to deploy live until 1.27 is on the main grid. I completely understand however that timing is always important and it may not be feasible to delay your product deployment until the undefined release of 1.27. It is also difficult to transfer complex objects / systems from beta to main grid. Which is why I'm just trying to give as much information to you as I have.

• Kelly

The current project I'm looking to release will have XML-RPC communications about twice / day / object ... current number of objects in world is about 100 ... so the load is very low. I've got several other products reliant on XML-RPC (as it's currently the only way to initiate comms from the web), but they are similarly rare in communication frequency (most average around 10 comms / day).

I'll re-build my web-comms script for HTTP-IN and test all my products on Aditi so that I'm at least ready for the switch whenever it happens - but 4 weeks is a bit too long for me to wait for product release (especially because the 10 and 20 after make me a bit leery).

Thanks again for the info.

Quote: Kenn Nilsson

Seeing the functions listed as "currently active" on the LSL Wiki has me feeling stalemated on my updating of web-communications in-world scripts and web-end programming ... not to mention that I'm sure having the functions listed as "currently active" has confused any number of scripters who are probably wondering why they can't get their scripts to compile with those functions in them

I'm not sure why this was listed as currently active but i have marked all with the preview grid note and add the (U) in the LSL_Functions overview

if done well I believe it will quite fault-tolerant, resilient and scalable

First of all congrats on getting this feature implemented. I was testing it extensively on the beta gird now that 1.27 server there and it works very well. The only crippling problem I encounter is limiting query string to 255 characters. Let me try to explain a bit.

The way to get data to LSL scripts can be via POST request where you can send up to 2K of data. The problem with POST however is that its not "redirectable". Because we would need to have some external name service, it would be optimal to use it with a GET request which will transparently return a 302 with Location: of the last registered llGetURL().

I have implemented such a service and are testing it atm on the beta. So for example no matter how many times I request:

http://gridurl.appspot.com/go/4bf16fdc-4968-40b6-8e39-dca23e4b0767/?foo=bar

It will always send the request to the appropriate URL including passing on parameters passed in the query string. So I can have another object using llHttpRequest() always use this url and pass the data to the destination object. This transparent redirection with POST is not possible, therefore limiting the query string to 255 becomes more problematic.

Are there any reasons why we cannot have query string header have the same limit of at least 2K since GET request don't have a body. Or if the request type is GET pass the query string as the body (doing this will actually make it transparent to the script no matter if data was sent via POST or GET).

http://purl.org/

and would the usage by scripts be bellow the abuse threshold?

Latif: Transparent redirection with POST is possible, just harder. Well, you may lose some header information if for example you have the persistent url service make the full request on behalf of the requester instead of just returning a redirect. The capability server that hosts the url does this - acts as a real proxy to forward requests, even posts, puts and deletes, to the simulator host.

Unfortunately I don't think we can raise the limit on the query string. To do so we would have to lower the max body size or have a 'max total size' shared by them which I think would just be too confusing in practice.

Lex: I just added a new blog which you saw, but yes http-in should be real soon now. Assuming everything goes smoothly during the pilot roll of course.

The example wiki page has been expanded:
https://wiki.secondlife.com/wiki/LSL_http_server/examples

And there was a recent blog post as well:
https://blogs.secondlife.com/community/technology/blog/2009/07/07/http-in-and-lsl-communications

Yay!

I related this issue to SVC-3297 because it's such a pain to use only global variables with the numerous state changes required for the app I'm working on. State-scope variables would make it so much more convenient and portable.

I was trying the PHP example for a update server system that uses my PHP and MySQL - even the example script and PHP seems not to on the live grid. Have I missed something?

• Resolved *****
  Talked with my web service host : the big onces like one.com block any fsocket use beyond port 80 - need to working around that with:
• redirection and GET
• or proxy host that supports fsocketopen on the SL ports in the 1200ish range with POST

We are close to rolling out a whole new freebies system for NCI. We are now using http-in. It is not something that is novel, but in our initial tests the latencies and script over head have dropped dramatically. On the http-in side of things there have been almost no difficulties, and it has been amazingly smooth. Kudos to a job well done. Your efforts have made our jobs a lot easier, thank you!

PS: As a bonus, the http-in has also made it much easier for us to include automatic updating that will be a great help to us in the future.