How would you like the feature to work?

Say, I created an app to be used as HUD and I need the app to store some data persistently. Settings, usage statistics, whatever.

I also like to store the data outside the linkset. Then it persists script crashes or linkset deleting. The customer just uses an other copy and it restores all the stored data. Easy for updates, too.

We can not use LL Experiences (they not work grid-wide but my customers want use the app everywhere) and we can not use external database because it is a bottleneck and can be down when SL is not and then the apps are dead. Also, both approaches require me to pay for the database when I want that the copies of my app that my customers own keep working.

Instead, we need databases that are available grid-wide and are owned to the customers. There are two ways to achieve this:

Global KVP database

LL provides a KVP database that is not bound to any experience and is {}available to both, premium and basic residents{}. The database most likely will be limited, be it by the volume of stored data, the traffic limit or be it by taking a subscription fee. The KVP database will be bound to the customer account. As long they keep the subscription (if there is one), their personal copy of the app will be working, even if I myself hit the buss and am no more in SL.

The database is also then accessible, when the database owner is not online or not on the region. Scripts owned by them will still be able to read or write into the database. The disadvantage is this database limitation. But when keeping it reasonable the benefit of the persistent data storage may overweight.

Local KVP database

{}This database is held by the viewer itself{}. When a script stores a persistent data, the data is sent to the viewer and the viewer stores it e.g. in a XML file within the user' account folder.

The storage will cost nothing and is potentially unlimited in size. And as long the XML file is backed up, the data remains persistent.

The disadvantage is, the data is exposed to external programs. Also one can move the XML file from one to an other account. To face this, the data in this file must be encrypted by using the avatar UUID and an other hidden string as passwords. Still, super secret information should not go in there.

Also, this database will be only accessible when the database owner is online and on the region, as the scripts have to talk with the viewer. For a HUD it is always given, for inworld installations not.

Database selection

It would be perfect if one could have choice of database: The always present but limited global database or the only online present but unlimited local database.

The API (suggested below) is designed with this choice in mind and allows even to select two more databases, the experience-based KVP store and an external database server, which were dismissed for the feature first.

Why is this feature important to you? How would it benefit the community? Just a few examples.

Data survives disconnects and crashes. When those happen, scripts may loose the changes taken short before the crash. While persistently stored data can be simply loaded after relog.

You can store data like script settings, usage modes and others. For updating or replacing the device, you not need to import these data into the new copy, just rez/wear it and the data is there.

Stored data can be shared among scripts residing in different products of even different brands. Imagine you choose 'red' as your favorite colour and now cloth huds crafted by different creators suggest the red colour theme for you.

However, when allowing different scripts to access the same data, data security should be implemented, either by the database itself or by scripts storing this data.

Data can be shared between devices when the devices are not all inworld or worn at the same time.

For example you can create roleplay items. By using one item, e.g. an axe, the axe stores into the database how much wood you chopped. You unwear the axe and when you come to your fireplace, the fireplace still knows how much wood is available.

Yes, you can do this by keeping the data in a roleplay hud which is worn all the time. But then this HUD needs to store the data somewhere persistently to avoid you to chop the wood again, should the hud crash when you TP home.

When the database is global and not local then you even can access the data while the database owner is not online or not on the region.

For example you can create devices that display your status ("busy", "AFK", "available".) You choose the status on your hud and all the online display devices you own and keep everywhere over the grid update your status live.

However, using this technology for a direct inter-sim communication should be avoided. But you can store the URL of a scripted HTTP server into the KVP database and then use the HTTP server for the actual communication. The online status display should better use this approach as well.

API suggestion

The feature uses a KVP-database and there is already a syntax for this. Thus I adapted the syntax of some of Experience Persistent Storage functions for this feature where it was possible.

Data organization

As noted above, a script will be able to select the database (local, global etc.) But the data (the key value pairs) is not stored in there directly. Because when you allow the army of scripters to write in one place you'll need to define somewhere the syntax and semantics of each keyword and the associated data. Or you'll earn chaos.

Instead, the databases are subdivided in tables. A table (also called 'storage' in following) keeps a small number of keywords and is used in one or few scripted projects. The project developers can define the syntax and semantics of these keywords and data and keep at this definition more easy than if they would work with the whole database.

When compared with Experience Persistent Storage, the KVP store of a single experience is such a single table within of a large database of such tables. Since we start with a database, we'll need four more functions to create, open, close and delete a table, which are not in the Experience Persistent Storage functions list.

Run-time permissions

It is quite possible that the introduced functions require run-time permissions (which are TBD) to be requested and granted beforehand.

Storage level functions

These are the extra functions needed to create, open, close or delete the tables (storages).

Creating a persistent storage

key {*}llCreatePersistentStorage{*}(string name, string read_pwd, string write_pwd, string del_pwd, list options);

Starts an asynchronous transaction to create a persistent storage with the specified name and the read, write and deletion passwords.

Storage name

name - An unique name for the storage on the database selected via options. Any string, limited to 160 characters. To be able to use the same storage for various projects, I'd suggest to build it of words connected together by colon ':'.

• For example when I want to use the storage in my own projects only, I would start the name by my avatar key, followed by a string I choose, e.g. the unique project number in my store: "eb66b7b7-7ddb-4c5a-95ad-bfeb9837ae29:12365".

• While when I'd like to share the storage with other's people projects, then I'll use a hierarchical tree of prefixes: "RP:SciFi:Stargate" for a "Stargate" roleplay project. The list of such prefixes and even public storage names may be published in any public place.

  Passwords

  Using passwords allows us to control the access to the storage and stored data to scripts crafted by different creators. The read and write password are used to open the storage for exchanging data. For public storages they may be empty strings.

  The deletion password is used to delete the storage only. It shall not be empty or known. This prevents a script to delete a public storage because only the creator of the storage would know the deletion password.

  Options

  The options list selects the database and provides further information.

  Database options

  These options select the database in which to create the storage in question:

• PST_DATABASE_LOCAL - Local database, the data will be kept in the viewer, as suggested above.

• PST_DATABASE_GLOBAL - Global database, the data should be held by a grid-wide KVP database suggested above.

• PST_DATABASE_EXPERIENCE - Experience database: The data is held by the KVP database within the Experience, the script is compiled under.

• PST_DATABASE_EXTERNAL - External database. In this case the data is held by an external server.

  The default database is local, it is also used when the options list is empty.

  The external database must not be implemented yet. In this case the data is held by an external server managed by the user or someone else. The server URL, auth parameters, access methods, etc. must also be provided in the options list.

  Expiration options.

  We'll need them to avoid heaping of storages which are no more needed. Especially when the database volume is limited and the inspection instruments hard to use.

• PST_LIFETIME - The storage is auto-deleted after specified time in seconds.

• PST_INACTIVITY - The storage is auto-deleted after specified time of inactivity.

  Both require an integer parameter providing the time: [{}PST_INACTIVITY{}, 24*3600] will delete the storage when there was no access to it for one day.

  When the auto-deletion kicks in but the table is open (for data exchange), the auto-deletion is delayed until the table is closed.

Examples

// Creating a storage with a fully public access
request = llCreatePersistentStorage("public-data", "", "", "password", []);

// Creating a storage with a read public access
request = llCreatePersistentStorage("other-data", "", "secret", "password", []);

// Creating a storage with a fully private access
request = llCreatePersistentStorage("private-data", "super", "secret", "password", []);

// The same but with an explicit databbase selection
request = llCreatePersistentStorage("private-data", "super", "secret", "password", [PST_DATABASE_LOCAL]);

// Private global database with auto-deletion ater one week fixed time
request = llCreatePersistentStorage("private-data", "super", "secret", "password", [PST_DATABASE_GLOBAL, PST_LIFETIME, 7*24*3600]);

Deleting a persistent storage

key {*}llDeletePersistentStorage{*}(string name, string del_pwd, list options);

Starts an asynchronous transaction to delete a storage with defined name.

Removing the storage requires the valid deletion password to be provided.

Opening a persistent storage

key {*}llOpenPersistentStorage{*}(string name, integer read_only, string password, list options);

Starts an asynchronous transaction to open a storage with defined name.

When read_only = TRUE, the storage is open if the read_pwd is provided as password. When read_only = FALSE, the storage is open in the read-write mode, if the write_pwd is provided as password.

The function call provides the key which is not only used in the dataserver event, but it is a handle for the data connection. The handle remains valid until the storage is closed and used in the functions exchanging data with the storage. Also to close the storage, this handle is used instead of the name and options.

Examples (notice the password usage in the opening functions)

// Creating a storage with a fully public access
request = llCreatePersistentStorage("public-data", "", "", "password", []);

// Opening it in the read-only mode
table = llOpenPersistentStorage("public-data", TRUE, "", []);

// Opening it in the read-write mode
table = llOpenPersistentStorage("public-data", FALSE, "", []);

// Creating a storage with a read public access
request = llCreatePersistentStorage("other-data", "", "secret", "password", []);

// Opening it in the read-only mode
table = llOpenPersistentStorage("other-data", TRUE, "", []);

// Opening it in the read-write mode
table = llOpenPersistentStorage("other-data", FALSE, "secret", []);

// Creating a storage with a fully private access
request = llCreatePersistentStorage("private-data", "super", "secret", "password", []);

// Opening it in the read-only mode
table = llOpenPersistentStorage("private-data", TRUE, "super", []);

// Opening it in the read-write mode
table = llOpenPersistentStorage("private-data", FALSE, "secret", []);

Closing the persistent storage

key {*}llClosePersistentStorage{*}(key storage);

Starts an asynchronous transaction to close the open storage with the specified data handle. This handle identifies the storage, so there are no other parameters needed.

Example

// Creating a storage with a fully private access
request = llCreatePersistentStorage("private-data", "super", "secret", "password", []);

// Opening it in the read-write mode
table = llOpenPersistentStorage("private-data", FALSE, "secret", []);

// Closing it again
request = llClosePersistentStorage(table);

Auto-closing

The open storage is closed automatically upon script state change, stopping the script, unwearing or deleting the object, other events of loosing connection to the database, e.g. the script opened a local storage but the owner (database holder) teleports away.

Storage access functions

These four functions are exchanging data with an open storage. They work similar to the Experience Persistent Storage functions and have thus a similar syntax. The main difference is they take the handle of the open storage they access to.

This allows a script to keep more than one storage open at the same time.

Creating key-value-pairs within the open storage

key {*}llCreatePersistentKeyValue{*}(key storage, string k, string v);

Starts an asynchronous transaction to create a key-value pair associated with the given storage using the given key (k) and value (v). Similar to {}llCreateKeyValue{}(k, v).

Updating key-value-pairs into the open storage

key {*}llUpdatePersistentKeyValue{*}(key storage, string k, string v, integer checked, string origin_value);

Starts an asynchronous transaction to update a key-value pair associated with given storage using the given key (k) and value (v). Similar to {}llUpdateKeyValue{}(k, v, checked, original_value).

When the key did not exist in the storage yet, it will be created ({}llUpdateKeyValue{} behaviour).

Reading key-value-pairs from the open storage

key llReadPersistentKeyValue(key storage, string k);

Starts an asynchronous transaction to read the value associated with the specified key (k) and the given storage. Similar to {}llReadKeyValue{}(k).

Deleting key-value-pairs from the open storage

key llDeletePersistentKeyValue(key storage, string k);

Starts an asynchronous transaction to delete a key-value pair associated the given storage with the given key (k). Similar to {}llDeleteKeyValue{}(k).

Dataserver event.

All the eight functions above trigger the dataserver event which uses the key the functions provided and reports the success or failure of the operation, as well provides the requested information in the data param. The dataserver event has the same definition for all eight functions.

{*}dataserver{*}(key request, string data);

The parameters are:

• request - key returned from llCreatePersistentStorage, llDeletePersistentStorage, llOpenPersistentStorage, llClosePersistentStorage, llCreatePersistentKeyValue, llUpdatePersistentKeyValue, llReadPersistentKeyValue, llDeletePersistentKeyValue.

• data - string containing a comma-delimited list

  while

• on failure: data = {}llDumpList2String{}([ {}FALSE{}, error ], ",");

• on success: data = {}llDumpList2String{}([ {}TRUE{}, value ], ",");

  and

• integer error - An PST_ERROR_* flag that describes why the operation failed (error constants).

• string value - Data being actually delivered. Can contain further commas.

  Error constants

  The error constants are used for the dataserver event to describe failure of the storage operations. Some of them, more TBD while implementation.

• PST_ERROR_NONE - No error was detected.

• PST_ERROR_THROTTLED - The call failed due to too many recent calls.

• PST_ERROR_SOURCE_DISABLED - The access to the selected database is disabled.

• PST_ERROR_INVALID_PARAMETERS - Function call with invalid parameters.

• PST_ERROR_ACCESS_DENIED - The access to the selected storage was denied (e.g. bad password).

• PST_ERROR_WRITE_PROTECTED - Attempt to open a write-locked storage in read-write mode failed.

• PST_ERROR_READ_ONLY - Write access failed, the storage was open in the read-only mode.

• PST_ERROR_UNKNOWN_ERROR - An unknown error not covered by any of the other predetermined error states.

• PST_ERROR_QUOTA_EXCEEDED - An attempt to write data to the key-value store failed due to the data quota being met.

• PST_ERROR_KEY_NOT_FOUND - The requested key does not exist in the connected storage.

• PST_ERROR_NAME_NOT_FOUND - No storage with this name exists in the selected database.

• PST_ERROR_ALREADY_EXISTS - Failed to create a table or key as it already exist.

  Write protection

  Experience Persistent Storage use a 'soft lock': A script sets or removes the write protection via llUpdateKeyValue function and the checked argument. The accessing scripts have to update and respect this lock themselves.

  This delegates the responsibility to not overwrite a write-protected database to the scripters. I not know how good it can be reproduced when the database is open to public. Even if there are passwords, the list of scripters writing accessing scripts is potentially unmanageable.

  For this reason I'd suggest to use a 'hard lock': Whenever a script opens a table in the read-write mode, the table is write-protected and opening the table in the same mode by an other script is rejected with the PST_ERROR_WRITE_PROTECTED error.

  This way only one single script will be able to write into the open storage at the same time.

  Inspections

  The databases should allow inspections. For this, the viewer needs a new floater that allows listing of all available databases (local, global, experience) and listing of all saved tables in them.

  For each table the floater would show

• the creation date,

• date or age of the last access,

• name of the created script,

• name of the item holding the script (for locating in the inventory),

• name of the script and item creator,

• number of the stored keys and the used memory (very important when the database is limited).

  Also there must be a way to delete a table (no deletion password required).

  Listing of the stored keys and especially the associated values is probably not possible (due encryption).

  Complex Example

  As example we create a table for a medieval role-play system and fill it with required keywords.

  key     table;
  key     write;
  integer step;
  
  // character name is chosen behind the scenes
  string  charName;
  
  default
  {
  	state_entry() {
  		table = llCreatePersistentStorage("RP:Medieval:Asgard", "", "", "Yggdrasil", []);
  	}
  	
  	dataserver(key request, string data) {
  		if (request != table) return;
  		
  		// table created successfully - preparing it in an other state
  		if((integer)llGetSubString(data, 0, 0)) {
  			state Writing;
  		}
  	}
  }
  
  state Writing 
  {
  	state_entry() {
  		// we open the table to write into
  		table = llOpenPersistentStorage("RP:Medieval:Asgard", FALSE, "", []);
  	}
  	
  	dataserver(key request, string data) {
  		// table successfully open? create the first key
  		if (request == table) {
  			if((integer)llGetSubString(data, 0, 0)) {
  				step  = 0;
  				write = llCreatePersistentKeyValue(table, "name", charName);
  			}
  		}
  		
  		// table is being written
  		else if (request == write) {
  			// operation failed
  			if (!(integer)llGetSubString(data, 0, 0)) {
  				llClosePersistentStorage(table);
  			}
  			
  			// other keys follow one by one
  			else {
  				step++;
  				
  				if (step == 1) {
  					write = llCreatePersistentKeyValue(table, "level", "1");
  				}
  				else if (step == 2) {
  					write = llCreatePersistentKeyValue(table, "XP", "0");
  				}
  				else if (step == 3) {
  					write = llCreatePersistentKeyValue(table, "HP", "100");
  				}
  				else if (step == 4) {
  					write = llCreatePersistentKeyValue(table, "SP", "100");
  				}
  				else if (step == 5) {
  					write = llCreatePersistentKeyValue(table, "MP", "0");
  				}
  				
  				// writing completed
  				else {
  					llClosePersistentStorage(table);
  				}
  			}
  		}		
  	}
  }

{
  'Build Id': 'unset',
  'Business Unit': ['Platform'],
  'How would you like the feature to work?': 'how',
  'ReOpened Count': 0.0,
  'Severity': 'Unset',
  'Target Viewer Version': 'viewer-development',
  'Why is this feature important to you? How would it benefit the community?': 'why',
}