I'm also linking to SVC-114, because that's the main inventory loss "umbrella" we're keeping an eye on.

https://wiki.secondlife.com/wiki/Bug_triage/2007-08-20/Transcript

[15:11] Squirrel Wood: VWR-2001 may be related
[15:12] Aric Linden: I think this can be closed as a duplicate or tracked in the umbrella that Torley has linked to
[15:12] Aric Linden: I know this is being actively worked this week.
[15:12] Aric Linden: I had conversations about this today.
[15:13] Rosen Janus: I do happen to get a lot of "gesture missing from database" errors
[15:13] Goldvald Enoch: Which bug is it duplicating, if you don't mind telling?
[15:13] Aric Linden: Shall we leave it linked and move on?
[15:13] Goldvald Enoch: Same here, Rosen.
[15:13] Kooky Jetaime: I'm getting the gesture missing from database, and I'm not even loading any gestures..
[15:13] Aric Linden: It's duplicating the bug that torley linked to: the umbrella inventory issue.
[15:14] Squirrel Wood: get it on login every time
[15:14] Rosen Janus: Same here
[15:14] Goldvald Enoch: Does that cover "gesture missing", when you log on?
[15:14] Squirrel Wood: ctrl-shift-4 may show something on the missing issue
[15:15] Rosen Janus: On this login it happened five times
[15:15] Goldvald Enoch: It happens at least five times to me as well.
[15:15] Jacek Antonelli: It has some linked issues that aren't linked to by the umbrella meta-issue (SVC-114)
[15:15] Rosen Janus: x.x Waiting for the page to come up
[15:16] Aric Linden: I'm happy to import this and link it to the extant issue. will that suffice?
[15:16] Squirrel Wood: I guess that's the best thing to do
[15:16] Goldvald Enoch: The repro for me, is simply to relog, and wait for the errors ... lots of errors on the console too, I guess.
[15:16] Aric Linden: Let's add it to the existing issue and move on. I'll discuss this with the team working on it.

This issue does duplicate a lot of issues that I linked to, but as I mentioned in the description, most of these have been ignored, incorrectly marked as duplicates of unrelated issues, or resolved as duplicates in favor of EACH OTHER. I opened this issue in hopes of getting it treated as a definitive collection of all information available on this topic, because I don't feel that a real coherent case about this specific issue has been created before.

So please, don't be distracted by SVC-114 just because it also relates to a kind of loss of items. This is a distinct issue, and it's serious.

This is significant because this landmark worked yesterday.

While I have gestures in my inventory, I don't USE them, so a missing gesture to me is not really something I give a rat's toosh about. But we've both gotten the message, and there's a ton of tickets about simaler issues, so obviously this is part of a larger problem.

And, missing badly enough I can't get a UUID for them either!

ARGH!

While I have a ton of non-clothing prims in my inventory (or....I did....) most of what I'm seeing affected is CLOTHING.

Whole avatars, half of an outfit...annoyingly, often skirts will be missing, while the rest of the (mesh/non-prim) outfit will be fine. Not exactly PG rated...

Seriously.... This needs to be fixed!

The items lost span from shapes and bald, to clothing, objects and gestures.

Please revisit what is wrong with the asset server!

Usually the skirts. This problem needs to be addressed. I am loosing inventory that costs thousands.

Hopefully this will get fixed, because this is SERIOUSLY bad!!!

So far, this time, I have thousands of L$ worth inventory missing. Full perm building materials such as animated pose balls, scripts, etc .. will not rez. I consistently get the "Inventory missing from database" error. My personal furnishings, my own builds, jewelry, prim clothing, clothing, hair .. "Inventory missing from database" error. I also have 4 "gestures missing" messages at start up.

This is a horrid situation and it's a waste of money on items that no longer work and combined have a substantial real life monetary value. This needs to be fixed asap.

Fix it.

Now.

Do. Not. Add. A. Single. New. Feature. Until. You. Fix. This.

I have lost scripts, objects, notecards...a complete sampling of items from my inventory. I would like each and every single one of these items returned. Failing that...at least return the items missing that I have purchased.

And provide a secure way for residents to back up their databases off-site in case this ever happens again.

The configuration notecard for my ZHAO went missing. I handn't touched in almost a year, I went to add an animation, but its gone.

I've also been getting "missing gesture" alerts for months now. It of course doesn't tell you which gesture is missing.

I haven't really gone through my inventory lately. For all I know there's tons more missing.

Like others I cannot continue to purchase items that may or may not be there when I want or need them.

For the past 2-3 weeks now, I've been getting random "missing" messages while trying to put on clothing or objects. While I'm also having problems with gestures and the like like others, the majority of my issues are with outfits (clothes and objects).

I've found OCCASIONALLY that several cache dumps / relogs into practically empty SIMs will apparently force some of the inventory items to load... But the problem usually just manifests itself into another batch of items later on when I try to change. Also I've found (just moments ago) that trying to wear the items repeatedly MAY force it to load and rez. However, I'd been trying with the same piece of inventory for the last half hour or so, and that is on the ridiculous side of things.

Please address this issue.

Apparently this is not a new issue (http://blog.secondlife.com/2006/11/10/inventory-issues-its-not-just-you/). So it's well known to the Lindens and they don't seem to be willing or able to fix it, which makes this even more frustrating.

So now everything is so messy, that I had no choice and my store is now temporary closed since 5 days. Very good for business and reputation. A filled a support ticket a week from now and no new at all. Very bad customers service!!! Unforgiven for a big business like Linden. They have seriously to think about a kind of backup system for important inventory stuff or at least a way to re-upload those textures when they are suddenly missing..

"If you realize that you have missing inventory, follow the steps below as soon as possible.

1. If you do not yet have it installed, install the latest Beta Client.

secondlife.com/community/preview.php

2. Login to the Beta Grid and find a quiet place where no one will annoy you while you are trying to solve a problem.

DO NOT REZ ANYTHING WHILE ON THE BETA GRID. GO INTO BUSY MODE IMMEDIATELY. DO NOT ACCEPT ITEMS, DO NOT DELETE ITEMS, DO NOT PASS OR RENAME ITEMS.

3. Open your inventory, type in a search term (I used Tree) and allow your inventory to populate.

4. Got your goods back? Awesome! Quit the Beta client, login to the Main Grid and do the same thing. Once you have retrieved your items on the Main Grid and your inventory has fully populated, clear cache, logout and see cache'ing as you ...

5. Log back into the Main Grid, recheck inventory, smile a lot, raise Hell as usual.

This technique has been proven to work more than half the time so it may save your hide. Please bear in mind that you should only expect positive results if you attempt this workaround as soon as possible upon discovery of your missing items. I am not guaranteeing that you will be able to recover your items but having at least a chance at doing so is much better than no chance at all. I hope this helps some of the residents that fall victim to this issue."

Does not matter if i want to wear or rezz it on the ground... no way to use them. I hope they are not really lost!

This is scary.

I hope we can find a solution soon.

This may be related to another issue I have had for a number of months - my "Recent Inventory" folder never purges after logout. Could this be related? I did not report it, as it was simply a nuisance to me, but perhaps it was a precursor to a more severe inventory problem? If not, should I open a new ticket for this?

PLEASE help!!

Obviously people reporting the problem doesn't help get it fixed. They've been doing that for, literally, years.

Here's hoping people voting to get the problem solved will get it fixed, but I'm not holding my breath.

Maybe Linden Labs has no idea how to solve the problem... So here's pseudocode for my proposed solution.

Record every operation a user performs that will have a permanent effect on a object (before someone claims this is too much data, or too hard, to record, I refer you to the detailed linden transfer log already maintained). This is basically building (including the user deleting the object), and placing attachments. Since most users don't build at all, the amount of data recorded should be quite small. Each time, BEFORE the inventory server is backed up (and if it's not being backed up on a regular schedule now, you should start), load a copy of the /last/ backup and "replay" the building/attachment-adjusting operations on them. Then compare to the current inventory. They should match. If they do /not/ match inventory corruption or loss occurred (i.e. objects were altered or deleted, but the user did not do it). If inventory corruption/loss occurred whoever is in charge of maintaining the inventory server's code should be alerted, and a copy of the damaged objects, along with their good versions, extracted for them to study to aid in debugging (it is important for them to be able to look for patterns in what was corrupted/lost). Further, any items that were corrupted or lost should be loaded from the last backup, have any building operations that have occurred since then applied to them, and then put back into the user's inventory or on the land where they were placed.

If you write code to do this, then you should get notification when inventory loss occurs (thus you will know when it is /really/ resolved, and when any changes you make damage the inventory server code), and the users won't really lose items (or rather, they will automatically get them back every so often).

Doing it this way also avoids needing to make most of your infrastructure work correctly. If the garbage collection is still unreliable, this will catch it, and undo the damage. Further, it will probably give you a very good indication of /when/ garbage collection goes wrong, since you will be able to see large numbers of items that were lost, and look for what they have in common, that differs from items that were not lost. This should make debugging easier.

To avoid unnecessary computation, the inventory of any user who has not logged in since the last time a backup was made would not need to be checked outside of making sure that it had not changed (i.e. no "log replay" operation needs to be carried out for these). Obviously if they have not logged in, they could not have done any building, and therefore their inventory should be exactly the same. We all know that only about 100,000 or so players really use Second Life in a month or so, given the concurrent users never goes much over 50,000, so that should be much, much less work than the total number of avatars. You could save a lot of time by only storing/comparing database changes, as well. I believe this is traditionally referred to as "delta dumps".

The only /honest/ reason I can see this might be difficult is the amount of data involved. Remember that you (Linden Labs) syphon off a part of the cash value of lindens each time they're purchased. This means basic account users /still/ pay you for any objects they purchased, thus they have the right to expect inventory protection. If the amount you are skimming off linden purchases and premium accounts and sim rentals is too small to cover it, increase the amount you take until you can afford this. There is no excuse for losing things people paid you for.

Finally, get rid of coalesced objects. They have no value except in occasionally tricking users into deleting their objects unintentionally. Please don't allow user interface sadists to design similar mal-features in the future. This is bad enough to be considered a 'inventory loss bug' even though it was intentionally written in.

There ARE plenty of old avatars, and they haven't "rotted away". I'm less than a month from my third SL-birthday.

In fact, your entire comment is full of guesses based on incorrect assumptions. Nearly everything you've said here is predicated on a misunderstanding of how SL is built, so it doesn't apply.

I also know of at least 3 avatars that have survived several years (one was, I believe, about 3 years old, the oldest I've seen). I know of a great many others that are missing textures, or nearly gone (one person was left with just the AO portion). Many /do/ 'rot away'.

In any case, if you think I don't understand the infrastructure well enough, the least you could do is enlighten the rest of us rather than criticizing unconstructively. At least I bothered to propose a fix, something I haven't seen anyone else do so far. If there /is/ something wrong with it, more information could allow me, or someone else, to suggest something better.

As to the rest, sorry, I wrote that comment shortly after waking up and wasn't up to the task of listing everything I was speaking of. Here's what I meant:

• Your suggestion of recording all operations on objects is predicated on a false assumption: that the inventory loss we're speaking of here is actually loss of items in someone's inventory. In fact, "inventory loss" is a misnomer; what we're talking about here is asset loss. The inventory server still has entries for the items, but the asset server (which stores the actual object data for ALL inventory items on the grid, including those in the inventory of objects rezzed in-world) loses the items. The inventory server stores "pointers" to specific assets, but the actual data for the assets is stored in the asset server. Several people can have pointers to the same exact asset, which is why I can pass you an inventory item that points to a missing asset and you can also see the "missing from database" error.

So this means that storing a log of everything that happens to an object won't be helpful; the inventory server is correctly keeping track of inventory items. It's the asset server that's losing data. Furthermore, the log of data you're suggesting would truly be untenably huge; it would be many times the size of the current asset database, and it would have to encompass much more than just the changes you list. It'd have to keep a running list of all modifications that happen due to, for example, an object being physical, and even the constant inherent modifications implied by a script running. It's infeasible.

It's not necessary, either. The data for assets taken into inventory is stored correctly into the asset server and pointers are properly made in the inventory system. It's just that assets seem to be getting discarded later on when there are still inventory pointers to them.

• And, in fact, there IS already code to notify LL when inventory loss occurs. See the recent blog post I linked to at the end of this issue's description. Every time you see "...missing from database.", LL's backend adds your item to a list of lost things.

• Your suggestion won't help with garbage collection. The asset server and the inventory system are decoupled. Incorrect garbage collection causes the asset data to disappear when there's still a pointer to it. Think of it like having a card catalog at a library that lists a specific book that should be in the library, but it was taken off the shelves and lost.

• People who have not logged in in the past month could nevertheless have had their inventories change, due to object returns.

• LL doesn't siphon off money from lindex purchases. I believe I remember seeing them post that they actually LOSE money on the lindex because the fees they charge don't quite cover the labor and resources they put into it, but they continue to run it for the benefit of users.

• Coalesced objects do have a very distinct value, which is why they were implemented despite the confusion they added. Objects are coalesced to reduce load on the inventory server, by allowing multiple prims that are returned simultaneously to take up only one inventory entry rather than one entry per prim. The latter resulted in many simultaneous returns that hurt performance. It wasn't a UI sadist that designed this, it was a developer looking to solve a performance bottleneck (Kelly Linden, I believe). I agree that object coalescing really hurts usability, though.

So ultimately, due to the way the system is designed, I don't think your idea will really help. I don't know what exactly will help, because I'm not behind the scenes at LL and I don't know more about how the infrastructure of the system works. I DOknow that LL is full of incredibly talented developers, and that the fact that this issue hasn't been resolved immediately means there isn't a solution any of us on the OUTSIDE can come up with in an afternoon given our lack of full knowledge of how the system works. The best we can do is help them debug by providing information.

These are scripts that reside in my sunglasses, which acts as my toolbelt. The scripts perform various useful actions for me like controlling my camera, letting me sit on people, letting me set text above my head, etc. Each task is in a different script, but it seems like almost ALL of them are missing from database now.

I'm pretty sure that every script I've lost so far existed in an attachment that I use regularly. To me, this suggests a place to look for errors in the garbage-collection system. I think there might be some kind of race condition involving worn items, or items being attached or detached AS the garbage collection system builds its list of assets that are in use. Somehow, items in attachments are prone to being missed and mistakenly considered unused.

• incorrect garbage collection
• incorrectly-handled transfer corner cases
• incorrectly-handled simulator corner cases
  If Linden Labs has mentioned any other causes of inventory loss, I
  would like to read about it.

You mention several real and imagined problems with my first
suggestion of how to protect against inventory loss.

Real problems include:

• objects owned by accounts that have not logged in can change
  without it indicating error (for example, by allowing a friend
  to modify their objects, or objects being returned)
• scripts can modify some permanent aspects of an object, which
  would be incorrectly interpreted as an error using my initial
  suggestion of how to protect against inventory loss
• there are more efficient ways to detect and repair corruption
  and loss than loading the last backup, replaying all
  transformations on a object, and comparing against current
  inventory

Imagined problems:

• I have no concern for protecting script state (restarting a
  script rarely does much harm; it might be annoying in rare cases
  such as security orbs)
• I am not concerned with protecting location/orientation in world
  (so recording script and physics alterations to
  location/orientation is irrelevant; rezzed objects recovered
  from loss could just be placed in "Lost and Found"; it would
  only be annoying for multi-object things such as large
  buildings, and still less upsetting than total loss)

This is my revised suggestion, with corrections made for the valid
points you brought up:

• incorrect garbage collection - Similar to my first suggestion,
  record operations, by users or scripts, that could change the
  number of references to an object. Examples include deleting an
  object from their inventory or changing the texture on an
  object. Before the next backup of the asset/inventory servers
  is made, or objects are to be garbage collected, the previous
  backup should be loaded, and the log 'replayed' against it, then
  compared to the current state looking for discrepancies in the
  reference counts. When one is found, the one with the higher
  number of references should be used, since both the garbage
  collector and backup-comparer could be buggy (better to
  incorrectly retain objects than incorrectly delete them). The
  log should not be very bulky by modern standards. The only
  thing I can imagine bloating the log much is scripts which swap
  sculpt maps for animation. The exact size of the log depends on
  how long you go between 'compare the current state against the
  last back up + the log since then, then make a new backup'
  cycles. Backups should be being made monthly, if not weekly,
  and the log need not be retained after the comparison and any
  inventory loss is repaired. The only reason a garbage collector
  would delete an item early is if references are not being
  reliably recorded, which is why this is needed. It's "second
  guessing" the reference count. Measure twice, cut once. Linden
  Labs has not, as far as I know, reported the asset server just
  losing information. They /have/ said the garbage collector is
  removing things when it should not.
• incorrectly-handled transfers - Before deleting an object from
  the source, confirm that it exists in the destination (i.e. that
  the write was successful). This handles the fairly frequently
  occurring cases of objects failing to copy (or if a no-copy
  object, disappearing during transfer) when sending them to
  another user, and things like trying to send an object to
  someone marked busy. Luskwood's bat update had transfers fail
  to many people, losing the right hip attachment.
• incorrectly-handled simulator corner cases - Like with
  transferring, rezzing an object should check that the object now
  exists in world before deleting it from inventory. The other
  problem I know of involving the simulators and losing objects is
  when a region rollback occurs. All copyable objects, and any
  no-copy objects not inside the backup about to be restored,
  should be returned to the owner's Lost and Found. Ideally when
  objects are returned for this reason they would be grouped under
  a sub-directory named something like "<region name> Rollback
  <time the rollback occurred>" (the things inside angle brackets
  should be replaced with their associated values).
• everything else - Script state often inexplicably becomes
  corrupted, especially on vehicles or attachments. Prim drift
  occasionally occurs. The only good solution I can think of for
  this is to allow users to back up their things, using strong
  encryption to prevent piracy, if necessary. See VWR-358 for a
  discussion of this. It would allow users without region
  rollback authority to restore corrupted objects to a usable
  state, and protect against whatever future causes of object
  corruption or loss there may be. I don't consider this, alone,
  to be sufficient since users should not be expected to back
  things up constantly, but given personal experience with support
  staff refusing to restore lost objects, even when they were lost
  only days ago, and still there on the beta grid, I would rather
  users be able to restore their own objects.

The existing method of restoring a object if it is accessed within
a certain period of time, after being set aside for deletion by the
garbage collector, is a poor solution, since it only works if the
lost object is accessed in time, and the user doesn't even know
what they might lose if they don't access it (i.e. it's pure luck,
since you don't know what you need to pull up periodically). It's
better than nothing, but I think double-checking the reference
counts is a better solution. Of course both measures could be
left in place for added reassurance.

Linden Labs supposedly makes money off Basic accounts. Given the
only way this could occur is by Basic accounts purchasing Lindens
to make groups, form partnerships, and buy objects in world, I
don't see any other way they could be making money from them
except by skimming value off the LindenX. How, exactly, Linden
Labs makes their money is neither here nor there in terms of
preventing inventory loss, outside my attempting to make the point
that they obviously do make money, and if they can't afford the
hardware or programmers to fix this problem, they can skim more
off Linden purchases, and/or hike prices for Premium accounts or
sim rentals, to pay for it.

Coalesced objects are hard to explain outside of user-interface
sadism. Were this not so, at the very least, they could be given
a very distinct icon, and not be misleadingly named based on one,
seemingly randomly selected, item inside. Instead of calling a
coalesced object containing a belt, a mirror, and a bed, "Belt",
which might have been something you were fiddling with modifying
for fun and didn't want to keep, it would be named something like
"Coalesced Object" and you would get a special warning and have to
confirm it when deleting them. Thus we have established that,
even were coalesced objects a good performance solution, the
interface is unnecessarily bad; bad in ways which can't be excused
by performance concerns. What remains is to show that they are
also not necessary. Returns may well demanding on performance.
In the past, Linden Labs considered putting a 10-minute lower
bound on auto-return. This also reveals a more sane solution to
the problem coalesced objects supposedly were intended to solve.
Simply rate-limit the movement of objects from the sim to the
returnee's inventory to one which does not negatively impact the
system.

Lex Neva, if you, or anyone else, has corrections/additions to make to these suggestions, I'll try to revise them again, but may be away for a few weeks soon. I apologize for the odd formatting.

• The current garbage collector doesn't use reference counting, at least, not as I understand the term. Reference counting means keeping track of the number of times a given asset is referred to, in the inventory of any user, or in the inventory of objects rezzed in-world. Rather than do that, I believe the current garbage collection system periodically goes through every user's inventory, every object's inventory, and quite a few other sources (sounds referenced in gestures, textures referenced in clothing, sound UUIDs mentioned in script source code, etc), producing a list of assets that are still in use, and then it deletes everything else. It's logically pretty sound, theoretically much more efficient (read: feasible) than reference-counting... but in practice, something's wrong in the process and some items get missed.

• As to incorrectly handled transfers, I know LL's quite aware of this huge problem, and I think their ultimate goal is to make all inventory operations transactional to avoid loss. This source of loss isn't the subject of this JIRA issue, though.

• I think you underestimate how much data your log is going to take up, even in modern terms. The asset database was recently reported to contain over one billion assets. Every single BYTE could translate to a gigabyte. Think large scale here.

• I'm pretty sure the system that catches "missing from database" objects and tries to restore them if they weren't garbage collected too long ago is a stopgap measure, and is considered only that and not a particularly happy solution.

Right now 1 gigabyte of hard drive space costs less than a dollar.
I seriously doubt the reference change log would be too large in 1
week to manage with modern technology at a price Linden Labs can
afford, but if it is, one alteration to the plan would be to allot
each object so many reference changes. Say allow it 1000
reference changes within a 'compare last backup + log since then
to current state, then make new backup and clear log' cycle. If
it goes past that, don't record any more changes, and ignore it
when checking it against the last backup. That means things doing
rapid texture changes, such as objects swapping sculpt maps for
animation, won't be protected, but at least everything else would.
Maintaining a counter is definitely /not/ too expensive given each
object contains at least 3 numbers indicating it's point in space.
Adding one more won't break anybody's bank.

A friend of mine mentioned a problem where you could rez a item,
the sim could crash soon after, and the item would not be there
when the sim came back up, even though there had been no rollback.
In case my transfer check, and rez check, explanations didn't make
it clear, a transfer or rez must be /written to disk/ before it is
considered successful and the object deleted from its source
location if no-copy.

They also mentioned that resetting a script in the head of one of
their avatars causes it to lose all the texture UUIDs, so that is
an example of where losing a script's state can do serious harm.
Perhaps I should be more concerned about script state than I was.
As mentioned before, the only good solution to this seems to be to
allow users to back their items up to their hard drives.

Although it has no bearing on preventing inventory loss, the
comments I made about how Linden Labs makes their money draws a
lot of attention. Several people pointed out that I left out
being charged for uploads. One pointed out that in the case of
group creation, partnering, and uploads, Linden Labs doesn't skim
off the value of the lindens, they take the entire value.
Skimming off value only occurs when lindens are bought, then sold,
such as when a builder cashes out some of the lindens they made
(selling them on the LindenX) from other people buying (with
lindens they bought on the LindenX) their works.

The friend who gave the most feedback also pointed out one valid
use (probably the only one) for coalesced objects. You can use it
to move several objects while retaining their position and
orientation relative to each other. For instance, if you have a
building with several bits of furniture inside, you can select
them all in edit mode, then take them, turning them into a
coalesced object, then go to another sim, and rez the whole thing,
avoiding having to unpack and place everything again. This seems
like enough of an excuse to keep them to me, but there is simply
no excuse for the interface. The icon should be a different
color, and generally look less like the standard tan cube used for
normal objects. I would suggest something like a white glowing
paperwad. The name should have "Coalesced Object" in it (similar
to how the name for a normal object is "Object"). Ideally some of
the contents would be listed after. A coalesced object containing
a table, 4 chairs, and a tea kettle might get the default name
"Coalesced Object: table, chair, chair, chair, chair, tea kettle".
You could, of course, rename it afterward, to, say, "Diningroom
Set". Coalesced Objects should have an appropriate warning and
deletion confirmation. Too many objects get deleted by accident
because the coalesced object is named something like "Chair", and
you delete it thinking you have several copies of that already.
This /does/ merit it being included in the 'inventory loss'
section.

------------------------------------------------------------------

In response to what Lex just posted (as I wrote this), whatever the garbage collection algorithm is (reference counting, mark and sweep...), it should be possible to load a backup into a data structure just like the one the real reference tree is in, replay operations recorded on it, and compare the tree produced by loading the last backup and replaying the log, to the current tree. Somehow on the real tree some items are becoming de-linked, even though no user or script operation should have de-linked them (e.g. a texture may become de-linked, and then get garbage collected, even though someone has a item in their inventory with that texture on it).

The total amount of objects in the database doesn't effect the log size, it's the rate of change. A small number of objects rapidly changing (e.g. avatars swapping sculpt maps to achieve fancy animation) will produce more log data than thousands of static items (e.g. buildings, cloths, most avatar parts, etc). The counter I mentioned in this post should eliminate all arguments against at least trying the log idea. Unfortunately I'm sure it will make handling the 'replay and compare' algorithm much more complex (probably it would look at the items in the 'real' tree, see if they'd hit their max change count, then ignore them and any branches 'past' them when looking for differences between itself and the 'backup+log' tree). I'd rather have the vast majority of my inventory protected, even if I may still lose a few things that rapidly change references, than have no protection at all. What's more, if I /know/ that certain things are more prone to be lost, I'll be more likely to make sure I keep multiple copies around, or avoid buying them.

I don't deny that it will take both some programming, and hardware resources, to make double checking the garbage collector possible. However, to stop the data loss, either it will have to be double checked /somehow/ (if not by my method, some other, and I'm having trouble imagining another that could work), or it will need to be shown to be correct, and I personally can't forsee Linden Labs ever formally proving their garbage collector correct. That's why I didn't even suggest that.

Looks like we agree on the rest though.

Coaldust, if you do want to start solving LL's problems (and PLEASE don't take this as sarcasm, it's a geniune suggestion), I recommend applying for a job with them. Then you'd be able to learn how the system works internally, and you'd be able to present (and perhaps implement) your solution if it's valid or revise it if it isn't. Barring that, I suggest going to LL developers' office hours and seeing if you can engage one of them in a conversation about your suggestions above.

I would like to avoid filling this (already long) conversation area with suggestions of how LL should solve the problem, or complaints that they haven't. I'd rather see us gather more information on this specific type of inventory loss here in hopes that we can help them with their detective work in solving the problem.

A different set of programmers could write another garbage collector (by different set, I mean people who are not involved in the design or maintenance of the original garbage collector, but they can still be from Linden Labs) being given only information on the input and output data-formats and procedure calls.

Wrapper code could then be written to call both garbage collectors when it is time to garbage collect. It should intercept what each marks for deletion. Only if both agree that something should be deleted, should it be deleted. Discrepancies (both disagreements to retain, or to delete, not just disagreements to delete) should be reported to the appropriate maintenance staff to aid in debugging (hopefully revealing patterns in what would be lost, or incorrectly retained).

This idea was inspired by the redundant computers on the space shuttle, and how they 'vote' on decisions, reducing single points of failure. It's unlikely that any two separately developed algorithms will share the same design and implementation flaws.

I believe this would be much more time and space efficient than my original suggestion for correcting the data loss due to garbage collection errors.

I'd like to thank Lex for arguing with me until I came up with a cleaner solution.

Before rezzing one couple poseball, i move the Folder that came after buying the item to my working folder.
On First trial, it said "Failed to place object in specified location"
As it happen very often, and the pose ball just return to the right folder.
On second trial for the same poseball, it said "Item is missing from database".

Then i try to logout and relogin Second Life,
Checking my working folder, then i cannot find the folder for items i bought, so i'm totally loss 13 couple pose balls.

Trying, to clear chace, update to 1.18.4, still i can't found the folder for the items i just bought.

I think this means that we can see a fairly consistent connection between this problem and attachments. I'll update the issue description.

1) 3 textures went missing. The textures's names and UUIDs are in my inventory but when I look at the textures they have no pixels.

Critically for me, all builds using these textures now display missing images instead of the correct textures. I have obtained new copies of the textures from their sources and I asked if I could have the UUIDs replaced with the correct copies. I was told this could not be done. Well, I recommend you figure out how to do it because this could destroy builds all over SL if it happened to more widely used textures than mine. Meanwhile I am having to repair and replace no-mod items.

2) a script that is used in my door locks also went missing. This script has a zero UUID when I try to look at it in my inventory. Again, any doors using this copy of the script will cease to work properly.

3) A house that I packaged using a rezzer contains 14 copies of the same door. It now wont rez because the rezzer gives up after encountering 3 objects missing from the database. It appears that out of the 14 doors, only one is still in the database.

It would be nice to get an update on progress toward identifying and resolving the root cause of the specific class of asset loss described in this jira.

I cleared cache and relogged. I waited from the afternoon to the evening. Nothing worked. I might try the Beta Grid trick just out of curiousity.

I bought these shoes recently (1 month ago) from a vendor in a mall. Given my understanding of how the asset system works - that the objects themselves exist on an asset server and my inventory (and everyone elses) is just a pointer to the same asset - does this mean that every single person that ever bought these shoes has now lost use of them?

It seems like most of the examples of items lost are things that only 1 or 2 people may have had in their inventory (custom created scripts, notecards and objects). But how can something sold through a vendor be lost in this fashion? Wouldn't dozens or hundreds of people have references to that same object?

http://blog.secondlife.com/2008/01/02/resolved-various-inventory-problems-affecting-some-residents/

[9:23] Gesture is missing from database.
[9:23] Gesture is missing from database.
[9:23] Gesture is missing from database.
[9:23] Gesture is missing from database.
[9:24] Gesture is missing from database.
[9:24] Gesture is missing from database.
[9:24] Gesture is missing from database.
[9:24] Gesture is missing from database.
[9:26] Unable to load gesture.
Please try again.
[9:26] Unable to load gesture.
Please try again.
[9:26] Unable to load gesture.
Please try again.
[9:26] Unable to load gesture.
Please try again.
[9:26] Unable to load gesture.
Please try again.
[9:26] Unable to load gesture.
Please try again.
[9:26] Unable to load gesture.
Please try again.
[9:26] Unable to load gesture.
Please try again.
[9:26] Unable to load gesture.
Please try again.
[9:26] Unable to load gesture.
Please try again.
[9:26] Unable to load gesture.
Please try again.
[9:26] Unable to load gesture.
Please try again.
[9:26] Unable to load gesture.
Please try again.
[9:26] Unable to load gesture.
Please try again.
[9:26] Unable to load gesture.
Please try again.
[9:26] Unable to load gesture.
Please try again.
[9:26] Unable to load gesture.
Please try again.
[9:26] Unable to load gesture.
Please try again.
[9:26] Unable to load gesture.
Please try again.
[9:26] Unable to load gesture.
Please try again.
[9:26] Unable to load gesture.
Please try again.
[9:26] Unable to load gesture.
Please try again.

"
Showstopper: ONLY the most severe, confirmed issues which demand immediate attention from Linden Lab. For example, inability for many Residents to login. IMPORTANT: Abusing this setting will cause revocation of Issue Tracker access. If in doubt, mark "Critical" instead.

Critical: Generally, most crashes (particularly if they're easy to reproduce and affect many), content loss, significant memory leaks, greatly reduced performance, etc."

[2008/01/09 23:28] Unable to create requested object. Object is missing from database.

So, count one more for attachment loss. I believe this ussue needs "show stopper" status, as attachments are more often than not things we get from other people (often paid) with no permissions. Second Life is less fun when we lose things we've become attached to (no pun intended) and can't get back.

Also, everyone, I recommend backing up scripts with modify permissions on your own computer. That's the one asset you CAN back up (well, you can back up images too, but that costs money to upload back to SL, so you decide yourself whether you want to), I do that with every single script I have the permissions for that I cannot personally replicate within 10 minutes.

This is a HUD flight assistant with adjustments. I've used it for over a year now, and it is free + open source. All of a sudden i cannot view the script inside it... I can still use the HUD i just can't edit it.

i had this horrible message unabale to create objct object missing form data base

no recovery as clearing cache after waiting solved the problem .
each tim ei lost moeny and a lot of time ... data base problems should be a priority for linden lad and data integrity is essential in such an appilcation.

I use second Life 1.18.5 (3) Nov 28 2007 13:59:53 (Second Life Release) and sometimes also windlight as i remeber i meet this problem sincei use both wiewer may be a coincidence but ...

i need an answer and a solution cost too much

Problem is...you cant do it anymore...because a significant number of the embedded scripts result in "script is missing from database". I have not found a work around for this, but it appears that if ANYTHING is embedded into a notecard...an object, a gesture, a sound, another notecard, or a script, you stand a good chance of losing it.

I will be happy to give a copy of a notecard that demonstrates this, to any linden that actually addresses this issue, after it having been here in the jira for six months.

Thanks

The most common vector of this is definitely related to attachment inventory.

I suspect gestures are treated similarly as attachments with respect to this problem, and might be a good cross-reference in the code to locating the problem.

I did try clearing the cache, re-installing the client, removing removing the user preferences/directory and just doing a fresh install but none of all that made any difference.

When using my seconday ava the same error shows.
However, my wife's account on this very same computer does not seem to show this kind of problem.

Have not yet tried logging in at a different computer (a bit difficult as i am living far away from my closest neighbors), but i do not expect to make any difference. I'm also a bit apprehensive of testing this with other objects.

I ran into this problem with a settings notecard for a script inside a wings attachment. I had to rewrite all the settings. Minor really, but god knows how much more of my inventory this affects.
Cleared cache, restarted SL, restarted and logged into another sim, tried passing notecard to a friend, etc. etc. ad infinitum. No luck. "Notecard is missing from database."

Is anyone asking the important question of "why hasn't this been addressed yet"? Unbelievable and unacceptable, given how much we as a user base have spent on SL over the past year.

--------------------------------------
I think, perhaps a special extra FREE land for all shop owners, so they can rez their products in world... as so far all items I have rezzed (walls, boxes, flowers) have remained safe. (EDIT I refrase that... some TEXTURES on the items I have rezzed are gone now)

If I was to rez all my hairdos (more than 100 prims each) it would eat up all my prim allowances and more

-----------------------------

or how about, instead of the current "unused items are garbaged" system a regular forced inventory clean up day, like every three months?

People would get a popup -warning in advance to TAG each item they REALLY want to preserve on the properties window. And all items untagged would floosh.

Or that ability only for the original creator, so at least SHE would have original items safe.

How about if there were a free public storage sim people could go and rez their objects for safe-keeping?

Or a forced regular - like once every three months- an inventory cleaning day. And there could be a tag-box on permissions one could tag if they wanted to keep the item. A warning in advance and the all items without the tag would floosh. The creators could make it so no DEMO could ever be tagged... in that way the inventory would always be clear from demos

If not a grid-wide perhaps just for the content creators to keep their originals safe.

EDITED: Annnd now the previous comment is here again... odd. I have interviewed several of my friends, and they all seem to have issues... some bigger than others. Also my business partner has lost a lot of the items she made... I REALLY REALLY would want to mark this as a "Showstopper"

Imagine all content creators loosing their inventory and thus be unable to answer customer requests about replacement objects.

If all content creators leave SL what is left??

It may not be as face-hitting as a "multiple people not able to log in right now"
but it will eventually lead to people not WANTING to log in... as they have lost all their tools (scripts, textures)
all their creations (imagine an artist to loose their paintings) everything what they are (shapes, skins, gestures, attachements) so far all what is left are the friendships... but I imagine those to shatter as when the content creators go
the friends who have nothing left will also go.

A real transaction management middle tier system might help too.

But if the databases are actually losing records then it is time to get a real database with real data architects and real DBAs
(assuming of course there are none of these resources in place at LL. We can't tell since there is no published staff/resume to accountability matrix. I have never seen any posted vacancies for DBAs or Data Architects. Programmers are not capable of performing these roles effectively because they think differently and simply are not allocated to such 24*7 mandatory work.)

I would love to be a part and rework the data architecture myself. That would be a real challenge worth doing right.

Isn't that hard to do.

The best way to clean out the scum from SL is to lay traps and let them walk into them.

Oh and it (local database and encryption/access libs) can't be open source either. Someday LL will realize not all code has to be open source. In this case the open source viewers would not have the feature of backed up inventory and people using them would still be subject to the inventory loss problems. That is the price they pay for using a non secure viewer. Customer choice.

She came to the conclution the sim I was in (my home sim which is a mainland parcel) had to be reseted.
She reseted my sim... and poof all my inventory items I have checked so far have come back working again!!

I figured it was because in some way the parcel servers (= the sims) act as a router from inventory requests to the asset servers...

as each copy of a item in your inventory is not the actual item-- it is merely a "fast icon" that refers to the original, which is saved on the asset server. So there is only one item of a kind in the whole SL... but there can be zillion copies of fasticons... or something like that.

And if the copy is changed enough, it would create a new original... nevertheless, the sim acts as a router... and if it has it´s wires crossed the link fails.

I suggest grid-wide sim resetting

it affects not only my gowns but also things i want to rezz like trees furniture effects animals pictures ect ect ect .
I would say more than 1/3 of my inventory is not usable anymore.

I investigated and sevral of my freinds experience the same , i had it before but now since last two updates it is really critical
and absolut not funny anymore, it pulls my nerves and makes me angry , after all there is money behind it !!!!!!! I sure hope it will be resolved soon