• All submissions to this site are governed by Second Life Project Contribution Agreement. By submitting patches and other information using this site, you acknowledge that you have read, understood, and agreed to those terms.
Issue Details (XML | Word | Printable)

Key: SVC-1149
Type: New Feature New Feature
Status: Open Open
Priority: Critical Critical
Assignee: Unassigned
Reporter: Escort DeFarge
Votes: 3
Watchers: 6
Operations

If you were logged in you would be able to see more operations.
2. Second Life Service - SVC

Data Integrity and the Implementation of the MySQL RDBMS Instances

Created: 04/Jan/08 08:12 AM   Updated: 21/Jun/09 01:52 PM
Return to search
Component/s: None
Affects Version/s: 1.18.6 Server
Fix Version/s: None

Environment: All
Issue Links:
Relates
 


 Description  « Hide
MySQL is known to sacrifice data integrity constraint checks in favor of speed (i.e. speed at low to medium user load). Second Life has moved well beyond servicing a "medium user load", and thus forcing reliance into application-level constraint checking code is both bug-prone and no longer appropriate. Additionally, at high load, the speed advantage offered by MySQL's traditional paradigm (of ignoring or not implementing referential constraints) is no longer applicable.

This is a request that ALL OF the following information be carefully reviewed and action taken where necessary:

http://dev.mysql.com/tech-resources/articles/mysql-data-integrity.html
http://wiki.postgresql.org/wiki/Why_PostgreSQL_Instead_of_MySQL:_Comparing_Reliability_and_Speed_in_2007
http://use.perl.org/~Smylers/journal/34246
http://tweakers.net/reviews/657/6

Questions to verify during the requested review are:
1) Do LL's database deployments use InnoDB not MyISAM tables?
2) If so, do the DB schema deployed utilize MySQL 5.x referential constraints?
3) Do the DB schema definitions take account of the fact that MySQL may ignore certain specified constraints 'silently'?



 All   Comments   Change History      Sort Order: Ascending order - Click to sort in descending order
Escort DeFarge added a comment - 04/Jan/08 08:31 AM - edited
Added a link to the Meta-Issue of Inventory Loss for which the above may point to the Meta-Solution.

Mercia Mcmahon added a comment - 04/Jan/08 09:33 AM
To prevent this being closed as "nothing to do here" I suggest making the Questions requests to implement referential constraints etc

Escort DeFarge added a comment - 04/Jan/08 10:52 AM
Minor edits to clarify the request as suggested by Mercia.

Lex Neva added a comment - 05/Jan/08 10:26 AM
Reading the first link, it looks to me like you're overstating the problem here. The way you phrased it makes it seem like MySQL is trying as hard as it can to process queries as quickly as possible, and might end up, yanno, corrupting the database on you in the name of speed. Oopsies! That's not the case at all, though. The first article clearly explains that mysql < 5.0 just doesn't actually throw errors if you give it a string instead of a number. A careful programmer should ALWAYS validate data provided by users before sending it to the database, so this is a non-issue. It's not like the database is going to spontaneously corrupt itself when you're not looking.

I'm pretty sure I remember an announcement from LL awhile back that their MySQL databases were being upgraded to 5.0. I also know that the main asset database, which stores all of the underlying data for images, objects, gestures, etc, is NOT running on MySQL like many people seem to think.

I think you're being alarmist here, and raising an alarm when you don't have enough information to do so. It's pretty likely that LL already handles this situation either through application-level checks or mysql 5.0 strict data type constraints. Why not leave this to the professionals at LL who certainly have already made this design decision?


Escort DeFarge added a comment - 05/Jan/08 12:03 PM
Goodness me, Lex! It's postings like your above comment that can start mini flame wars! I think you are reading a critical tone into my request that simply is not there.

On Topic:
1) MySQL, historically and paradigmatically, does sacrifice integrity checks for speed. I really don't think anybody with any experience of RDBMS systems would argue otherwise.
2) For sure, "a careful programmer" should validate data at the presentation and business tiers, however in large scale distributed application development it is impractical to think that reliance on this form of validation is ENOUGH. Hence, I do not think that integrity checks in the database are a "non-issue" at all.
3) Implementing MySQL 5.0 does not mean that referential checks are added to the schema without additional work. You'll note also that STRICT mode can be bypassed on a per conection basis, something that realistically WILL BE DONE at some point by a developer on a deadline.
4) If the information has been reviewed and the "design decisions" already taken then the Lindens can validly close this request. However, the existence of SVC-114 and the large number of directly related issues very strongly suggests a serious data integrity issue.

Off Topic:
1) I am trying to be a solution provider here not a critic. I am offering a valid first step towards the long-term solution of SVC-114 et al. If you have better information and can offer a more effective first step then please offer that as an alternative.
2) Like most other users, you are correct that I have no insight into the platform beyond publicly available information.
3) I am a professional enterprise application developer with over 15 years of experience of large-scale systems.


Fluf Fredriksson added a comment - 05/Jan/08 12:23 PM
Meanwhile... I'm happily watching this issue for the eventual LL feedback it deserves.
Like Lex and Escort, I have no idea if all, some or none of the issues raised have been dealt with already.
I also have no idea if the asset server system is running on MySQL, or if it is, which version of MySQL.
But the entry does raise (in my view) valid concerns that are worth double checking even if it's just a trivial matter for LL to say "done that already", or "not relevant but thanks for asking".

Lex Neva added a comment - 05/Jan/08 12:32 PM
Sorry... I guess I did get a little antagonistic in that. I didn't mean to start a flame war.

I guess I just mean to get across these basic ideas:

1) Neither of us really knows what's going on behind the scenes, like you said.
2) I tend to assume that LL developers have enough professional experience to know about basic best practices like this.
3) From the outside, I don't think there's any evidence that a lack of strict data checking in the database is what's causing any of the various forms of inventory loss.
4) It might actually be hurtful and/or distracting to make suggestions like this, because it implies that LL developers haven't taken obvious steps that any responsible professional system designer would.

I'm reacting less to you, and more to a specific kind of comment I've often seen directed from users to LL. Maybe this issue isn't even an example of it. I'm talking about the way that SL users with technical experience look at all of the problems happening on the grid and assume that, from the outside, without any specific knowledge of how the backend is implemented, and in a very short period of time, they can quickly point out what LL must be doing wrong. Often, when reading comments like that, I think that if it were really that simple, LL would have come up with it just as quick as the commenter. I'd much rather see people like that apply to work at LL and get in the backend and make much more educated suggestions

Then again, I don't have 15 years of enterprise experience... so maybe this isn't as obvious a suggestion as I think.


Escort DeFarge added a comment - 05/Jan/08 12:50 PM
Take for example a valid SQL97 column declaration like...
agent_id varchar(36) references agent(id),
...which would be parsed, throw no warnings or errors, but the foreign key constraint defined there would be ignored by MySQL!

So I don't really think that the issue is necessarily all that obvious. I collected the pertinent evidence together for review, so it's not unlikely that some of these "quirks" have been missed previously. With regard to actually "fixing" SVC-114, I believe that it will very likely be a multi-step process. The first issue is to discover the actual cause. A famous fictional character once said: "Eliminate the impossible, and whatever remains, however improbable, must be the truth".


Lex Neva added a comment - 05/Jan/08 05:12 PM
I actually was aware that foreign keys in MySQL are ignored, even though I'm not a heavy MySQL user (just personal projects).

Anyway, I've raised my concerns, so I'm willing to let them rest


Escort DeFarge added a comment - 05/Jan/08 05:39 PM - edited
Actually if you declare...

agent_id varchar(36),
foreign key (agent_id) references agent(id)

...then the FK declaration isn't ignored by MySQL (4.x and later, anyhow).

Just a little history: I've been at the merchant end of inventory loss issues daily for months. I had hoped and trusted (as you seem to now) to see a resolution - but if anything the issue is apparently getting worse as the concurrent user base grows. All I can do (without changing jobs and moving my family to San Francisco) is offer what I can to assist what must be a hugely time consuming effort inside Linden Lab development and that daily absorbs the time of my customers inside SL (most of whom are merchants).


Lex Neva added a comment - 05/Jan/08 05:52 PM
Oh, interesting. I thought even foreign key lines like that were ignored.

And yeah... I'm a merchant too, and I've gotten a fair amount of pain from inventory loss, not to mention losing the only copy of a couple of scripts that were VERY important I'm not trying to detract from the need for a resolution to inventory loss... I'm just trying to say that I'm not sure how helpful it is for LL to have to answer "Did you try ___?" from a lot of talented engineers who don't have access to the backend... especially when, in my opinion, it's likely that they've thought of everything we would and more. I'd rather see them spend that time thinking of more things to try.


Delerium Hannibal added a comment - 31/Jan/08 04:07 AM
After reading the comments back and forth, I think Lex is taking one thing for granted.

1. You can't just sit there and assume that they have thought of it. Remember that they teach you back in early early school, "The only stupid question is the one not asked."

2. Collective ideas can be better. Just because Linden Labs has talented staff doesn't mean that they have thought of EVERYTHING. If that was the case then SL would be perfect and flawless... Don't discount the ideas of others by saying "I'm sure they have already thought of that" without siting sources where they have stated that they thought of that already. If the comments from talented engineers aren't welcomed then linden labs wouldn't even have this bug reporting system put into place.

this line caught my eye the most:
"especially when, in my opinion, it's likely that they've thought of everything we would and more. I'd rather see them spend that time thinking of more things to try."

It reminds me of a line I once heard. "Whenever you think you are the best, someone better proves you wrong."

I'm not saying that in regards to you, but more in regards to your view of linden labs. I assure you that it's more likely that they haven't thought of everything we would and more, simply because "we" is a much bigger collective with alot more ideas than the few of them. I'd rather them hear out our ideas and judge which ones are worth trying with the added knowledge they have of their inner system.


Escort DeFarge added a comment - 28/Sep/08 09:31 AM
Updated moved links

Ann Otoole added a comment - 21/Jun/09 01:52 PM
It is wise to disconnect any assumptions of assets residing in a properly architected database. They live in home grown XML called LLSD on a fast file server system.

The inventory databases are mysql. I doubt there is any RI anyway given the complete vacuum of data architecture resources at LL.

The rest cannot be said because of what cannot be said.

All that can be said is LL now has someone in a senior position at LL that used to work for Oracle. Whatever actions are required to introduce actual metadata and data architecture are not out of order.