Improve database consistency

[appgurueu]

Our database consistency is bad currently. In case of a hard crash, the different databases can easily become inconsistent. The root of the problem here is that everything lives in a separate database - auth, players, mod storage, map - and sometimes even differs in save logic.

The consequences from this range from duplication bugs (player inventory is already persisted; node inventory is not, server is made to crash) to problems with rideable creatures (#15605) etc.

For consistency, I believe it would be ideal if everything used a single database such that we can properly use database consistency features (transactions) to ensure that either all the changes during a map save interval or none of them are saved.

Of course, there are practical problems with this since we want to support multiple, different backends, but given that by default everything uses SQLite, we could make it work at least for this case.

(Additionally, we should encourage modders to avoid storing data in separate files: They should use mod storage for consistency instead.)

[sfan5]

problems with rideable creatures (#15605) etc.

not a good example as that "bug" only involves a single database (the map)

[appgurueu]

not a good example as that "bug" only involves a single database (the map)

Does it? Player positions are stored in the player database, entity positions are stored in the map. These can't be updated in the same transaction.

[sfan5]

Does it? Player positions are stored in the player database, entity positions are stored in the map. These can't be updated in the same transaction.

Yes. This is just an edge case of re-save decision logic for entities. Not a consistency issue or race condition.

[lhofhansl]

Unless you write everything in a single transaction you'd have the same problem, no?

[appgurueu]

Well, it probably wasn't the best idea to subsum save logic consistency and database consistency problems in this issue.

I think what sfan is saying is that the entity issue is currently so bad primarily because the re-save logic differs.

I bet we have more inconsistent logic since our current architecture doesn't discourage it. Forcing everything into a single transaction would pretty much force consistency.

The issue can't technically be fully fixed without doing everything in the same transaction, but it would probably be far less likely, since you'd need Luanti to be killed while one transaction is in progress and another has already completed, which is unlikely since most of the time the databases should be between save intervals.

[sfan5]

The issue in the opening text is real, I am just here to disagree with one of the examples given.