Annotations Lost After Network failure for around 2 mins.

[abhijith96]

Hi I have around 10000 images in an azure blob storage container. There are intermittent network connection issues and some times the the call to azure blob store fails. But during that time some of the annotations are lost.

To Reproduce
Steps to reproduce the behaviour:

Setup data import form azure blob store. Use a container prefix path.
Annotate a few files.
Simulate a network error (Turn off the internet connection so that the get request from azure blob fails)
Upto 50% of the annotations will be missing after refreshing.
My assumption was that all the annotations will be persisted in the sql lite database in my local storage. Losing half of the data that I annotated is causing great pain. During the first network connection error the task number I was annotating was 2003. Then after refresh the annotated task numbers where from 1 to 500 only. Rest of the tasks did not have any annotations.

OS: Macos
Label Studio Version 1.14.0.post0

[heidi-humansignal]

Hello,

This is when you have target storage setup? or it just source storage that causes these issue?

I would also suggest:

Switch to a PostgreSQL Database:
Label Studio uses SQLite by default, which might not be ideal for large projects with many tasks. SQLite can have limitations with concurrent access and large datasets, potentially leading to data integrity issues during network interruptions. Switching to a PostgreSQL database can provide better performance and reliability.
You can find instructions on how to set up Label Studio with PostgreSQL in our documentation:

https://docs.humansignal.com/guide/storedata.html#PostgreSQL-database

Thank you,
Abu

Comment by Abubakar Saad
Workflow Run

[abhijith96]

Hi ,
Thanks for getting back to me.
I have an azure blob storage as source.
Here is a snippet from source storage settings:
"Status Completed
Tasks 10734 (0 new)
Last Sync November 15, 2024 ∙ 22:37:38"

I also have azure blob container in the same storage account as target storage.
"Status Completed
Annotations 0 (0 total)
Last Sync November 15, 2024 ∙ 22:33:17"
"
I have not synced it yet after starting annotations.
I also checked the sqlite tables for annotation information. It seems that some of the annotations that are missing have been cleared from the sql lite database. And there are some duplicate entries for some annotated files. All this happened after the network disconnection.
I will try Postgres SQL if it helps. But I am not sure whether switchting to Postgres SQL will solve the issue.

[heidi-humansignal]

Hello,

The problem may be related to using SQLite as the backend database. SQLite is suitable for small projects and testing purposes but can encounter issues with larger datasets and network instability. Since you're working with over 10,000 tasks, switching to a more robust database like PostgreSQL is highly recommended.

Let us know how that goes.

Thank you,
Abu

Comment by Abubakar Saad
Workflow Run

[Bradly-s]

I use PostgreSQL for over 25000 tasks.
However, this issue still exists.
How to solve it?

[heidi-humansignal]

Hello,

Ohh, sorry I think I misunderstood that. If you had target storage attach, it should have most of the annotation because it should sync after every submit. Depending on how long did disconnection lasted for, the rest of them should have stored in sqlite. But, if they were too many, I think thats where the lost of data occurred? I'm not sure if you can retreive those, let me find out about it

Thank you,
Abu

Comment by Abubakar Saad
Workflow Run

[joycepg]

we have the same problem.
azure deployment using script from here
setup using postgresql database
configured source and target azure blob storage
synced, created around 250 annotations from 2000 tasks.
downloaded the annotations (thankfully)
came back a few days later, and the web app had been automatically stopped by azure through inactivity. it took over 10 minutes to start up, but when it did, it had lost everything (user signups, the source and target connections, and therefore all our work).
this re-initializing after idle has happened multiple times. it makes the software close to useless.

(edit) all the information is in the azure blob storage still, and we can reconnect and resync the images, but if we reconnect and resync the target container, it does not pick up the work already done, even though the blobs are there.

[heidi-humansignal]

Hello,

(edit) all the information is in the azure blob storage still, and we can reconnect and resync the images, but if we reconnect and resync the target container, it does not pick up the work already done, even though the blobs are there.

Did the postgres db reinitialized as well? how was that setup? I think it doesn't pick up because the name might have changed for those images if you had to resync it.

Thank you,
Abu

Comment by Abubakar Saad
Workflow Run