Synchronization of Local Storage Results in Database Error #7170
Comments
|
Hello, Thank you for contacting Label Studio, I understand that you're experiencing a "database is locked" error during the synchronization process in Label Studio, particularly after approximately 34 minutes. I appreciate the detailed information you've provided. This issue is likely occurring due to SQLite’s handling of concurrent write operations, especially given the large the high volume of tasks (over 163,000 files).Since SQLite locks the entire database during writes, prolonged operations can lead to conflicts, preventing the sync from completing successfully. to mitigate this, we can recommend to divide the workload into smaller batches. Instead of importing all tasks at once, try breaking them into smaller sets and uploading them incrementally. This approach will reduce the likelihood of database locks and improve stability. Also, you can switch your database backend to a production-grade system such as PostgreSQL. PostgreSQL is designed to handle multiple concurrent transactions more efficiently and will help prevent these locking issues. Please let us know if you need further assistance.
|
Describe the bug
During the synchronization process, an error consistently occurs after approximately 34 minutes, indicating that the database is locked and cannot be accessed. This prevents the successful completion of the synchronization.
To Reproduce
Steps to reproduce the behavior:
Curl Response
{ "id": 2, "type": "localfiles", "synchronizable": true, "path": "/mnt/files_ki/predict", "regex_filter": ".*(jpe?g|png)", "use_blob_urls": true, "last_sync": "2025-03-05T13:18:25.399310Z", "last_sync_count": 0, "last_sync_job": null, "status": "completed", "traceback": "Traceback (most recent call last): File \"/label-studio/.venv/lib/python3.12/site-packages/django/db/backends/utils.py\", line 105, in _execute return self.cursor.execute(sql, params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File \"/label-studio/.venv/lib/python3.12/site-packages/django/db/backends/sqlite3/base.py\", line 354, in execute return super().execute(query, params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sqlite3.OperationalError: database is locked The above exception was the direct cause of the following exception: Traceback (most recent call last): File \"/label-studio/label_studio/io_storages/base_models.py\", line 461, in sync import_sync_background(self.__class__, self.id) File \"/label-studio/label_studio/io_storages/base_models.py\", line 490, in import_sync_background storage.scan_and_create_links() File \"/label-studio/label_studio/io_storages/localfiles/models.py\", line 100, in scan_and_create_links return self._scan_and_create_links(LocalFilesImportStorageLink) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File \"/label-studio/label_studio/io_storages/base_models.py\", line 401, in _scan_and_create_links task = self.add_task(data, self.project, maximum_annotations, max_inner_id, self, key, link_class) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File \"/label-studio/label_studio/io_storages/base_models.py\", line 325, in add_task task = Task.objects.create( ^^^^^^^^^^^^^^^^^^^^ File \"/label-studio/.venv/lib/python3.12/site-packages/django/db/models/manager.py\", line 87, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File \"/label-studio/.venv/lib/python3.12/site-packages/django/db/models/query.py\", line 679, in create obj.save(force_insert=True, using=self.db) File \"/label-studio/label_studio/tasks/models.py\", line 531, in save super().save(*args, update_fields=update_fields, **kwargs) File \"/label-studio/.venv/lib/python3.12/site-packages/django/db/models/base.py\", line 892, in save self.save_base( File \"/label-studio/.venv/lib/python3.12/site-packages/django/db/models/base.py\", line 998, in save_base updated = self._save_table( ^^^^^^^^^^^^^^^^^ File \"/label-studio/.venv/lib/python3.12/site-packages/django/db/models/base.py\", line 1161, in _save_table results = self._do_insert( ^^^^^^^^^^^^^^^^ File \"/label-studio/.venv/lib/python3.12/site-packages/django/db/models/base.py\", line 1202, in _do_insert return manager._insert( ^^^^^^^^^^^^^^^^ File \"/label-studio/.venv/lib/python3.12/site-packages/django/db/models/manager.py\", line 87, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File \"/label-studio/.venv/lib/python3.12/site-packages/django/db/models/query.py\", line 1847, in _insert return query.get_compiler(using=using).execute_sql(returning_fields) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File \"/label-studio/.venv/lib/python3.12/site-packages/django/db/models/sql/compiler.py\", line 1836, in execute_sql cursor.execute(sql, params) File \"/label-studio/.venv/lib/python3.12/site-packages/sentry_sdk/utils.py\", line 1796, in runner return sentry_patched_function(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File \"/label-studio/.venv/lib/python3.12/site-packages/sentry_sdk/integrations/django/__init100 5711 100 5711 0 0 2 0 0:47:35 0:34:53 0:12:42 1396^^^^^^^^^^^^ File \"/label-studio/.venv/lib/python3.12/site-packages/django/db/backends/utils.py\", line 79, in execute return self._execute_with_wrappers( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File \"/label-studio/.venv/lib/python3.12/site-packages/django/db/backends/utils.py\", line 92, in _execute_with_wrappers return executor(sql, params, many, context) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File \"/label-studio/.venv/lib/python3.12/site-packages/django/db/backends/utils.py\", line 100, in _execute with self.db.wrap_database_errors: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File \"/label-studio/.venv/lib/python3.12/site-packages/django/db/utils.py\", line 91, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File \"/label-studio/.venv/lib/python3.12/site-packages/django/db/backends/utils.py\", line 105, in _execute return self.cursor.execute(sql, params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File \"/label-studio/.venv/lib/python3.12/site-packages/django/db/backends/sqlite3/base.py\", line 354, in execute return super().execute(query, params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ django.db.utils.OperationalError: database is locked", "meta": { "attempts": 15, "time_queued": "2025-03-05 12:43:32.588594+00:00", "time_in_progress": "2025-03-05 12:43:32.592662+00:00", "time_last_ping": "2025-03-05 13:18:25.114872+00:00", "duration": 2092.806653, "tasks_existed": 150324, "time_completed": "2025-03-05 13:18:25.399315+00:00" }, "title": "Input", "description": null, "created_at": "2025-02-19T09:07:57.571395Z", "project": 1 }Trace
Screenshots
If applicable, add screenshots to help explain your problem.
Environment (please complete the following information):
Additional context
File Count in Folder: 163348
Synced Count: 150324
Storage Configuration:
nginx additional configuration:
The text was updated successfully, but these errors were encountered: