File Management Service

Table of Contents

1. Introduction
2. Technology Stack and Features
3. Why a Separate File Management Service?
4. How to Use it?
5. API Endpoints
6. Contributing

Introduction

This microservice is designed to manage all file-related tasks. It uses MinIO for object storage and MySQL for managing file metadata. We support chunk uploads for handling large files efficiently, with Celery running background tasks to ensure smooth performance.

Technology Stack and Features

• ⚡ FastAPI for the Python backend API.
• 🧰 SQLAlchemy for the Python SQL database interactions (ORM).
• 🔍 Pydantic, used by FastAPI, for the data validation and settings management.
• 🗄️ MYSQL as the SQL database.
• 🔄 Alembic for database migrations.
• 🔧 Celery with RabbitMQ for task queue management and background processing.
• 💾 MinIO for scalable object storage with chunk upload support.
• ✅ Pytest for testing to ensure code reliability and functionality.
• 🐋 Docker Compose for development and production.

Why a Separate File Management Service?

1. Centralizes file operations, making management and maintenance easier.
2. Enables scaling file handling independently of other services.
3. Simplifies updates and changes to file handling without impacting other parts of the system.
4. Reduces code duplication by keeping file upload and retrieval logic in one place, resulting in cleaner code.

How to Use it?

1. Complete the .env File:

   • Copy the contents of .env.example to a new file named .env.
   • Fill in the required environment variables based on your setup.

2. Build the Docker Image:

   • Run the following command to build the Docker image:

     docker compose build

3. Run the Containers:

   • After the build is complete, start the containers in detached mode with:

     docker compose up -d

4. Migrate the Database:

   • Access the running container to perform the database migration:

     docker compose exec filemanager bash

   • Inside the container, run the migration using Alembic:

     alembic upgrade head

5. Access the Service:

   • The project is now up and running, accessible on port 8000.
   • You can access the project documentation by navigating to /docs on your browser.

API Endpoints

Here’s a quick reference guide to the available API endpoints, their methods, and what they do:

Method	URL	Description
POST	/api/v1/file/upload/init/	Initialize a new file upload session.
POST	/api/v1/file/upload/chunk/	Upload a file chunk.
POST	/api/v1/file/upload/complete/	Complete the file upload process.
GET	/api/v1/file/get/{file_id}	Retrieve a file by its ID.
GET	/api/v1/file/status/{file_id}	Check the upload status of a file.
POST	/api/v1/file/upload/retry	Retry uploading a file.

A Postman collection export is also available for testing these endpoints. You can import it into Postman to quickly get started with API testing.

Contributing

We welcome contributions from everyone! If you have ideas for improvements, new features, or bug fixes, feel free to contribute to this project. Here's how you can get involved:

1. Create an Issue:

   • If you find a bug, have a question, or want to suggest a feature, please open an issue. This helps us track and discuss your ideas.

2. Send a Pull Request (PR):

   • Fork the repository, make your changes in a new branch, and then create a pull request.
   • Please make sure your code follows the project's coding standards and passes all tests.

We appreciate your contributions and will do our best to review and merge your pull requests promptly. Thank you for helping us improve this project!