Introduction

In today's digital world, file storage and management is one of the fundamental requirements of every application. Choosing and managing between different storage solutions like AWS S3, MinIO, and local storage can be a complex process. In this article, we will examine a flexible bucket service developed with Node.js that supports multiple storage providers.

Project Overview

This bucket service is a comprehensive solution designed to meet the file management needs of modern web applications. The service supports three different storage types:

• AWS S3: Cloud-based, scalable storage

• MinIO: Open-source, S3-compatible object storage

• Local Storage: For development and testing environments

Technical Architecture

Technology Stack

{
  "dependencies": {
    "@aws-sdk/client-s3": "^3.812.0",
    "@aws-sdk/s3-request-presigner": "^3.823.0",
    "express": "^4.18.2",
    "sequelize": "^6.37.7",
    "minio": "^8.0.5",
    "multer": "^1.4.5-lts.2",
    "jsonwebtoken": "^9.0.2",
    "uuid": "^11.1.0"
  }
}

Database Model

The service works on two main models using PostgreSQL database:

Bucket Model

{
  id: UUID (Primary Key),
  bucketName: String (Not Null),
  isActive: Boolean (Default: true)
}

FileStore Model

{
  id: UUID (Primary Key),
  ownerId: UUID (Not Null),
  fileId: UUID,
  originalName: String (Not Null),
  mimeType: String,
  extension: String,
  size: Integer,
  bucketName: String,
  path: String,
  isPublic: Boolean (Default: false),
  isActive: Boolean (Default: true),
  status: String
}

Key Features

1. Multi-Provider Support

The service can dynamically switch between different storage providers based on the BUCKET_TYPE parameter determined by the environment variable:

const bucketType = process.env.BUCKET_TYPE;
const validBucketTypes = ["aws", "minio", "local"];

if (!validBucketTypes.includes(bucketType)) {
  throw new Error(`Invalid BUCKET_TYPE: ${bucketType}`);
}

2. Secure File Upload

The service secures file upload operations using a JWT-based authorization system:

const operatingBucket = buckets?.find((b) => b.bucketName === bucketName);
if (!operatingBucket || !operatingBucket.action?.some((a) => a === "write")) {
  return next(NotAuthorizedError("youDontHaveAnyPermission"));
}

3. Automatic Bucket Management

Buckets are automatically checked and created when necessary:

async function checkBucket(bucketName) {
  try {
    await client.send(new HeadBucketCommand({ Bucket: bucketName }));
  } catch (err) {
    if (isNotFound(err)) {
      await createBucket({ bucketName });
    }
  }
}

API Endpoints

1. Health Check

Checks the service status and connection information.

Response:

{
  "status": "ok",
  "message": "S3 connection successful",
  "bucketType": "aws",
  "bucketCount": 5
}

2. File Upload

Supports multiple file upload operations (maximum 10 files).

Request Parameters:

• userId: User ID

• buckets: Authorization information

• files: Files to upload (multipart/form-data)

Response:

{
  "success": true,
  "data": [
    {
      "fileId": "uuid-string",
      "originalName": "document.pdf",
      "mimeType": "application/pdf",
      "size": 1024000,
      "status": "uploaded",
      "downloadUrl": "http://localhost:3900/download/uuid-string"
    }
  ]
}

3. File Download

Secure file download operation. Provides temporary access using signed URLs.

Security Features

JWT-Based Authorization

The service performs user authentication and authorization using JSON Web Token:

const token = jwt.sign({
  userId: "user-uuid",
  buckets: [{
    action: ["read", "write"],
    bucketId: "bucket-uuid",
    bucketName: "user-bucket",
    isPublic: false
  }]
}, secretKey);

Permission Controls

Granular permission control is performed for each operation:

• Read: File read/download permission

• Write: File write/upload permission

Signed URLs

File download operations are performed using temporary and secure signed URLs.

Docker Development Environment

The project includes ready Docker Compose configuration for MinIO:

version: "3.8"
services:
  minio:
    image: minio/minio:latest
    container_name: minio
    ports:
      - "9000:9000"
      - "9001:9001"
    environment:
      MINIO_ROOT_USER: minioadmin
      MINIO_ROOT_PASSWORD: minioadmin
    volumes:
      - ./data:/data
    command: server /data --console-address ":9001"

Installation and Usage

1. Install Dependencies

npm

2. Set Environment Variables

# .env file
NODE_ENV=development
BUCKET_TYPE=minio  # aws, minio, local
BUCKET_NAME=your-bucket-name
BASE_URL=http://localhost:3900

# For AWS (if aws is selected)
AWS_REGION=us-east-1
AWS_ACCESS_KEY_ID=your-access-key
AWS_SECRET_ACCESS_KEY=your-secret-key

# For MinIO (if minio is selected)
MINIO_ENDPOINT=localhost
MINIO_PORT=9000
MINIO_ACCESS_KEY=minioadmin
MINIO_SECRET_KEY=minioadmin

# Database
DB_HOST=localhost
DB_PORT=5432
DB_NAME=bucket_service
DB_USER=postgres
DB_PASSWORD

3. Start the Service

npm start

Advantages

1. Flexibility

• Support for three different storage types

• Easy provider switching

• Environment-based configuration

2. Security

• JWT-based authentication

• Granular permission system

• Secure access with signed URLs

3. Scalability

• Multiple file uploads

• Asynchronous operations

• Database-based metadata management

4. Developer Friendly

• RESTful API design

• Comprehensive error handling

• Docker support

Use Cases

1. E-commerce Platforms: Product images and documents

2. Content Management Systems: Media files

3. Enterprise Applications: Document archiving

4. SaaS Applications: User files

Future Enhancements

• File Preview: PDF, image preview support

• File Compression: Automatic compression features

• CDN Integration: Faster file access

• Bulk Operations: Bulk file operations

• Audit Logging: Detailed operation logs

Conclusion

This bucket service offers a comprehensive and flexible solution that meets the file management needs of modern web applications. It provides great convenience to developers with its multi-provider support, secure API design, and easy integration features.

Thanks to its open-source nature and modular architecture, the project can be customized and extended according to different needs. It is an ideal file management solution especially for teams working in microservice architecture.