File upload and storage

Files are not like regular API payloads. A JSON request body is a few kilobytes. A file upload can be gigabytes. This difference in scale changes how you design the backend. You cannot stream a 2 GB video through your application server the same way you process a login request. This article covers the patterns that make file storage reliable, fast, and secure.

Direct upload vs presigned URLs

There are two fundamental approaches to handling file uploads.

Direct upload

The client sends the file to your application server, which validates it and forwards it to object storage (S3, GCS, Azure Blob).

sequenceDiagram
  participant C as Client
  participant API as API Server
  participant S3 as Object Storage

  C->>API: POST /upload (file in request body)
  API->>API: Validate file type and size
  API->>S3: PutObject (file bytes)
  S3-->>API: 200 OK
  API->>API: Save metadata to database
  API-->>C: 201 Created (file URL)

Direct upload. The file passes through the application server.

Advantages: simple to implement, easy to add validation and virus scanning inline.

Disadvantages: your application server handles all the bandwidth. A 1 GB file ties up a worker process for minutes. Ten concurrent uploads can exhaust your server’s memory.

Presigned URL upload

The client asks your API for a presigned URL. Your API generates a URL that grants temporary write access to a specific object storage location. The client uploads directly to object storage, bypassing your server entirely.

sequenceDiagram
  participant C as Client
  participant API as API Server
  participant S3 as Object Storage

  C->>API: POST /uploads/initiate (filename, content_type, size)
  API->>API: Validate request, generate object key
  API->>S3: Generate presigned PUT URL (expires in 15 min)
  S3-->>API: Presigned URL
  API->>API: Save upload record (status: pending)
  API-->>C: 200 OK (presigned URL + upload ID)

  C->>S3: PUT file (using presigned URL)
  S3-->>C: 200 OK

  C->>API: POST /uploads/{id}/complete
  API->>S3: HeadObject (verify file exists and matches expected size)
  S3-->>API: Object metadata
  API->>API: Update upload record (status: completed)
  API-->>C: 200 OK (file URL)

Presigned URL upload. The file goes directly to object storage. The API server only handles metadata.

Advantages: your server handles zero file bytes. Upload bandwidth scales with object storage (practically unlimited), not your application servers.

Disadvantages: more complex flow, harder to add inline validation (you validate after upload).

For any production system handling files larger than a few megabytes, presigned URLs are the right choice.

Chunked upload for large files

Large files (hundreds of megabytes to gigabytes) should not be uploaded in a single HTTP request. If the connection drops at 95%, the entire upload must restart. Chunked upload splits the file into parts and uploads each independently.

S3’s multipart upload works like this:

1. Initiate: call CreateMultipartUpload to get an upload ID.
2. Upload parts: upload each chunk (5 MB to 5 GB) with a part number. Each chunk gets its own presigned URL.
3. Complete: call CompleteMultipartUpload with the list of part numbers and ETags.

// Server-side: generate presigned URLs for each chunk
async function initiateChunkedUpload(fileSize, chunkSize = 10 * 1024 * 1024) {
  const numChunks = Math.ceil(fileSize / chunkSize);
  const { UploadId } = await s3.createMultipartUpload({
    Bucket: 'my-bucket',
    Key: objectKey,
  });

  const presignedUrls = [];
  for (let i = 1; i <= numChunks; i++) {
    const url = await getSignedUrl(s3, new UploadPartCommand({
      Bucket: 'my-bucket',
      Key: objectKey,
      UploadId,
      PartNumber: i,
    }), { expiresIn: 3600 });
    presignedUrls.push({ partNumber: i, url });
  }

  return { uploadId: UploadId, presignedUrls };
}

Chunked uploads keep the effective upload time nearly linear with file size, even when failures occur. Single uploads waste all progress on every failure.

Resumable uploads

Resumable uploads build on chunked uploads by tracking which chunks have been successfully uploaded. If the client disconnects, it can query the server for the list of completed parts and resume from where it left off.

The tus protocol is an open standard for resumable uploads. It uses HTTP headers to communicate upload progress:

// Client creates an upload
POST /files HTTP/1.1
Upload-Length: 1073741824
Tus-Resumable: 1.0.0

// Server responds with upload URL
HTTP/1.1 201 Created
Location: /files/upload-abc123

// Client uploads a chunk
PATCH /files/upload-abc123 HTTP/1.1
Upload-Offset: 0
Content-Length: 10485760

[bytes 0-10485759]

// After disconnect, client queries progress
HEAD /files/upload-abc123 HTTP/1.1

HTTP/1.1 200 OK
Upload-Offset: 10485760
Upload-Length: 1073741824

// Client resumes from byte 10485760
PATCH /files/upload-abc123 HTTP/1.1
Upload-Offset: 10485760

Implement tus on the server side or use a tus-compatible proxy (tusd) in front of your object storage.

Virus scanning pipeline

Every file uploaded by users is a potential vector for malware. Serving an infected file back to other users is a liability. Virus scanning must happen before the file is made available.

graph LR
  Upload["File uploaded<br/>to quarantine bucket"] --> Scan["Virus scanner<br/>(ClamAV / cloud service)"]
  Scan -->|Clean| Move["Move to<br/>public bucket"]
  Scan -->|Infected| Quarantine["Keep in quarantine<br/>+ alert"]
  Move --> CDN["Serve via CDN"]

  style Upload fill:#f39c12,color:#fff
  style Scan fill:#3498db,color:#fff
  style Move fill:#2ecc71,color:#fff
  style Quarantine fill:#e74c3c,color:#fff
  style CDN fill:#9b59b6,color:#fff

Virus scanning pipeline. Files start in quarantine and only move to the public bucket after passing the scan.

Implementation details

1. Quarantine bucket: all uploads land in a quarantine bucket that is not publicly accessible.
2. Scan trigger: an S3 event notification triggers a Lambda function or background job that scans the file.
3. Scanner: ClamAV (open source, self-hosted) or a cloud service (AWS GuardDuty, Google Cloud Security Scanner).
4. Clean files: moved to the public bucket and the upload record is updated with the final URL.
5. Infected files: stay in quarantine. The upload record is marked as rejected. An alert fires.
6. Client notification: the client polls the upload status or receives a webhook when the scan completes.

Scan time depends on file size. Budget 5 to 30 seconds for most files. Show a “processing” state in the UI rather than making the user wait.

CDN integration for serving files

Once files are in the public bucket, serve them through a CDN for low-latency delivery worldwide.

Signed URLs for private content

Not all files should be publicly accessible. For private files (user documents, invoices), generate signed CDN URLs that expire after a short period:

function getSignedFileUrl(objectKey, expiresInSeconds = 3600) {
  const policy = {
    Statement: [{
      Resource: `https://cdn.example.com/${objectKey}`,
      Condition: {
        DateLessThan: {
          'AWS:EpochTime': Math.floor(Date.now() / 1000) + expiresInSeconds
        }
      }
    }]
  };
  return signUrl(policy, privateKey);
}

Cache headers

Set appropriate cache headers for different file types:

Image transformations

Instead of storing multiple sizes of each image, use an image transformation service (Imgix, Cloudinary, or a custom Lambda@Edge function) that resizes on the fly and caches the result:

https://cdn.example.com/images/photo-abc.jpg?w=300&h=300&fit=crop

This approach is simpler than pre-generating every size at upload time and adapts to new size requirements without reprocessing existing files.

Storage cost management

Object storage is cheap per GB but costs add up at scale. Manage costs with:

• Lifecycle policies: automatically move old files to cheaper storage classes (S3 Glacier, GCS Coldline) after 90 days.
• Deduplication: hash files on upload and reuse existing objects for duplicates.
• Cleanup jobs: delete orphaned files (upload initiated but never completed) after 24 hours.

What comes next

The next article covers search integration: keeping search indexes in sync, synchronous vs event-driven indexing, handling deletes, relevance tuning, and autocomplete. Search often indexes file metadata and content, making it a natural companion to file storage.