File Uploads Look Simple. Here’s How to Design One That Doesn’t Bite Back

Every app I've worked on eventually grows a paperclip icon. Profile avatars, invoice attachments, CSV imports, log drops, video posts. Uploads are so common that we treat them as boilerplate: add a multipart endpoint, save to disk, ship it.

That's exactly why they hurt.

The hidden complexity

An upload is the one place where strangers hand your system arbitrary bytes and ask you to keep them. Do it naively and a single request can eat your server's memory, fill your disk, or run an attacker's script. The happy path takes an afternoon. The unhappy paths take months.

Four ways it goes wrong

• Security. Trusting the file extension, serving uploads from your main domain, or storing files where they can execute invites malicious scripts and, in the worst case, remote code execution.
• Server strain. Proxying every upload through your API ties up threads and memory. A few large files at once can take the service down.
• Storage and cost. No size limits, no expiry, no cleanup. Abandoned uploads and forgotten files quietly pile up on your bill.
• Poor UX. No progress bar, no retry, no resume. One dropped connection on a 500 MB video and the user starts over.

The architecture, step by step

1. Keep the bytes off your server. Your API shouldn't receive the file. It should authorize the upload and hand the client a short-lived URL scoped to one object, so the client uploads straight to object storage. That's a presigned URL on AWS S3, a signed URL on Google Cloud Storage, or a SAS token on Azure Blob Storage. Where the cloud supports it (S3 presigned POST policies can enforce a size range), cap the size at signing time, and always re-check after the upload.

2. Validate twice. In the browser, check size and extension so users get instant feedback. On the server, trust none of it. Re-check the limits and verify the real file type from the file's contents (magic bytes) with libmagic, Apache Tika or the file-type library, not from the filename or the Content-Type header.

3. Chunk big files. Use S3 multipart upload, GCS resumable uploads, Azure block blobs, or the open tus protocol with the Uppy client. Failed chunks retry on their own and interrupted uploads resume instead of restarting.

4. Quarantine, then process asynchronously. Land every file in a private, unverified bucket or prefix. The storage's object-created event (S3 Event Notifications, GCS Pub/Sub notifications, Azure Event Grid) puts a job on a queue such as SQS, RabbitMQ, BullMQ or Celery. Workers then:

• scan for malware with ClamAV, Amazon GuardDuty Malware Protection for S3, or Microsoft Defender for Storage;
• confirm the real file type;
• compress, transcode or thumbnail with libvips (sharp) or FFmpeg;
• move only clean files to the serving location.

For multi-step pipelines with retries and branching, an orchestrator such as AWS Step Functions or Temporal keeps the state for you.

5. Make it idempotent. Give every upload an ID and a database record (Postgres works well) with a status: pending, uploaded, processing, ready or rejected. Retried requests and duplicate storage events then do no harm. Hash files with SHA-256 to catch duplicates and verify integrity.

6. Clean up automatically. Add lifecycle rules (S3 Lifecycle, GCS Object Lifecycle Management, Azure Blob lifecycle management) to delete abandoned multipart uploads and unverified files after a day or two. Serve files from a separate domain or CDN such as CloudFront, Cloud CDN or Azure Front Door, with Content-Disposition set so browsers download rather than run them.

What the big clouds do

The three major clouds converge on the same pattern. Your server signs, the client talks to storage.

• Amazon Web Services (AWS): Uses S3 presigned URLs and presigned POST policies to grant fine-grained, time-limited upload permissions directly to the client. For large files, AWS relies on S3 Multipart Uploads, breaking payload streams into distinct chunks that upload concurrently and reassemble upon completion.
• Google Cloud Platform (GCP): Employs Cloud Storage signed URLs to securely route client uploads straight to storage buckets. To manage large or unstable uploads, GCP utilizes Resumable Uploads, allowing byte-stream transfers to pause and resume automatically without losing progress upon connection drops.
• Microsoft Azure: Leverages Blob Storage Shared Access Signature (SAS) tokens for direct client-to-storage authorization. Azure manages large file ingestions using Block Blob Staging and Commit, splitting files into uncommitted blocks uploaded in parallel before issuing a single final commit call to assemble the complete blob.

The end-to-end flow

Wrapping up

Treat an upload as a small workflow, not a single request: authorize, transfer, quarantine, verify, publish, clean up. Start with presigned URLs and a hard size limit. Add chunking, scanning and lifecycle rules as you grow. Your future self, and your cloud bill, will thank you.