Ownership Story: Building a 5GB File Upload Service Using Streams

Interview Question

"Tell me about a project where you took complete ownership."

Answer (STAR(situation,Task,Action,Result) Format)

Situation

In one of our applications, customers needed to upload very large files — up to 5 GB. The existing implementation loaded the entire file into memory before processing it. While it worked for smaller files, large uploads frequently caused memory spikes, request timeouts, and server crashes, making the feature unreliable.

The upload flow was becoming a bottleneck as more enterprise customers started using it.

Task

I was responsible for redesigning the upload service to support large files efficiently, without exhausting server memory, while ensuring the solution was reliable, scalable, and production-ready.

Action

I started by analyzing the existing implementation and found that buffering the entire file in memory was the main issue.

Instead of reading the complete file at once, I redesigned the upload flow using Node.js streams.

The key decisions I made were:

• Used streaming so the file was processed in small chunks instead of loading it entirely into RAM.
• Added backpressure handling so the upload speed matched the speed at which the server could process or store data, preventing memory buildup.
• Streamed the data directly to storage instead of writing temporary files whenever possible.
• Implemented upload size validation and file type validation before processing.
• Added proper error handling so partially uploaded files were cleaned up if a client disconnected or an error occurred.
• Added logging and metrics to monitor upload duration, failures, and throughput.
• Performed load testing with multiple concurrent uploads to verify memory usage and overall system stability.

I also collaborated with the frontend team to expose upload progress so users could see the percentage completed instead of waiting without feedback.

Result

The new implementation successfully supported uploads up to 5 GB while keeping memory usage almost constant because data was processed as a stream rather than being buffered.

The API became significantly more reliable under concurrent uploads, upload failures due to memory exhaustion were eliminated, and the overall user experience improved with upload progress tracking.

The streaming architecture also gave us a scalable foundation for handling even larger files in the future without major code changes.