Post cover image

July 23, 2026

System Design Interview: How Would You Build a Reliable Resumable Upload System for Huge Files?

Uploading a small image is easy.

By Anna Likhachova

16 min read

Uploading a 100 GB file over an unstable internet connection is a completely different problem.

A production-ready system must survive interrupted connections, retries, duplicated requests, browser restarts, application restarts, partial uploads, corrupted chunks, and concurrent clients.

This is not simply a question of splitting a file into smaller pieces.

It is a distributed systems problem involving durability, idempotency, consistency, storage, retries, security, and lifecycle management.

In this article, I will approach the problem the way I would during a senior backend or system design interview.

The Interview Question

Design a system that allows users to upload very large files, potentially hundreds of gigabytes, even when the internet connection is unstable or temporarily disconnected.

The system should allow the user to resume the upload without restarting from the beginning.

1. Clarifying the Requirements

Before proposing an architecture, I would ask several questions.

Functional requirements

  • What is the maximum file size?
  • Are files uploaded from browsers, mobile applications, desktop clients, or all three?
  • Should users be able to pause and resume manually?
  • Should uploads continue after the application restarts?
  • Can chunks be uploaded in parallel?
  • Should users see upload progress?
  • Do we need a content validation?
  • Should the file be processed after upload?

Non-functional requirements

  • How many users may upload files concurrently?
  • How much data is uploaded per day?
  • How long should incomplete uploads be retained?
  • What availability level is required?
  • Is eventual consistency acceptable?
  • Which storage systems are available?
  • Should uploads work across several data centers or regions?

For the rest of the design, I will assume the following:

  • Maximum file size: 100 GB.
  • Thousands of concurrent uploads.
  • Clients may disconnect and return later.
  • Uploads may be resumed after browser or application restart.
  • Chunks may be uploaded in parallel.
  • Files must be stored durably.
  • After upload, files may be processed by another system such as Spark.

2. Why a Single HTTP Upload Is Not Enough

The simplest implementation might look like this:

POST /files
Content-Type: multipart/form-data

The client opens one HTTP connection and sends the entire file.

This works for small files, but it becomes unreliable for very large ones.

If a 100 GB upload fails after 99 GB, the user may need to restart from the beginning.

A single upload request also creates additional problems:

  • reverse proxy timeout;
  • application server timeout;
  • connection reset;
  • excessive memory or temporary disk usage;
  • difficulty retrying safely;
  • poor progress tracking;
  • inability to resume after application restart.

Therefore, the file should be divided into independently uploadable chunks.

3. High-Level Architecture

A production architecture may look like this:

Client
   |
   | 1. Create upload session
   v
Upload API
   |
   | 2. Store metadata
   v
Metadata Database
   |
   | 3. Return upload ID and chunk configuration
   v
Client
   |
   | 4. Upload chunks
   v
Object Storage
   |
   | 5. Complete multipart upload
   v
Final File
   |
   | 6. Publish file-uploaded event
   v
Message Broker
   |
   | 7. Process file
   v
Spark / Processing Service

The important design decision is that the application server should not necessarily proxy all file bytes.

In a scalable design, the backend manages the upload session, permissions, metadata, and completion workflow, while the client uploads chunks directly to object storage through temporary authorized URLs.

4. Upload Workflow

Step 1: Create an upload session

The client sends file metadata to the backend.

POST /uploads
Content-Type: application/json
{
  "fileName": "transactions-2026.csv",
  "fileSize": 107374182400,
  "contentType": "text/csv",
  "checksum": "optional-sha256"
}

The backend validates:

  • file size;
  • file type;
  • user permissions;
  • quota;
  • allowed storage location.

It then creates an upload session.

{
  "uploadId": "upl_8f0d7e9c",
  "chunkSize": 16777216,
  "totalChunks": 6400,
  "expiresAt": "2026-07-24T10:00:00Z"
}

A chunk size between 8 MB and 64 MB is often practical, but the exact value depends on file size, client memory, storage limitations, network quality, and object storage requirements.

Step 2: Persist upload metadata

The backend stores the upload session in a database.

CREATE TABLE upload_session (
    upload_id           VARCHAR(100) PRIMARY KEY,
    user_id             VARCHAR(100) NOT NULL,
    original_file_name  VARCHAR(500) NOT NULL,
    content_type        VARCHAR(200),
    total_size          BIGINT NOT NULL,
    chunk_size          BIGINT NOT NULL,
    total_chunks        INTEGER NOT NULL,
    uploaded_chunks     INTEGER NOT NULL DEFAULT 0,
    status              VARCHAR(30) NOT NULL,
    storage_key         VARCHAR(1000),
    file_checksum       VARCHAR(128),
    created_at          TIMESTAMP NOT NULL,
    updated_at          TIMESTAMP NOT NULL,
    expires_at          TIMESTAMP NOT NULL
);

Possible statuses:

CREATED
UPLOADING
COMPLETING
COMPLETED
FAILED
EXPIRED
ABORTED

The database stores upload state, not necessarily the file contents.

Step 3: Upload chunks

The client splits the file locally.

Conceptually:

long offset = (long) chunkNumber * chunkSize;
long length = Math.min(chunkSize, fileSize - offset);

Each chunk receives:

  • upload ID;
  • chunk number;
  • byte range;
  • chunk checksum;
  • idempotency key.

The client can upload chunks sequentially or in parallel.

PUT /uploads/upl_8f0d7e9c/chunks/154
Content-Range: bytes 2583691264-2600468479/107374182400
X-Chunk-Checksum: 7637c68...

The important point is that each chunk is independently retryable.

5. Direct Upload to Object Storage

A scalable architecture usually avoids sending every byte through the backend.

Instead, the backend generates a temporary URL.

Client
   |
   | Request upload URL for chunk 154
   v
Backend
   |
   | Generate signed URL
   v
Client
   |
   | PUT chunk directly
   v
S3 / MinIO / Azure Blob / GCS

Example API:

POST /uploads/upl_8f0d7e9c/chunks/154/url

Response:

{
  "chunkNumber": 154,
  "uploadUrl": "https://storage.example.com/temporary-signed-url",
  "expiresAt": "2026-07-22T12:15:00Z"
}

The client then uploads directly to storage.

Advantages:

  • application servers do not carry large file traffic;
  • lower backend CPU and memory usage;
  • easier horizontal scaling;
  • object storage handles large uploads efficiently;
  • storage-specific multipart upload features can be used.

The backend remains responsible for authorization and metadata.

6. How Resume Works

When the client reconnects, it asks the backend which chunks already exist.

GET /uploads/upl_8f0d7e9c

Response:

{
  "uploadId": "upl_8f0d7e9c",
  "status": "UPLOADING",
  "totalChunks": 6400,
  "uploadedChunks": [0, 1, 2, 3, 5, 6, 8],
  "missingChunks": [4, 7, 9, 10]
}

The client resumes only the missing chunks.

For very large files, returning thousands of chunk numbers may be inefficient.

Alternatives include:

  • ranges of uploaded chunks;
  • compressed bitmaps;
  • pagination;
  • querying only a subset;
  • relying on the multipart upload state maintained by object storage.

Example range response:

{
  "uploadedRanges": [
    {
      "from": 0,
      "to": 153
    },
    {
      "from": 155,
      "to": 318
    }
  ]
}

7. Idempotency

Retries are unavoidable.

Suppose the client uploads chunk 154.

The server stores the chunk successfully, but the HTTP response is lost.

The client does not know whether the upload succeeded, so it retries.

Without idempotency, the system may create duplicates or inconsistent state.

A chunk should be uniquely identified by:

uploadId + chunkNumber

A relational representation could be:

CREATE TABLE upload_chunk (
    upload_id       VARCHAR(100) NOT NULL,
    chunk_number    INTEGER NOT NULL,
    size_bytes      BIGINT NOT NULL,
    checksum        VARCHAR(128) NOT NULL,
    storage_part_id VARCHAR(500),
    status          VARCHAR(30) NOT NULL,
    created_at      TIMESTAMP NOT NULL,
    updated_at      TIMESTAMP NOT NULL,
    PRIMARY KEY (upload_id, chunk_number)
);

If the same chunk arrives again with the same checksum, the server may return success.

If the same chunk number arrives with a different checksum, the server should reject it as a conflict.

409 Conflict

Idempotency means that repeating the same request produces the same logical result.

8. Checksums and Data Integrity

A successful HTTP response does not automatically guarantee that the entire file is correct.

The system should support checksums at two levels.

Chunk-level checksum

Each chunk contains a checksum such as SHA-256.

checksum(chunk)

The server or object storage validates the uploaded content.

If the checksum does not match, only that chunk must be uploaded again.

File-level checksum

After all chunks are assembled, the system may validate the checksum of the complete file.

checksum(complete file)

This protects against:

  • corrupted chunks;
  • incorrect chunk ordering;
  • missing chunks;
  • duplicated chunks;
  • client-side slicing bugs.

For extremely large files, calculating a full checksum may be expensive, so the choice depends on security and correctness requirements.

9. Completing the Upload

After uploading all chunks, the client asks the backend to finalize the file.

POST /uploads/upl_8f0d7e9c/complete

The backend performs several checks:

  1. The upload session exists.
  2. The user owns the upload.
  3. The session has not expired.
  4. Every required chunk exists.
  5. Chunk sizes are correct.
  6. Checksums are valid.
  7. The upload is not already completed.

The status should be changed atomically.

UPLOADING → COMPLETING → COMPLETED

A compare-and-set update can prevent two clients from completing the same upload simultaneously.

UPDATE upload_session
SET status = 'COMPLETING',
    updated_at = CURRENT_TIMESTAMP
WHERE upload_id = ?
  AND status = 'UPLOADING';

If zero rows are updated, another process may already be completing or may have completed the upload.

After validation, the storage system assembles the final file.

With object storage multipart upload, the storage provider usually combines the uploaded parts.

The backend should not read 100 GB into application memory.

10. Should the Backend Assemble the File?

Usually, no.

A naive implementation may do this:

try (OutputStream output = Files.newOutputStream(finalFile)) {
    for (int chunk = 0; chunk < totalChunks; chunk++) {
        Files.copy(chunkPath(chunk), output);
    }
}

This can work on a single server, but it creates several problems:

  • heavy disk I/O;
  • long-running operations;
  • local storage dependency;
  • failure during assembly;
  • difficulty scaling horizontally;
  • large temporary storage requirements.

If object storage supports multipart uploads, the storage layer should assemble the file.

If HDFS is the final destination, a separate controlled ingestion process may stream the completed file into HDFS.

11. Where Should the File Be Stored?

Option 1: Database BLOB

Technically possible, but usually not ideal for huge files.

Problems include:

  • large transaction logs;
  • expensive backups;
  • database replication overhead;
  • poor separation between metadata and binary storage;
  • increased database cost;
  • difficult operational maintenance.

A database is usually better suited for metadata.

Option 2: Local server disk

Suitable for:

  • prototypes;
  • small internal applications;
  • single-node systems.

Problems:

  • server failure may lose the file;
  • containers may use ephemeral storage;
  • requests may reach different instances;
  • autoscaling becomes difficult;
  • shared access is not guaranteed.

Option 3: Shared file system

Examples:

  • NFS;
  • network-attached storage;
  • distributed file system.

This can work, but scalability and operational complexity must be considered.

Option 4: HDFS

HDFS is appropriate when:

  • the organization already operates a Hadoop cluster;
  • files will be processed by Spark or MapReduce;
  • large sequential reads are expected;
  • the environment is internal and controlled.

However, HDFS is not usually exposed directly to browsers or mobile clients.

An upload service should authenticate users and control writes.

HDFS also performs poorly with millions of tiny files because each file consumes NameNode metadata.

Option 5: Object storage

Examples:

  • Amazon S3;
  • MinIO;
  • Azure Blob Storage;
  • Google Cloud Storage.

This is often the best fit because object storage provides:

  • high durability;
  • multipart upload;
  • lifecycle policies;
  • horizontal scalability;
  • direct client upload;
  • temporary signed URLs;
  • low operational complexity.

12. Follow-Up Interview Questions

Now let us look at the questions an interviewer may ask after the initial design.

Follow-Up 1: What happens if the same chunk is uploaded twice?

The upload endpoint must be idempotent.

The unique key should be:

uploadId + chunkNumber

If the chunk already exists and the checksum is identical, the server returns success.

200 OK

If the chunk number exists but the checksum is different, the server returns a conflict.

409 Conflict

This prevents silent file corruption.

The database should enforce uniqueness, not only application code.

PRIMARY KEY (upload_id, chunk_number)

Follow-Up 2: What happens if the backend crashes after storing the chunk but before updating the database?

This is a distributed consistency problem.

Storage and database updates may not belong to one transaction.

Possible sequence:

1. Store chunk in object storage.
2. Backend crashes.
3. Database still says the chunk is missing.

The system should support reconciliation.

When the client retries, the backend checks whether the chunk already exists in storage.

Alternatively, a background reconciliation job compares:

  • database metadata;
  • multipart upload state;
  • stored parts.

The system should treat the storage layer as the source of truth for actual bytes and the database as the source of truth for workflow state.

The update must remain idempotent.

Follow-Up 3: What happens if the database is updated but the chunk was not stored?

The chunk status should not be marked as completed before storage confirms success.

A safer sequence is:

1. Create chunk record with status UPLOADING.
2. Upload chunk to storage.
3. Receive storage confirmation.
4. Update chunk status to COMPLETED.

If the process crashes before step 4, reconciliation can verify whether the storage part exists.

A chunk should never be counted as successfully uploaded based only on the start of the operation.

Follow-Up 4: How many chunks should be uploaded in parallel?

Parallel uploads improve throughput, but unlimited concurrency can overload:

  • the client;
  • the network;
  • object storage;
  • the backend;
  • the user's memory.

A common client may upload between three and eight chunks concurrently.

The optimal number depends on:

  • client bandwidth;
  • latency;
  • chunk size;
  • device type;
  • server capacity.

The system may dynamically adjust concurrency.

For example:

  • poor mobile network: two parallel chunks;
  • fast desktop connection: eight parallel chunks;
  • server under load: lower concurrency.

The backend should also enforce per-user and global rate limits.

Follow-Up 5: How do you choose chunk size?

Small chunks improve retry efficiency because less data must be retransmitted.

However, they increase:

  • request count;
  • database records;
  • storage part metadata;
  • network overhead;
  • checksum calculations.

Large chunks reduce metadata overhead but make retries more expensive.

For example:

100 GB file with 5 MB chunks:
approximately 20,480 chunks
100 GB file with 64 MB chunks:
approximately 1,600 chunks

A practical design may choose a dynamic chunk size.

For example:

File below 1 GB: 8 MB chunks
File between 1 GB and 20 GB: 32 MB chunks
File above 20 GB: 64 MB chunks

Object storage multipart constraints must also be considered.

Follow-Up 6: How do you prevent one user from consuming all storage?

The system should enforce quotas.

Possible limits:

  • maximum file size;
  • maximum active uploads per user;
  • maximum total stored bytes;
  • maximum chunk upload rate;
  • maximum concurrent connections.

Quota validation should occur when the upload session is created.

The system must also account for incomplete uploads, because users could intentionally start many uploads without finishing them.

Follow-Up 7: What happens to abandoned uploads?

Incomplete uploads should expire.

Each session has an expiration time.

expiresAt = createdAt + retentionPeriod

A background cleanup job identifies expired sessions.

SELECT upload_id
FROM upload_session
WHERE status IN ('CREATED', 'UPLOADING', 'FAILED')
  AND expires_at < CURRENT_TIMESTAMP;

The cleanup process:

  1. Marks the session as expired.
  2. Aborts the multipart upload.
  3. Deletes temporary chunks.
  4. Releases reserved quota.
  5. Records audit information.

Cleanup must also be idempotent because the job may run more than once.

Object storage lifecycle policies can provide an additional safety net.

Follow-Up 8: How do you calculate upload progress?

The simplest approximation is:

uploaded bytes / total bytes

However, progress should distinguish between:

  • bytes sent by the client;
  • bytes confirmed by storage;
  • chunks verified by the backend;
  • final file completion;
  • post-upload processing.

A client may report:

Uploading: 76%
Verifying: 100%
Processing: 42%

Only confirmed chunks should count as durable upload progress.

Client-side bytes sent are not enough because the request may still fail.

Follow-Up 9: How do you support resume after a browser restart?

The client must persist upload metadata locally.

For a browser, this may be stored in:

  • IndexedDB;
  • local storage for small metadata;
  • application state storage.

The client stores:

{
  "uploadId": "upl_8f0d7e9c",
  "fileName": "transactions-2026.csv",
  "fileSize": 107374182400,
  "lastModified": 1784712000000,
  "chunkSize": 16777216
}

After restart, the client asks the user to reselect the file.

It then verifies that the file matches the original metadata.

Possible checks:

  • file name;
  • file size;
  • last modified timestamp;
  • checksum of selected sections;
  • complete file checksum.

The server is responsible for durable upload state.

The browser only keeps enough information to reconnect to that state.

Follow-Up 10: What happens if two devices use the same upload ID?

The backend must authenticate every request and verify upload ownership.

The upload ID alone is not authorization.

A request should be accepted only when:

authenticatedUserId == uploadSession.userId

The system may permit several devices intentionally, but concurrent writes to the same chunk must still be controlled.

For strict behavior, the upload session may contain a client token or version.

For flexible behavior, identical chunk uploads remain safe because of idempotency.

Follow-Up 11: How do you secure signed upload URLs?

Signed URLs should:

  • expire quickly;
  • allow only one specific operation;
  • target one specific object or part;
  • restrict content length;
  • validate content type when possible;
  • be issued only after authorization;
  • not expose permanent storage credentials.

A signed URL should not give the client permission to list the bucket or access other users' files.

The object key should not be derived solely from the original file name.

A safer key may look like:

temporary/{userId}/{uploadId}/parts/{chunkNumber}

Follow-Up 12: Can the client upload directly to HDFS?

Usually, this is not recommended for external clients.

HDFS authentication and authorization are designed for internal cluster environments.

Direct client access would expose:

  • cluster topology;
  • HDFS credentials;
  • internal network;
  • security configuration;
  • NameNode and DataNode endpoints.

A safer architecture is:

External Client
      |
      v
Upload API or Object Storage
      |
      v
Internal Ingestion Service
      |
      v
HDFS

If the organization already uses Hadoop, the completed file can be transferred into HDFS after upload validation.

Follow-Up 13: Can the file be stored directly in SQL?

Yes, but it is usually not the best design for huge files.

A database BLOB may be acceptable when:

  • files are relatively small;
  • file count is limited;
  • strict transaction coupling is required;
  • operational simplicity is more important than scale.

For hundreds of gigabytes, object storage or HDFS is usually more appropriate.

The database should contain:

  • upload state;
  • owner;
  • storage key;
  • checksum;
  • file size;
  • processing status.

The binary file should normally live in specialized storage.

Follow-Up 14: What happens if completion is requested twice?

The completion operation must also be idempotent.

Possible behavior:

UPLOADING → COMPLETING → COMPLETED

If the session is already COMPLETED, the server returns the existing file result.

{
  "status": "COMPLETED",
  "fileId": "file_19d84",
  "storageKey": "completed/file_19d84"
}

It should not create a second final file.

The transition should be protected by a conditional database update or distributed lock.

A conditional update is usually simpler and safer than a long-running distributed lock.

Follow-Up 15: What if one chunk is corrupted?

Chunk checksums allow the system to detect corruption before finalization.

The server marks the specific chunk as invalid.

{
  "chunkNumber": 154,
  "status": "CHECKSUM_MISMATCH"
}

The client uploads only that chunk again.

The full upload does not need to restart.

Follow-Up 16: How do you notify the processing system?

After the file reaches COMPLETED, the backend publishes an event.

{
  "eventId": "evt_76a8",
  "eventType": "FILE_UPLOAD_COMPLETED",
  "uploadId": "upl_8f0d7e9c",
  "fileId": "file_19d84",
  "storageKey": "completed/file_19d84",
  "checksum": "4c92...",
  "occurredAt": "2026-07-22T11:46:00Z"
}

The event may be sent through:

  • Kafka;
  • Amazon SQS;
  • RabbitMQ;
  • ActiveMQ;
  • cloud event services.

The processing consumer must be idempotent because the same event may be delivered more than once.

A common approach is to store the event ID or file ID in a processed-events table.

Follow-Up 17: What if the event is not published after the database transaction commits?

This is the classic dual-write problem.

The backend might:

  1. Mark the upload as completed in the database.
  2. Crash before publishing the event.

The file is complete, but processing never starts.

The transactional outbox pattern solves this.

Within one database transaction, the backend writes:

  • the completed upload state;
  • an outbox event record.
BEGIN;
UPDATE upload_session
SET status = 'COMPLETED'
WHERE upload_id = ?;
INSERT INTO outbox_event (
    event_id,
    event_type,
    aggregate_id,
    payload,
    status
)
VALUES (?, 'FILE_UPLOAD_COMPLETED', ?, ?, 'NEW');
COMMIT;

A separate publisher reads new outbox records and sends them to the broker.

If publishing fails, it retries.

This avoids losing the event.

Follow-Up 18: How would Spark process the uploaded file safely?

Spark should read only completed files.

A file should become visible to Spark only after finalization.

Possible strategies:

  • write into a temporary prefix and move to a completed prefix;
  • publish an event only after successful completion;
  • maintain a metadata table with status READY_FOR_PROCESSING;
  • use a manifest file;
  • use atomic rename in HDFS.

Example HDFS structure:

/uploads/in-progress/upl_8f0d7e9c/
/uploads/completed/file_19d84/

Spark reads only:

/uploads/completed/

The Spark job should also be idempotent.

It may use the file ID as a processing key.

PRIMARY KEY (file_id, processing_version)

This prevents duplicate output if the job is retried.

Follow-Up 19: What if Spark fails halfway through processing?

The uploaded file should remain immutable.

Spark should write results into a temporary destination.

/processing/tmp/file_19d84/run_01/

Only after successful completion should the output be promoted.

For systems such as Delta Lake, Iceberg, or transactional databases, the final commit can be atomic.

The processing status may look like:

PENDING
RUNNING
COMPLETED
FAILED

A failed job can be retried without re-uploading the original file.

Upload lifecycle and processing lifecycle should remain separate.

Follow-Up 20: What should be monitored?

Important metrics include:

  • active upload sessions;
  • upload creation rate;
  • chunk upload latency;
  • chunk retry rate;
  • checksum failure rate;
  • incomplete upload count;
  • expired upload count;
  • uploaded bytes per second;
  • storage errors;
  • completion failures;
  • abandoned temporary storage;
  • processing queue size;
  • Spark processing duration;
  • failed processing jobs.

Useful alerts include:

  • sudden increase in failed chunks;
  • high completion latency;
  • rapidly growing abandoned storage;
  • upload API saturation;
  • event publishing failures;
  • processing backlog.

Logs should include:

uploadId
userId
chunkNumber
requestId
storagePartId
checksum
status

Sensitive file contents should not be logged.

14. API Design

A possible REST API may include:

POST   /uploads
GET    /uploads/{uploadId}
POST   /uploads/{uploadId}/chunks/{chunkNumber}/url
PUT    /uploads/{uploadId}/chunks/{chunkNumber}
GET    /uploads/{uploadId}/chunks
POST   /uploads/{uploadId}/complete
DELETE /uploads/{uploadId}

Not every system needs both direct upload URLs and a backend chunk endpoint.

A small internal application may proxy chunks through the backend.

A large public system should normally prefer direct object storage upload.

15. Example Spring Boot Session Service

A simplified service may look like this:

@Service
@RequiredArgsConstructor
public class UploadSessionService {
    private final UploadSessionRepository uploadSessionRepository;
    private final StorageService storageService;
    @Transactional
    public UploadSession createSession(
            String userId,
            CreateUploadRequest request
    ) {
        validateFileSize(request.fileSize());
        validateContentType(request.contentType());
        long chunkSize = calculateChunkSize(request.fileSize());
        int totalChunks = Math.toIntExact(
                (request.fileSize() + chunkSize - 1) / chunkSize
        );
        UploadSession session = new UploadSession(
                UUID.randomUUID().toString(),
                userId,
                request.fileName(),
                request.fileSize(),
                chunkSize,
                totalChunks,
                UploadStatus.CREATED,
                Instant.now(),
                Instant.now().plus(Duration.ofDays(2))
        );
        return uploadSessionRepository.save(session);
    }
    public UploadStatusResponse getStatus(
            String uploadId,
            String userId
    ) {
        UploadSession session = getOwnedSession(uploadId, userId);
        List<Integer> uploadedChunks =
                storageService.findUploadedChunks(uploadId);
        return new UploadStatusResponse(
                session.getUploadId(),
                session.getStatus(),
                session.getTotalChunks(),
                uploadedChunks
        );
    }
    private UploadSession getOwnedSession(
            String uploadId,
            String userId
    ) {
        UploadSession session = uploadSessionRepository
                .findById(uploadId)
                .orElseThrow(() -> new UploadNotFoundException(uploadId));
        if (!session.getUserId().equals(userId)) {
            throw new AccessDeniedException("Upload does not belong to user");
        }
        return session;
    }
}

This is only a simplified example.

A real implementation would also include:

  • authorization;
  • quota validation;
  • idempotency;
  • presigned URLs;
  • optimistic locking;
  • retries;
  • cleanup;
  • checksums;
  • observability;
  • rate limiting.

16. Failure Scenarios

A strong system design answer should explicitly discuss failures.

Client disconnects

The upload session remains active.

The client later queries the current state and uploads missing chunks.

Client retries a successful chunk

The operation returns success because it is idempotent.

Backend instance crashes

Session state remains in the database.

Uploaded bytes remain in durable storage.

Another backend instance can continue the workflow.

Object storage temporarily fails

The client retries with exponential backoff.

The backend may issue a new signed URL.

Database temporarily fails

The backend must not pretend the chunk is durably registered.

Reconciliation repairs inconsistencies.

Completion fails

The session remains in COMPLETING or moves to FAILED.

A retry or recovery job completes or aborts the operation.

Message broker is unavailable

The outbox event remains in the database.

The publisher retries later.

Spark processing fails

The original file remains available.

The processing job can restart independently.

17. Final Architecture

A robust production design may look like this:

+----------------------+
                     |       Client         |
                     +----------+-----------+
                                |
                                | Create upload session
                                v
                     +----------------------+
                     |     Upload API       |
                     +-----+-----------+----+
                           |           |
                           |           | Store session state
                           |           v
                           |     +-------------+
                           |     | Metadata DB |
                           |     +-------------+
                           |
                           | Generate signed URLs
                           v
                     +----------------------+
                     |    Object Storage    |
                     |  Multipart Upload    |
                     +----------+-----------+
                                |
                                | Complete upload
                                v
                     +----------------------+
                     |   Completed Object   |
                     +----------+-----------+
                                |
                                | Transactional outbox
                                v
                     +----------------------+
                     |    Message Broker    |
                     +----------+-----------+
                                |
                                v
                     +----------------------+
                     | Spark / Processor    |
                     +----------+-----------+
                                |
                                v
                     +----------------------+
                     | Processed Data Store |
                     +----------------------+

18. Final Interview Answer

A concise interview answer could be:

I would not upload the entire file through one HTTP request. I would create a resumable multipart upload workflow.

The backend first creates an upload session and stores metadata such as the user, file size, chunk size, total chunk count, status, and expiration time.

The client splits the file into chunks and uploads them independently, preferably directly to object storage using short-lived signed URLs.

Each chunk is identified by upload ID and chunk number, making retries idempotent. Chunk-level checksums detect corruption.

When the client reconnects, it queries the upload session and sends only the missing chunks.

After all chunks are present, the client requests completion. The backend atomically changes the session status, validates all parts, and asks the storage system to finalize the multipart upload.

Incomplete sessions are removed by a cleanup job and storage lifecycle policies.

After completion, the backend publishes an event using a transactional outbox. A downstream Spark job may then process the completed file.

Spark is useful for processing the file, but it should not manage the resumable network upload itself.

Conclusion

Reliable huge-file upload is not primarily about splitting a file into chunks.

Chunking is only the beginning.

The real design must address:

  • resumability;
  • idempotency;
  • checksums;
  • durable state;
  • direct storage upload;
  • completion consistency;
  • abandoned session cleanup;
  • authorization;
  • quotas;
  • retries;
  • event delivery;
  • downstream processing.

The most important architectural separation is this:

Upload Service = reliable file transfer and session management
Object Storage or HDFS = durable file storage
Message Broker = workflow handoff
Spark = distributed processing after upload

Once these responsibilities are separated, the system becomes easier to scale, recover, monitor, and reason about.