How a Scalable Code Executor Works and How I Built One

It looks like a simple action, but several services work together behind the scenes.

The system must receive the request, validate it, place it in a queue, execute the code safely, store the result, and return it to the user — all without blocking the API.

This is how I built that flow.

The Main Architecture

The complete flow is:

User
  ↓
NGINX
  ↓
Node.js API Servers
  ↓
Amazon SQS
  ↓
Worker Servers
  ↓
Docker Containers
  ↓
DynamoDB
  ↓
Result Returned to User

User
  ↓
NGINX
  ↓
Node.js API Servers
  ↓
Amazon SQS
  ↓
Worker Servers
  ↓
Docker Containers
  ↓
DynamoDB
  ↓
Result Returned to User

The architecture separates request handling from code execution.

The API servers stay lightweight and responsive, while separate worker servers perform the more demanding task of running submitted code.

Services I Used

NGINX

NGINX works as the main entry point.

It receives requests and distributes them between multiple Node.js API servers.

This prevents the application from depending on only one API instance.

Node.js and Express.js

The API is built with Node.js and Express.js.

Its job is to:

• Receive submitted code
• Validate the request
• Apply rate limiting
• Generate an execution ID
• Send the job to Amazon SQS
• Return an immediate response

The API does not execute the submitted code directly.

Amazon SQS

Amazon SQS connects the API layer with the worker layer.

The API places every new execution request into the queue. Available workers continuously pull jobs from that queue.

This allows the system to accept more requests even when all workers are busy.

Worker Servers

Worker servers perform the actual processing.

Each worker:

• Reads a job from SQS
• Creates an isolated Docker container
• Executes the submitted code
• Captures the output or error
• Saves the result
• Removes the completed message from the queue

Docker

Docker creates an isolated environment for each execution.

Instead of running user code directly on the worker machine, the code runs inside a temporary container with a selected language runtime.

Execution time, memory, CPU usage, and access to the host system can be restricted.

DynamoDB

DynamoDB stores the basic execution status and final result.

The client uses the execution ID to check whether the job is still waiting, currently running, completed, failed, or timed out.

1. The User Submits Code

The client sends a request:

POST /execution

POST /execution

For example:

{
  "language": "python",
  "code": "print('Hello World')"
}

{
  "language": "python",
  "code": "print('Hello World')"
}

2. NGINX Routes the Request

NGINX receives the public request and forwards it to one of the available API servers.

3. The API Validates the Input

The API checks that:

• Code is provided
• A supported language is selected
• The request is within the allowed size
• The client has not exceeded the rate limit

4. An Execution ID Is Created

The system creates a unique execution ID.

This ID is used to track the job through the queue, worker, and result-retrieval process.

5. The Job Is Added to SQS

The API sends the submitted code, language, and execution ID to Amazon SQS.

The client immediately receives:

202 Accepted
{
  "executionId": "0190f3a7...",
  "status": "queued"
}

202 Accepted
{
  "executionId": "0190f3a7...",
  "status": "queued"
}

The client does not have to wait for the code to finish.

6. A Worker Picks Up the Job

An available worker receives the message from SQS.

The execution status changes from:

queued → processing

queued → processing

7. The Code Runs Inside Docker

The worker creates a temporary Docker container using the required language runtime.

Inside the container, the system:

• Adds the submitted code file
• Runs the selected language
• Captures output and errors
• Measures execution time
• Stops programs that exceed the timeout

8. The Result Is Saved

The result is saved with a final status such as:

completed
failed
timeout

completed
failed
timeout

The processed SQS message is then deleted so it is not executed again.

9. The Client Retrieves the Result

The client requests:

GET /execution/{executionId}

GET /execution/{executionId}

The system returns the latest status and, when available, the final output.

Why the Queue Makes the System Scalable

The queue is the most important part of the architecture.

Without SQS, every incoming request would need an available worker immediately.

With SQS, requests can wait safely until processing capacity becomes available.

More requests arrive
        ↓
More jobs wait in SQS
        ↓
Available workers process jobs
        ↓
Additional workers can be added
        ↓
The backlog is processed faster

More requests arrive
        ↓
More jobs wait in SQS
        ↓
Available workers process jobs
        ↓
Additional workers can be added
        ↓
The backlog is processed faster

The API and worker layers can therefore scale independently.

More API servers can be added when HTTP traffic increases, while more worker servers can be added when the execution queue grows.

Load Testing the Architecture

After completing the architecture, I performed load tests to understand how the system behaved under multiple simultaneous requests.

The purpose of the load tests was to observe:

• How many requests the API could accept
• Whether NGINX distributed traffic correctly
• How SQS handled sudden request spikes
• How quickly workers processed the queue
• How response time changed under higher load
• Whether executions completed successfully under pressure

The tests helped confirm the main advantage of the asynchronous design:

Even when worker capacity is fully occupied, the API can continue accepting valid requests and place them safely in the queue.

The execution time may increase while the queue is busy, but the API itself does not need to remain blocked waiting for every program to finish.

Here are some Load Test images :

Using 1 container on Worker Machine

Using 2 container on worker machine:

Why I Did Not Run Code Inside the API

Running code directly inside the API would create several problems:

• Long programs would keep requests open
• High traffic could block the API
• A failed execution could affect request handling
• Scaling would require scaling the entire application
• Unsafe code would run too close to the public API

By separating the layers, each part has one clear responsibility:

NGINX       → Routes traffic
API servers → Accept and validate requests
SQS         → Holds execution jobs
Workers     → Process queued jobs
Docker      → Isolates submitted code
DynamoDB    → Stores status and results

NGINX       → Routes traffic
API servers → Accept and validate requests
SQS         → Holds execution jobs
Workers     → Process queued jobs
Docker      → Isolates submitted code
DynamoDB    → Stores status and results

This makes the system easier to scale, test, maintain, and understand.

GitHub Repository

Add your project repository here:

https://github.com/saif8364/Runit-Cloud

https://github.com/saif8364/Runit-Cloud

The complete API, worker implementation, Docker configuration, infrastructure files, and load-testing setup are available in the GitHub repository.