August 28, 2026

The FTP Bug That Only Azure Could Give Us (And Why We’re Grateful It Did)

A story about legacy protocols, cloud migrations, and the humbling experience of finally reading the manual

By Sandeep

9 min read

A story about legacy protocols, cloud migrations, and the humbling experience of finally reading the manual

Every engineering team has at least one piece of infrastructure that nobody talks about — not because it's broken, but because it works. It sits quietly in a corner of the architecture diagram, doing its unglamorous job, and everyone silently agrees not to poke it.

For us, that piece was FTP.

The Protocol Nobody Wanted, But Everyone Needed

We run a video telematics platform — thousands of connected devices in the field, ours built around hardware from Howen, streaming and uploading footage and telemetry data continuously. Somewhere in that pipeline, a file has to move from a device to our backend. And the device, like a lot of embedded hardware out there, speaks FTP. Not S3 APIs, not SFTP, not some modern object-storage protocol. Plain, old-fashioned FTP.

If you've spent any time in infrastructure, you already know why that raises an eyebrow. FTP was designed in 1971 — before TLS, before NAT, before anyone had thought seriously about credential security. Using it today comes with baggage:

  • No encryption by default. Credentials and file contents travel in the clear unless you bolt on FTPS.
  • A connection model that doesn't scale gracefully. Every single file transfer needs its own dedicated, ephemeral network port — which we'll get into in detail, because it's the entire reason this story exists.
  • Firewall and NAT hostility. Because of that dynamic port requirement, FTP is notoriously difficult to run cleanly behind modern cloud load balancers, which were mostly designed with the assumption that a service listens on one predictable port.

None of that mattered, though, because we didn't get to choose the protocol — the device did. Our job wasn't to pick the ideal transport layer; it was to build a backend that speaks the language our hardware already speaks. So we wrote our own lightweight FTP server, deployed it, pointed our devices at it, and it worked. Beautifully. Quietly. For a long time.

And when something works quietly for a long time, you stop thinking about how it works. That complacency is where this story really begins.

The Migration Nobody Asked For

Every infrastructure team eventually faces a moment where "it's fine where it is" stops being an acceptable answer. For us, that moment came when a significant cloud-provider incident made single-cloud dependency feel like a liability rather than a convenience. Multi-cloud resilience stopped being a nice-to-have on a roadmap and became an urgent, board-level priority. We needed our platform running — reliably — on Azure, not just AWS.

For most of our stack, this was refreshingly uneventful. We run everything in Kubernetes, and Kubernetes is Kubernetes — the same manifests, the same Helm charts, the same mental model, regardless of which cloud is underneath. Our APIs came up. Our data pipelines came up. Our dashboards came up. One by one, every service in our platform crossed over to Azure and just… worked.

Except one.

The FTP server started. It accepted TCP connections. Devices could log in. And then — nothing. Uploads simply hung. No error. No rejection. Just silence, until something eventually timed out.

The "If It Ain't Broke" Trap

Here's the uncomfortable truth we had to sit with: we had been running this FTP server successfully for a long time without actually understanding how FTP works under the hood. We knew it as a black box that took files in on one side and produced files on S3 on the other. It had never given us a reason to look closer.

That's a trap every engineer falls into eventually. When something works, curiosity has no foothold — there's no incentive to dig into internals when the thing in front of you is doing its job. It's only when it breaks, in a way that resists every quick fix you throw at it, that you're finally forced to actually read the protocol instead of trusting that it'll keep behaving the way it always has.

We tried the obvious things first. Restarted pods. Re-checked our application config. Compared environment variables. Nothing. The application code was byte-for-byte identical to what was running successfully on AWS. Whatever was wrong wasn't in our code — which meant it had to be something about the network path itself. And to understand a network path problem in FTP, you first have to understand what FTP actually asks of a network.

FTP 101: Why One Port Was Never Going to Be Enough

Here's the thing about FTP that a lot of engineers — us included, until this week — never have to think about: it isn't one connection. It's two, running at the same time, doing completely different jobs.

1. The control connection — one persistent, always-open port.

This is where a client logs in and issues commands: USER, PASS, PWD, CWD, STOR, and so on. Think of it as the phone line between client and server — always up, always listening, ready to receive instructions at any moment. In our setup, this lives on port 7575.

2. The data connection — a temporary, on-demand port, opened fresh per transfer.

Here's the part that trips people up: commands alone don't move file bytes. Once the client says "I want to upload a file," the control connection isn't used to carry the actual data. Instead, the server dynamically opens a brand-new, temporary port, tells the client exactly which port that is, and the client opens a second, independent connection to it — just for that one file. Once the transfer finishes, that port closes again, freed up for the next session to use.

This is called passive mode, and it's the industry-standard way of doing FTP data transfers today (versus the older active mode, where the server connects back to the client — a model that fails outright behind almost any modern firewall or NAT). In passive mode, the client always initiates both connections, which plays nicer with firewalls, but introduces a new requirement: the server needs a whole range of ports available for these on-demand data connections, because multiple sessions might be transferring files simultaneously and each one needs its own dedicated data port for the duration of its transfer.

In our config, that range is 7576–7586 — eleven ports, meaning up to eleven concurrent file transfers before a new session has to wait for one to free up. (And yes — you're right to clock that as a scaling concern. A narrow passive port range becomes a real bottleneck the moment your device fleet grows large enough to be uploading concurrently. It's one more entry on FTP's list of sins, right alongside the lack of built-in encryption.)

Watching It Happen, Live

The best way to actually see this two-connection dance is to do it by hand. Here's how we walked through it ourselves, and how you can too, against any FTP server:

telnet ftp.upload.example.com 7575
USER s3
PASS s3
EPSV

The server responds with something like:

229 Entering Extended Passive Mode (|||7577|)

That's the server saying: "For this session's file transfer, connect to port 7577." That number isn't fixed — it's chosen dynamically, on the spot, from the configured passive range, and it belongs exclusively to this one session until the transfer completes.

While that control session is still open, open a second terminal and try:

telnet ftp.upload.example.com 7577

It connects — because right now, in this exact moment, something is actually listening there. Let the session finish (or close the control connection), and try that same telnet again: it'll fail, because the port has already been released back into the pool for the next session to use.

This little experiment is the whole story in miniature: the passive port only exists, and only accepts traffic, for the lifetime of one active transfer. That single fact — obvious once you've seen it, invisible until you go looking — turned out to be exactly what broke us in Azure.

Same Code, Same Ports, Two Different Clouds

Here's where it got genuinely strange. We confirmed, port by port, that both environments were configured identically:

  • Same persistent port (7575) — open and reachable in both.
  • Same passive port range (7576–7586) — declared identically in both Kubernetes Service manifests, verified line by line.
  • Same application binary, same startup config, same everything.

On AWS: uploads worked flawlessly, every time. On Azure: the control connection worked fine — login, PWD, CWD, all of it — but the moment a transfer tried to use one of the passive ports, the connection would just... hang. Not "connection refused." Not an error. A dead, silent nothing, until something eventually gave up and timed out.

That distinction — refused versus silent hang — turned out to be the clue that cracked the whole thing open. "Refused" means a packet reached its destination and got an active response saying "nothing's here." A silent hang means the packet never got a response at all — something upstream, before it ever reached our application, was simply swallowing it.

That pointed us away from our own code entirely, and straight at the one layer we hadn't thought to question: what each cloud's load balancer does by default, automatically, without ever being asked.

The Default Nobody Told Us About

Every cloud load balancer needs a way to decide whether it's safe to send traffic to a given backend port. That mechanism is a health probe — a periodic, automated check that asks, in effect, "is anything actually listening here right now?" If the answer is no, the load balancer marks that port unhealthy and quietly stops routing traffic to it. No error to the client. No log entry anyone would think to check. Just silence — which, in hindsight, is exactly what we'd been staring at.

Here's the mismatch: our passive ports are only ever "listening" while a transfer is actively in progress. The rest of the time — which, statistically, is nearly always — they're closed by design. That's not a bug. That's the entire architecture of FTP passive mode working exactly as intended.

AWS's load balancer, as it turned out, was configured (via our own Helm chart, without us fully appreciating why) to health-check the entire listener set against the one port that's reliably, permanently alive — the control port. It never independently questioned whether each individual passive port was "up" at that instant, because that was never the right question to ask in the first place.

Azure's load balancer, by contrast, defaults to probing every declared port independently, against itself. Ask "is 7577 listening right now?" often enough, and — because 7577 is only supposed to be alive during an active transfer — you'll get "no" almost every time. Azure took that "no" at face value, marked the port unhealthy, and stopped forwarding traffic to it entirely. Not because anything was actually broken. Because we were, unknowingly, asking the load balancer the wrong question.

The Fix: Teaching Azure to Ask the Right Question

Once we understood what was being asked, the fix became almost anticlimactic. We didn't need to open new ports, rewrite the application, or change a single line of Go. We needed to tell Azure's load balancer: "Don't check whether each passive port is alive on its own. Check whether the persistent control port is alive, and treat that as good enough for all of them."

Azure's AKS integration exposes exactly this as a per-port annotation on the Kubernetes Service object:

apiVersion: v1
kind: Service
metadata:
  name: ftp-server
  namespace: iot-platform
  annotations:
    service.beta.kubernetes.io/port_7576_health-probe_port: "7575"
    service.beta.kubernetes.io/port_7576_health-probe_protocol: "Tcp"
    service.beta.kubernetes.io/port_7577_health-probe_port: "7575"
    service.beta.kubernetes.io/port_7577_health-probe_protocol: "Tcp"
    # ...repeated for every port in the passive range...
    service.beta.kubernetes.io/port_7586_health-probe_port: "7575"
    service.beta.kubernetes.io/port_7586_health-probe_protocol: "Tcp"

Every passive port, individually redirected to check the one port that's genuinely, reliably always up. In effect, we're telling Azure: "7575 being alive is a trustworthy proxy for the whole server being healthy and capable of opening any of these passive ports on demand — even if none of them happen to be in active use right this second."

Nine-ish repeated annotation lines. That's the entire fix. Redeploy, and every earlier symptom reverses itself in order: telnet to a live passive port connects instantly, curl -v sails past the EPSV exchange without hanging, and the transfer completes with a clean success response, exactly the way it always had on AWS.

The Real Lesson Wasn't the Nine Lines

It would be easy to write this whole story off as "Azure load balancer gotcha, annotation fixes it, moving on." And technically, sure — that's the fix. But that undersells what actually happened here.

The real story is that we had been operating a piece of infrastructure for a long time without deeply understanding its core mechanics — not out of negligence, but because it had simply never needed to be understood. It worked. There was no incentive to look closer. It took a cloud migration, a genuinely confusing failure, and a fair amount of dead-end troubleshooting before we sat down and actually read up on how FTP's control and data connections interact — knowledge that, frankly, we should have had on day one of writing an FTP server, not two years and one cloud migration later.

There's a broader habit worth naming here, especially in a moment where it's so easy to reach for a quick AI-assisted fix or a Stack Overflow answer and move on: sometimes the fastest fix is the enemy of actually understanding your system. We didn't get to shortcut our way out of this one. The only way through was going back to fundamentals — actually learning what EPSV means, what a passive port range is for, why it exists at all — before the why behind the fix made any sense.

So if there's a takeaway here, it's not "add these annotations to your Azure load balancer." It's this: the next time something that's "just always worked" suddenly doesn't, resist the urge to just patch around it. Go find out why it worked in the first place. You might be surprised how much you didn't know about a system you've been running successfully for years.

Cheers — and may your passive ports always find a healthy home. 🚀

If you've hit something similar migrating FTP, SFTP, or any dynamic-port protocol between clouds, I'd love to hear how you solved it — drop a comment.

#FTP #SFTP #Azure #AWS #IOT #Howen