Post cover image

August 17, 2026

The Evolution of Authentication: From Passwords to Refresh Tokens (Part 1)

Why did we ever need Sessions, API Keys, Bearer Tokens, JWT, and Refresh Tokens? Here’s the real story — told in the order it actually…

By prasanna kml

13 min read

Why did we ever need Sessions, API Keys, Bearer Tokens, JWT, and Refresh Tokens? Here's the real story — told in the order it actually happened, not the way tutorials usually explain it.

Most authentication articles start with a definition: "JWT is a compact, URL-safe token used for authentication." That's technically correct — and almost useless. It tells you what JWT is, not why it had to exist.

Here's the truth: every authentication mechanism in use today exists because the one before it broke under some new kind of pressure. Plaintext passwords failed, so we hashed them. Hashing didn't protect the wire, so we added HTTPS. Re-sending passwords was wasteful, so we invented sessions. Sessions didn't fit non-browser clients, so we standardized headers. Headers still leaked real passwords, so we built API Keys and Bearer Tokens. Tokens still needed a database lookup on every request, so we built JWT. And JWT's biggest strength — being stateless — turned out to be its biggest weakness the moment you need to revoke one early. That's what refresh tokens fix.

This is Part 1 of a two-part series. Before diving in, here's the full landscape of authentication methods in use today — so you can see where this article fits, and what's coming next.

Credential-based

  1. Username & Password — the baseline credential check
  2. Password Hashing — protecting what's stored, not a standalone auth method but foundational to every credential system
  3. One-Time Passwords (OTP/TOTP) — a short-lived code from an app like Google Authenticator or sent via SMS, usually paired with a password as a second factor

Transport & session layer

  1. HTTPS (TLS) — protecting credentials and tokens in transit; not an auth method itself, but a prerequisite for all of them
  2. Sessions (cookie-based) — stateful, server-remembers-you, browser-native

Header/token-based (stateless)

  1. Basic Authentication — credentials Base64-encoded in every request
  2. Digest Authentication — an older, less common HTTP standard that hashes credentials with a server-issued nonce instead of sending them in the clear like Basic Auth; mostly superseded today but still found in legacy systems and some IoT/embedded devices
  3. API Keys — identifies the application, not the user
  4. Bearer Tokens — identifies the user, without cookies
  5. JWT (JSON Web Token) — a self-contained, signed bearer token format
  6. Refresh Tokens — long-lived tokens used to silently obtain new short-lived access tokens

Delegated & federated identity

  1. OAuth 2.0 — an authorization framework (not authentication) for letting one app act on a user's behalf without handling their password
  2. OpenID Connect (OIDC) — an authentication layer built on top of OAuth 2.0, which OAuth alone doesn't actually provide
  3. SAML — XML-based single sign-on, still dominant in enterprise and government systems
  4. Social Login ("Sign in with Google/GitHub/Apple") — OAuth/OIDC applied to consumer-facing login
  5. Single Sign-On (SSO) — the umbrella concept that OIDC and SAML both implement: one login, many applications

Multi-factor & biometric

  1. Multi-Factor Authentication (MFA/2FA) — combining two or more of "something you know, have, or are"
  2. Biometric Authentication — fingerprint, face, or voice, almost always paired with device-level secure hardware rather than sent over the network
  3. Passkeys (WebAuthn / FIDO2) — public-key based, phishing-resistant, passwordless login now pushed by Apple, Google, and Microsoft

Cryptographic & machine identity

  1. Mutual TLS (mTLS) / Client Certificates — both client and server present certificates; common in banking APIs and service meshes
  2. HMAC Request Signing — the client signs each request with a shared secret (used by AWS Signature v4, many webhook providers); proves integrity and origin without sending a reusable credential
  3. OAuth 2.0 Client Credentials Grant — the machine-to-machine flavor of OAuth, for service-to-service calls with no human user involved

Enterprise & network-level

  1. Kerberos — ticket-based authentication for internal enterprise networks, the backbone of Windows Active Directory environments
  2. LDAP-based Authentication — centralized directory lookups, often the system sitting behind enterprise SSO

Adaptive

  1. Risk-based / Adaptive Authentication — not a mechanism on its own, but a layer that adjusts requirements (e.g., triggers MFA) based on signals like device, location, or behavior

That's the real breadth of the field. This series can't do justice to all 25 in depth, so Part 1 covers items 1, 2, 4, 5, 6, 8, 9, 10, and 11 — the full historical chain from a plaintext password through to refresh tokens, which rounds out the entire header/token-based category. Part 2 continues with MFA, OAuth 2.0, OIDC, SAML, and passkeys. The rest — Digest Auth, OTP/TOTP, mTLS, HMAC signing, Kerberos, LDAP, and adaptive authentication — are flagged above because they're genuinely part of the landscape, even though they fall outside this series' scope.

Each section below follows the same shape — the problem, the fix, and the catch — so you can jump straight to whichever one you need.

1. Username & Password

Type: Credential-based, stateless

The problem: Before the modern web, applications only had to answer one question — who is this? The simplest possible answer was a username and password, stored directly.

Alice → password123

The catch: The database itself became the weak point. If it ever leaked, the attacker had the real password with no further work required. And because people reuse passwords, one leak could cascade into email, banking, and every other account sharing that password.

The fix wasn't better authentication logic — it was changing what the database was allowed to store at all.

2. Password Hashing

Type: Storage security layer

The problem: Storing the real password meant a single database leak was a full compromise.

The fix: Stop storing the password. Store a one-way hash of it instead, using a slow, salted hashing algorithm — bcrypt, scrypt, or, per current OWASP guidance, Argon2id.

password123 → $2b$12$KixQ4z8jY... (bcrypt hash)

A proper hashing algorithm automatically generates and stores a unique random salt per password. That matters — without a salt, two users with the same password would produce identical hashes, and attackers could precompute lookup tables (rainbow tables) to reverse them in bulk. The server never needs the original password again; at login, it re-hashes the input using the stored salt and compares the results.

The catch: This protects the password at rest. It does nothing while the password is still traveling from the browser to the server.

3. HTTPS (TLS)

Type: Transport security layer

The problem: Over plain HTTP, a login request is readable by anyone on the same network.

POST /login HTTP/1.1
username=alice
password=MyPassword123

Tools like Wireshark make this trivial to capture. Worse, a man-in-the-middle attacker can modify the request, not just read it.

The fix: Wrap the connection in TLS. An eavesdropper now sees only encrypted bytes.

The catch: This is table stakes, not a weakness — but it only solves the transit problem. It says nothing about how often a password has to travel in the first place.

With hashing and HTTPS in place, passwords are finally safe at rest and in transit. The next problem is about frequency, not exposure.

4. Sessions

Type: Stateful, cookie-based, browser-native

The problem: Browsing an e-commerce site means dozens of requests — login, view products, cart, orders, checkout. Resending your password on every single one is unnecessary exposure, and slow.

The fix: Authenticate once. The server creates a session ID and stores it server-side; the browser only holds a cookie.

Login → Server verifies → Creates Session ID → Stored server-side
Server → sends Session ID as a cookie
Browser → auto-resends that cookie on every request

The browser never sees "Alice" again, just a random ID like 9f3d81a4c5e8, mapped to Alice in a server-side store (commonly Redis or a database table).

The catch: Session hijacking. If an attacker steals the session ID — say, via an XSS attack reading cookies — the server can't distinguish them from the real user, because possession of the ID is the proof. The standard mitigation is cookie hardening:

  • HttpOnly — JavaScript can't read the cookie, blocking most XSS-based theft
  • Secure — the cookie is only ever sent over HTTPS
  • SameSite — blocks the cookie from being sent on many cross-site requests, reducing CSRF risk

A common follow-up question: why not just store the username and password in the browser and send it every time, instead of a session? You could — it's close to Basic Authentication, next — but it undoes everything hashing and sessions just fixed: the real password travels constantly, never expires on its own, and a leak means full account takeover instead of one revocable session.

Sessions work beautifully for browsers. Then the client landscape changed.

5. Basic Authentication

Type: Stateless, header-based, client-agnostic

The problem: Cookies need something to store and auto-resend them. Browsers do this natively — a Python script, a Java service, or a cron job doesn't. As APIs and non-browser clients multiplied, the industry needed one standard way for any HTTP client to send credentials.

The fix: A standardized Authorization header, understood by every HTTP library out of the box.

Authorization: Basic YWxpY2U6cGFzc3dvcmQxMjM=

That's just Base64 for alice:password123:

import base64
decoded = base64.b64decode("YWxpY2U6cGFzc3dvcmQxMjM=")
print(decoded.decode())  # alice:password123

The catch: Base64 is encoding, not encryption — anyone can reverse it instantly. Basic Auth doesn't fix credential security, it fixes interoperability. Real credentials still travel on every request, HTTPS is still mandatory, and there's no way to revoke access without changing the user's actual password.

Worth a myth-check here: Basic Auth isn't a "more secure evolution" of sessions. They solve different problems and largely coexisted — sessions for browsers, Basic Auth for any client that needed standardization without one.

6. API Keys

Type: Application identity, stateless

The problem: Suppose you run a Weather API and a Travel Website wants to call it. Giving that company a real username and password is dangerous — a leaked credential could log into dashboards, change billing, or delete resources, not just call one endpoint.

The fix: Issue a purpose-built, low-privilege secret instead.

X-API-Key: abc123xyz987
api_key = request.headers.get("X-API-Key")
if api_key in API_KEYS:
    print("Valid Application")
else:
    print("Unauthorized")

This key can only call the API, and it can be rotated or revoked instantly without touching any real password. In production, treat API keys the way you treat passwords: store a hash of the key, not the raw value, and scope each key to the minimum permissions and rate limits it actually needs — a leaked key should have a small, well-defined blast radius.

The catch: An API Key identifies the application, not the user. A Weather API doesn't care who clicked the button — everyone gets the same weather. A Banking API cares a lot. If ten million customers share one mobile app, they also share one API Key — it proves this is the official Bank app, but can't tell Alice's balance from Bob's.

That gap — proving which user, not just which app — is exactly what Bearer Tokens were built for.

7. Bearer Tokens

Type: Stateless, header-based, user identity

The problem: As mobile apps, SPAs, and microservices took over, sessions showed real cracks outside the browser. Mobile developers had to manually manage cookie storage, expiration, and domains. And in a microservices chain — API Gateway, User Service, Order Service, Payment Service — every service has to ask a shared store like Redis "who owns this session ID?", thousands of times a second.

The fix: Let the client carry proof of identity itself, instead of the server storing it centrally.

Authorization: Bearer abc123xyz

The catch: Bearer Tokens and API Keys aren't competitors — they're often used together, answering different questions.

POST /orders
X-API-Key: mobile-app-key           →  which application is calling?
Authorization: Bearer eyJhbGci...   →  which user is authenticated?

Authentication actually evolved along two parallel tracks, not one straight line:

  • User authentication: Password → Sessions → Bearer Tokens → JWT → OAuth/OIDC
  • Application authentication: Password → Basic Auth → API Keys → OAuth Client Credentials

Bearer Tokens solve "who's the user, without cookies." But the HTTP spec never defines what's inside the token — Authorization: Bearer <token> only says how to send it, not what format it takes. If the token is a random opaque string, verifying it still means a lookup — either a direct database check or a call to a token introspection endpoint. That undefined gap is exactly where JWT enters.

8. JWT (JSON Web Token)

Type: Self-contained, stateless, signed

The problem: Early Bearer Tokens were just random opaque strings. The server had to store and look one up on every single request.

xyz123 → Token Store → Alice

Fine at small scale. At Netflix or Google scale — where one client request can fan out across five or six microservices — that's thousands of lookups per second, and every service now depends on a shared store being fast and available.

The fix: Make the token carry the user data itself, instead of pointing to it.

Header.Payload.Signature

Header — describes the token:

{ "alg": "HS256", "typ": "JWT" }

Payload — the actual claims:

{
  "sub": "alice",
  "role": "admin",
  "exp": 1750000000
}

Signature — a cryptographic hash of header + payload + a secret key. Change "role": "admin" to "role": "superadmin" and the signature breaks instantly — the server rejects the token.

In Python, with PyJWT:

import jwt
import datetime
SECRET = "my-secret-key"
payload = {
    "sub": "alice",
    "role": "admin",
    "exp": datetime.datetime.utcnow() + datetime.timedelta(hours=1)
}
token = jwt.encode(payload, SECRET, algorithm="HS256")
print(token)
decoded = jwt.decode(token, SECRET, algorithms=["HS256"])
print(decoded)
# {'sub': 'alice', 'role': 'admin', 'exp': 1750000000}

No database round-trip needed. Any microservice holding the signing secret — or the public key, for asymmetric algorithms — can verify the token and read the claims independently.

What JWT actually solved:

  • Eliminated per-request token lookups
  • Let every microservice validate independently, with no shared store
  • Scales horizontally — ten servers or a thousand, validation logic doesn't change

The catch — and this is where most real-world JWT bugs live:

  • JWT doesn't eliminate the database. It removes the need to look up who owns the token, not the need to fetch application data — like Alice's actual orders — once you know who she is.
  • JWT is signed, not encrypted. The signature stops tampering, but the payload is just Base64URL-encoded. Anyone holding the token can decode and read it. Never put a password or secret inside a JWT payload.
  • Always validate the signing algorithm server-side, explicitly. One of the most exploited real-world JWT bugs is the alg: none attack — some early libraries would accept a token with "alg": "none" and no signature at all, treating it as valid. Modern libraries reject this by default, but always pin an explicit algorithm allowlist when verifying rather than trusting whatever alg the token claims.
  • Choose your signing algorithm deliberately. HS256 is symmetric — one secret both signs and verifies, so every service that needs to verify tokens also has the power to forge them. RS256 or ES256 are asymmetric — one private key signs, and any number of services can verify with a public key they hold safely, without ever being able to issue tokens themselves. For a multi-service architecture, asymmetric signing is usually the safer default.
  • Decide where the token lives on the client, deliberately. Storing a JWT in localStorage is convenient but readable by any JavaScript on the page — a single XSS vulnerability leaks every active token. An HttpOnly cookie (with Secure and SameSite, same as sessions above) keeps it out of reach of JavaScript entirely, at the cost of reintroducing CSRF considerations. There's no universally "correct" answer here, but it should be a deliberate choice, not a default.
  • Allow a small clock skew when validating exp. Servers rarely have perfectly synchronized clocks; a few seconds of tolerance avoids rejecting genuinely valid tokens.
  • Revocation is structurally hard. Because the token is self-contained, you can't easily invalidate it early. If Alice logs out, her JWT stays valid until exp expires unless you build extra infrastructure for revocation. That's exactly why refresh tokens exist — covered next.

One clarification that resolves a lot of confusion: Bearer and JWT are not the same thing. Bearer describes how the token is sent — an HTTP authentication scheme. JWT describes what the token looks like — its structure and encoding. JWT didn't replace Bearer Tokens; it became the most popular format for them.

9. Refresh Tokens

Type: Long-lived, paired with a short-lived access token

The problem: JWT leaves you stuck between two bad options. Make the access token long-lived, and a stolen token stays dangerously valid for days or weeks with no easy way to cut it off. Make it short-lived instead, and now the user gets logged out every 15 minutes and has to re-enter credentials constantly — which just drags everyone back toward the password-on-every-request problem this whole chain was trying to escape.

The fix: Stop trying to make one token do both jobs. Issue two tokens at login instead.

Login → Server issues:
  Access Token   (short-lived,  e.g. 15 minutes, usually a JWT)
  Refresh Token  (long-lived,   e.g. 7–30 days, opaque and stored server-side)

The client uses the access token for every API call, exactly as before. When it expires, the client doesn't ask the user to log in again — it silently sends the refresh token to a dedicated endpoint and gets a brand-new access token back.

Access Token expires
        │
        ▼
Client → POST /token  { refresh_token: "..." }
        │
        ▼
Server validates refresh token → issues new Access Token

This is a deliberate reversal of the earlier JWT trade-off. The access token stays stateless and short-lived, so a theft only matters for a few minutes. The refresh token is stored server-side, specifically so it can be revoked — logging Alice out everywhere just means deleting her refresh token record, the same clean revocation sessions always had.

The catch: A refresh token is a higher-value target than any single access token, precisely because it's long-lived — if it leaks, an attacker can keep minting fresh access tokens until someone notices. Two mitigations are now standard practice:

  • Refresh token rotation — issue a brand-new refresh token every time one is used, and invalidate the old one immediately. A refresh token becomes single-use.
  • Reuse detection — if a refresh token that's already been rotated out gets used again, that's a strong signal it was stolen and used in parallel by an attacker. The correct response is to revoke the entire token family immediately, not just that one token.

Storage matters here just as much as it did for JWTs: a refresh token belongs in an HttpOnly, Secure cookie, never in localStorage, for the same XSS reasons discussed earlier.

The full comparison

  • Username & Password — no state; solves basic identity checks; main weakness is plaintext storage risk if not hashed
  • Password Hashing — protects stored credentials; doesn't protect data in transit
  • HTTPS — protects data in transit; doesn't reduce how often credentials are sent
  • Sessions — stateful; best for browser-based web apps; weak to session hijacking and server-side lookup costs at scale
  • Basic Authentication — stateless; best for simple scripts, CLI tools, automation; sends credentials on every request
  • API Keys — stateless; best for service-to-service and application identity; can't distinguish individual users
  • Bearer Tokens — stateless; best for mobile apps, SPAs, distributed systems; format isn't defined, so opaque tokens still need lookups
  • JWT — stateless; best for microservices and horizontal scale; hard to revoke early, and the payload is readable, not secret
  • Refresh Tokens — long-lived, server-side; best for keeping users logged in safely without long-lived access tokens; main risk is token theft, mitigated by rotation and reuse detection

Where this leaves us

Username & Password → Password Hashing → HTTPS → Sessions
→ Basic Authentication → API Keys → Bearer Tokens → JWT → Refresh Tokens

None of these methods are obsolete in an absolute sense. Sessions are still the right call for a traditional browser app. API Keys are still exactly right for service-to-service calls. The real skill isn't chasing the newest method, it's recognizing which problem you actually have.

Together, JWT and refresh tokens round out the entire header/token-based category — stateless validation for speed, paired with a revocable long-lived token for control. What's still missing is everything Part 2 picks up: multi-factor authentication, and the federated identity layer — OAuth 2.0, OIDC, SAML, and passwordless passkeys — that lets one login work safely across many applications.