Overview

Anthos Service Mesh security helps you mitigate insider threats and reduce the risk of a data breach by ensuring that all communications between workloads are encrypted, mutually authenticated, and authorized.

In this lab, you learn how PERMISSIVE mode mTLS allows services to receive both plaintext and mTLS traffic from clients, allowing you to incrementally adopt mTLS. You also enabled STRICT mode mTLS across your service mesh effectively blocking plaintext traffic to all your Istio injected services and then you scoped STRICT mode mTLS down to a single namespace

Objectives

In this lab, you learn how to perform the following tasks:

  • Enforce STRICT mTLS mode across the service mesh
  • Enforce STRICT mTLS mode on a single namespace
  • Explore the security configurations in the Anthos Service Mesh Dashboard
  • Add authorization policies to enforce access based on a JSON Web Token (JWT)
  • Add authorization policies for HTTP traffic in an Istio mesh

Setup and requirements

login to gcp console, open cloud shell and authenticate to project by running the command

gcloud config set project "PROJECT_ID"

Task 1. Confirm Anthos Service Mesh setup

  • This lab environment has already been partially configured. A GKE cluster with 2 nodes has been provisioned for you. If you would like to investigate how the cluster creation process happened, look at the setup-vm startup-script.

Configure cluster access for kubectl

  • Set the Zone environment variable:
CLUSTER_ZONE="Zone added at lab start"
  • Set environment variables for the zone and cluster name:
export CLUSTER_NAME=gke
  • In Cloud Shell, configure kubectl command line access by running:
# get the project id
export GCLOUD_PROJECT=$(gcloud config get-value project)

# configure kubectl
gcloud container clusters get-credentials $CLUSTER_NAME \
    --zone $CLUSTER_ZONE --project $GCLOUD_PROJECT

Verify cluster and Anthos Service Mesh installation

  • Check that your cluster is up and running:
gcloud container clusters list

Output:

NAME: gke
LOCATION: "ZONE"
MASTER_VERSION: 1.28.8-gke.1095000
MASTER_IP: 35.186.181.121
MACHINE_TYPE: e2-standard-2
NODE_VERSION: 1.28.8-gke.1095000
NUM_NODES: 3
STATUS: RUNNING
  • Ensure the following Kubernetes istiod services are deployed:
kubectl get service -n istio-system

Output:

NAME                      TYPE             CLUSTER-IP
istiod-asm-1153-6         ClusterIP        10.7.249.134
istiod                    ClusterIP        10.7.249.95
  • Ensure the corresponding Kubernetes istiod-* pods are deployed and all containers are up and running:
kubectl get pods -n istio-system

Output:

NAME                                      READY   STATUS
istiod-asm-1153-6-6d66584796-865d8        1/1     Running
istiod-asm-1153-6-6d66584796-bph5h        1/1     Running
...

Task 2. Deploy sleep and httpbin services

  • In this task, you create a set of namespaces to host the httpbin, and sleep services. You will use those services to explore the impact of mTLS on traffic. The sleep service acts as the client and will call the httpbin service, which acts as a server.
  • Setup the authentication example
  • You will deploy this example configuration and then use it to explore the authentication options that Istio offers:
None
  • In Cloud Shell, create namespaces for the example clients and services. Traffic in the legacy-* namespaces takes place over plain text, while traffic in the mtls-* namespaces happens over mTLS:
kubectl create ns mtls-client
kubectl create ns mtls-service
kubectl create ns legacy-client
kubectl create ns legacy-service

kubectl get namespaces
  • Output:
NAME              STATUS   AGE
...
legacy-client     Active   5m50s
legacy-service    Active   5m34s
mtls-client       Active   5m56s
mtls-service      Active   5m40s
  • Deploy the legacy services in the legacy-* namespaces. You call them legacy because they are not part of the mesh:
#configurations are stored in Github

kubectl apply -f \
https://raw.githubusercontent.com/istio/istio/release-1.6/samples/sleep/sleep.yaml \
-n legacy-client

kubectl apply -f \
https://raw.githubusercontent.com/istio/istio/release-1.6/samples/httpbin/httpbin.yaml \
-n legacy-service
  • Enable auto-injection of the Istio sidecar proxy on the mtls-* namespaces:
# get the revision label
export DEPLOYMENT=$(kubectl get deployments -n istio-system | grep istiod)
export VERSION=asm-$(echo $DEPLOYMENT | cut -d'-' -f 3)-$(echo $DEPLOYMENT \
    | cut -d'-' -f 4 | cut -d' ' -f 1)

# enable auto-injection on the namespaces
kubectl label namespace mtls-client istio.io/rev=${VERSION} --overwrite
kubectl label namespace mtls-service istio.io/rev=${VERSION} --overwrite
  • Deploy the services in the mtls-* namespaces:
kubectl apply -f \ https://raw.githubusercontent.com/istio/istio/release-1.6/samples/sleep/sleep.yaml \ -n mtls-client kubectl apply -f \ https://raw.githubusercontent.com/istio/istio/release-1.6/samples/httpbin/httpbin.yaml \ -n mtls-service
  • Verify that the sleep service and the httpbin service are each deployed in both the mtls-service and legacy-service namespaces:
kubectl get services - all-namespaces
  • Output:
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE … legacy-client sleep ClusterIP 10.7.248.186 80/TCP 31s legacy-service httpbin ClusterIP 10.7.248.121 8000/TCP 14s mtls-client sleep ClusterIP 10.7.252.127 80/TCP 99s mtls-service httpbin ClusterIP 10.7.247.200 8000/TCP 50s
  • Verify that a sleep pod is running in the mtls-client and legacy-clientnamespaces and that an httpbin pod is running in the mtls-service and legacy-service namespaces:
kubectl get pods - all-namespaces

Output:

NAMESPACE NAME READY STATUS RESTARTS AGE … legacy-client sleep-f8cbf5b76–5v8f9 1/1 Running 0 65s legacy-service httpbin-779c54bf49–5fr9k 1/1 Running 0 47s mtls-client sleep-64d4546bdb-gnpc5 2/2 Running 0 2m13s mtls-service httpbin-5d7bdc9d78–7mfn5 2/2 Running 0 84s
  • Verify that the two sleep clients can communicate with the two httpbin services
  • Use Cloud Shell to run this nested command loop:
for from in "mtls-client" "legacy-client"; do for to in "mtls-service" "legacy-service"; do kubectl exec $(kubectl get pod -l app=sleep -n ${from} -o jsonpath={.items..metadata.name}) -c sleep -n ${from} - curl "http://httpbin.${to}:8000/ip" -s -o /dev/null -w "sleep.${from} to httpbin.${to}: %{http_code}\n" done done

Output:

sleep.mtls-client to httpbin.mtls-service: 200 sleep.mtls-client to httpbin.legacy-service: 200 sleep.legacy-client to httpbin.mtls-service: 200 sleep.legacy-client to httpbin.legacy-service: 200
  • Now you're ready to enforce security policies for this application.

Task 3. Understand authentication and enable service to service authentication with mTLS

Anthos Service Mesh

  • In the console, go to Navigation Menu > Kubernetes Engine > Service Mesh.
  • In the Anthos Service Mesh dashboard, you will see the 2 services that were created in the mtls-service and mtls-client/sleep namespaces. You don't see the legacy services because you did not label their namespace, and therefore they remain outside of the mesh.
  • In the list view section Click on > icon next to httpbin (mtls-service/httpbin) namespace.

Click on Go to dashboard

  • In the left side panel, go to Connected Services.
  • Notice that you have 2 services:
  • The sleep service in the mtls-client namespace, which is part of the mesh and has a sidecar proxy. Therefore you see the real name and communication goes over mTLS, as it's the default behavior in Istio.
  • An unknown service, which represents the sleep service in the legacy-client, which is not part of the mesh and has no sidecar proxy. Therefore you do not see the real name and the communication goes over plain text.
  • Use your mouse to hover over the lock symbol in the Request port column, and verify that green means mTLS and red means plain text.
None
  • Now check out the Security tab in the left side panel. It shows you that the httpbin service has received both plaintext and mTLS traffic.
None
  • Test auto mutual TLS
  • By default, Istio configures destination workloads in PERMISSIVE mode. WhenPERMISSIVE mode is enabled a service can accept both plaintext and mTLS traffic. mTLS is used when the request contains the X-Forwarded-Client-Cert header.
  • Use the Cloud Shell to send a request from the sleep service in the mtls-clientnamespace to the httpbin service in the mtls-service namespace:
kubectl exec $(kubectl get pod -l app=sleep -n mtls-client -o jsonpath={.items..metadata.name}) -c sleep -n mtls-client - curl http://httpbin.mtls-service:8000/headers -s | grep X-Forwarded-Client-Cert
  • Output:
"X-Forwarded-Client-Cert": "By=spiffe:// …
  • The traffic included the X-Forwarded-Client-Cert header and therefore was mutually authenticated and encrypted
  • Now send a request from the sleep service in the mtls-client namespace to the httpbin service in the legacy-service namespace:
kubectl exec $(kubectl get pod -l app=sleep -n mtls-client -o jsonpath={.items..metadata.name}) -c sleep -n mtls-client - curl http://httpbin.legacy-service:8000/headers -s | grep X-Forwarded-Client-Cert
  • The X-Forwarded-Client-Cert header isn't present so the traffic was sent and received in plaintext.
  • Finally, send a request from the sleep service in the legacy-client namespace to the httpbin service in the mtls-service namespace:
kubectl exec $(kubectl get pod -l app=sleep -n legacy-client -o jsonpath={.items..metadata.name}) -c sleep -n legacy-client - curl http://httpbin.mtls-service:8000/headers -s | grep X-Forwarded-Client-Cert
  • The X-Forwarded-Client-Cert header isn't present so the traffic was sent and received in plaintext
  • Note: The httpbin service in the mtls-service namespace accepted mTLS traffic from the sleep service in the mtls-client namespace and plaintext from the sleepservice in the legacy-client namespace.
  • Enforce STRICT mTLS mode across the service mesh
  • In STRICT mode, services injected with the Istio proxy will not accept plaintext traffic and will mutually authenticate with their clients.
  • You can enforce STRICT mTLS mode across the whole mesh or on a per-namespace basis by creating PeerAuthentication resources.
None
  • Create a Peer Authentication resources for the entire Service Mesh:
kubectl apply -n istio-system -f - <<EOF apiVersion: "security.istio.io/v1beta1" kind: "PeerAuthentication" metadata: name: "mesh-wide-mtls" spec: mtls: mode: STRICT EOF
  • Run this nested command loop:
for from in "mtls-client" "legacy-client"; do for to in "mtls-service" "legacy-service"; do kubectl exec $(kubectl get pod -l app=sleep -n ${from} -o jsonpath={.items..metadata.name}) -c sleep -n ${from} - curl "http://httpbin.${to}:8000/ip" -s -o /dev/null -w "sleep.${from} to httpbin.${to}: %{http_code}\n" done done
sleep.mtls-client to httpbin.mtls-service: 200 sleep.mtls-client to httpbin.legacy-service: 200 sleep.legacy-client to httpbin.mtls-service: 000 command terminated with exit code 56 sleep.legacy-client to httpbin.legacy-service: 200
  • Note: The httpbin service in the mtls-service namespace now rejects the plaintext traffic it receives from the sleep client in the legacy-client namespace.
  • Remove the mesh wide mTLS PeerAuthentication resource by running this command in Cloud Shell:
kubectl delete pa mesh-wide-mtls -n istio-system
  • Enforce STRICT mTLS mode on a single namespace
  • In Cloud Shell create a namespace for STRICT mTLS:
kubectl create ns strict-mtls-service
  • Enable auto-injection of the Istio sidecar proxy on the new namespace:
# get the revision label export DEPLOYMENT=$(kubectl get deployments -n istio-system | grep istiod) export VERSION=asm-$(echo $DEPLOYMENT | cut -d'-' -f 3)-$(echo $DEPLOYMENT \ | cut -d'-' -f 4 | cut -d' ' -f 1) # enable auto-injection on the namespaces kubectl label namespace strict-mtls-service istio.io/rev=${VERSION} - overwrite
  • Use Cloud Shell to deploy another instance of the httpbin service in the strict-mtls-service namespace:
kubectl apply -f \ https://raw.githubusercontent.com/istio/istio/release-1.6/samples/httpbin/httpbin.yaml \ -n strict-mtls-service
  • Create a PeerAuthentication resource for the strict-mtls-service namespace:
kubectl apply -n strict-mtls-service -f - <<EOF apiVersion: "security.istio.io/v1beta1" kind: "PeerAuthentication" metadata: name: "restricted-mtls" namespace: strict-mtls-service spec: mtls: mode: STRICT EOF
  • Verify that the httpbin service in the mtls-service namespace still accepts plaintext traffic:
kubectl exec $(kubectl get pod -l app=sleep -n legacy-client -o jsonpath={.items..metadata.name}) -c sleep -n legacy-client - curl "http://httpbin.mtls-service:8000/ip" -s -o /dev/null -w "sleep.legacy-client to httpbin.mtls-service: %{http_code}\n"
  • Output:
sleep.legacy-client to httpbin.mtls-service: 200
  • Now check to see that the strict-mtls-service namespace httpbin service does not accept plaintext traffic:
kubectl exec $(kubectl get pod -l app=sleep -n legacy-client -o jsonpath={.items..metadata.name}) -c sleep -n legacy-client - curl "http://httpbin.strict-mtls-service:8000/ip" -s -o /dev/null -w "sleep.legacy-client to httpbin.strict-mtls-service: %{http_code}\n"
  • Output:
sleep.legacy-client to httpbin.strict-mtls-service: 000 command terminated with exit code 56
  • Verify that the httpbin service in the strict-mtls-service namespace does accept mTLS traffic:
kubectl exec $(kubectl get pod -l app=sleep -n mtls-client -o jsonpath={.items..metadata.name}) -c sleep -n mtls-client - curl "http://httpbin.strict-mtls-service:8000/ip" -s -o /dev/null -w "sleep.mtls-client to httpbin.strict-mtls-service: %{http_code}\n"
  • Output:
sleep.mtls-client to httpbin.strict-mtls-service: 200
  • In the Google Cloud console, select Navigation Menu > Kubernetes Engine > > Service Mesh.
  • In Anthos Service Mesh dashboard, you will see that you have 3 services now.
  • In the list view section Click on > icon next to strict-mtls-service/httpbinnamespace.
  • Click on Go to dashboard
  • In the left side panel, click on Connected Services.
  • Use your mouse to hover over the lock symbol in the Request Port column to see that only mTLS traffic has been received. Traffic might take a couple of minutes to be reflected in the dashboard. If you don't see the sleep service, wait 1-2 minutes and refresh the dashboard.
  • Now check out the Security tab in the left side panel. Again only mTLS traffic has been received.
  • Remove the strict-mtls-service peer authentication policy by running this command in Cloud Shell:
kubectl delete pa restricted-mtls -n strict-mtls-service

Task 4. Leverage RequestAuthentication and AuthorizationPolicy resources

  • This task shows you how to set up and use RequestAuthentication and AuthorizationPolicy resources. Ultimately, you will allow requests that have an approved JWT, and deny requests that don't.

RequestAuthentication

  • A RequestAuthentication resource defines the request authentication methods that are supported by a workload. Requests with invalid authentication information will be rejected. Requests with no authentication credentials will be accepted but will not have any authenticated identity.
  • Create a RequestAuthentication resource for the httpbin workload in themtls-service namespace. This policy allows the workload to accept requests with a JWT issued by testing@secure.istio.io.
kubectl apply -f - <<EOF apiVersion: "security.istio.io/v1beta1" kind: "RequestAuthentication" metadata: name: "jwt-example" namespace: mtls-service spec: selector: matchLabels: app: httpbin jwtRules: - issuer: "testing@secure.istio.io" jwksUri: "https://raw.githubusercontent.com/istio/istio/release-1.8/security/tools/jwt/samples/jwks.json" EOF
  • Verify that a request with an invalid JWT is denied:
kubectl exec "$(kubectl get pod -l app=sleep -n mtls-client -o jsonpath={.items..metadata.name})" -c sleep -n mtls-client - curl "http://httpbin.mtls-service:8000/headers" -s -o /dev/null -H "Authorization: Bearer invalidToken" -w "%{http_code}\n"
401
  • Verify that a request without any JWT is allowed:
kubectl exec "$(kubectl get pod -l app=sleep -n mtls-client -o jsonpath={.items..metadata.name})" -c sleep -n mtls-client - curl "http://httpbin.mtls-service:8000/headers" -s -o /dev/null -w "%{http_code}\n"
  • Output:
200

AuthorizationPolicy

  • Create an AuthorizationPolicy resource for the httpbin workload in the mtls-service namespace:
kubectl apply -f - <<EOF apiVersion: security.istio.io/v1beta1 kind: AuthorizationPolicy metadata: name: require-jwt namespace: mtls-service spec: selector: matchLabels: app: httpbin action: ALLOW rules: - from: - source: requestPrincipals: ["testing@secure.istio.io/testing@secure.istio.io"] EOF
  • The policy requires all requests to the httpbin workload to have a valid JWT with requestPrincipal set to testing@secure.istio.io/testing@secure.istio.io. Istio constructs the requestPrincipal by combining the iss and sub of the JWT token with a / separator as shown:
  • Download a legitimate JWT that can be used to send accepted requests:
TOKEN=$(curl https://raw.githubusercontent.com/istio/istio/release-1.8/security/tools/jwt/samples/demo.jwt -s) && echo "$TOKEN" | cut -d '.' -f2 - | base64 - decode -
  • Output:
{"exp":4685989700,"foo":"bar","iat":1532389700,"iss":"testing@secure.istio.io","sub":"testing@secure.istio.io"}
  • Note that the iss and sub keys are set to testing@secure.istio.io. This causes Istio to generate the attribute requestPrincipal with the value testing@secure.istio.io/testing@secure.istio.io:
  • Verify that a request with a valid JWT is allowed:
kubectl exec "$(kubectl get pod -l app=sleep -n mtls-client -o jsonpath={.items..metadata.name})" -c sleep -n mtls-client - curl "http://httpbin.mtls-service:8000/headers" -s -o /dev/null -H "Authorization: Bearer $TOKEN" -w "%{http_code}\n"
  • Output:
200
  • Verify that a request without a JWT is denied:
kubectl exec "$(kubectl get pod -l app=sleep -n mtls-client -o jsonpath={.items..metadata.name})" -c sleep -n mtls-client - curl "http://httpbin.mtls-service:8000/headers" -s -o /dev/null -w "%{http_code}\n"
  • Output:
403

Task 5. Authorizing requests based on method and path

  • This task shows you how to control access to workloads by using an AuthorizationPolicy that evaluates the request type and URL.
  • Update the require-jwt authorization policy for the httpbin workload in the mtls-service namespace. The new policy will still have the JWT requirement that you set up in the previous task. In addition, you are going to limit the type of HTTP requests, so that clients can only perform GET requests to the /ipendpoint:
kubectl apply -f - <<EOF apiVersion: security.istio.io/v1beta1 kind: AuthorizationPolicy metadata: name: require-jwt namespace: mtls-service spec: selector: matchLabels: app: httpbin action: ALLOW rules: - from: - source: requestPrincipals: ["testing@secure.istio.io/testing@secure.istio.io"] to: - operation: methods: ["GET"] paths: ["/ip"] EOF
  • Verify that a request to the httpbin's /ip endpoint works:
kubectl exec "$(kubectl get pod -l app=sleep -n mtls-client -o jsonpath={.items..metadata.name})" -c sleep -n mtls-client - curl "http://httpbin.mtls-service:8000/ip" -s -o /dev/null -H "Authorization: Bearer $TOKEN" -w "%{http_code}\n"
  • Output:
200
  • Verify that a request to the httpbin's /headers endpoint is denied:
kubectl exec "$(kubectl get pod -l app=sleep -n mtls-client -o jsonpath={.items..metadata.name})" -c sleep -n mtls-client - curl "http://httpbin.mtls-service:8000/headers" -s -o /dev/null -H "Authorization: Bearer $TOKEN" -w "%{http_code}\n"
  • Output:
403
  • Remove the require-jwt authorization policy by running this command:
kubectl delete AuthorizationPolicy require-jwt -n mtls-service

Review

  • In this lab, you explored mutual TLS authentication in Istio. You saw how PERMISSIVE mode mTLS allows services to receive both plaintext and mTLS traffic from clients, allowing you to incrementally adopt mTLS. You also enabled STRICT mode mTLS across your service mesh effectively blocking plaintext traffic to all your Istio injected services and then you scoped STRICT mode mTLS down to a single namespace.
  • In addition, you explored RequestAuthentication and AuthorizationPolicy resources in Istio.