Using cloudflared and Terraform to expose services in Kubernetes

My setup is a little unconventional, but since I wanted to do this, I’m sure there’ll be someone else who wants to do this as well.

At the end of this, you will have configuration for exposing services from a Kubernetes cluster without setting up ingress controllers or cert-manager, as that will all be taken care of by Cloudflare. Adding configuration for new services will also be as minimal as possible, and you will be able to expose services in different namespaces with ease.

Prerequisites

Sign up for Cloudflare Zero Trust in order to be able to use Tunnels. There is a free tier with all required functionality, so you do not need to pay for this.
A Kubernetes cluster, somewhere. It does not need to be publicly accessible from the internet, it could even be on your home network. You will also need some service running on it to verify that your setup is working.
Terraform. I personally use Terraform Cloud, a service provided by Hashicorp, but as long as you can plan and apply, you can run it from anywhere.

Configuration

Providers

We’re going to be creating resources in both Cloudflare and Kubernetes, so both of those providers will be needed. In addition, the hashicorp/random provider will be used to generate a secret for the tunnel.

hcl
terraform {
  required_providers {
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "2.7.1"
    }
    cloudflare = {
      source  = "cloudflare/cloudflare"
      version = "3.8.0"
    }
    random = {
      source  = "hashicorp/random"
      version = "3.1.0"
    }
  }
}
provider "kubernetes" {
  # Add your own cluster configuration here
}
provider "cloudflare" {
  email      = var.cloudflare_email
  api_key    = var.cloudflare_api_key
  account_id = var.cloudflare_account_id
}
provider "random" {}
variable "cloudflare_email" {
  type        = string
}
variable "cloudflare_api_key" {
  type        = string
  sensitive   = true
}
variable "cloudflare_account_id" {
  type        = string
}

The Cloudflare account ID is necessary for some of the resources created later.

Creating the tunnel

The first real step is to create the tunnel in Cloudflare.

hcl
locals {
  tunnel_name = "<your-tunnel-name>"
}
resource "random_id" "tunnel_secret" {
  byte_length = 35
}
resource "cloudflare_argo_tunnel" "tunnel" {
  account_id = var.cloudflare_account_id
  name       = local.tunnel_name
  secret     = random_id.tunnel_secret.b64_std
}

Since the tunnel name is referenced in a couple of locations, I’ve created a local variable for it.

One note about the random_id resource: it’s not marked as sensitive in Terraform, and so will be visible in your Terraform state. I think this is a bit of an oversight, given that it is often used in the community specifically to generate secrets like this. To get around this, the best option is likely to generate the secret outside of Terraform, and pass it in as a variable.

One other curious thing is the length of the secret generated: 35 bytes. I chose this for two reasons:

The minimum required lenght is 32 bytes.
An example in one of Cloudflare’s blogs used 35, so I copied that. 🤷

Running cloudflared in Kubernetes

Now that the tunnel is created, we need to run cloudflared in the cluster to be able to route traffic to our services. This is also where we define which services we want to expose, as that gets added to the configuration for cloudflared.

hcl
locals {
  service_pairs = {
    "<subdomain.your.zone>" = "http://<service-name>.<namespace>.svc.cluster.local:80",
  }
  cfd_config_ingress = join("\n", [for host, service in local.service_pairs : "- hostname: ${host}\n  service: ${service}"])
}
resource "kubernetes_namespace_v1" "cloudflared" {
  metadata {
    name = "cloudflared"
    labels = {
      "app" = "cloudflared"
    }
  }
}
resource "kubernetes_config_map_v1" "config" {
  metadata {
    name      = "config"
    namespace = kubernetes_namespace_v1.cloudflared.metadata.0.name
    labels = {
      "app"  = "cloudflared"
      "tier" = "app"
    }
  }
  data = {
    "config.yaml" = <<EOF
# Name of the tunnel you want to run
tunnel: ${local.tunnel_name}
credentials-file: /etc/cloudflared/creds/credentials.json
metrics: 0.0.0.0:2000
no-autoupdate: true
ingress:
${local.cfd_config_ingress}
- service: http_status:404
EOF
  }
}
resource "kubernetes_secret_v1" "creds" {
  metadata {
    name      = "creds"
    namespace = kubernetes_namespace_v1.cloudflared.metadata.0.name
    labels = {
      "app"  = "cloudflared"
      "tier" = "app"
    }
  }
  data = {
    "credentials.json" = <<EOF
{
  "AccountTag"   : "${var.cloudflare_account_id}",
  "TunnelID"     : "${cloudflare_argo_tunnel.tunnel.id}",
  "TunnelName"   : "${cloudflare_argo_tunnel.tunnel.name}",
  "TunnelSecret" : "${random_id.tunnel_secret.b64_std}"
}
EOF
  }
}
resource "kubernetes_deployment_v1" "cloudflared" {
  depends_on = [
    cloudflare_argo_tunnel.tunnel
  ]
  metadata {
    name      = "cloudflared"
    namespace = kubernetes_namespace_v1.cloudflared.metadata.0.name
    labels = {
      "app"  = "cloudflared"
      "tier" = "app"
    }
  }
  spec {
    replicas = 2
    selector {
      match_labels = {
        app  = "cloudflared"
        tier = "app"
      }
    }
    template {
      metadata {
        labels = {
          app  = "cloudflared"
          tier = "app"
        }
      }
      spec {
        container {
          image = "cloudflare/cloudflared:${var.cloudflared_image_tag}"
          name  = "cloudflared"
          args = [
            "tunnel",
            "--config",
            "/etc/cloudflared/config/config.yaml",
            "run",
          ]
          liveness_probe {
            http_get {
              path = "/ready"
              port = 2000
            }
            failure_threshold     = 1
            initial_delay_seconds = 10
            period_seconds        = 10
          }
          resources {
            limits = {
              "memory" = "500Mi"
              "cpu"    = "500m"
            }
            requests = {
              "memory" = "100Mi"
              "cpu"    = "50m"
            }
          }
          volume_mount {
            name       = "config"
            mount_path = "/etc/cloudflared/config"
            read_only  = true
          }
          volume_mount {
            name       = "creds"
            mount_path = "/etc/cloudflared/creds"
            read_only  = true
          }
        }
        volume {
          name = "config"
          config_map {
            name = kubernetes_config_map_v1.config.metadata[0].name
            items {
              key  = "config.yaml"
              path = "config.yaml"
            }
          }
        }
        volume {
          name = "creds"
          secret {
            secret_name = kubernetes_secret_v1.creds.metadata[0].name
          }
        }
      }
    }
  }
}

That’s a fair chunk of configuration, but it can be broken down into its resources:

Create a namespace to put anything cloudflared related into, so it’s not bloating any other namespaces.
Create a ConfigMap with the configuration file inside. Most importantly, this configuration file lists the ingress rules for the tunnel. These rules are generated from the local.service_pairs variable defined at the top, which is just a map of the domain name you want your service to have to the service that traffic should be routed to.
Create a Secret that contains the secret passed when creating the tunnel.
Create a deployment that runs cloudflared in a pod, mounting the ConfigMap and Secret into the correct locations.

Configuring DNS records

To wrap up the configuration, all we need to do is configure the correct DNS records to point at the tunnel. We are already using the Cloudflare Terraform provider, and we have local.service_pairs defined with the hostnames we want, so we can just create the DNS records we need now.

hcl
data "cloudflare_zone" "zone" {
  name = "<your.zone>"
}
resource "cloudflare_record" "tunnel" {
  zone_id = data.cloudflare_zone.zone.id
  type    = "CNAME"
  name    = local.tunnel_name
  value   = cloudflare_argo_tunnel.tunnel.cname
  ttl     = 1
  proxied = true
}
resource "cloudflare_record" "services" {
  for_each = local.service_pairs
  zone_id = data.cloudflare_zone.zone.id
  type    = "CNAME"
  name    = each.key
  value   = cloudflare_record.tunnel.hostname
  ttl     = 1
  proxied = true
}

This configuration creates a CNAME record for the tunnel, which is a little redundant. If you don’t want it, just remove it and refer to cloudflare_argo_tunnel.tunnel.cname in the other block instead. The second resource block uses for_each to create a new record for every service you define in local.service_pairs.

If you’re using the tunnel for more than just HTTP, then you will want to set proxied = false on both records, otherwise your DNS records will be pointing to Cloudflare’s server instead of your own.

That’s it. Copy all of the code blocks, put them in a file (or multiple files to keep your Terraform configuration tidier), fill out any variables you need to, plan, and apply. Cloudflare Tunnels make exposing services incredibly easy if you don’t need the additional routing features if a Kubernetes ingress. Although, I’m sure it’s possible to point the tunnel at an ingress instead of a service directly.

Stuart Thomson