Terraform on GCP: Remote State in Cloud Storage and the IAM Bootstrap Problem Solved

by

Every GCP project starts the same way. You write some Terraform, run terraform apply, and watch the resources spin up. Then a colleague joins, clones the repo, and immediately destroys your state by running from their laptop. Or you rotate a service account key, and now your pipeline can’t authenticate. Or — the classic — you lose your terraform.tfstate file and pray to whatever gods watch over infrastructure engineers.

These aren’t edge cases. They’re the predictable failures of treating Terraform state as a local concern on a cloud-native platform that was built for distributed teams from day one.

This article covers two things that belong together: storing Terraform state in Google Cloud Storage with proper locking, and the IAM bootstrap pattern — the approach that solves the chicken-and-egg problem of "I need Terraform to manage IAM, but I need IAM to run Terraform."

Official Terraform GCS backend docs: https://developer.hashicorp.com/terraform/language/backend/gcs

Why Local State Is a Trap

The default terraform.tfstate file on your workstation works fine for solo projects and tutorials. The moment you add a second person, a CI/CD pipeline, or multiple environments, it falls apart:

  • No locking. Two concurrent apply runs will corrupt state. No warning, no error, just corruption.
  • Secrets in plaintext. State files often contain sensitive values (passwords, private keys) that shouldn’t live in a git repo.
  • No audit trail. Who ran apply at 2am on Friday? No idea.
  • Machine dependency. State tied to one laptop is a single point of failure.

GCS solves all of this. It gives you versioning, server-side encryption, access logging, and — critically — state locking via Google Cloud Storage’s native object locking (or via a separate lock file approach Terraform manages automatically).

The Bootstrap Problem

Here’s the real challenge on GCP: before you can use Terraform, you need a GCS bucket for state and a service account with the right permissions. Both of those are GCP resources. You could create them with gcloud commands, but then they’re not managed by Terraform and you’ve got manual infrastructure floating around. You could manage them with Terraform, but then you need state for that Terraform — which requires the bucket you haven’t created yet.

This is the IAM bootstrap loop, and the answer is: accept one layer of manually bootstrapped infrastructure, keep it tiny and well-documented, and manage everything else with Terraform from that point forward.

The pattern looks like this:

bootstrap/ (manual or one-shot script)
  └─ Creates: GCS bucket, bootstrap SA, OIDC/WIF config
infra/ (Terraform)
  └─ Backend: gs://your-tf-state-bucket/infra
  └─ Creates: all real infrastructure

Step 1: Bootstrap the Prerequisites

You only run this once per GCP project. Use a shell script so it’s repeatable and auditable. Keep it in your repo under bootstrap/.

#!/usr/bin/env bash
# bootstrap/setup.sh
# Run once: creates the GCS bucket and service account Terraform needs.
# Prerequisites: gcloud auth login, correct project set.

set -euo pipefail

PROJECT_ID="${1:?Usage: ./setup.sh <project-id>}"
REGION="europe-west1"
BUCKET_NAME="tf-state-${PROJECT_ID}"
SA_NAME="terraform-runner"
SA_EMAIL="${SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com"

echo "==> Setting project"
gcloud config set project "${PROJECT_ID}"

echo "==> Enabling required APIs"
gcloud services enable \
  cloudresourcemanager.googleapis.com \
  iam.googleapis.com \
  storage.googleapis.com \
  iamcredentials.googleapis.com

echo "==> Creating GCS bucket for Terraform state"
gcloud storage buckets create "gs://${BUCKET_NAME}" \
  --location="${REGION}" \
  --uniform-bucket-level-access \
  --public-access-prevention

# Enable versioning — lets you roll back to a previous state if something breaks
gcloud storage buckets update "gs://${BUCKET_NAME}" \
  --versioning

echo "==> Creating Terraform service account"
gcloud iam service-accounts create "${SA_NAME}" \
  --display-name="Terraform Runner"

echo "==> Granting SA access to state bucket"
gcloud storage buckets add-iam-policy-binding "gs://${BUCKET_NAME}" \
  --member="serviceAccount:${SA_EMAIL}" \
  --role="roles/storage.objectAdmin"

# Terraform needs to read/write project-level resources.
# Scope this down if your project allows it.
echo "==> Granting SA project-level roles"
gcloud projects add-iam-policy-binding "${PROJECT_ID}" \
  --member="serviceAccount:${SA_EMAIL}" \
  --role="roles/editor"

gcloud projects add-iam-policy-binding "${PROJECT_ID}" \
  --member="serviceAccount:${SA_EMAIL}" \
  --role="roles/iam.securityAdmin"

echo ""
echo "Bootstrap complete."
echo "Bucket: gs://${BUCKET_NAME}"
echo "Service account: ${SA_EMAIL}"

Run it:

chmod +x bootstrap/setup.sh
./bootstrap/setup.sh my-gcp-project-id

Gotcha: roles/editor is broad. It’s fine for a first pass, but in production you want a custom role that covers exactly what your Terraform manages. Start with editor + iam.securityAdmin, then audit what permissions your apply actually exercises and tighten from there.

Step 2: Configure the GCS Backend

In your Terraform root module:

# infra/backend.tf

terraform {
  backend "gcs" {
    bucket = "tf-state-my-gcp-project-id"
    prefix = "infra/state"   # acts as a folder path inside the bucket
  }
}

The prefix field is important. One bucket can hold state for multiple environments or modules — just use different prefixes:

tf-state-my-project/
  infra/state/default.tfstate
  infra-staging/state/default.tfstate
  infra-prod/state/default.tfstate

Initialize the backend:

terraform init

If you’re migrating from local state, Terraform will ask whether to copy the existing state to the new backend. Say yes.

Gotcha: Once you set a backend, the bucket value is not interpolatable — you cannot use var.project_id or any Terraform variable inside a backend {} block. The backend is initialized before variables are resolved. Use -backend-config flags or a .tfbackend file to parameterize it:

# infra/environments/prod.tfbackend
bucket = "tf-state-my-gcp-project-id-prod"
prefix = "infra/state"

terraform init -backend-config=environments/prod.tfbackend

Step 3: Authenticate Without Static Keys

Service account JSON keys are the old way. They expire, get leaked in repos, and create a rotation headache. For CI/CD, use Workload Identity Federation instead. For local dev, use Application Default Credentials (ADC).

Local Development

gcloud auth application-default login

Terraform’s Google provider picks up ADC automatically. No keys needed. Your personal gcloud identity needs the same roles as the Terraform service account (or you impersonate it — see below).

To impersonate the Terraform service account from your local machine without creating a key:

# infra/providers.tf

provider "google" {
  project = var.project_id
  region  = var.region

  # Impersonate the Terraform SA — no key file required
  impersonate_service_account = "terraform-runner@${var.project_id}.iam.gserviceaccount.com"
}

Your personal account needs roles/iam.serviceAccountTokenCreator on that SA:

gcloud iam service-accounts add-iam-policy-binding \
  "terraform-runner@${PROJECT_ID}.iam.gserviceaccount.com" \
  --member="user:[email protected]" \
  --role="roles/iam.serviceAccountTokenCreator"

CI/CD with Workload Identity Federation

Add this to your bootstrap script (or as a second bootstrap step once your CI provider is known):

# Add to bootstrap/setup.sh or a separate bootstrap/wif.sh

WIF_POOL_NAME="github-actions-pool"
WIF_PROVIDER_NAME="github-provider"
GITHUB_ORG="your-github-org"
GITHUB_REPO="your-repo-name"

# Create the WIF pool
gcloud iam workload-identity-pools create "${WIF_POOL_NAME}" \
  --location="global" \
  --display-name="GitHub Actions Pool"

# Create the OIDC provider
gcloud iam workload-identity-pools providers create-oidc "${WIF_PROVIDER_NAME}" \
  --location="global" \
  --workload-identity-pool="${WIF_POOL_NAME}" \
  --issuer-uri="https://token.actions.githubusercontent.com" \
  --attribute-mapping="google.subject=assertion.sub,attribute.repository=assertion.repository" \
  --attribute-condition="assertion.repository=='${GITHUB_ORG}/${GITHUB_REPO}'"

# Get the WIF provider resource name
WIF_PROVIDER=$(gcloud iam workload-identity-pools providers describe "${WIF_PROVIDER_NAME}" \
  --location="global" \
  --workload-identity-pool="${WIF_POOL_NAME}" \
  --format="value(name)")

# Allow GitHub Actions to impersonate the Terraform SA
gcloud iam service-accounts add-iam-policy-binding \
  "${SA_EMAIL}" \
  --role="roles/iam.workloadIdentityUser" \
  --member="principalSet://iam.googleapis.com/${WIF_PROVIDER}/attribute.repository/${GITHUB_ORG}/${GITHUB_REPO}"

echo "WIF Provider: ${WIF_PROVIDER}"
echo "Add this to your GitHub Actions workflow."

Your GitHub Actions workflow then looks like:

# .github/workflows/terraform.yml

name: Terraform

on:
  push:
    branches: [main]
  pull_request:

permissions:
  contents: read
  id-token: write   # required for WIF

jobs:
  terraform:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - id: auth
        uses: google-github-actions/auth@v2
        with:
          workload_identity_provider: "projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/github-actions-pool/providers/github-provider"
          service_account: "terraform-runner@PROJECT_ID.iam.gserviceaccount.com"

      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: "~> 1.9"

      - name: Terraform Init
        run: terraform init -backend-config=environments/prod.tfbackend
        working-directory: infra

      - name: Terraform Plan
        run: terraform plan -out=tfplan
        working-directory: infra

      - name: Terraform Apply
        if: github.ref == 'refs/heads/main'
        run: terraform apply tfplan
        working-directory: infra

Gotcha: You need the project number (not ID) in the WIF provider path. Get it with:

gcloud projects describe ${PROJECT_ID} --format="value(projectNumber)"

Step 4: State Locking in Practice

GCS backend uses a lock file (default.tflock) stored in the same bucket prefix. When a plan or apply starts, Terraform writes this file. If another process tries to run concurrently, it sees the lock file and aborts.

This works, but it has a failure mode: if Terraform crashes mid-run, the lock file stays. You end up with:

Error: Error locking state: Error acquiring the state lock

Force-unlock with the lock ID shown in the error:

terraform force-unlock LOCK_ID

Don’t automate force-unlock in CI. It exists for manual recovery. If your pipeline is hitting lock contention regularly, the real problem is overlapping pipeline runs — fix that with concurrency controls in your CI system (GitHub Actions concurrency key, for instance).

Step 5: Manage the Bootstrap Resources with Terraform (Eventually)

Once everything is running, you can import the bootstrap resources into Terraform and start managing them declaratively. This closes the loop — you get to a state where Terraform owns everything, including its own prerequisites.

# bootstrap/main.tf — manages the bootstrap bucket and SA

resource "google_storage_bucket" "tf_state" {
  name                        = "tf-state-${var.project_id}"
  location                    = var.region
  uniform_bucket_level_access = true
  public_access_prevention    = "enforced"

  versioning {
    enabled = true
  }

  lifecycle_rule {
    condition {
      num_newer_versions = 10   # keep last 10 versions of state
    }
    action {
      type = "Delete"
    }
  }
}

resource "google_service_account" "terraform_runner" {
  account_id   = "terraform-runner"
  display_name = "Terraform Runner"
}

resource "google_storage_bucket_iam_member" "tf_state_access" {
  bucket = google_storage_bucket.tf_state.name
  role   = "roles/storage.objectAdmin"
  member = "serviceAccount:${google_service_account.terraform_runner.email}"
}

Import the existing resources:

terraform import google_storage_bucket.tf_state tf-state-my-project-id
terraform import google_service_account.terraform_runner \
  projects/my-project-id/serviceAccounts/[email protected]

The bootstrap module stores its own state somewhere small and stable — you can use a separate bucket prefix (bootstrap/state) or even keep it local for the one file that creates the bucket. That second option is acceptable here because the bootstrap state only contains two or three resources and almost never changes.

Production Checklist

Before you call this setup production-ready, verify these:

Bucket hardening:

  • Uniform bucket-level access is on (prevents ACL bypasses)
  • Public access prevention is enforced
  • Versioning is enabled (state rollback capability)
  • Object retention or lifecycle rules to cap state version accumulation
  • No direct human write access to the bucket outside break-glass procedures

IAM hygiene:

  • Terraform SA has the minimum roles needed, not blanket owner
  • No SA JSON keys exist (use WIF in CI, ADC + impersonation locally)
  • SA key creation is blocked via org policy if you have Organization-level GCP

Operational:

  • State bucket is in the same project as the infrastructure it describes, or in a dedicated "ops" project with cross-project access
  • Bucket access logging is enabled (who read/wrote state and when)
  • Alerts on unexpected state changes or access from new principals

Gotcha: Bucket versioning does not automatically clean up old state versions. Add a lifecycle rule (shown above) or you’ll accumulate hundreds of versions over time. Each is billed as storage. Not expensive, but messy.

The Module Structure That Actually Works

For a real project with multiple environments, this is the layout that causes the least pain:

.
├── bootstrap/
│   ├── setup.sh           # one-time bootstrap script
│   ├── wif.sh             # WIF setup for CI
│   └── main.tf            # optional: import bootstrap resources later
├── modules/
│   ├── networking/
│   ├── gke/
│   └── cloud-sql/
├── infra/
│   ├── backend.tf
│   ├── providers.tf
│   ├── variables.tf
│   ├── main.tf
│   └── environments/
│       ├── dev.tfvars
│       ├── staging.tfvars
│       ├── prod.tfvars
│       ├── dev.tfbackend
│       ├── staging.tfbackend
│       └── prod.tfbackend
└── .github/
    └── workflows/
        └── terraform.yml

One backend file per environment, one vars file per environment. terraform init -backend-config=environments/prod.tfbackend + terraform apply -var-file=environments/prod.tfvars. Clean, no magic.

The Takeaway

The GCS backend with WIF authentication is the correct default setup for any GCP project that’s more than a personal experiment. The manual bootstrap step is unavoidable — accept it, script it, document it, and move on. Everything after that bootstrap can and should be Terraform-managed.

The SA impersonation pattern for local dev is worth the extra five minutes of setup. It means no one on the team has a key file sitting around, and rotating credentials becomes a non-event.

Get this plumbing right once, and you’ll stop thinking about it. That’s the goal.

👁 Views: 202,848 · Unique visitors: 57,406