Orlando Thöny
Orlando's Blog

Follow

Orlando's Blog

Follow
How to move Terraform root module state inside other root module

How to move Terraform root module state inside other root module

Orlando Thöny's photo
Orlando Thöny
·Aug 3, 2021·

4 min read

I ran into the problem that I wanted to make an existing root module (I'll call this source) a child module of another root module (I'll call this destination). Both modules used a remote state backend. In my case a GCP storage bucket.

Example

Let's say we have 2 root modules. redis & databases. The databases module contains a postgresql child module. We want to make the redis module a child module of the databases module.

This requires that we move the state of the existing source module into the destination module's state.

Our before & after Terraform module state:

(SVG version for less pixel artifacts) Terraform migrate module state into other module

Before

File structure

~/
├── databases/
│   └── main.tf
├── redis/
│   └── main.tf
└── modules/
    └── postgresql/
        └── main.tf

Code

~/databases/main.tf:

...

resource "module" "postgresql" {
  source = "../modules/postresql"
}

~/redis/main.tf:

...

resource "google_redis_instance" "my_redis_instance" {
  ...
}

resource "random_password" "my_redis_pw" {
...
}

~/modules/postgresql/main.tf:

...

resource "google_sql_database_instance" "postgresql" {
  ...
}

resource "random_password" "my_sql_pw" {
...
}

After

File structure

~/
├── databases/
│   └── main.tf
└── modules/
    ├── redis/
    │   └── main.tf
    └── postgresql/
        └── main.tf

Code

~/databases/main.tf:

...
// Has the resource address "module.redis" 
resource "module" "redis" {
  source = "../modules/redis"
}

resource "module" "postgresql" {
  source = "../modules/postresql"
}

~/modules/redis/main.tf:

...

resource "google_redis_instance" "my_redis_instance" {
  ...
}

resource "random_password" "my_redis_pw" {
...
}

~/modules/postgresql/main.tf:

...

resource "google_sql_database_instance" "postgresql" {
  ...
}

resource "random_password" "my_sql_pw" {
...
}

Making module child of other module

Resolving the problem described above can be achieved by running a combination of commands:

  • terraform state pull: Pull the remote states of the source & destination modules down as state files to your local drive.
  • terraform state mv: Move all resources from the pulled down local source module state to the local destination module state.
  • terraform state push: Push the local destination module state file back up to the remote.

As explained in this Stackoverflow answer.

This approach works fine for small modules that don't contain a whole lot of resources. It can be cumbersome if that's not the case for you. When you're working with large modules. Especially if you need to move multiple such modules.

I created a small script that automates this process.

It will also create a backup of your existing state before changing anything. Just in case (Terraform additionally also creates backup files before every terraform state mv).

It takes in a couple of arguments:

  1. The directory of our destination module. In our example: ~/databases.
  2. The directory of our source module. In our example: ~/redis.
  3. The resource address inside the destination module where the source module should be moved to. In our example: module.redis.

And the optional option -dry-run. Which does what it indicates. Does a dry run. No state is moved, but you can see what the script will do on an actual run.

migrate-state.sh:

#!/usr/bin/env bash
set -euo pipefail

# Support aborting via SIGINT, without this bash will not exit the for loop until it's finished
trap 'exit 0' INT

# Usage example:
# bash migrate-state.sh '/home/myuser/terraform/destination-module' '/home/myuser/terraform/source-module' 'module.source' 'config/dev_backend.tfvars' 'config/dev_backend.tfvars'
# Arguments:
# $1 - destinationModuleDirectory: The directory in which the module resides where state should be moved to
# $2 - sourceModuleDirectory: The directory in which the module resides where state should be moved from
# $3 - destinationModuleSourceParentAddress: The resource address inside the destination module that is used as parent for all resources that are moved from the source module
# $4 (optional) - sourceModuleBackendConfig: The value that should be used for the "-backend-config" parameter when running  terraform init for the source module
# $5 (optional) - destinationModuleBackendConfig: The value that should be used for the "-backend-config" parameter when running  terraform init for the destination module
# Options:
# -dry-run
#   Show changes that will be made, does not actually change anything

function migrateState() {
  local destinationModuleDirectory="${1:?'The destinationModuleDirectory argument is missing'}"
  local sourceModuleDirectory="${2:?'The sourceModuleDirectory argument is missing'}"
  local destinationModuleSourceParentAddress="${3:?'The destinationModuleSourceParentAddress argument is missing'}"
  local sourceModuleBackendConfig="${4:-''}"
  local destinationModuleBackendConfig="${5:-''}"

  local isDryRun=0
  for arg in "${@}"
  do
    if [[ "${arg}" == '-dry-run' ]];
    then
      isDryRun=1
      break
    fi
  done

  local lightBlue='\e[38;5;26m'
  local colorEnd='\e[0m'

  local destinationModuleLocalStateFile="${destinationModuleDirectory}/destination-module.tfstate"

  cd "${destinationModuleDirectory}" || false

  local destinationTerraformInitArgs=''
  if [[ "${destinationModuleBackendConfig}" != '' ]];
  then
    destinationTerraformInitArgs="-backend-config=${destinationModuleBackendConfig}"
  fi
  terraform init "${destinationTerraformInitArgs}"

  terraform state pull > "${destinationModuleLocalStateFile}"
  cp "${destinationModuleLocalStateFile}" "${destinationModuleLocalStateFile}.bak"

  cd "${sourceModuleDirectory}" || false
  local sourceTerraformInitArgs=''
  if [[ "${sourceModuleBackendConfig}" != '' ]];
  then
    sourceTerraformInitArgs="-backend-config=${sourceModuleBackendConfig}"
  fi
  terraform init "${sourceTerraformInitArgs}"
  terraform state pull > "${sourceModuleDirectory}/source-module.tfstate.bak"

  for sourceResourceAddress in $(terraform state list)
  do
    local destinationResourceAddress="${destinationModuleSourceParentAddress}.${sourceResourceAddress}"
    printf "Moving ${lightBlue}%-100s${colorEnd} to ${lightBlue}%-120s${colorEnd}\n" "${sourceResourceAddress}" "${destinationResourceAddress}"

    if [[ ${isDryRun} == 1 ]];
    then
      terraform state mv -state-out="${destinationModuleLocalStateFile}" -dry-run "${sourceResourceAddress}" "${destinationResourceAddress}"
    else
      terraform state mv -state-out="${destinationModuleLocalStateFile}" "${sourceResourceAddress}" "${destinationResourceAddress}"
    fi

    printf '\n'
  done

  if [[ ${isDryRun} == 0 ]];
  then
    cd "${destinationModuleDirectory}" || false

    terraform init "${destinationTerraformInitArgs}"
    terraform state push "${destinationModuleLocalStateFile}"
  fi
}

migrateState "${@}"

The script is also available on GitHub.

To get back to our example. We'd run

bash migrate-state.sh '~/databases' '~/redis' 'module.redis' -dry-run

to see what will happen.

And then remove the -dry-run option:

bash migrate-state.sh '~/databases' '~/redis' 'module.redis'
 
Share this