I ran into the problem that I wanted to make an existing root module (I'll call this source) a child module of another root module (I'll call this destination). Both modules used a remote state backend. In my case a GCP storage bucket.
Example
Let's say we have 2 root modules. redis & databases. The databases module contains a postgresql child module. We want to make the redis module a child module of the databases module.
This requires that we move the state of the existing source module into the destination module's state.
Our before & after Terraform module state:
(SVG version for less pixel artifacts)
Before
File structure
~/
├── databases/
│ └── main.tf
├── redis/
│ └── main.tf
└── modules/
└── postgresql/
└── main.tf
Code
~/databases/main.tf:
...
resource "module" "postgresql" {
source = "../modules/postresql"
}
~/redis/main.tf:
...
resource "google_redis_instance" "my_redis_instance" {
...
}
resource "random_password" "my_redis_pw" {
...
}
~/modules/postgresql/main.tf:
...
resource "google_sql_database_instance" "postgresql" {
...
}
resource "random_password" "my_sql_pw" {
...
}
After
File structure
~/
├── databases/
│ └── main.tf
└── modules/
├── redis/
│ └── main.tf
└── postgresql/
└── main.tf
Code
~/databases/main.tf:
...
// Has the resource address "module.redis"
resource "module" "redis" {
source = "../modules/redis"
}
resource "module" "postgresql" {
source = "../modules/postresql"
}
~/modules/redis/main.tf:
...
resource "google_redis_instance" "my_redis_instance" {
...
}
resource "random_password" "my_redis_pw" {
...
}
~/modules/postgresql/main.tf:
...
resource "google_sql_database_instance" "postgresql" {
...
}
resource "random_password" "my_sql_pw" {
...
}
Making module child of other module
Resolving the problem described above can be achieved by running a combination of commands:
terraform state pull: Pull the remote states of the source & destination modules down as state files to your local drive.terraform state mv: Move all resources from the pulled down local source module state to the local destination module state.terraform state push: Push the local destination module state file back up to the remote.
As explained in this Stackoverflow answer.
This approach works fine for small modules that don't contain a whole lot of resources. It can be cumbersome if that's not the case for you. When you're working with large modules. Especially if you need to move multiple such modules.
I created a small script that automates this process.
It will also create a backup of your existing state before changing anything. Just in case (Terraform additionally also creates backup files before every terraform state mv).
It takes in a couple of arguments:
- The directory of our destination module. In our example:
~/databases.
- The directory of our source module. In our example:
~/redis.
- The resource address inside the destination module where the source module should be moved to. In our example:
module.redis.
And the optional option -dry-run. Which does what it indicates. Does a dry run. No state is moved, but you can see what the script will do on an actual run.
migrate-state.sh:
#!/usr/bin/env bash
set -euo pipefail
# Support aborting via SIGINT, without this bash will not exit the for loop until it's finished
trap 'exit 0' INT
# Usage example:
# bash migrate-state.sh '/home/myuser/terraform/destination-module' '/home/myuser/terraform/source-module' 'module.source' 'config/dev_backend.tfvars' 'config/dev_backend.tfvars'
# Arguments:
# $1 - destinationModuleDirectory: The directory in which the module resides where state should be moved to
# $2 - sourceModuleDirectory: The directory in which the module resides where state should be moved from
# $3 - destinationModuleSourceParentAddress: The resource address inside the destination module that is used as parent for all resources that are moved from the source module
# $4 (optional) - sourceModuleBackendConfig: The value that should be used for the "-backend-config" parameter when running terraform init for the source module
# $5 (optional) - destinationModuleBackendConfig: The value that should be used for the "-backend-config" parameter when running terraform init for the destination module
# Options:
# -dry-run
# Show changes that will be made, does not actually change anything
function migrateState() {
local destinationModuleDirectory="${1:?'The destinationModuleDirectory argument is missing'}"
local sourceModuleDirectory="${2:?'The sourceModuleDirectory argument is missing'}"
local destinationModuleSourceParentAddress="${3:?'The destinationModuleSourceParentAddress argument is missing'}"
local sourceModuleBackendConfig="${4:-''}"
local destinationModuleBackendConfig="${5:-''}"
local isDryRun=0
for arg in "${@}"
do
if [[ "${arg}" == '-dry-run' ]];
then
isDryRun=1
break
fi
done
local lightBlue='\e[38;5;26m'
local colorEnd='\e[0m'
local destinationModuleLocalStateFile="${destinationModuleDirectory}/destination-module.tfstate"
cd "${destinationModuleDirectory}" || false
local destinationTerraformInitArgs=''
if [[ "${destinationModuleBackendConfig}" != '' ]];
then
destinationTerraformInitArgs="-backend-config=${destinationModuleBackendConfig}"
fi
terraform init "${destinationTerraformInitArgs}"
terraform state pull > "${destinationModuleLocalStateFile}"
cp "${destinationModuleLocalStateFile}" "${destinationModuleLocalStateFile}.bak"
cd "${sourceModuleDirectory}" || false
local sourceTerraformInitArgs=''
if [[ "${sourceModuleBackendConfig}" != '' ]];
then
sourceTerraformInitArgs="-backend-config=${sourceModuleBackendConfig}"
fi
terraform init "${sourceTerraformInitArgs}"
terraform state pull > "${sourceModuleDirectory}/source-module.tfstate.bak"
for sourceResourceAddress in $(terraform state list)
do
local destinationResourceAddress="${destinationModuleSourceParentAddress}.${sourceResourceAddress}"
printf "Moving ${lightBlue}%-100s${colorEnd} to ${lightBlue}%-120s${colorEnd}\n" "${sourceResourceAddress}" "${destinationResourceAddress}"
if [[ ${isDryRun} == 1 ]];
then
terraform state mv -state-out="${destinationModuleLocalStateFile}" -dry-run "${sourceResourceAddress}" "${destinationResourceAddress}"
else
terraform state mv -state-out="${destinationModuleLocalStateFile}" "${sourceResourceAddress}" "${destinationResourceAddress}"
fi
printf '\n'
done
if [[ ${isDryRun} == 0 ]];
then
cd "${destinationModuleDirectory}" || false
terraform init "${destinationTerraformInitArgs}"
terraform state push "${destinationModuleLocalStateFile}"
fi
}
migrateState "${@}"
The script is also available on GitHub.
To get back to our example. We'd run
bash migrate-state.sh '~/databases' '~/redis' 'module.redis' -dry-run
to see what will happen.
And then remove the -dry-run option:
bash migrate-state.sh '~/databases' '~/redis' 'module.redis'
