How to use Gitlab's remote terraform state in CI with "developer" accounts background

How to use Gitlab's remote terraform state in CI with "developer" accounts

# Problem in question

You have your app.

You have some "head maintainers" and some developers on your team.

Your developers have the "developer role" in the repository.

Your infra guy sets up CI that automatically creates preview deployments in merge requests.

Everything works when he works on the feature - he has maintainer role.

Everything works when you merge the MR - you have the owner role.

Then one of the devs creates a new MR and the pipeline fails with the following error


│ Error: Error acquiring the state lock

│ Error message: HTTP remote state endpoint invalid auth

│ Terraform acquires a state lock to protect the state from being written
│ by multiple users at the same time. Please resolve the issue above and try
│ again. For most commands, you can disable locking with the "-lock=false"
│ flag, but this is not recommended.

Uh-oh! Your devops guy forgot about the fact that gitlab only allows Maintainer (or higher) accounts to access the terraform state - and when jobs are running in CI they inherit the triggerer's permissions - in this case your dev's.

Now you need to figure out how to have shared remote state, without adding too many other external dependencies. Will you switch to using S3? That would be wasteful - you've already set everything up to work with Gitlab ...

# The Solution

A rather dumb and simple one. Create another repository (eg. my-project-preview-tfstate), disable all features except "Infrastructure", create a maintainer-level access token scoped to api (you have disabled all other features in the repository so nothing much to manipulate with this token), save the token as a CI variable in your main repository and use it to authenticate with terraform.