Azure DevOps multi-stage pipeline for Terraform

Azure DevOps is hosted service which helps you to create CICD pipeline, you can deploy your azure Devops source code repository or you can bring existing yaml pipeline from external DevOps services such as GitHub, Bitbucket etc, today we are going to discuss how to create multi-stage CICD pipeline for your Terraform code in Azure DevOps.

As outcome we are expecting a plan (build) stage of each of environment and a apply (deployment), at plan stage review can review the changes which are going to deploy and at apply stage there will be a approve gate in place where approve can approve the changes and it will be deployed to respective environments.

In this article, I am going to share all the steps you need to know to create the multi-stage pipeline in few minutes.

Prerequisite:

• The Basic of Azure DevOps project, you can find on my YouTube channel if you are interested to learn.
• Basics of Yaml.
• Terraform commands.
• Some basic Azure CLI commands.
• Some of Linux basic commands (such as copy ‘cp).

Another approach to setup CICD

Code Setup:

Before I start creating the pipeline I would like to show you the Terraform code structure which I have for this configuration.

Code structure

in the pipeline section, we are going to define our YAML file pipeline.

inside the ‘src’ we have ‘code’ and ‘module’ folder …. inside we have all our variables, maps, and provider defined and inside the module, i have Terraform modules for my azure resources.

my terraform code will be executed from ‘code’ folder which will call the respective modules

The source code structure is totally up to you, defining pipeline is not dependent on the structure, this the structure I prefer for my personal use if you like you can follow the same.

I have also created a few empty folders for each of my environment

These folders location where my pipeline will perform the Plan stage (build).

Creating Pipeline:

Section-1 : In the pipeline folder we are creating the azure-pipeline.yaml file

here at the top section, we have defined the pipeline trigger in the trigger I am specifying when my pipeline should auto-trigger the code above says trigger this pipeline when there is an update into ‘src/code/input’ folder

I have also created yaml to storage the variables which we are going to use in our pipeline, variables.yaml file has the following variables list

if you want you to store the variables into the azure variables group as a secret, for this demo I am keeping these variables into yaml files only ;).

[NOTE]: Here to run my Terraform commands I am using ‘ubuntu-latest’ agent pool, hence I need to use linux command only, if you are using any other platform then you need to replace the appropriate commands according to your operating system type.

• Section 2: In this step, we need to add a step to install Terraform in your build agent only if you are using Microsoft hosted agent because as of now MS hosted agent doesn’t come with terraform installed on it hence we need to use the task to install the terraform.

Here we are installing Terraform version ‘0.13.2’.

[NOTE]: If your project uses self-hosted and the agent has terraform already installed on it then you can ignore this step.

• Section 3: Terraform plan step on Dev environment, the objective of this step is to create the terraform plan, which can be reviewed by review, the plan will show what going to add/edit/delete on your subscription.

just like the below screenshot

Sample plan file screenshot

Here is the sample code for the same:

here we have created a new YAML pipeline stage (Plan_Dev), this stage depends on the previous stage (Setup).

Next, we are defining the job collection object in which we have created a job Build job (Plan_Dev), this job has multiple steps each step can have one or more tasks.

overall structure will look like this:

Yaml pipeline structure

In this step, we have 3 different task

→ Task 1: Script - this task will following activities

→ Copy content of ‘code’ folder to ‘dev’ folder move the execution pointer to ‘dev’ folder

→ Run terraform init and Terraform plan command

→ Task 2: copy the content of ‘dev’ folder into the staging directory, just to create artifacts.

→ Task 3: publish the artifacts to Azure DevOps, we are providing the name of our artifact as per our environment name (e.g.: dev_artifacts)

Section 4: Apply Stage, here in this stage we are going to define the steps to run the Terraform apply command which will create the infrastructure into the azure subscription.

Apply stage is a bit different from the Plan stage, apply stage uses the artifacts which are generated from the plan stage hence we are using the ‘deployment’ block

YAML schema — Azure Pipelines | Microsoft Docs

In the step we couple of tasks, before we discuss those tasks have a look at the ‘-deployment’ tag, here in tag we are defining the strategy to run the deployment using artifacts of build stage.

we are also using environment tag, using the environments we have to define the conditional deployment, wherein reviewer can review the stage and then they can make the decision to deploy.

Script — this task will following activities

→ Copy content of ‘code’ folder to ‘dev’ folder move the execution pointer to ‘dev’ folder

→ Run ‘Terraform init’ command.

→ Run ‘Terraform plan’ command.

→ Run ‘Terraform apply’ command

[NOTE]: here additional to build stage we are using ‘Terraform apply’ command to apply the changes.

[Important]: Our terraform plan command is using input variables that are defined in our source code as Json file, that’s why we are referring the path in our command.

terraform plan -var-file=”input/dev.tfvars.json” -out=”out.plan” -var=”client_id=$(client_id)” -var=”client_secret=$(client_secret)” -var=”tenant_id=$(tenant_id)” -var=”subscription_id=$(subscription_id)”

This is it, using the above-mentioned step you should be able to deploy infrastructure to the dev environment in your Azure subscription.

Question 1: What about deploying changes to multiple environments?

Well, the answer is simple…

Repeat the same step for the rest of your environments :) happy days.

Question 2: Why do we need a separate build stage after each environment?

Answer: As we have multiple environments and for each environment, we are maintaining the terraform state into its own environment-specific blob storage file (which will be stored into azure storage account), it might possible that different environments have different changes to be deployment from the last deployment…hence for each specific environment we want its own build so that reviewer can review the plan of that specific environment, deploying single plan of one environment into all the environment will not be the correct implementation because your plan of one environment might destroy something into another environment.

full YAML pipeline can be found here:

If you like this article please give a clap.

I have my Youtube channel where I post videos on Azure/Azure DevOps/ Terraform / DotNet technologies every week.

Please Subscribe to my Youtube channel

If you prefer the learn using video tutorials, you subscribe me on my youtube channel using the link below

Bee-a-Learner

References :

YAML schema - Azure Pipelines

Terraform Build & Release Tasks - Visual Studio Marketplace

YAML schema - Azure Pipelines