IaaC by Pulumi

This is recognition post of Pulumi quite new open source solution to manage infrastructure using code. Pulumi is a new type of IaaC located somewhere in the middle between DSLs and SDKs. It has many nice feature. Let's have a look.

Types

Shell

There are many types of IaaC tools on the market. The oldest one, I guess, is shell script combined with SSH. For simply tasks (for instance just check ps -ef or df -h is good enough.

Agent based

Then there are mamy agent-based configuration management tools like puppet or chef where you don't have to reinvent wheel again and you can use modules (provided by tool maintainers of community). Such tools provided some level of abstraction. For instance managing of configuration files of FW rules can be handled by simple object with attributes in the language of these tools. This is big improvements in compare to sed, grep and awk.

Agent-less

Quite similar to previous ones is Ansible with one exception: Ansible is agent-less. This makes Ansible such popular because is light, sshd server is everywhere. And to be honest this is high level shell script and that's why learning curve is really steep.

DSL

Next Terraform comes in as a standard from DSL-based IaaC tool from cloud management. Terraform with state and backends do the job. There are many modules in *hubs ready to use (some of the are official). In DSL YAMLs/HCLs or any other JSONs describe the infrastructure and engine takes care on how to modify or keep in sync infra. State keeps infra in sync with desired configuration. Another pros is self-documentary. From tools side (DB/PaaS, IaaS) there are SDKs, REST APIs and CLI clients provided (which can be consumed by mentioned tools on directly from terminal).

SDK

SDK is a library for specified language where you can write a code to deal with infra like boto3 for AWS. This let DevOps possibility to create IaaC well tested (in many cases) in language of choice. But this is procedural and dependency and life cycle have to be managed by developer.

REST API vs pulumi

Rest API - every single task can be done using curl or any other HTTP client (like requests form python) this usable for small clients where only few API calls is enough. All client part is created by you. Including error handling.

CLI vs pulumi

CLI is a set of commands to use in the terminal. It's good for troubleshooting of CI/CD pipeline.

CDK vs pulumi

CDK is similar to the Pulumi but its only for AWS and is transformed to AWS Cloud Formation. There is no backend and service equivalent.

Pulumi characteristics

Roughly speaking Pulumi is like Terraform but instead using HCL to describe infrastructure it uses regular programming languages. Currently there are four supported platform: node.js, go, python, .NET

Example of code which creates VM in GCP

package main

import (
        "github.com/pulumi/pulumi-gcp/sdk/v6/go/gcp/compute"
        "github.com/pulumi/pulumi-gcp/sdk/v6/go/gcp/serviceaccount"
        "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
        pulumi.Run(func(ctx *pulumi.Context) error {
                ctx.Log.Info("Start", nil)
                defaultAccount, err := serviceaccount.NewAccount(ctx, "defaultAccount", &serviceaccount.AccountArgs{
                        AccountId:   pulumi.String("service-account-id"),
                        DisplayName: pulumi.String("Service Account"),
                })
                if err != nil {
                        return err
                }
                _, err = compute.NewInstance(ctx, "mypulumiinstance", &compute.InstanceArgs{
                        MachineType: pulumi.String("e2-micro"),
                        Zone:        pulumi.String("europe-central2-a"),
                        Tags: pulumi.StringArray{
                                pulumi.String("managed-by"),
                                pulumi.String("pulumi"),
                        },
                        BootDisk: &compute.InstanceBootDiskArgs{
                                InitializeParams: &compute.InstanceBootDiskInitializeParamsArgs{
                                        Image: pulumi.String("debian-cloud/debian-9"),
                                },
                        },
                        NetworkInterfaces: compute.InstanceNetworkInterfaceArray{
                                &compute.InstanceNetworkInterfaceArgs{
                                        Network: pulumi.String("default"),
                                        AccessConfigs: compute.InstanceNetworkInterfaceAccessConfigArray{
                                                nil,
                                        },
                                },
                        },
                        Metadata: pulumi.StringMap{
                                "foo": pulumi.String("bar"),
                        },
                        MetadataStartupScript: pulumi.String("echo hi > /test.txt"),
                        ServiceAccount: &compute.InstanceServiceAccountArgs{
                                Email: defaultAccount.Email,
                                Scopes: pulumi.StringArray{
                                        pulumi.String("cloud-platform"),
                                },
                        },
                })
                if err != nil {
                        return err
                }

                return nil
        })
}

Ecosystem is made of many elements let's briefly look at it:

Project

Project is a basically folder with Pulumi.yaml file. That file contains following fields:

name: pulumi-go-code-reuse
runtime: go
description: <DESCRIPTION>

You can create it by running pulumi new

Stack

Stack is like definition of infrastructure for programming stage like dev, staging, production. For different stage infrastructure looks different. Of course if totally different app has similar architecture it can be deployed as a different stack.

But physically, what is a stack? This is Pulumi.<stackname>.yaml file. How to create stack?

pulumi stack init <stackname>

You can jump between stacks during operations

pulumi stack select <stanc nAME>

You can list stacks

pulumi stack ls

stack is an items which we are actually deploy

pulumi up

With stack you can do many more actions, here I've listed only few of them.

State and backend in pulumi

State is a description if current infrastructure manage by pulumi . Each stake has its own state. This is similar to terraform state.

Backend on the other hand is a place there state is being stored. There are many options which can to group in to main two:

• Service
• Self-managed
  • Cloud (AWS S3, Azure Blob Storage, Google Cloud Storage)
  • Local filesystem (default ~/.pulumi)

Service

Hosted at app.pulumi.com. It provides out-of-the-box:

• checkpointing for fault tolerance
• concurrent state locking
• deployment history
• encryption
• secure access
• integration

It is possible to have self-hosted service locally as well!

After signing up you can just login to the backend from your terminal using below command

pulumi login

Self-managed

Self-managed backends comes with trade offs. For instance, you will need to manually configure secure access, encryption, and history. It's feasible but require additional effort and maintenance.

In order to login to the backend stored in Google Store run following command

pulumi login gs://<my-pulumi-state-bucket>

Local file is a special kind of self-managed backend. Right for development and for learning. It stores all data on the local FS

pulumi login file:///app/data

Migration:

login ti one
pulumi stack export --show-secrets --file my-app-production.checkpoint.json
logout

login to another
pulumi stack import --file my-app-production.checkpoint.json

Resources

Resources represent the fundamental units of infrastructure like VM, storage, DNS, k8s clusters etc.

Every Resorce has following arrtibutes:

• name to reference the resource
• options:
  • parent - to create hierarchy
  • provider
  • depends on - to have priority during creation
  • deletebeforereplace
  • etc.
• components - group of resources
• etc.

Pulumi - Conclusion

Pulumi is a provisioning tool Similar to terraform with state, backends and DSL. What distinguishes it from terraform is lack o HCL. Instead you can use language of your choice (go, python, typescript, js) to describe infrastructure. It lead to better testing, code reuse etc. I see changes in the horizon in the field of infrastructure as a code cause even Terraform and CDK creates CDKTF which is similar to Pulumi.