Expanding our API for better Terraform provisioning

Recently, a customer reached out to us, expressing their desire to provision Transloadit with Hashicorp's Terraform. We already have the Terraform Provider Plugin available, which leverages our API to set up an account and Templates. However, as the customer pointed out, one key feature was missing. Template Credentials still had to be created and managed through the console UI.

This was the only way to do it... until today! We have now added an extra endpoint, making it possible to manage your third-party credentials, without venturing to our website to do this in the good ol' fashioned point-and-click style.

If you are already looking ahead, here is the documentation for the Template Credentials API endpoint.

Why is this necessary?

Head of Engineering at Tastemade Bryan McLemore made us aware of this use case, describing how our API at the time led to more people than necessary being exposed to API secrets.

They were left with two choices:

We could break our principals around automation and manually create the IAM keys, which need to get copied over to the Transloadit console by hand. This process itself would increase the number of people who need to have access to the secret as well as force more people to see it. Neither is good.

Alternatively, we inject the secrets into the Templates as they're generated in Terraform. This preserves automation, but makes it so that anyone who can see the Template definition, can also see the secrets as they're stored in plain text. They'd have to go out of their way to see them, but it's still possible.

Neither of these options are ideal. Instead, we decided to take on board Bryan's feedback and introduce the /template_credentials API endpoint to get around this awkward secret exchange. This is how he imagined that taking place:

The general pattern I have is that IAM keys (and other secrets) are generated by Terraform and then stored directly where they're needed. For example, our CI variables and in this case our new Transloadit Templates. To mitigate risks associated with generating IAM keys in Terraform, we have the Terraform state set up in a secure and remote place that requires special permissions to gain access to it at all.

After introducing the endpoint, any customer could make HTTP calls to our API to automatically provision credentials. However, to help Bryan's use case, we also asked lead maintainer Etienne Carriere to roll out support for this endpoint to the Transloadit Terraform Provider Plugin.

In Terraform, you define your desired Transloadit configuration locally in Git. You evolve it with your team, and you can roll back, just like with the code of your app. When you run Terraform, it presents you with the differences between the actual and desired Transloadit configuration, and will propose to make whatever changes necessary for your actual Transloadit configuration to match your desired configuration.

This protects against drift from human errors, Transloadit data loss, and other issues that plague modern IT operations. It's not unlike when, back in the day, we collectively moved from managing websites directly on the server disk with FTP, to using Git and CI/CD.

Provisioning in Terraform

Now that we know why we might want to use Terraform with Transloadit, let's take a look at an example below on how we can implement some basic infrastructure.

provider "aws" {
  region = "us-east-1"
}
provider "transloadit" {}

resource "aws_iam_user" "transloadit" {
  name     = "transloadit"
}

resource "aws_iam_access_key" "transloadit" {
  user     = aws_iam_user.transloadit.name
}
resource "transloadit_template_credential" "s3_credentials" {
  name     = "my-terreaform-credentials"
  type     = "s3"

  content = jsonencode({
    bucket        = "uploads-bucket"
    bucket_region = "us-east-1"
    key           = aws_iam_access_key.transloadit.id
    secret        = aws_iam_access_key.transloadit.secret
  })
}

resource "transloadit_template" "my-terraform-template" {
  name     = "my-terraform-template"
  template =
    {
      "steps": {
        ":original": {
          "robot": "/upload/handle"
        },
        "exported": {
          "use": [ ":original" ],
          "credentials": "${transloadit_template_credential.s3_credentials.name}",
          "robot": "/s3/store"
        }
      }
    }
}

In our sample code, we define two providers: Transloadit and AWS. From this, we create two resources aws_iam_user and aws_iam_access_key. These represent our AWS IAM user, from which we can then access our S3 key and secret. The key is shared under the transloadit_template_credential resource; to which we then pass the name of our Template Credentials, the type of credentials (S3 in this case, but many other third parties that we integrate with are supported), as well as the actual key and secret for the credentials from the AWS IAM user.

The main advantage to using this pattern is that it allows you to integrate with an encrypted storage option, such as Hashicorp Vault, resulting in secure storage of our Terraform state, since our IAM keys would be stored in plaintext by default.

After this, we can specify a resource to define the Template that we want to run – and we're off and away.

You can read more about the parameters in the documentation for our Terraform plugin. And if you're looking for platform-specific parameters for your credentials, go to the documentation for one of our many file importing and exporting Robots.

Now, just to confirm it works as expected this first time, you can check out your Transloadit Console, and you should be able to see your new S3 Credentials, ready for integration.

The power to change Transloadit is in your hands

All this went from a humble feature request to being a part of our API in under a month. We take great pride in still having the same agility that we first started with years ago. On top of that, we are excited that this has allowed us to showcase the continued impact our customers have on the future of Transloadit. So, if there is any way in which we could improve your current workflow – we hope this goes to show that you should not hesisate to reach out and let us know. Ostensibly simple feature requests like these can prove to have a lasting effect on Transloadit.

If there is something you would like to bring to our attention, leave a GitHub issue on one of our SDKs or reach out to support.