Launching Transloadit's Terraform Provider plugin
Today, we are very happy to announce Transloadit's very own Terraform Provider Plugin! In this blog post, we'll go about explaining why that has got us so excited, and how you can set it up for yourself.
What is Transloadit?
Transloadit is a file processing service for companies. Developers describe desired media outputs with a declarative JSON recipe and Transloadit then encodes video, resizes and describes images, normalizes audio, makes thumbnails of documents or screenshots of websites, and so much more.
What is Terraform?
Terraform is a code-based tool for provisioning infrastructure (Infrastructure as Code (IaC)). Developers describe their desired infrastructure with a declarative HCL recipe, and Terraform then compares and modifies the actual infrastructure's state to ensure that it adheres to the desired described state. It may create an S3 bucket, rename a load balancer, etc.
Using Terraform in Transloadit
We have been using Terraform internally to provision 460 AWS resources across three regions while
keeping them in sync. It has been working well. If a clumsy colleague (or, more likely, I myself)
accidentally changes something, or forgets to also sync intended changes to our
Asia datacenter,
Terraform will notice and correct when I run terraform apply. What's more, since the recipes are
in Git, we can see what our infrastructure for software looked like six years ago — and if needed,
we can even roll back to that state. Powerful stuff!
Using Transloadit in Terraform
There's a true wealth of Provider Plugins available, so you can even set up a 1Password Vault with Terraform these days. We thought it would be a good idea to also offer a Transloadit Terraform Provider that would allow you to provision Transloadit recipes (Templates) with Terraform. And here's why.
Let's say you're using Transloadit to automatically encode uploaded videos. You'll need to save the results somewhere, typically in a bucket on a cloud storage service like S3. This means that Transloadit needs write access to this bucket. If you were already using Terraform to create and configure this bucket and the associated IAM permissions, you can now, from the same infra recipe, provision and configure Transloadit to save files in that bucket, and configure AWS to allow us to do so.
With a Transloadit Terraform Provider Plugin, you could orchestrate all of your media processing pipelines declaratively. This allows you to:
- review and collaborate on changes in PRs
- roll back to previous configurations of your infrastructure
- auto-heal configuration mistakes
- sync changes to other regions
- ensure compatibility between neighboring infrastructure components
Setting out to build the Transloadit Terraform Provider
Having tasted the power of Terraform and wanting to offer its functionality to customers of our service as well, we decided to build a Transloadit Terraform Provider.
Unfortunately, we had never written a Terraform Provider ourselves, and we've been rather busy, too. So, we asked around on Hacker News to see if any external freelancer would be up for the job. And sure enough, Etienne Carriere got in touch with us. Etienne was not only willing to help out, he also asked that his fee would be donated to the Red Cross instead 🏥
Etienne, thank you so much for doing a phenomenal job! We're not sure if we are more impressed by the thoroughness, the speed of execution, or the fact that you won't take a penny. If you didn't have a fan base already, you can be sure that you do now!
Without further ado, let's take a look at what Etienne built and how to use it.
Walkthrough
Install
After signing up for Transloadit and noting down your credentials, you'll need to install:
Both are just single binaries that you need to put in the appropriate place.
Define
After that, you can add the recipe for your Transloadit Templates, for example in a file
called main.tf:
cd ~/code
mkdir transloadit-infra
$EDITOR main.tf
With the following contents:
provider "aws" {
# This needs AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in your env
version = "2.62.0"
}
provider "transloadit" {
# This needs TRANSLOADIT_AUTH_KEY and TRANSLOADIT_AUTH_SECRET in your env
version = "0.1.0"
}
resource "aws_s3_bucket" "uploads" {
bucket = "uploads.example.com"
acl = "private"
}
data "aws_iam_policy_document" "transloadit-policy" {
statement {
sid = "AllowTransloaditToStoreFilesInUploadsExampleComBucket"
effect = "Allow"
actions = [
"s3:GetBucketLocation",
"s3:ListBucket",
"s3:PutObject",
"s3:PutObjectAcl",
]
resources = [
"arn:aws:s3:::${aws_s3_bucket.uploads.id}",
"arn:aws:s3:::${aws_s3_bucket.uploads.id}/*",
]
}
}
resource "aws_iam_user_policy" "transloadit-policy" {
name = "transloadit-policy"
user = aws_iam_user.transloadit.name
policy = data.aws_iam_policy_document.transloadit-policy.json
}
resource "aws_iam_user" "transloadit" {
name = "transloadit"
}
resource "aws_iam_access_key" "transloadit" {
user = aws_iam_user.transloadit.name
}
resource "transloadit_template" "to-ipad-with-thumbnails" {
name = "to-ipad-with-thumbnails"
template = <<EOT
{
"steps": {
":original": {
"robot": "/upload/handle"
},
"encoded": {
"use": ":original",
"robot": "/video/encode",
"preset": "ipad-high",
"ffmpeg_stack": "v3.3.3"
},
"thumbed": {
"use": ":original",
"robot": "/video/thumbs",
"count": 4,
"ffmpeg_stack": "v3.3.3"
},
"exported": {
"use": [ ":original", "encoded", "thumbed"],
"bucket": "${aws_s3_bucket.uploads.id}",
"key": "${aws_iam_access_key.transloadit.id}",
"secret": "${aws_iam_access_key.transloadit.secret}",
"robot": "/s3/store"
}
}
}
EOT
}
output "to-ipad-with-thumbnails-id" {
value = transloadit_template.to-ipad-with-thumbnails.id
}
Note that this will write the IAM user's secret access key to the state file, so please protect your back-end state file judiciously. Alternatively, you may supply a
pgp_key, which will prevent the secret from being stored in plaintext, at the cost of preventing the use of the secret key in automation. From https://www.terraform.io/docs/providers/aws/r/iam_access_key.html.
Provision
Now back on the CLI, run:
# You can get your Transloadit key & secret from https://transloadit.com/c/template-credentials/
export TRANSLOADIT_AUTH_KEY="********"
export TRANSLOADIT_AUTH_SECRET="********"
# You can get your AWS key & secret from https://console.aws.amazon.com/iam/home
# Under users->user->security credentials
export AWS_ACCESS_KEY_ID="********"
export AWS_SECRET_ACCESS_KEY="********"
terraform init # only required on the first run after using a new Provider (version)
terraform apply
Terraform will now propose to make changes to your infra.
Assuming you inspected carefully and chose yes, you have now provisioned a back-end that can
handle user upload of any video. It will encode those videos for optimal display on iPads, extract
four preview thumbnail images, and save all of the results to your S3 bucket: uploads.example.com
(you'll want to change this bucket name).
On the AWS side, all the IAM users and policies have been provisioned so that Transloadit can save the files onto S3.
You can reference the value of the output to-ipad-with-thumbnails-id as your template_id (for
instance in your Uppy webbrowser file uploading integration), and, that's it 😄
💡 If you're interested in the JSON media encoding recipe used here, you can check out our Integration Docs.
More detailed examples of the Transloadit Terraform Provider are available in the provider's website directory.
Test
You'll be able to see your to-ipad-with-thumbnails Template here. You
can click on it, toggle the Testing Area, and upload a video to see how everything is processed.
What's next?
We are trying to get this provider listed as an official Plugin with HashiCorp. If we succeed, it
will be easier to discover the Provider Plugin, read its documentation, and even install it
automatically by simply referencing transloadit in your HCL and running terraform init.
We hope you'll find this plugin useful already and can't wait to see what you manage to build with it. Do let us know!