"Cloud Alchemy" - Pulumi, standing up infrastructure with actual code

Jason Smith Jason Smith

June 8, 2022

cloud-alchemy-logo-1-black-scaled.jpg

Alright, it is time for me to migrate my on-prem application to the cloud. If we are using “alchemy” terms, we want to “transmute” the applications. Moving to the cloud doesn’t mean that my applications will just float in the aether. My applications will be on someone else’s infrastructure. This can be a pain…. but it doesn’t have to be.

Infrastructure as Code has an inherent problem

At this point, you have likely heard of infrastructure as code (IaC). In case you haven’t, think of it as declaring the resources you need as code instead of hardware configuration files. It makes it pretty simple to create some markup, pass it along to a IaC tool and spin up whatever cloud resources you need.

The problem that I always had with the term “Infrastructure as Code” is the “code” part. Now granted, code is a really large term. No one seems to agree on what’s coding and what’s not. Don’t believe me, just go into StackOverflow and say that HTML is a programming language and see what happens.

My main problem with many IaC is a little thing called DSL. No, I am not talking about fast internet, I am talking about “Domain-specific Languages”. What are DSLs? They are (programming) languages with a very specific use case, or domain. A good example would be that Progress Chef’s “recipes”. This represents a language that can only be run natively with the Progress Chef’s configuration management tool.

The problem with modern DSLs in relation to infrastructure is two-fold.

  1. You have to learn a new language with a very specific use-case. While most all of these languages use a key:value style format similar to that of YAML, they are all very unique in their syntax. You are effectively learning a skill for a very specific tool. This is additional technical debt to ensure that everyone is trained and up to date.

  2. The language isn’t transferable. Let’s say you change companies or even teams. This new group of developers are using a different IaC configuration tool. You understand the high level concepts of IaC but the syntax and what not to execute is so different than everything else.

Wouldn’t it be nice if we could just use the programming language that we use to write our applications to also define our infrastructure?

Pulumi: The “Coding” in Infrastructure as Code

I discovered Pulumi back in 2020 when I was reading a CNCF newsletter. The thing that really spoke to me was the idea that I could deploy a GKE cluster with a CloudSQL instance on a given VPC using Python and I didn’t have to write my own tool to do it.

I personally got to know some of the Pulumi team and learned about how it worked and was blown away. With Cloud Alchemy, a basic rule I want to follow is to keep things as simple as possible. If we can reuse assets or knowledge to accomplish a goal, we absolutely should. The idea that I can use the language that I am using to also spin up my cloud environment was a game change.

Another positive is that it’s very platform agnostic. Naturally, I like to use Kubernetes clusters and Google Cloud but you will see that they support many platforms. You can actually see how extensive their registry is right here.

Give me a run down

The best way to learn is to do so I will share their “Getting Started” guide. That being said, I wanted to point out some additional features that I like besides the universal language support.

For one, it’s Open Source. While being open source doesn’t automatically make a product “good”, in my mind, it does encourage community adoption. Pulumi is released under the Apache License 2.0 and has 12.7k stars. I, as a rule, try to use open source software as it addresses my belief in community owned software and, in my experience, it tends to be better than commercial software as you have more viewpoints contributing.

The CI/CD integration is a win for me. If you are an application developer, you are familiar with CI/CD. You are likely pushing code to your git repository and it is undergoing a ton of checks before it is integrated into the base code and pushed to production. Since you are using “normal code” such as Python or Typescript, you can include this code in your CI/CD pipeline as you deploy your cloud environment.

Pulumi Crossguard is another cool feature. It is what we in the cloud computing world call “policy as code”. Being able to define cloud resources using Python or Go (or whatever language Pulumi supports) is amazing but in a world where web application security is a must, it is equally as amazing to be able to define policies for security and compliance within the same tool.

Pulumi Packages is an addition that is useful as well. The simple idea is, “why reinvent the wheel”. I am sure that there are many functions that are essentially boilerplate such as spinning up a CloudSQL instance or a storage bucket or something. There is no point in manually creating these code packages for common use-cases. Pulumi Packages lists a variety of reusable components for multiple architectures in many languages.

Then of course there is the Pulumi Service. Early in my life, I tried to host everything myself. In my mind, it was cheaper than the $XX/mo that they were offering. In a sense, it was. However, it didn’t come without its own challenges. I spent way too much time maintaining the system or I would, quite frankly, neglect it due to it being too burdensome. The Pulumi Service is a an easy-to-use managed version of Pulumi. This way to don’t have to store the engine on your own laptop or setup a server while still reaping the benefits.

Any downsides?

If anyone ever tells you that they have the perfect tool that is 100% everything you need and is perfect, walk the other way. Everything has their pros and cons and, quite frankly, what’s most important is the use case.

The obvious con is that they are the new kid on the block. They don’t have as established a track record as many other IaC platforms do so some people may be iffy on adopting it. They have come a long way and show a lot of promise though. That being said, they aren’t as “wired into” the Cloud Native world as some of the others.

There is also a learning curve and a smaller community of support. Now, with any new tool there will be a learning curve but some people may be turned off by the fact that since its adoption is smaller, there aren’t as many tutorials or users to provide support. It may, arguably, be harder to find talent who understand it.

That being said, because you can BYOL, that learning curve is much flatter. Your developers are likely already familiar with concepts around programming a certain language so they just need to learn the libraries and they are good to go. If they already have a high level understanding of IaC, that’s even better.

All in all, I think Pulumi has a lot of promise and you will definitely be seeing me using more of it in my Cloud Alchemy demos. That being said, I will also say that you should use the tool that works best for your organization’s culture. Maybe it’s Pulumi, maybe it’s something else. I would definitely recommend doing your research and trying it out to see if it fits for you.