, May 14, 2021

0 results found in this keyword

My Stack - Senpai Stack


  •   5 min reads
My Stack - Senpai Stack

Stacks evolve and are ever changing. Right now the pattern I like is immutable and micro service architecture backed by GitOps for automation. GitOps pushes the idea of using the version control system to allow deployments to happen. Typically some tool will be responsible for monitoring the code repository and triggering work flows when someone creates a pull request. I use a minimalistic approach with as little between me and the end goal as possible while still keeping a separation of concern for virtual resources, software dependencies, and deployment options / work flows.

Senpai Stack

The stack layers

Each layer represents a different GitHub repository with code in it. The Core Tools section represents the tools which are used to work with their respective layers. Decoupling the layers allows for you work in isolation from the other components when you are working on updating things like a ruby version in the iso layer. The infrastructure and service layer don't care about what's going on and will not trigger a build in the automation pipelines. We can test, build, and ship each layer on its own respective time line and converge the other layers when the time is ready.

We're going to go in reverse for the explanation.

Infrastructure Layer

devops-miami/infraLayer
This is a demo repo for teaching folks how to build using the Senpai Stack I put together for daily DevOps - devops-miami/infraLayer

The Infrastructure layer (Infra for short) is responsible for spinning up your cloud resources. We use Terraform as the core tool to write infrastructure as code which we can check into version control. Terraform works for many different cloud providers and allows us to write a standard language and pattern to control a multi cloud architecture.

I use my own pattern when I write my Terraform. It's highly opinionated but what isn't now a days. This is what has worked in production for me over the last 4 years of my career. You aren't going to learn this any where else.

  • Reuseable Resources - A modules folder containing all my resources each in its respective folder.
  • For Loops and Maps - Rather than redefining modules we pass maps which will iterate and create all the resources we need in one shot.
  • Main.tf is simple and shows the types of resources / modules being used and their dependencies.
  • Lookup() for mapping dependencies in modules.
  • Outputs are done via maps so Lookup() works for dependencies.
  • Vars file stores all the components with a reference to its dependency when it applies.
  • Dry Code Base - Only settings are being passed from the vars file the code base itself is static and the same across prod, qa, and poc (you won't fat finger something in between because its in another folder)
  • Version Numbers - The infrastructure code base is typically split into static and dynamic infrastructure so I can do blue green deployments on my infrastructure.
  • Terragrunt is used as a wrapper for SOPS and generating providers.

ISO Layer

ISO Layer Example

The next layer in the mix is the image layer which is responsible to wrangling the software dependencies we need to deploy. Is it Java Script, Ruby, or Python? The types of packages you need and the version that they can be neatly bundled into an ISO image which can layer be deployed by the Service Layer. This an immutable software work flow where the images are never altered after they are created. If we need an update for an image or software package installed on it we iterate to a new version and build a new image with the changes and push this to are image repository. The ISO Layer isn't always required as we use helm to deploy the charts and they come with a default Docker image. This example shows how the ISO layer is supplemental to the service layer. The emphasis here is how the layers are GitHub repos which are monitored by CI/CD apps running in the service layer.

Some of the patterns you will see:

  • ISO Layer is managed with Docker.
  • Secrets are encrypted with SOPS.
  • Jenkins on k8 used for builds and pushes to cloud providers.
  • Supports docker-compose up for local development.
  • GitOps work flow with tests posted to Github on PR submission.

Service Layer

Service Layer Example

This layer is responsible for load balancing, certificates, and uses the resources created by the Infrastructure and ISO layer to serve applications. We use Helm which simplifies deploying software to Kubernetes clusters to work in the Service Layer. Helm can run a one liner that will grab a pre-built set of instructions called a Helm Chart for running an ISO image (maybe a custom one from the ISO Layer) and allows us to pass a values file which will adjust all the settings for that particular deployment. Helm Charts are maintained and created by the community and it makes it so you don't need to reinvent the wheel every time you want to get some software rolling. The pattern here is we will go to the Helm Charts repo and find a chart for software we want and follow those instructions to get going. I like to keep track of all the steps for deploying a chart in the README.md in the Service Layer in the folder name of the respective software. When we run our Helm command it will tell the Kubernetes cluster we configured in the Infrastructure Layer that we want a Deployment/Service/Pod/Container running the ISO image I specify.  

  • Service Layer is managed with Helm.
  • Service Layer uses the images from the ISO Layer and the resources from the Infrastructure Layer to deploy your software.
  • Secrets are managed with SOPS.
  • Argo handles deployments in Blue Green fashion.
  • The service layer doesn't care about the cloud you're running on. Although a different values and ingress file might be needed for cloud specific annotations.

I'll start creating posts on how to create the individual layers to get some of the points across in code.

Related News

You've successfully subscribed to Devops Miami Blog
Great! Next, complete checkout for full access to Devops Miami Blog
Welcome back! You've successfully signed in
Success! Your account is fully activated, you now have access to all content.
Success! Your billing info is updated.
Billing info update failed.
Your link has expired.