![Book cover of Dev ops handbook](devopshandbook.png)

Here are some point-form takeaways from my notes from reading [The DevOps Handbook](https://app.thestorygraph.com/books/70e99c96-fc97-465a-bf1b-058072e18bc5) by John Willis, Gene Kim, and Patrick Debois.

## Index

[Core premise](#core-premise)
[First way of DevOps: Flow](#first-way-of-devops-flow)
- [Make work visible](#make-work-visible)
- [Limit WIP](#limit-wip)
- [Small batch sizes](#small-batch-sizes)
- [Elevate constraints in value streams](#elevate-constraints-in-value-streams)
- [Categories of waste and hardship](#categories-of-waste-and-hardship)
- [Developer productivity vs number of developers](#developer-productivity-vs-number-of-developers)
[Second way of DevOps: Feedback](#second-way-of-devops-feedback)
[Third way of DevOps: Continuous Learning and Experimentation](#third-way-of-devops-continuous-learning-and-experimentation)
[Practices to improve the three ways](#practices-to-improve-the-three-ways)
- [Immutable infrastructure](#immutable-infrastructure)
- [Infrastructure as code](#infrastructure-as-code)
- [Testing pyramid](#testing-pyramid)
- [Continuous integration](#continuous-integration)
- [No more pull requests](#no-more-pull-requests)
- [Git strategies](#git-strategies)
- [Continuous delivery](#continuous-delivery)
- [Metrics](#metrics)
- [A/B testing](#ab-testing)
- [Chatbots for automation](#chatbots-for-automation)
- [Single company-wide repository](#single-company-wide-repository)

## Core premise

At the core of this book are the three "ways of DevOps".

1. Left-to-right flow of work from development to operations to the customer.
	- Speeding up this channel gets work from idea to the customer faster.
	- Decreasing lead times through better process leads to higher value delivered to customers and higher developer satisfaction.
	- Resulting practices from improving this channel include continuous integration, continuous deployment, and infrastructure-as-code, among others.
2. Right-to-left constant feedback from customers to operations to developers.
	- Prevent problems from happening again.
	- Enable faster detection of and recovery from issues.
	- Resulting practices from improving this channel are easy to access and understand metrics, and making your systems more observable.
3. Cycles of learning. Creating a generative, high-trust culture.
	- Scientific approach to risk taking.
	- Organizational learning. Multiply the effects of new knowledge. Transform local discoveries into global improvements.
	- Amplifying positive feedback loops creating ever-safer systems.

## First way of DevOps: Flow

### Make work visible

Make work visible. Kanban boards allow you to easily see the state of a current projects and the team in general. Keep work from development and operations on the same kanban board so that both teams can see the in-progress work of the other.

### Limit WIP

Multi-tasking significantly degrades efficiency. Limit the number of WIP items to keep teams focused. You can do this by controlling the swim lane size in your kanban boards.

Reference: [Making Work Visible: Exposing Time Theft to Optimize Work & Flow](https://app.thestorygraph.com/books/6c5c24e2-6006-43b9-9e30-569400cd1581) by Dominica Degrandis

### Small batch sizes

Small batch sizes lead to faster lead times for individual projects, less WIP, faster detection of issues, and less rework.

### Elevate constraints in value streams

Focus efforts on optimizing bottlenecks in your value streams. The most value and effect can be had by remediating bottlenecks rather than steps before or after a bottleneck.

### Categories of waste and hardship

These categories of work are where inefficiencies are introduced and time and effort are wasted.

- Partially done work: WIP, work waiting in queues, etc. Partially done work loses value as time passes.
- Extra processes: Additional work that doesn't add value to the customer. Increases lead time.
- Extra features: Output that is not valued by the customer. Adds complexity and increases lead time.
- Task switching: Multitasking adds effort and time to the value stream.
- Waiting: Delays or queues increase lead time.
- Motion: Friction in handoffs adds ambiguity into the value stream and can increase lead time.
- Defects: Bugs, miscommunications, etc. add lead time and rework.

Reference: [Implementing Lean Software Development: From Concept to Cash](https://app.thestorygraph.com/books/a18e5dee-89e4-4c79-948e-ada45f61b476) by Mary Poppendieck and Tom Poppendieck

Added categories from Damon Edwards:

- Nonstandard or manual work: Any manual work that can be automated. Increases lead time and can lead to human error.
- Heroics: Individuals and teams should never be put in positions where they must perform unreasonable acts. Don't let heroics become the norm. When an team member needs to perform heroics, a breakdown of process has occurred. Should be remedied such that it's never required again.
### Developer productivity vs number of developers

When we increase the number of developers, individual developer productivity often significantly decreases due to communication, integration, and testing overhead

## Second way of DevOps: Feedback

Feedback and feedforward loops allow us to detect and remediate problems while they're smaller, cheaper, and easier to fix, to avert problems before they cause a catastrophe, and create organizational learning that we integrate into future work.

![Nesting circles showing the time difference of feedback from different components of the value stream](feedback-cycle-times.png)
^ Time it takes to get feedback from different stages of feedback. Taken from DevOps Handbook

## Third way of DevOps: Continuous Learning and Experimentation

Couple of quotes:

> When accidents and failures occur, instead of looking for human error, we look for how we can redesign the system to prevent the accident from happening again.

^ DevOps Handbook

> Even more important than daily work is the improvement of daily work.

^ Mike Orzen, author of Lean IT

You want to create an environment of trust where experimentation and learning are the norm. You want to have simple process in place to elevate local learnings to global company-wide knowledge.

## Practices to improve the three ways

### Immutable infrastructure

Never allow changes directly to infrastructure. Any changes must be checked into version control in our infrastructure-as-code, then deployed as new infrastructure that replaces the existing infrastructure, rather than tweaking it in-place.

You wouldn't let someone log into app servers and make changes directly to the code. So why let that happen for infrastructure?

### Infrastructure as code

- Version control for your infrastructure. Blame and change history.
- Single source of truth for the state of your infrastructure.
- Replicate environments in a quick, repeatable, documented way.
- Make environments easier to rebuild than repair, makes it simpler and quicker to resolve issues.

### Testing pyramid

Should be able to discover the majority of issues in the quickest running tests, to facilitate fast feedback and iteration.

![Pyramid shape showing the different levels of testing comparing their speed to receive feedback](TestingPyramid.jpg)

If an issue is found during integration or end-to-end testing, which usually take a long time to run, a unit test should be created that catches this issue. That way if the same issue happens again, it will be caught sooner and the developer can get faster feedback.

Not only should you improve your code coverage over time, you should aim to move tests down the pyramid so that issues continue to be found faster.

Manual tests can be run in parallel with automated tests (which also could be running in parallel with each other) to improve testing efficiency.

![Flow chart showing the stages of testing and which portions can be done in parallel](AutomatedandManualParallelTesting.jpg)

### Continuous integration

The purpose of creating and working in branches is so that multiple developers can work in parallel and minimize conflicts.

Developers then need to manually "integrate" their code with other work. This process can be error prone. Especially when you don't understand someone else's code that is causing a conflict. This issue is exasperated the more developers are working in parallel.

Continuous Integration solves this problem by getting everyone to integrate into the main branch daily as part of their normal work.

### No more pull requests

Pull requests were designed by GitHub to facilitate accepting contributions from unknown and untrusted developers on open source projects. They give maintainers an opportunity to confirm the code matches the style and ethos of the project.

Developers within an organization should already share the style and ethos of the team. They should be trusted to continue to promote that ethos in their code.

Pull requests sacrifice performance and quality. This is a worth while tradeoff for open source projects. They force the author to wait for review, causing their work to become stale. And they force reviewers to context switch to review code. Often they don't take the time necessary to actually understand the code because of time pressures, so errors slip though anyway even though all that time was spent!

Using pull request internally is like forcing your family through airport security just to enter your house. It's a costly solution to a different problem.

If your worry is having multiple pairs of eyes on the code before it's deployed, that should happen way earlier in the project, before hard to change decisions are solidified. For example, having multiple devs on a project, Pari programming, frequent code reviews, etc. These can replace pull requests and assist in automating the integration and deployment pipeline.

Related reference not from DevOps Handbook: [Pull Requests from Infrastructure as Code](https://infrastructure-as-code.com/book/2021/01/02/pull-requests.html) by Kief Morris

### Git strategies

**Optimizing for individual productivity**: individual branches. Quick progress in the beginning, but offload integration work to the end, or sporadically through the project. This is more like waterfall development.

**Optimizing for team productivity**: always integrating and working off of main. Main is always up to date with the latest work and in a deployable state. This is more like agile development.

When merging is difficult because of large batch sizes, developers become incentivized not to refactor code so that they keep their integration simple. This contributes to tech debt in the codebase growing over time.

### Continuous delivery

Any code that passes all tests, builds, and is deployable in a *production-like* environment will be automatically deployed to production.

Our code and environments should be designed and built in such a way that the release of new functionality to users does not require a change to our application code. We should decouple the ideas of releasing features to users and deploying code to an environment. They are two different acts serving two different stakeholders.

When we're deploying code all the time through continuous deployment and dark launching, releasing new functionality to users becomes simply a business and marketing decision.

Reference: [Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation.](https://app.thestorygraph.com/books/77eb0975-4194-42e7-9db3-005d4250940f) by Jez Humble and David Farley

### Metrics

We should enable every developer to create metrics in their daily work and be able to display and analyze them.

Metrics should be visible and digestible for anyone in the company, and perhaps even publicly.

### A/B testing

If you don't have data about how whether a feature or change is performing compared to a baseline, then you don't know if the change has zero impact or even negative impact.

A study at Microsoft showed that through AB testing, 2/3 of proposed changed showed zero or negative impact.

Reference: [Release It!: Design and Deploy Production-Ready Software](https://app.thestorygraph.com/books/1bd756aa-7a60-4af1-a407-6311f9741421) by Michael T. Nygard

### Chatbots for automation

Example:
> @hubot deploy owl to production

The results of chatbot commands are returned to the chat.

- Everyone can see everything that is happening
- Engineers, on their first day of work, can see what daily work looked like and how it was performed
- People were more apt to ask for help when they saw others helping each other
- Rapid organizational learning was enabled and accumulated

### Single company-wide repository

Everyone has access to all the most up to date code easily.

Randy Shoup described the single repository as the most powerful mechanism for preventing failures at Google.

Tom Limoncelli described it as a competitive advantage for Google.


Book Notes: DevOps Handbook, The


Here are some things that I'd like to learn more about in 2022. This is kindof a reference for myself. So I can look back at this next year and reflect on what I learned and what I wanted to learn. [Let me know on twitter](https://twitter.com/tyreldelaney) if you are learning these same things or you have any tips or resources to share!

![Logo montage of all the technologies outlined in this article](what-im-learning-in-2022.jpg)

## Index

[Learning from scratch](#learning-from-scratch)

* [Docker](#docker)
* [Vim/NeoVim](#vimneovim)

[Learning more of](#learning-more-of)

* [Github Actions](#github-actions)
* [AWS DynamoDB](#aws-dynamodb)
* [AWS Client VPN, VPNs in general, and SSO](#aws-client-vpn-vpns-in-general-and-sso)
* [SSH](#ssh)
* [HTTPS/SSL](#httpsssl)
* [Raspberry Pi](#raspberry-pi)
* [Home Network](#home-network)

[Things I'm interested in but might not get to this year](#things-im-interested-in-but-might-not-get-to-this-year)

* [A better static site generator / CMS](#a-better-static-site-generator--cms)
* [Webhooks](#webhooks)

## Learning from scratch

### Docker
Docker really feels like one of those technologies that I've been conveniently ignoring for a little too long. I think what I'll do to start is simply try to containerize some smaller projects I'm working on this year to dip my toes in.

I acquired a handful of second-hand Raspberry Pis not long ago and I've got a few ideas for some small home automation & monitoring projects that might be perfect simple projects to dockerize.

A stretch goal would be to work up to containerizing our monolithic app at my day job.

### Vim/NeoVim
I've always just barely snuck by with my fluency in the terminal. The more I get into networking and Dev Ops the more this deficiency is showing. In general I'd just like to work in the terminal a bit more this year.

Specifically I'm going to try to develop a couple Node and Python projects I'm cooking up remotely on the Raspberry Pis I hope to run them on.

Really two main goals I'd like to accomplish with this practice:
1. Get proficient and comfortable using vim.
2. Learn more about making a portable and lightweight dev environment. Something that I can easily spin up on a Raspberry Pi or AWS EC2 instance over SSH and feel comfortable doing real work in.

## Learning more of

### Github Actions
Automation is a big focus for me this year. And really just because I rely on Github both personally and professionally, Github Actions is a natural choice to skill up in.

We're now looking into automating and outsourcing our test suite at work and since we use Github for source control and code reviews/pull requests it makes total sense to use Github Actions.

I've automated a few things using Github Actions already, namely [tests and deployment of my](https://github.com/tyrelh/battlesnake-typescript-node) [Battlesnake](https://play.battlesnake.com/) to AWS Elastic Beanstalk. Automating even just tests for our app at work will be a bit more of a challenge since it's a much older stack and has many more moving parts.

### AWS DynamoDB
I wasn't actually planning on learning DynamoDB at the start of the year. But a project idea came to mind and the possibility of using Dynamo to store data rather than something like S3 sounded intriguing.

Essentially what I'm planning is building out a few Raspberry Pi temperature sensors for my house. Maybe 4 or 5 in total monitoring a few rooms in the house as well as the outdoor temperature.

The idea would be to collect data from all these sensors using Raspberry Pis maybe every 5 mins or so, and ship that data off to Dynamo. Then building an infrastructure to rotate through tables. Basically make a new table each day with a high write provision, and rotate down the previous days tables read and write provisions. Maybe after some period archive the data in S3.

Might be over engineering for the task, but it's for learning.

[Best Practices for Handling Time Series Data in DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-time-series.html)

### AWS Client VPN, VPNs in general, and SSO
With my day job going fully remote this year, setting up secure remote tools has been high on our priority list. Part of that will be learning about AWS Client VPN which we may use to grant our devs access to some of our AWS infrastructure.

Part of this work is also moving to more zero trust architecture, and leaning into SSO for authentication.

Also I recently acquired a Synology NAS for my home (which we actually use to use in our office before we gave that space up). I'm looking to centralize alot of my personal cloud use to the Synology and move away from services like Google Drive (office and documents), iCloud (photos), and Notion (notes). Part of that will be to setup a VPN at home so that I can use those services hosted in my home from anywhere. As well as accessing some of my Raspberry Pis for remote development.

### SSH
Learning more about SSH and tunneling goes hand-in-hand with a few other learning priorities I have. Learning best practices for connecting securely and crafting secure infrastructure. Also using SSH to practice remote development and learning terminal based tools.

### HTTPS/SSL
Same as above. Learning more about best practices around secure remote communication. Getting more confident understanding certs and how they work.

### Raspberry Pi
I've owned a Raspberry Pi 3 for a handful of years now, but I've never really had a project for it. I experimented a bit with retro game emulation on it, but those projects are pretty trivial to setup.

Now I have a handful of Raspberry Pi 2s and 3s and I have some plans for some smaller single purpose projects for the house that will be perfect for them.

### Home network
This kinda just happened organically, but I have a lot of computers in my house now! What I'd like to do is centralize a bunch of things into a proper rack. I have a few computers that are mining cryptocurrency that I'd like to transfer to rack-mountable server cases. Along with my new Synology NAS and a few Raspberry Pis.

Wiring my house with Cat-6 probably won't happen any time soon, but centralizing a bunch of things in a rack will let me hard-wire them all to a small switch without needing to run cables through all my walls.

Like I mentioned above, also setting up network storage and remote access for my home network.

## Things I'm interested in but might not get to

### A better static site generator / CMS
My website currently is hand-built in Javascript and React with a few plugins that let me render my articles from markdown. There is still a fair amount of manual process to get each markdown file wired up to display as a page and route, but at least I don't need to reformat the article. I can just paste it in from my notes app and then wire it up.

Ideally I'd like to learn a better static site generator to simplify this process even more. I've been thinking of learning [Gatsby](https://www.gatsbyjs.com/) for a while now, but I've just found it a bit cumbersome and hard to get into.

Some alternatives I'm considering learning instead of Gatsby are [Jeckyll](https://jekyllrb.com/) or [Hexo](https://hexo.io/).

### Webhooks
Just something I'm interested in learning more about.

## End

[@ me on twitter](https://twitter.com/tyreldelaney) if you are learning these same things or you have any tips or resources to share!

What I'm Learning in 2022

Node CI/CD Pipeline using GitHub Actions & AWS Elastic Beanstalk


During a recent project to convert my JavaScript & Node [Battlesnake](https://play.battlesnake.com/) over to TypeScript & [Deno](https://deno.land/), I decided I would also create a more robust suite of tests for it.

![Deno and Github logos with an addition symbol between them](DenoTestsAndGithubActionsCI.png)

If you are unfamiliar with [Battlesnake](https://play.battlesnake.com/), it is an online and in-person programming competition where you build a AI in the form of a REST API that can play a multiplayer version of the classic game [Snake](https://en.wikipedia.org/wiki/Snake_(video_game_genre). You can check out details on their [website](https://play.battlesnake.com/) to learn more, and even watch some past events on their [Twitch channel](https://www.twitch.tv/BattlesnakeOfficial).

## Tl;dr
* [Deno Tests](#deno-tests): Deno includes a very simple test runner framework.
* [Github Actions](#github-actions): Github actions allow you to automate tasks based on triggers within your Github repo.
* [Create the workflow](#create-the-workflow): A .yml file describes a series of actions you would like to perform based on a trigger.
* [Automate Deno tests](#automate-deno-tests): Just add a `run: deno test` command to your workflow .yml.

## Deno Tests
[Deno](https://deno.land/) includes a simple test runner for tests in js, ts, jsx, or tsx files. It can be run with:

```bash
deno test
```

To create tests, all you need to use is use the `Deno.test()` method included with Deno within files ending in *test* (`{*_,}test.{js,ts,jsx,tsx}`) anywhere in your project structure. I put mine in *tests/*.

Example of a basic test:

```typescript
import { assert } from "https://deno.land/std/testing/asserts.ts";
import { root } from "../app/main.ts";

Deno.test("Root response contains expected fields", () => {
  // when
  const result = root();
  // then
  assert(result?.apiversion);
  assert(result?.author);
  assert(result?.head);
  assert(result?.tail);
  assert(result?.color);
});
```

This test is simply asserting that values exist for these specified fields.

You can add many tests to single files by calling `Deno.test()` over and over again. You can also make as many test files as you like, Deno will automatically execute all the files it recognizes ending in *test* (`{*_,}test.{js,ts,jsx,tsx}`).

Visit the [Deno Testing docs](https://deno.land/manual/testing) for more info about creating tests.

Easy! So now I can create a robust set of unit tests for my Battlesnake AI.

Since the game Battlesnake works by [sending you the entire game state for every move](https://docs.battlesnake.com/references/api), I can also save the game state JSON from particularly tricky moves and use them within their own tests. The output can be compared against the single best move, or even a set of equally good moves if their are multiple. Creating tests like this tests the behaviour of the AI overall and ensures that changes to the behaviour aren't making the AI perform worse on moves it previously was able to do correctly.

So now that I have tests up and working, I wanted to automate them using Github Actions.

## Github Actions
Github Actions are still quite new to me and I hadn't touched them at all before this project. I just assumed it would be complicated to set up.

For what I wanted to do for this project it was actually trivial. Couldn't have been easier.

What I wanted was to run these tests automatically whenever pushing to my repository. I figured this Deno app is pretty simple all things considered, so I shouldn't have to mess around with Docker or anything like that.

I'll step you through what I did.

## Create the workflow
Github Actions are based around these things called workflows. They are essentially .yml scripts that describe the series of events you want to perform in your action.

If you click the Actions tab within the repo you want to create the Action you will be presented with a Get started with GitHub Actions page. This page has many example actions you get you started for so many different languages, frameworks, and platforms. It will even suggest actions to you based on the contents of your repo. I chose the *Skip this asn set up a workflow yourself* option.

![Get started with Github Actions](DenoAndGithubActions-get-started-with-github-actions.png)

This plops you into a web editor and gives you a basic example workflow .yml file to start with. It looks something like this:

```yaml
# This is a basic workflow to help you get started with Actions

name: CI

# Controls when the action will run. Triggers the workflow on push or pull request
# events but only for the master branch
on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  # This workflow contains a single job called "build"
  build:
    # The type of runner that the job will run on
    runs-on: ubuntu-latest

    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
    # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
    - uses: actions/checkout@v2

    # Runs a single command using the runners shell
    - name: Run a one-line script
      run: echo Hello, world!

    # Runs a set of commands using the runners shell
    - name: Run a multi-line script
      run: |
        echo Add other actions to build,
        echo test, and deploy your project.

```

With this you can actually click the *Start commit* button and that will guide you to commit this .yml file to your repo within the *.github/workflows/* directory.

You can see near the top of that file that there are triggers for `on push` and `on pull_request` attached to the master branch. You can try to push something to master and see it work!

If you visit the Actions tab of your repo now you will see your Actions and the history of all the times they have run. You can click into them for more details and to see the console outputs. Also by default it will email your Github account email whenever an action fails. I imagine this can be changed somewhere if you don't like that.

![Github Actions](DenoAndGithubActions-github-actions.png)

## Automate Deno tests
Now I wanted to try to run my tests using Deno. I knew this would involve installing Deno somehow so I searched around.

I found [this repo](https://github.com/denolib/setup-deno) from [denolib](https://github.com/denolib) with exactly what I wanted.

Really all you need to do is add the `uses` line pointing to this repo to your workflow .yml file under the `steps` section. You can also include the Deno version you prefer. Then within the `run` section you now have access to `deno` and all of its functions.

So now my workflow .yml file looks like this:

```yaml
name: CI

on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@master
    - uses: denolib/setup-deno@master
      with:
        deno-version: v1.x
    - run: deno run https://deno.land/std/examples/welcome.ts
```

One really cool feature of Deno is that you can run scripts from the web right in the terminal, and you can also do this right within Github Actions. That is what the URL is doing in

```yaml
- run: deno run https://deno.land/std/examples/welcome.ts
```

You can run this and then look within your Github Actions for the console output from this example *welcome.ts* file.

So now to run my tests, all I needed to do was change that `deno run` command to `deno test`. My final .yml workflow looks something like this:

```yaml
name: CI

on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

jobs:
  run-tests:
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@master
    - uses: denolib/setup-deno@master
      with:
        deno-version: v1.x
    - run: deno test
```

Now every time I push to my master branch all my tests will be run, including any new tests I just committed, and I will be notified both on the Github UI and in my email if any tests fail. I can then log into Github and look at the output to see what failed.

I hope this small guide helped you out and showed you just how simple both Deno and Github Actions are!

#cicd

Book Notes: DevOps Handbook, The

August 27, 2023 8 Minute Read

What I'm Learning in 2022

March 23, 2022 5 Minute Read

Node CI/CD Pipeline using GitHub Actions & AWS Elastic Beanstalk

May 30, 2020 24 Minute Read

Deno Tests & GitHub Actions CI

May 23, 2020 6 Minute Read

Get in touch