AWS Dynamic DNS

Battlesnake Programming Competition 2018

Book Notes: DevOps Handbook, The

Dark Mode React Toggle

Deno Tests & GitHub Actions CI

Deploy a Next.js Static Site to GitHub Pages

Migrating Frontend Assets During Grails 2 → 3 Upgrade

Lightshift Game

Building a Mechanical Keyboard from Scratch

My Introduction to Game Development

Next.js Tag Pages

Node CI/CD Pipeline using GitHub Actions & AWS Elastic Beanstalk

Resources I Found Valuable in 2022

Small Form Factor PC Build Part 1

Small Form Factor PC Part 2 - Water Cooling

Static Site Search Part 3 - Optimizing the Index

Static Site Search Part 1 - Preprocessing Articles

Static Site Search Part 2 - Search Component

How I use my Synology NAS as my personal cloud

What is a Synology NAS?

What I'm Learning in 2022

What is DevOps?

![Book cover of Dev ops handbook](devopshandbook.png)

Here are some point-form takeaways from my notes from reading [The DevOps Handbook](https://app.thestorygraph.com/books/70e99c96-fc97-465a-bf1b-058072e18bc5) by John Willis, Gene Kim, and Patrick Debois.

## Index

[Core premise](#core-premise)
[First way of DevOps: Flow](#first-way-of-devops-flow)
- [Make work visible](#make-work-visible)
- [Limit WIP](#limit-wip)
- [Small batch sizes](#small-batch-sizes)
- [Elevate constraints in value streams](#elevate-constraints-in-value-streams)
- [Categories of waste and hardship](#categories-of-waste-and-hardship)
- [Developer productivity vs number of developers](#developer-productivity-vs-number-of-developers)
[Second way of DevOps: Feedback](#second-way-of-devops-feedback)
[Third way of DevOps: Continuous Learning and Experimentation](#third-way-of-devops-continuous-learning-and-experimentation)
[Practices to improve the three ways](#practices-to-improve-the-three-ways)
- [Immutable infrastructure](#immutable-infrastructure)
- [Infrastructure as code](#infrastructure-as-code)
- [Testing pyramid](#testing-pyramid)
- [Continuous integration](#continuous-integration)
- [No more pull requests](#no-more-pull-requests)
- [Git strategies](#git-strategies)
- [Continuous delivery](#continuous-delivery)
- [Metrics](#metrics)
- [A/B testing](#ab-testing)
- [Chatbots for automation](#chatbots-for-automation)
- [Single company-wide repository](#single-company-wide-repository)

## Core premise

At the core of this book are the three "ways of DevOps".

1. Left-to-right flow of work from development to operations to the customer.
	- Speeding up this channel gets work from idea to the customer faster.
	- Decreasing lead times through better process leads to higher value delivered to customers and higher developer satisfaction.
	- Resulting practices from improving this channel include continuous integration, continuous deployment, and infrastructure-as-code, among others.
2. Right-to-left constant feedback from customers to operations to developers.
	- Prevent problems from happening again.
	- Enable faster detection of and recovery from issues.
	- Resulting practices from improving this channel are easy to access and understand metrics, and making your systems more observable.
3. Cycles of learning. Creating a generative, high-trust culture.
	- Scientific approach to risk taking.
	- Organizational learning. Multiply the effects of new knowledge. Transform local discoveries into global improvements.
	- Amplifying positive feedback loops creating ever-safer systems.

## First way of DevOps: Flow

### Make work visible

Make work visible. Kanban boards allow you to easily see the state of a current projects and the team in general. Keep work from development and operations on the same kanban board so that both teams can see the in-progress work of the other.

### Limit WIP

Multi-tasking significantly degrades efficiency. Limit the number of WIP items to keep teams focused. You can do this by controlling the swim lane size in your kanban boards.

Reference: [Making Work Visible: Exposing Time Theft to Optimize Work & Flow](https://app.thestorygraph.com/books/6c5c24e2-6006-43b9-9e30-569400cd1581) by Dominica Degrandis

### Small batch sizes

Small batch sizes lead to faster lead times for individual projects, less WIP, faster detection of issues, and less rework.

### Elevate constraints in value streams

Focus efforts on optimizing bottlenecks in your value streams. The most value and effect can be had by remediating bottlenecks rather than steps before or after a bottleneck.

### Categories of waste and hardship

These categories of work are where inefficiencies are introduced and time and effort are wasted.

- Partially done work: WIP, work waiting in queues, etc. Partially done work loses value as time passes.
- Extra processes: Additional work that doesn't add value to the customer. Increases lead time.
- Extra features: Output that is not valued by the customer. Adds complexity and increases lead time.
- Task switching: Multitasking adds effort and time to the value stream.
- Waiting: Delays or queues increase lead time.
- Motion: Friction in handoffs adds ambiguity into the value stream and can increase lead time.
- Defects: Bugs, miscommunications, etc. add lead time and rework.

Reference: [Implementing Lean Software Development: From Concept to Cash](https://app.thestorygraph.com/books/a18e5dee-89e4-4c79-948e-ada45f61b476) by Mary Poppendieck and Tom Poppendieck

Added categories from Damon Edwards:

- Nonstandard or manual work: Any manual work that can be automated. Increases lead time and can lead to human error.
- Heroics: Individuals and teams should never be put in positions where they must perform unreasonable acts. Don't let heroics become the norm. When an team member needs to perform heroics, a breakdown of process has occurred. Should be remedied such that it's never required again.
### Developer productivity vs number of developers

When we increase the number of developers, individual developer productivity often significantly decreases due to communication, integration, and testing overhead

## Second way of DevOps: Feedback

Feedback and feedforward loops allow us to detect and remediate problems while they're smaller, cheaper, and easier to fix, to avert problems before they cause a catastrophe, and create organizational learning that we integrate into future work.

![Nesting circles showing the time difference of feedback from different components of the value stream](feedback-cycle-times.png)
^ Time it takes to get feedback from different stages of feedback. Taken from DevOps Handbook

## Third way of DevOps: Continuous Learning and Experimentation

Couple of quotes:

> When accidents and failures occur, instead of looking for human error, we look for how we can redesign the system to prevent the accident from happening again.

^ DevOps Handbook

> Even more important than daily work is the improvement of daily work.

^ Mike Orzen, author of Lean IT

You want to create an environment of trust where experimentation and learning are the norm. You want to have simple process in place to elevate local learnings to global company-wide knowledge.

## Practices to improve the three ways

### Immutable infrastructure

Never allow changes directly to infrastructure. Any changes must be checked into version control in our infrastructure-as-code, then deployed as new infrastructure that replaces the existing infrastructure, rather than tweaking it in-place.

You wouldn't let someone log into app servers and make changes directly to the code. So why let that happen for infrastructure?

### Infrastructure as code

- Version control for your infrastructure. Blame and change history.
- Single source of truth for the state of your infrastructure.
- Replicate environments in a quick, repeatable, documented way.
- Make environments easier to rebuild than repair, makes it simpler and quicker to resolve issues.

### Testing pyramid

Should be able to discover the majority of issues in the quickest running tests, to facilitate fast feedback and iteration.

![Pyramid shape showing the different levels of testing comparing their speed to receive feedback](TestingPyramid.jpg)

If an issue is found during integration or end-to-end testing, which usually take a long time to run, a unit test should be created that catches this issue. That way if the same issue happens again, it will be caught sooner and the developer can get faster feedback.

Not only should you improve your code coverage over time, you should aim to move tests down the pyramid so that issues continue to be found faster.

Manual tests can be run in parallel with automated tests (which also could be running in parallel with each other) to improve testing efficiency.

![Flow chart showing the stages of testing and which portions can be done in parallel](AutomatedandManualParallelTesting.jpg)

### Continuous integration

The purpose of creating and working in branches is so that multiple developers can work in parallel and minimize conflicts.

Developers then need to manually "integrate" their code with other work. This process can be error prone. Especially when you don't understand someone else's code that is causing a conflict. This issue is exasperated the more developers are working in parallel.

Continuous Integration solves this problem by getting everyone to integrate into the main branch daily as part of their normal work.

### No more pull requests

Pull requests were designed by GitHub to facilitate accepting contributions from unknown and untrusted developers on open source projects. They give maintainers an opportunity to confirm the code matches the style and ethos of the project.

Developers within an organization should already share the style and ethos of the team. They should be trusted to continue to promote that ethos in their code.

Pull requests sacrifice performance and quality. This is a worth while tradeoff for open source projects. They force the author to wait for review, causing their work to become stale. And they force reviewers to context switch to review code. Often they don't take the time necessary to actually understand the code because of time pressures, so errors slip though anyway even though all that time was spent!

Using pull request internally is like forcing your family through airport security just to enter your house. It's a costly solution to a different problem.

If your worry is having multiple pairs of eyes on the code before it's deployed, that should happen way earlier in the project, before hard to change decisions are solidified. For example, having multiple devs on a project, Pari programming, frequent code reviews, etc. These can replace pull requests and assist in automating the integration and deployment pipeline.

Related reference not from DevOps Handbook: [Pull Requests from Infrastructure as Code](https://infrastructure-as-code.com/book/2021/01/02/pull-requests.html) by Kief Morris

### Git strategies

**Optimizing for individual productivity**: individual branches. Quick progress in the beginning, but offload integration work to the end, or sporadically through the project. This is more like waterfall development.

**Optimizing for team productivity**: always integrating and working off of main. Main is always up to date with the latest work and in a deployable state. This is more like agile development.

When merging is difficult because of large batch sizes, developers become incentivized not to refactor code so that they keep their integration simple. This contributes to tech debt in the codebase growing over time.

### Continuous delivery

Any code that passes all tests, builds, and is deployable in a *production-like* environment will be automatically deployed to production.

Our code and environments should be designed and built in such a way that the release of new functionality to users does not require a change to our application code. We should decouple the ideas of releasing features to users and deploying code to an environment. They are two different acts serving two different stakeholders.

When we're deploying code all the time through continuous deployment and dark launching, releasing new functionality to users becomes simply a business and marketing decision.

Reference: [Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation.](https://app.thestorygraph.com/books/77eb0975-4194-42e7-9db3-005d4250940f) by Jez Humble and David Farley

### Metrics

We should enable every developer to create metrics in their daily work and be able to display and analyze them.

Metrics should be visible and digestible for anyone in the company, and perhaps even publicly.

### A/B testing

If you don't have data about how whether a feature or change is performing compared to a baseline, then you don't know if the change has zero impact or even negative impact.

A study at Microsoft showed that through AB testing, 2/3 of proposed changed showed zero or negative impact.

Reference: [Release It!: Design and Deploy Production-Ready Software](https://app.thestorygraph.com/books/1bd756aa-7a60-4af1-a407-6311f9741421) by Michael T. Nygard

### Chatbots for automation

Example:
> @hubot deploy owl to production

The results of chatbot commands are returned to the chat.

- Everyone can see everything that is happening
- Engineers, on their first day of work, can see what daily work looked like and how it was performed
- People were more apt to ask for help when they saw others helping each other
- Rapid organizational learning was enabled and accumulated

### Single company-wide repository

Everyone has access to all the most up to date code easily.

Randy Shoup described the single repository as the most powerful mechanism for preventing failures at Google.

Tom Limoncelli described it as a competitive advantage for Google.



In my role as a software developer at my day job, I’ve been getting more interested in DevOps as a career trajectory. But what does that even mean? Do I even know?
  
This is a brief exploration into what DevOps is and how I see it relating to a small development team.

![DevOps Cycle from Atlassian](development-cycle.jpg)

## Index

* [Tl;dr](#tldr)
* [Initial Assumptions](#initial-assumptions)
* [Operations & Facilitators](#operations--facilitators)
* [Agile Development](#agile-development)
	* [Quick Iteration](#quick-iteration)
	* [Feedback & Data](#feedback--data)
* [Automation](#automation)
* [Not Only Software Development](#not-only-software-development)
* [Conclusion](#conclusion)
* [Resources](#resources)

## Tl;dr
Here are a few key ideas that I think define what DevOps is and what someone in a DevOps position should focus on:

* Automate things like dev environment setup and employ tools to make writing code simple. Make it easy and painless to write code.
* Automate testing so developers and teams are confident in the code being produced. Developers shouldn't have to "remember" to run tests.
* Automate infrastructure provisioning and blue/green deployments. Code that's reviewed and ready to go should be able to reach customers with a single click (with proper authentication). Make deploying new code trivial.
* Automate data collection and illumination for the dev team primarily but also for the business side depending on the size of your company.
* And I think in general, a DevOps role should facilitate software development by increasing confidence in work being done by both developers and product managers.

## Initial Assumptions

Earlier on in my career and in university, DevOps was a buzz word that was thrown around all the time. When I heard the term DevOps I associated that with “cloud infrastructure”. I was connecting the idea of “building things on cloud infrastructure” to the role of DevOps.
  
While using cloud infrastructure might be one aspect of the tools and responsibilities of a DevOps role, it doesn’t get to the root of what the role is about.

## Operations & Facilitators

Within a software company, big or small, there are various so-called “Operations” roles. Business Operations, Customer Operations, Insert Anything Operations. My perspective on these roles in general is that they are facilitator roles rather than managerial roles. “Managers” may often have an Officer title, while “facilitators” may have an Operations title.

What do I mean by Facilitators? These are roles that essentially amplify the work in a given area or team. Their job is to ensure process and procedure are in place so that work can get done effectively, impactfully, and efficiently. It’s not necessarily their job to oversee the work being done, but to boost the impact and productivity of the work being done.

DevOps is short for Development Operations. And in my context as a software developer, Development is actually short for Software Development (Development can mean many different things in different fields).
  
So in more plain language, the role of DevOps is related to the operations of the development team and the development lifecycle. Ok, so that sounds more appropriate than just “cloud infrastructure”, but what is involved in the operations of those things?

## Agile Development

While I’m no expert on Agile development practices, I’ve been working with variations of it for my entire software development career thus far.

In essence, Agile (with a capital A) is intended to make the development process more iterative and quick to react to user feedback. Rather than blocking out big periods of time for large projects that overhaul huge portions of the customer experience, we attempt to release work quicker and in smaller pieces to get feedback on particular features sooner.

This allows us to not waste time perfecting features that our customers don’t want or won’t use. Instead we can focus more directly on things that are impactful for our customers and our bottom line. Therefore, one aspect of a DevOps role is facilitating the Agile development process.

### Quick Iteration

Getting code written, reviewed, and deployed quickly helps you then iterate on that code sooner, leading to better experiences for the customer over time. Building and improving processes around code testing, review, and deployment is a key facilitator opportunity for a DevOps role. This can cover many different things, from automating testing and deployments to improving code review process.

### Feedback & Data

Timely feedback is also vital to the Agile development process. Both user experience feedback and systems operation feedback (through statistics and error reporting). This feedback informs subsequent development cycles and how features, tweaks, and bug fixes are prioritized against each other for the greatest impact. Developing systems to effectively and concisely collect and relay feedback to the appropriate developers and project leads allows for more informed decision making, and thus is another key responsibility of a DevOps role.

## Automation

There are many pieces of manual work and human intervention in the software development cycle, and the level of automation in place can vary wildly between different teams and companies. I entered my software development career from essentially a cloud-native perspective, having little to no experience deploying code in a different paradigm.

But today, with things like IaC (infrastructure as code) and remote dev environments being the norm, every aspect of the developer experience can be automated in some way. Including writing code believe it or not.

* __Dev environment setup can be automated with old tools like Docker, or emerging tools like [JetBrains Space](https://www.jetbrains.com/space/).__ Space is essentially a remote dev environment that helps teams standardize their development environments across a team. Space is in closed beta as of the writing of this article. GitHub & Microsoft are working on similar tools with GitHub Codespaces and VS Code.
* __Writing code can be automated with new tools like [GitHub Copilot](https://copilot.github.com/).__ Copilot is a machine learning tool that has learned how to write code in many popular languages. While it can't directly replace a human developer yet, it can greatly speed up the process of writing code from scratch.
* __Reviewing, unit-testing, and merging code can be automated with tools like GitHub and [GitHub Actions](https://github.com/features/actions).__ There is still a bit of human touch needed in the review process to ensure features are truly meeting the business needs of the brief, but reviewing security and dependencies for example can be greatly sped up with the help of automation tools. Running tests can be automated with a variety of CI tools. Merging code and resolving merge conflicts has become downright trivial with tools like git and GitHub.
* __Deploying code to infrastructure can be automated with a variety of CD tools.__ I've been mainly interested in GitHub Actions and automating deployments to AWS. But I'm now also interested in learning more about AWS CDK (cloud development kit). [This blog post from GitHub](https://github.blog/2021-01-25-improving-how-we-deploy-github/) has been quite influential in my current thinking about automated deployments.
* __App stats and error monitoring can be automated with tools like Sentry, Datadog, and NewRelic.__ Those stats can be viewed through there respective dashboards or made more accessible through things like Slack bots.

## Not Only Software Development

In a small company or on a small team, often certain roles wear many hats. In our case, business operations bleed into the development team all the time. People working with clients, company finances, product managers, or even the CEO, will often have inquiries for data about various aspects of our business operations.

Without adequate access to that data, requests can sometimes fall on developers who are familiar with the system and the data. Ad-hoc SQL queries to our database can sometimes be the norm, but this isn’t an efficient use of anyone's time.

Building systems and interfaces to our company’s data can be integral to operations and development. Depending on the size of the company and the other roles there, this idea of data illumination could fall on a Business Operations role, but often it can require integration and understanding of the technical stack that is foundational to your application. Therefore that responsibility can fall on a DevOps role, to build digestible interfaces and secure pipelines to get the relevant data to the team members that need it.

## Conclusion

In summary, I see DevOps as primarily a facilitator and multiplier role. Depending on the size of the team they could be simultaneously doing development themselves, but the focus is on how to make the development team more productive.

Facilitating the development team to be more productive means a few things:

* Make code easy and painless to write. Automate environment setup.
* Make code simple to test. Automate tests.
* Make code trivial to deploy when ready. Automate infrastructure and blue/green deployments.
* Make data about development, testing, and deployment visible to the team. Automate data collection and illumination with tools like Datadog.

My summary and this article represents my understanding and opinions right now (Spring 2022). I am constantly learning and improving on my career journey and my knowledge and opinions will undoubtedly change over time.

If you have any comments or suggestions please feel free to reach out to me on Twitter [@tyreldelaney](https://twitter.com/tyreldelaney)! 👋🏻

## Resources

* [The Phoenix Project](https://itrevolution.com/the-phoenix-project/) - Gene Kim, Kevin Behr, and George Spafford
* [Improving how we deploy GitHub](https://github.blog/2021-01-25-improving-how-we-deploy-github/) - GitHub Blog
* [DevOps principles](https://www.atlassian.com/devops/what-is-devops) - Atlassian

#automation

Book Notes: DevOps Handbook, The

August 27, 2023 8 Minute Read

What is DevOps?

April 10, 2022 7 Minute Read

Node CI/CD Pipeline using GitHub Actions & AWS Elastic Beanstalk

May 30, 2020 24 Minute Read

Get in touch