AWS access that revokes itself

AWS Dynamic DNS

AWS Solutions Architect Associate Certification

Battlesnake Programming Competition 2018

Book Notes: DevOps Handbook, The

Building deployment automation piece by piece

Dark Mode React Toggle

Deno Tests & GitHub Actions CI

Deploy a Next.js Static Site to GitHub Pages

Migrating Frontend Assets During Grails 2 → 3 Upgrade

Lightshift Game

Markdown based Code Block Syntax Highlighting

Building a Mechanical Keyboard from Scratch

My Introduction to Game Development

Next.js Tag Pages

Node CI/CD Pipeline using GitHub Actions & AWS Elastic Beanstalk

Resources I Found Valuable in 2022

Small Form Factor PC Build Part 1

Small Form Factor PC Part 2 - Water Cooling

Shipping ECS Fargate logs to Datadog with FireLens (and what broke)

Static Site Search Part 3 - Optimizing the Index

Static Site Search Part 1 - Preprocessing Articles

Static Site Search Part 2 - Search Component

How I use my Synology NAS as my personal cloud

What is a Synology NAS?

What I'm Learning in 2022

What is DevOps?

This past June I set up log shipping from ECS Fargate to Datadog using FireLens. It worked on the first deploy. Then I spent a week chasing two problems with it, and months later I found out I'd been wrong about what fixed one of them.

![Shipping ECS Fargate logs to Datadog with FireLens heading on a dark background](shipping-ecs-fargate-logs-to-datadog-with-firelens-and-what-broke-ecs-fargate-logs-hero.png)

## Tl;dr

[The setup](#the-setup): A new Go API on ECS Fargate, and three ways to get its logs into Datadog.
[The happy path](#the-happy-path): About twenty lines of Terraform for the log router and the app's log config.
[Breakage #1: our tags weren't applied (except they were)](#breakage-1-our-tags-werent-applied-except-they-were): I filed a bug on myself for mixing up tags and attributes.
[Breakage #2: every log is INFO](#breakage-2-every-log-is-info): We logged the HTTP status code as `status`, which is a reserved attribute. Datadog read `200` as a severity.
[What I'd do differently](#what-id-do-differently): Grep for reserved attribute names, and prove each log level on day one.
[So what did I learn?](#so-what-did-i-learn)
[Resources and links](#resources-and-links)

## The setup

I was standing up the platform side of a new Go API on ECS Fargate. Small team, SOC2 coming up, and we needed to see what the thing was doing in production. We already used Datadog, so the only question was how to get the logs there:

* Run the Datadog agent as a sidecar in every task. Well documented, but it's a second container to size, patch, and pay for in every task definition.
* Ship logs straight from our custom logger to [Datadog's HTTP intake endpoint](https://docs.datadoghq.com/logs/log_collection/?tab=http#additional-configuration-options). Tempting, but then our code owns retries, batching, and backpressure to a third party, and logging is the thing you want working *most* when everything else is broken.
* Use [AWS FireLens](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_firelens.html), ECS's built-in log routing on Fluent Bit, with Datadog as the output plugin.

We went with FireLens. [AWS](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_firelens.html) and [Datadog](https://docs.datadoghq.com/integrations/ecs_fargate/?tab=webui#log-collection) both point you at it for Fargate, it lives entirely in the task definition (so it's Terraform, not application code), and the application keeps writing JSON to stdout.

Logs were flowing on a Wednesday. That Saturday I filed two bugs against my own setup, and both were closed the following Thursday, so about a week from "it works" to "it actually works". Understanding why took a few more months.

## The happy path

FireLens is a log *router*. You add an `aws-for-fluent-bit` container to your task definition, mark it as the FireLens container, and point your application container's log driver at it. Your app keeps writing to stdout, and Fluent Bit forwards it to Datadog's intake.

The log router container, in Terraform:

```hcl
{
  name              = "datadog_log_router"
  image             = var.aws_for_fluent_bit_image   # pinned to a specific tag
  essential         = true
  cpu               = floor(var.cpu * var.log_router_container_resource_ratio)
  memoryReservation = floor(var.memory * var.log_router_container_resource_ratio)
  firelensConfiguration = {
    type    = "fluentbit"
    options = { enable-ecs-log-metadata = "true" }
  }
  user = "0"
}
```

`enable-ecs-log-metadata` attaches the ECS context (cluster, task ARN, task definition family, container name) to every log line. It defaults to on, but I'd rather state that default than rediscover it when it's missing.

The resource ratio is the part I couldn't find in any of the docs, which happily show a FireLens container with no CPU or memory on it at all. We give the log router a fixed fraction of whatever the main container gets, so a beefy task gets a beefy log router and a small one doesn't waste reservation. It's not sophisticated, but we never had to think about it again.

The app container's log configuration is where the Datadog behaviour lives:

```hcl
logConfiguration = {
  logDriver = "awsfirelens"
  options = {
    "Name"           = "datadog"
    "Host"           = var.datadog_host
    "dd_service"     = var.dd_service
    "dd_source"      = "ecs-task"
    "dd_tags"        = "environment:${terraform.workspace},stack:platform"
    "dd_message_key" = "log"
    "TLS"            = "on"
    "provider"       = "ecs"
  }
  secretOptions = [
    {
      name      = "apikey"
      valueFrom = "${local.datadog_api_key_secret_arn}:api_key::"
    }
  ]
}
```

The API key lives in Secrets Manager and ECS injects it through `secretOptions`, so it never lands in the task definition JSON or in state. The `:api_key::` suffix pulls a single key out of a JSON secret: the format is `<arn>:<json-key>:<version-stage>:<version-id>`, and the trailing colons are the empty version fields. It's in the [AWS docs](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/specifying-sensitive-data-tutorial.html), but it's easy to skim past, and getting it wrong gives you a container that won't start with a message about the secret, not the format.

Logs showed up in Datadog within minutes. It really was that easy, until Saturday.

## Breakage #1: our tags weren't applied (except they were)

`dd_tags` was setting `environment:<workspace>` and `stack:platform` on every log, and as far as I could tell in the Datadog UI, neither was there. We run two products in one Datadog org, and tags were how we'd separate them in dashboards, monitors, and eventually access. If tags didn't work, the whole logging story was blocked.

So I filed a bug. I was looking at the *Attributes* section of the log side panel, and tags are in the *Tags* section. They'd been applied the whole time. Five days later I closed it with a comment that amounts to "oh, this already works, I was looking at the wrong panel."

Datadog does make this easy to get wrong, because it splits log metadata into two things that look identical when you're new to it.

Tags are infrastructure-level metadata: `dd_tags`, the ECS metadata from `enable-ecs-log-metadata`, `service`, `source`, `env`. They share a namespace with your metrics and traces, which is what lets you pivot from a log to a dashboard. You query them bare, like `env:prod`.

Attributes are the parsed contents of your JSON log body, whatever your logger emitted. You query them with an `@` prefix, like `@env:prod`.

Ten minutes of poking at the log panel would have saved me the ticket.

## Breakage #2: every log is INFO

Every log line arrived in Datadog as INFO, warnings and errors included. The status facet showed exactly one value, no monitor on `status:error` could fire, and the error rate graph was a flat line at zero, which looks like good news right up until you find out what it means.

My hunch, written into the ticket, was that Datadog reads severity from an attribute called `status`, while Go's `slog.NewJSONHandler` emits `"level":"INFO"`. Different key, so everything defaults to INFO. We renamed the key, the severities came back, I closed the ticket, and that was the week.

That explanation is wrong, and I only found out months later while writing this up.

### What the docs actually say

Datadog [preprocesses JSON logs](https://docs.datadoghq.com/logs/log_configuration/pipelines/#preprocessing) before your pipelines see them. On status, it says:

> if a JSON formatted log file includes one of the following attributes, Datadog interprets its value as the log's official status: `status`, `severity`, `level`, `syslog.severity`

`level` is in that list, and it was before our incident. Our preprocessing config is stock, so Go's default `slog` output was always going to be read correctly. What actually made everything INFO is in two details of the docs that are easy to read past.

First, the list is ordered, and the order *is* the precedence. I'd read it as a set. Preprocessing takes the *first* of those attributes it finds, so with both `status` and `level` present, `level` is never consulted.

Second, unrecognized values don't fail, they become INFO. From the [Log Status Remapper docs](https://docs.datadoghq.com/logs/log_configuration/processors/log_status_remapper/), integers 0 through 7 map to Syslog severities, and "All others map to info (6)." A *missing* `status` sends the search on to `severity`, then `level`. A `status` that's present but unreadable ends the search right there, at INFO. Nothing errors, and nothing tells you Datadog didn't understand your value.

### The actual root cause

Our HTTP request logger did this:

```go
slog.Int("status", 200)
```

That's the HTTP status code, in the reserved attribute, on every request log. Preprocessing found `status`, read `200`, which isn't in 0-7, mapped it to INFO, and stopped looking. `level` was sitting right there in the same log line, correctly populated, and never got read.

It stayed quiet because an HTTP status code is a *plausible* integer, just never a valid Syslog severity. `200`, `404`, and `500` all map to INFO, so a 500 logged at the same severity as a 200, and a 500 is exactly what you'd want to alert on.

### The fix, and the half of it that did nothing

The PR shipped two changes:

```go
// The change we credited: rename slog's severity key to `status`.
stdoutHandler = slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
	ReplaceAttr: func(_ []string, a slog.Attr) slog.Attr {
		if a.Key == slog.LevelKey {
			a.Key = "status"
		}
		return a
	},
})

// The change that actually fixed it: stop squatting on the reserved attribute.
slog.Int("status_code", 200)   // was: slog.Int("status", 200)
```

The second one is the fix. With the HTTP code moved to `status_code`, preprocessing found no `status`, fell through to `level`, and read `"INFO"` / `"WARN"` / `"ERROR"` exactly as documented.

The `ReplaceAttr` block does nothing, because `level` was already supported, and it makes our output less portable by emitting a non-standard severity key. It's eight lines that should be deleted.

Both changes shipped together, the symptom went away, and the rename got the credit for months because it was the more interesting-looking change. We had a working fix and a wrong mental model, and nothing in the system was ever going to correct us.

We also considered a Datadog Status Remapper processor pointed at `level`, or a Fluent Bit filter renaming the key before it ships. Both add configuration to work around a problem we created in the application, while fixing the squatted attribute is one line.

### Grep your logger for these

Datadog reserves these attribute names and aliases, and silently prefers them over whatever you meant:

| Reserved | Aliases it also reads |
| --- | --- |
| `status` | `severity`, `level`, `syslog.severity` |
| `message` | `msg`, `log` |
| `service` | `syslog.appname`, `dd.service` |
| `host` | `syslog.hostname`, `dd.host` |
| `timestamp` | `date`, `_timestamp`, `published_date` |
| `trace_id` | `dd.trace_id`, `contextMap.dd.trace_id` |

`status` is the one that bites, because it's the obvious name for an HTTP status code. But `message`, `host`, and `date` are all names a reasonable logger emits for reasonable reasons. Grep your logging code for every one of these before your first deploy. It takes two minutes, and the alternative is a monitor that silently never fires.

If you got here from the symptom, Datadog has a guide: [Logs show INFO status for warnings or errors](https://docs.datadoghq.com/logs/guide/logs-show-info-status-for-warnings-or-errors/). It states the default: "By default, when Datadog's Intake API receives a log, an `INFO` status generates and appends itself as the `status` attribute."

## What I'd do differently

* Grep for reserved attribute names before the first deploy. It's the highest-value thing on this list.
* Verify every log level on day one. Emit a DEBUG, INFO, WARN and ERROR as soon as logs flow, and confirm all four in the UI. I found this three days late because "logs are showing up" felt like done.
* Decide your retention before you need it. We dual-wrote to CloudWatch and Datadog at first, paying twice for the same lines, and landed on 30 days in Datadog. That window is also why I couldn't audit this later: by the time I checked whether the rename had done anything, the original logs were gone, and I had to reconstruct it from the diff and the docs.
* Know when direct HTTP is the better answer. FireLens is the right default for Fargate because it's declarative and lives outside your application. If you're somewhere ECS doesn't reach, or need something a Fluent Bit output plugin doesn't support, Datadog's HTTP intake is right there. Just know your application now owns log delivery.

## So what did I learn?

The integration was twenty lines of config. The hard part was the semantics: which metadata is a tag and which is an attribute, and who owns the word `status`. Datadog won't tell you it preferred a different attribute than the one you meant, and a fix that ships next to the real one will take the credit for months, because the symptom went away and nobody had a reason to look closer.

So distrust the pipeline until you've personally seen one WARN and one tag in the UI. And when the symptom clears, make sure you know *which* change cleared it.

## Resources and links

* [Datadog: ECS Fargate log collection](https://docs.datadoghq.com/integrations/ecs_fargate/?tab=webui#log-collection)
* [AWS: Custom log routing with FireLens](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_firelens.html)
* [AWS: Specifying sensitive data with Secrets Manager](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/specifying-sensitive-data-tutorial.html)
* [Datadog: Preprocessing for JSON logs](https://docs.datadoghq.com/logs/log_configuration/pipelines/#preprocessing)
* [Datadog: Log Status Remapper](https://docs.datadoghq.com/logs/log_configuration/processors/log_status_remapper/)
* [Datadog: Logs show INFO status for warnings or errors](https://docs.datadoghq.com/logs/guide/logs-show-info-status-for-warnings-or-errors/)
* [Datadog: Log collection over HTTP](https://docs.datadoghq.com/logs/log_collection/?tab=http#additional-configuration-options)


## Backstory
I've worked with AWS professionally for around 5 years. I haven't touched most of what AWS offers, but I know a handful of services well.

I'd wanted an AWS certification for a while. In 2020, early in my career, I started a prep course for the AWS Certified Developer Associate. I'd only been working with AWS for a year and had neither the hands-on experience nor the time to devote to it. I spent several months on the Udemy course, made it 20-30% of the way through, and abandoned it. The content was just too much for me at the time.

Fast forward 4 years to September 2023. I'd been working with AWS that entire time and had a ton of hands-on experience: EC2, Elastic Beanstalk, Load Balancers and Auto Scaling, RDS & Aurora, CloudFront & WAF, Route53 & DNS, Security Groups & IAM. It felt like a good time to reattempt certification.

![AWS solutions architect title on a dark background](aws-solutions-architect-associate-certification-awssolutionsarchitecthero.png)
## Preparation
Folks online heavily recommend Stephane Maarek's AWS certification prep courses on Udemy. He has them for most if not all AWS certificates.

In 2020 I attempted his [prep course for the AWS Certified Developer Associate exam](https://www.udemy.com/course/aws-certified-developer-associate-dva-c01)) and only made it 22% of the way through according to Udemy.

This time, focusing more on Platform and Infrastructure professionally, I pursued Solutions Architect instead of Developer, and bought his [Ultimate AWS Certified Solutions Architect Associate SAA-C03](https://www.udemy.com/course/aws-certified-solutions-architect-associate-saa-c03/) in October 2023.

### Stephane Maarek's AWS Certification prep course

Stephane's courses are good and very thorough. The AWS Solutions Architect certification covers a wide array of topics and the questions vary, so a prep course needs to cover a lot.

27.5 hours of lectures in total, covering services at roughly 3 levels: in-depth, general, and surface-level.

**In-depth:**

- S3 (Simple Storage Service)
- EC2 (Elastic Compute Cloud)
- IAM (Identity and Access Management)
- VPC (Virtual Private Cloud) & Networking
	- Internet & NAT Gateways, NAT Instances
	- Subnets
	- Bastion Hosts
	- NACL & Security Groups
	- VPC Peering
	- VPC Endpoints
	- VPC Flow Logs
	- and more
- Route53

**General:**

- AWS CLI
- EBS (Elastic Block Store)
- AMI (Amazon Machine Image)
- EFS (Elastic File System)
- ALB (Application Load Balancer)
- NLB (Network Load Balancer)
- ASG (Auto Scaling Groups)
- ELB (general Elastic Load Balancing)
- RDS (Relational Database Service)
- Aurora
- CloudFront
- SQS (Simple Queue Service)
- Kinesis Data Streams and Data Firehose
- ECS (Elastic Container Service)
- Lambda
- CloudWatch
- KMS (Key Management Service)

**Surface-level:**

- GWLB (Gateway Load Balancer)
- ElastiCache
- Global Accelerator
- Snow Family
- FSx
- Storage Gateway
- Transfer Family
- DataSync
- SNS (Simple Notification Service)
- MQ (Apache ActiveMQ & RabbitMQ)
- ECR (Elastic Container Repository)
- EKS (Elastic Kubernetes Service)
- App Runner
- DynamoDB
- API Gateway
- Step Functions
- Cognito
- Document DB
- Neptune
- Keyspaces
- QLDB
- Timestream
- Athena
- Redshift
- OpenSearch
- EMR (Elastic MapReduce)
- QuickSight
- Glue
- Lake Formation
- Kinesis Data Analytics
- Rekognition
- Transcribe
- Polly
- Translate
- Lex & Connect
- Comprehend
- SageMaker
- Forecast
- Kendra
- Personalize
- Textract
- EventBridge
- CloudTrail
- Config
- Organizations
- Control Tower
- SSM (Systems Manager)
- Secrets Manager
- ACM (AWS Certificate Manager)
- WAF (Web Application Firewall)
- Shield
- Firewall Manager
- GuardDuty
- Inspector
- Macie
- DMS (Database Migration Service)
- Backup
- CloudFormation
- SES (Simple Email Service)
- Pinpoint
- Cost Explorer
- Batch
- AppFlow
- Amplify
- ElasticBeanstalk

Yes. There is a lot of different AWS services.

A few concepts are covered separate from individual services:

- High Availability
- 3-tiered Web Applications (Frontend, Backend, Database)
- Serverless
- Decoupling Applications
- Choosing the right database for the kind of data you have
- Disaster Recovery
- Caching Strategies
- Data & Event Processing

The course frames these services and topics through the AWS Well Architected Framework and its 6 pillars:

- Operational excellence
- Security
- Reliability
- Performance efficiency
- Cost optimization
- Sustainability

https://docs.aws.amazon.com/wellarchitected/latest/framework/definitions.html

**Course structure**

Each unit ends with a quiz, and about half the lectures include a "hands-on" video where the instructor uses the service in the AWS web console, noting what the free tier covers and what will cost you.

Working a few hours each week it took me about 4 months to complete.

The course includes a repeatable 65-question practice test. I took it after finishing and failed at about 65%. It gives you the results, which answers were correct, and a thorough description of each correct answer, which made great study material. I studied my weak areas for 2 weeks and retook it, passing at about 95%. Same test, so a better score was expected.

## The exam

I felt ready to attempt the certification exam at this point.

### Scheduling the exam

You can schedule the exam in-person or online these days. A few years ago my only option was an in-person exam in Vancouver, 4 hours away; there's a Victoria facility now, but I took mine online.

AWS uses the Pearson Vue platform for online tests. First time I'd used it, but I suspect it's common in this remote era.

Scheduling and paying was simple. Your first AWS exam costs $157.50 USD (~$215 CAD). Each one you complete earns a 50% voucher for the next, so only the first is full price.

Pearson Vue asks you to run a system test beforehand. Their proprietary software runs on your machine doing some kind of logging and surveillance, attempts to lock your screen to the test, and records your webcam for the entire exam. The system test checks compatibility and that your connection can stream video.

They want a private space and a reliable connection, and advise against a work laptop since MDM software and corporate firewalls interfere with the testing software. I only have wifi at home and my space isn't private, so I booked a booth at a co-working space with a wired connection and used my personal Windows 11 laptop.

### Taking the exam

The basic structure of taking an AWS certification exam remotely is:
- Check in 15 mins before
- Start the exam
- No breaks
- You have 2:10 to finish 65 questions
- You can mark questions for review to return to later

I hit a fair number of technical difficulties. I'm grateful the invigilators and I solved them ad-hoc and I wasn't disqualified.

**Technical difficulties**

First, signing in on my phone (you scan a QR code). The site asks for photos of your ID and of the room from several angles. Uploading them crashed the site and dumped me back to a login screen. Stressful, since I didn't know how strict the start time was. Luckily an identifier given to me by the software on my computer let me rejoin the check-in already in progress.

In the second part of check-in an invigilator gets on a call with you through the software on your computer. Fairly standard questions, then they ask you to show them your testing space by webcam. Using my laptop webcam meant picking up the laptop, at which point the janky USB-C Ethernet adapter from the co-working space disconnected and I lost internet. I was mortified, and swapped in my own adapter as fast as I could. The video chat never came back but the text chat did, and I was re-queued and picked up by a different invigilator. Same questions again, but this time I held the dongle in place while showing them around the room.

The last difficulty was quick to resolve but still stressful: about 4 questions in, the application crashed to a red warning message. I could still call the invigilator, they reset the system, my progress was intact and I lost a minute or two.

### The exam itself

There were 65 multiple choice questions with 3-6 possible answers given.

I'm not supposed to share the exam questions, and don't remember them that precisely anyway. But I do remember a few question types and the style in which they were asked.

I took the Solutions Architect certification exam, so the questions were heavily slanted that way.

The general format was "given this scenario/client/task, how would you solve it in the most high availability/redundant/lowest initial cost/lowest ongoing cost/most automated way". They'd present something like "a client would like to transfer 50 petabytes of time-series financial records from their on premises servers", then ask for the solution matching a qualifier: cheapest, most highly available, least operational overhead.

Most questions had multiple "correct" answers, meaning several would accomplish the task. But only 1 was "most correct" and also met the qualifier.

So the exam tests not just knowing what most AWS services do, but knowing precisely when you'd use one over another. NAT instances vs managed NAT Gateways, for example. Both have pros and cons.

A lot of questions focused on VPCs, particularly connecting multiple VPCs, and on transferring data in and out of AWS.

If I remember correctly you need 70% to pass. The exam is pass or fail, but you get a score at the end you can use to calculate how many questions you got right.

You can retake an failed exam after 2 weeks at the full cost.

## Takeaways

Obviously all the learning was done up front; the certification exam is just a goal to shoot for. Stephane Maarek's course was thorough and well laid out and I learned a lot from it, albeit mostly at a surface level. Actively using AWS since starting at Giftbit helped just as much.

I really enjoyed having the certification as a north star. It was a great motivator to study regularly and use my pro-d time at work (I also studied a bunch outside of work).

This exam was fairly difficult, but having experience with AWS and completing that prep course gave me what I needed to be successful.


![Book cover of Dev ops handbook](devopshandbook.png)

Here are some point-form takeaways from my notes from reading [The DevOps Handbook](https://app.thestorygraph.com/books/70e99c96-fc97-465a-bf1b-058072e18bc5) by John Willis, Gene Kim, and Patrick Debois.

## Index

[Core premise](#core-premise)
[First way of DevOps: Flow](#first-way-of-devops-flow)
- [Make work visible](#make-work-visible)
- [Limit WIP](#limit-wip)
- [Small batch sizes](#small-batch-sizes)
- [Elevate constraints in value streams](#elevate-constraints-in-value-streams)
- [Categories of waste and hardship](#categories-of-waste-and-hardship)
- [Developer productivity vs number of developers](#developer-productivity-vs-number-of-developers)
[Second way of DevOps: Feedback](#second-way-of-devops-feedback)
[Third way of DevOps: Continuous Learning and Experimentation](#third-way-of-devops-continuous-learning-and-experimentation)
[Practices to improve the three ways](#practices-to-improve-the-three-ways)
- [Immutable infrastructure](#immutable-infrastructure)
- [Infrastructure as code](#infrastructure-as-code)
- [Testing pyramid](#testing-pyramid)
- [Continuous integration](#continuous-integration)
- [No more pull requests](#no-more-pull-requests)
- [Git strategies](#git-strategies)
- [Continuous delivery](#continuous-delivery)
- [Metrics](#metrics)
- [A/B testing](#ab-testing)
- [Chatbots for automation](#chatbots-for-automation)
- [Single company-wide repository](#single-company-wide-repository)

## Core premise

At the core of this book are the three "ways of DevOps".

1. Left-to-right flow of work from development to operations to the customer.
	- Speeding up this channel gets work from idea to the customer faster.
	- Decreasing lead times through better process leads to higher value delivered to customers and higher developer satisfaction.
	- Resulting practices from improving this channel include continuous integration, continuous deployment, and infrastructure-as-code, among others.
2. Right-to-left constant feedback from customers to operations to developers.
	- Prevent problems from happening again.
	- Enable faster detection of and recovery from issues.
	- Resulting practices from improving this channel are easy to access and understand metrics, and making your systems more observable.
3. Cycles of learning. Creating a generative, high-trust culture.
	- Scientific approach to risk taking.
	- Organizational learning. Multiply the effects of new knowledge. Transform local discoveries into global improvements.
	- Amplifying positive feedback loops creating ever-safer systems.

## First way of DevOps: Flow

### Make work visible

Make work visible. Kanban boards allow you to easily see the state of a current projects and the team in general. Keep work from development and operations on the same kanban board so that both teams can see the in-progress work of the other.

### Limit WIP

Multi-tasking significantly degrades efficiency. Limit the number of WIP items to keep teams focused. You can do this by controlling the swim lane size in your kanban boards.

Reference: [Making Work Visible: Exposing Time Theft to Optimize Work & Flow](https://app.thestorygraph.com/books/6c5c24e2-6006-43b9-9e30-569400cd1581) by Dominica Degrandis

### Small batch sizes

Small batch sizes lead to faster lead times for individual projects, less WIP, faster detection of issues, and less rework.

### Elevate constraints in value streams

Focus efforts on optimizing bottlenecks in your value streams. The most value and effect can be had by remediating bottlenecks rather than steps before or after a bottleneck.

### Categories of waste and hardship

These categories of work are where inefficiencies are introduced and time and effort are wasted.

- Partially done work: WIP, work waiting in queues, etc. Partially done work loses value as time passes.
- Extra processes: Additional work that doesn't add value to the customer. Increases lead time.
- Extra features: Output that is not valued by the customer. Adds complexity and increases lead time.
- Task switching: Multitasking adds effort and time to the value stream.
- Waiting: Delays or queues increase lead time.
- Motion: Friction in handoffs adds ambiguity into the value stream and can increase lead time.
- Defects: Bugs, miscommunications, etc. add lead time and rework.

Reference: [Implementing Lean Software Development: From Concept to Cash](https://app.thestorygraph.com/books/a18e5dee-89e4-4c79-948e-ada45f61b476) by Mary Poppendieck and Tom Poppendieck

Added categories from Damon Edwards:

- Nonstandard or manual work: Any manual work that can be automated. Increases lead time and can lead to human error.
- Heroics: Individuals and teams should never be put in positions where they must perform unreasonable acts. Don't let heroics become the norm. When an team member needs to perform heroics, a breakdown of process has occurred. Should be remedied such that it's never required again.
### Developer productivity vs number of developers

When we increase the number of developers, individual developer productivity often significantly decreases due to communication, integration, and testing overhead

## Second way of DevOps: Feedback

Feedback and feedforward loops allow us to detect and remediate problems while they're smaller, cheaper, and easier to fix, to avert problems before they cause a catastrophe, and create organizational learning that we integrate into future work.

![Nesting circles showing the time difference of feedback from different components of the value stream](feedback-cycle-times.png)
^ Time it takes to get feedback from different stages of feedback. Taken from DevOps Handbook

## Third way of DevOps: Continuous Learning and Experimentation

Couple of quotes:

> When accidents and failures occur, instead of looking for human error, we look for how we can redesign the system to prevent the accident from happening again.

^ DevOps Handbook

> Even more important than daily work is the improvement of daily work.

^ Mike Orzen, author of Lean IT

You want to create an environment of trust where experimentation and learning are the norm. You want to have simple process in place to elevate local learnings to global company-wide knowledge.

## Practices to improve the three ways

### Immutable infrastructure

Never allow changes directly to infrastructure. Any changes must be checked into version control in our infrastructure-as-code, then deployed as new infrastructure that replaces the existing infrastructure, rather than tweaking it in-place.

You wouldn't let someone log into app servers and make changes directly to the code. So why let that happen for infrastructure?

### Infrastructure as code

- Version control for your infrastructure. Blame and change history.
- Single source of truth for the state of your infrastructure.
- Replicate environments in a quick, repeatable, documented way.
- Make environments easier to rebuild than repair, makes it simpler and quicker to resolve issues.

### Testing pyramid

Should be able to discover the majority of issues in the quickest running tests, to facilitate fast feedback and iteration.

![Pyramid shape showing the different levels of testing comparing their speed to receive feedback](TestingPyramid.jpg)

If an issue is found during integration or end-to-end testing, which usually take a long time to run, a unit test should be created that catches this issue. That way if the same issue happens again, it will be caught sooner and the developer can get faster feedback.

Not only should you improve your code coverage over time, you should aim to move tests down the pyramid so that issues continue to be found faster.

Manual tests can be run in parallel with automated tests (which also could be running in parallel with each other) to improve testing efficiency.

![Flow chart showing the stages of testing and which portions can be done in parallel](AutomatedandManualParallelTesting.jpg)

### Continuous integration

The purpose of creating and working in branches is so that multiple developers can work in parallel and minimize conflicts.

Developers then need to manually "integrate" their code with other work. This process can be error prone. Especially when you don't understand someone else's code that is causing a conflict. This issue is exasperated the more developers are working in parallel.

Continuous Integration solves this problem by getting everyone to integrate into the main branch daily as part of their normal work.

### No more pull requests

Pull requests were designed by GitHub to facilitate accepting contributions from unknown and untrusted developers on open source projects. They give maintainers an opportunity to confirm the code matches the style and ethos of the project.

Developers within an organization should already share the style and ethos of the team. They should be trusted to continue to promote that ethos in their code.

Pull requests sacrifice performance and quality. This is a worth while tradeoff for open source projects. They force the author to wait for review, causing their work to become stale. And they force reviewers to context switch to review code. Often they don't take the time necessary to actually understand the code because of time pressures, so errors slip though anyway even though all that time was spent!

Using pull request internally is like forcing your family through airport security just to enter your house. It's a costly solution to a different problem.

If your worry is having multiple pairs of eyes on the code before it's deployed, that should happen way earlier in the project, before hard to change decisions are solidified. For example, having multiple devs on a project, Pari programming, frequent code reviews, etc. These can replace pull requests and assist in automating the integration and deployment pipeline.

Related reference not from DevOps Handbook: [Pull Requests from Infrastructure as Code](https://infrastructure-as-code.com/book/2021/01/02/pull-requests.html) by Kief Morris

### Git strategies

**Optimizing for individual productivity**: individual branches. Quick progress in the beginning, but offload integration work to the end, or sporadically through the project. This is more like waterfall development.

**Optimizing for team productivity**: always integrating and working off of main. Main is always up to date with the latest work and in a deployable state. This is more like agile development.

When merging is difficult because of large batch sizes, developers become incentivized not to refactor code so that they keep their integration simple. This contributes to tech debt in the codebase growing over time.

### Continuous delivery

Any code that passes all tests, builds, and is deployable in a *production-like* environment will be automatically deployed to production.

Our code and environments should be designed and built in such a way that the release of new functionality to users does not require a change to our application code. We should decouple the ideas of releasing features to users and deploying code to an environment. They are two different acts serving two different stakeholders.

When we're deploying code all the time through continuous deployment and dark launching, releasing new functionality to users becomes simply a business and marketing decision.

Reference: [Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation.](https://app.thestorygraph.com/books/77eb0975-4194-42e7-9db3-005d4250940f) by Jez Humble and David Farley

### Metrics

We should enable every developer to create metrics in their daily work and be able to display and analyze them.

Metrics should be visible and digestible for anyone in the company, and perhaps even publicly.

### A/B testing

If you don't have data about how whether a feature or change is performing compared to a baseline, then you don't know if the change has zero impact or even negative impact.

A study at Microsoft showed that through AB testing, 2/3 of proposed changed showed zero or negative impact.

Reference: [Release It!: Design and Deploy Production-Ready Software](https://app.thestorygraph.com/books/1bd756aa-7a60-4af1-a407-6311f9741421) by Michael T. Nygard

### Chatbots for automation

Example:
> @hubot deploy owl to production

The results of chatbot commands are returned to the chat.

- Everyone can see everything that is happening
- Engineers, on their first day of work, can see what daily work looked like and how it was performed
- People were more apt to ask for help when they saw others helping each other
- Rapid organizational learning was enabled and accumulated

### Single company-wide repository

Everyone has access to all the most up to date code easily.

Randy Shoup described the single repository as the most powerful mechanism for preventing failures at Google.

Tom Limoncelli described it as a competitive advantage for Google.


Here are some things that I'd like to learn more about in 2022. This is kindof a reference for myself. So I can look back at this next year and reflect on what I learned and what I wanted to learn. [Let me know on twitter](https://twitter.com/tyreldelaney) if you are learning these same things or you have any tips or resources to share!

![Logo montage of all the technologies outlined in this article](what-im-learning-in-2022.jpg)

## Index

[Learning from scratch](#learning-from-scratch)

* [Docker](#docker)
* [Vim/NeoVim](#vimneovim)

[Learning more of](#learning-more-of)

* [Github Actions](#github-actions)
* [AWS DynamoDB](#aws-dynamodb)
* [AWS Client VPN, VPNs in general, and SSO](#aws-client-vpn-vpns-in-general-and-sso)
* [SSH](#ssh)
* [HTTPS/SSL](#httpsssl)
* [Raspberry Pi](#raspberry-pi)
* [Home Network](#home-network)

[Things I'm interested in but might not get to this year](#things-im-interested-in-but-might-not-get-to-this-year)

* [A better static site generator / CMS](#a-better-static-site-generator--cms)
* [Webhooks](#webhooks)

## Learning from scratch

### Docker
Docker really feels like one of those technologies that I've been conveniently ignoring for a little too long. I think what I'll do to start is simply try to containerize some smaller projects I'm working on this year to dip my toes in.

I acquired a handful of second-hand Raspberry Pis not long ago and I've got a few ideas for some small home automation & monitoring projects that might be perfect simple projects to dockerize.

A stretch goal would be to work up to containerizing our monolithic app at my day job.

### Vim/NeoVim
I've always just barely snuck by with my fluency in the terminal. The more I get into networking and Dev Ops the more this deficiency is showing. In general I'd just like to work in the terminal a bit more this year.

Specifically I'm going to try to develop a couple Node and Python projects I'm cooking up remotely on the Raspberry Pis I hope to run them on.

Really two main goals I'd like to accomplish with this practice:
1. Get proficient and comfortable using vim.
2. Learn more about making a portable and lightweight dev environment. Something that I can easily spin up on a Raspberry Pi or AWS EC2 instance over SSH and feel comfortable doing real work in.

## Learning more of

### Github Actions
Automation is a big focus for me this year. And really just because I rely on Github both personally and professionally, Github Actions is a natural choice to skill up in.

We're now looking into automating and outsourcing our test suite at work and since we use Github for source control and code reviews/pull requests it makes total sense to use Github Actions.

I've automated a few things using Github Actions already, namely [tests and deployment of my](https://github.com/tyrelh/battlesnake-typescript-node) [Battlesnake](https://play.battlesnake.com/) to AWS Elastic Beanstalk. Automating even just tests for our app at work will be a bit more of a challenge since it's a much older stack and has many more moving parts.

### AWS DynamoDB
I wasn't actually planning on learning DynamoDB at the start of the year. But a project idea came to mind and the possibility of using Dynamo to store data rather than something like S3 sounded intriguing.

Essentially what I'm planning is building out a few Raspberry Pi temperature sensors for my house. Maybe 4 or 5 in total monitoring a few rooms in the house as well as the outdoor temperature.

The idea would be to collect data from all these sensors using Raspberry Pis maybe every 5 mins or so, and ship that data off to Dynamo. Then building an infrastructure to rotate through tables. Basically make a new table each day with a high write provision, and rotate down the previous days tables read and write provisions. Maybe after some period archive the data in S3.

Might be over engineering for the task, but it's for learning.

[Best Practices for Handling Time Series Data in DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-time-series.html)

### AWS Client VPN, VPNs in general, and SSO
With my day job going fully remote this year, setting up secure remote tools has been high on our priority list. Part of that will be learning about AWS Client VPN which we may use to grant our devs access to some of our AWS infrastructure.

Part of this work is also moving to more zero trust architecture, and leaning into SSO for authentication.

Also I recently acquired a Synology NAS for my home (which we actually use to use in our office before we gave that space up). I'm looking to centralize alot of my personal cloud use to the Synology and move away from services like Google Drive (office and documents), iCloud (photos), and Notion (notes). Part of that will be to setup a VPN at home so that I can use those services hosted in my home from anywhere. As well as accessing some of my Raspberry Pis for remote development.

### SSH
Learning more about SSH and tunneling goes hand-in-hand with a few other learning priorities I have. Learning best practices for connecting securely and crafting secure infrastructure. Also using SSH to practice remote development and learning terminal based tools.

### HTTPS/SSL
Same as above. Learning more about best practices around secure remote communication. Getting more confident understanding certs and how they work.

### Raspberry Pi
I've owned a Raspberry Pi 3 for a handful of years now, but I've never really had a project for it. I experimented a bit with retro game emulation on it, but those projects are pretty trivial to setup.

Now I have a handful of Raspberry Pi 2s and 3s and I have some plans for some smaller single purpose projects for the house that will be perfect for them.

### Home network
This kinda just happened organically, but I have a lot of computers in my house now! What I'd like to do is centralize a bunch of things into a proper rack. I have a few computers that are mining cryptocurrency that I'd like to transfer to rack-mountable server cases. Along with my new Synology NAS and a few Raspberry Pis.

Wiring my house with Cat-6 probably won't happen any time soon, but centralizing a bunch of things in a rack will let me hard-wire them all to a small switch without needing to run cables through all my walls.

Like I mentioned above, also setting up network storage and remote access for my home network.

## Things I'm interested in but might not get to

### A better static site generator / CMS
My website currently is hand-built in Javascript and React with a few plugins that let me render my articles from markdown. There is still a fair amount of manual process to get each markdown file wired up to display as a page and route, but at least I don't need to reformat the article. I can just paste it in from my notes app and then wire it up.

Ideally I'd like to learn a better static site generator to simplify this process even more. I've been thinking of learning [Gatsby](https://www.gatsbyjs.com/) for a while now, but I've just found it a bit cumbersome and hard to get into.

Some alternatives I'm considering learning instead of Gatsby are [Jeckyll](https://jekyllrb.com/) or [Hexo](https://hexo.io/).

### Webhooks
Just something I'm interested in learning more about.

## End

[@ me on twitter](https://twitter.com/tyreldelaney) if you are learning these same things or you have any tips or resources to share!


This is a Dynamic DNS system built using AWS Route53 and a Node script running on a Raspberry Pi.

![](./aws-ddns.jpg)

## Index
* [What is a DDNS?](#what-is-a-ddns)
* [Prerequisites](#prerequisites)
* [Getting your current IP](#getting-your-current-ip)
* [Get the IP address from your DNS record to check if it needs to be updated](#get-the-ip-address-from-your-dns-record-to-check-if-it-needs-to-be-updated)
* [Update IP in DNS record](#update-ip-in-dns-record)
* [Run the script on a schedule](#run-the-script-on-a-schedule)
* [Resources](#resources)

## What is a DDNS?
A dynamic DNS is handy if you want to host or access things on your home network from the internet but you don't have a static IP address from your service provider.

Essentially what it does is update the IP address for a given DNS record to be the current public IP address of the machine running the script.

So for example, if you want to have a domain name always pointing to your home network, you run this script on a machine within your home network.

Let's say your domain is `example.com`, you could create an A record for `home.example.com` that points to your home network. You could manually check what your current public IP address is and set it in the AWS console. But likely you have a dynamic IP at home, and it may randomly change at any moment, breaking your DNS record.

Let's automate updating it

## Prerequisites
You'll need:
* An AWS account
	* With credentials set up in your environment of choice
* A domain and hosted zone set up in Route53
* Node installed
* AWS-SDK node package
* Some kind of machine running in your network of choice that will be always on so that it can run the script on a regular schedule. I use a Raspberry Pi as it's low power, easy to setup, and tiny.

First setup your project. I used NPM to init a new project, then added the `aws-sdk` and `node-fetch` packages. Also add `"type": "module"` to your _package.json_ so node doesn't yell at you for using `import`.

## Getting your current IP
There are many ways to do this. A trivial way that I chose is to use a service provided by AWS [checkip.amazonaws.com](http://checkip.amazonaws.com). Simply fetch that URL and trim the result to get your current public IP address.

It's probably a good idea to be validating the IP address returned is a valid IPV4 format. I make a simple helper function to validate an IP address

```javascript
import fetch from "node-fetch";

function validateIp(ipString) {
	const re = /^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$/;
	return !!ipString.match(re);
}

async function getCurrentIp() {
	console.log("Fetching current ip...");
	const response = await fetch("http://checkip.amazonaws.com/");
	const ip = (await response.text()).trim();
	if (validateIp(ip)) {
		console.log(ip);
		return ip;
	}
	console.error("Fetching ip failed");
	return false
}

async function main() {
	const ip = await getCurrentIp();
	if (!ip) {
		return 1;
	}
}

main();
```

## Get the IP address from your DNS record to check if it needs to be updated

This maybe isn't actually necessary. You're really just making a call to save yourself from making a call. But if this script got more complicated this might be a better idea.

Check out the [AWS Node SDK Docs](https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/getting-started-nodejs.html) and the [AWS JavaScript Route53 SDK Docs](https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Route53.html) for more details on their usage.

I made a function  `getIpFromDnsRecord` that fetches the current IP address from the A record of the given hostname and hosted zone id.

```javascript
import fetch from "node-fetch";
import AWS from "aws-sdk";

const hostedZoneId = "GETTHISIDFROMYOURAWSCONSOLE";
const hostname = "home.yourdomain.com";

const route53 = new AWS.Route53({ apiVersion: "2013-04-01" });

function validateIp(ipString) {
	const re = /^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$/;
	return !!ipString.match(re);
}

async function getIpFromDnsRecord() {
	console.log(`Fetching ip currently set for ${hostname}...`);
	const params = {
		HostedZoneId: hostedZoneId,
		StartRecordName: hostname,
		StartRecordType: "A",
		MaxItems: "2"
	}
	const recordSets = await route53.listResourceRecordSets(params).promise()
	for (let record of recordSets["ResourceRecordSets"]) {
		if (record["Name"] == hostname + ".") {
			if (record["ResourceRecords"].length == 1) {
				const ip = record["ResourceRecords"][0]["Value"].trim();
				if (validateIp(ip)) {
					console.log(ip);
					return ip;
				}
			}
		}
	}
	console.error(`Fetching ip for hostname ${hostname} failed`)
	return false;
}

async function getCurrentIp() {
	console.log("Fetching current ip...");
	const response = await fetch("http://checkip.amazonaws.com/");
	const ip = (await response.text()).trim();
	if (validateIp(ip)) {
		console.log(ip);
		return ip;
	}
	console.error("Fetching ip failed");
	return false
}

async function main() {
	const ip = await getCurrentIp();
	if (!ip) {
		return 1;
	}

	const previousIp = await getIpFromDnsRecord();
	if (!previousIp) {
		return 1;
	}

	if (ip == previousIp) {
		console.log("Ip hasn't changed. No action required")
	} else {
		console.log("Ip address has changed!")
	}
	
	return 0;
}

main();
```

## Update IP in DNS record
If the current public IP address is different than the one set in your DNS record, you'll want to update it!

I made a function `setIpInDnsRecord(ip)` that will update the DNS record in Route53.

```javascript
import fetch from "node-fetch";
import AWS from "aws-sdk";

const hostedZoneId = "GETTHISIDFROMYOURAWSCONSOLE";
const hostname = "home.yourdomain.com";

const route53 = new AWS.Route53({ apiVersion: "2013-04-01" });

function validateIp(ipString) {
	const re = /^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$/;
	return !!ipString.match(re);
}

async function setIpInDnsRecord(ip) {
	console.log(`Updating DNS record for ${hostname} to ${ip}...`);
	const date = new Date();
	const params = {
		ChangeBatch: {
			Changes: [{
				Action: "UPSERT", // this will update an existing record
				ResourceRecordSet: {
					Name: hostname,
					ResourceRecords: [
						{ Value: ip }
					],
					TTL: 300,
					Type: "A"
				}
			}],
			Comment: `Updated IP address on ${date.toString()}`
		},
		HostedZoneId: hostedZoneId
	};
	const response = await route53.changeResourceRecordSets(params).promise();
	// missing any error handling
	console.log(`DNS record for ${hostname} updated to ${ip}. Can take up to 60s to propogate`);
}

async function getIpFromDnsRecord() {
	console.log(`Fetching ip currently set for ${hostname}...`);
	const params = {
		HostedZoneId: hostedZoneId,
		StartRecordName: hostname,
		StartRecordType: "A",
		MaxItems: "2"
	}
	const recordSets = await route53.listResourceRecordSets(params).promise()
	for (let record of recordSets["ResourceRecordSets"]) {
		if (record["Name"] == hostname + ".") {
			if (record["ResourceRecords"].length == 1) {
				const ip = record["ResourceRecords"][0]["Value"].trim();
				if (validateIp(ip)) {
					console.log(ip);
					return ip;
				}
			}
		}
	}
	console.error(`Fetching ip for hostname ${hostname} failed`)
	return false;
}

async function getCurrentIp() {
	console.log("Fetching current ip...");
	const response = await fetch("http://checkip.amazonaws.com/");
	const ip = (await response.text()).trim();
	if (validateIp(ip)) {
		console.log(ip);
		return ip;
	}
	console.error("Fetching ip failed");
	return false
}

async function main() {
	const ip = await getCurrentIp();
	if (!ip) {
		return 1;
	}

	const previousIp = await getIpFromDnsRecord();
	if (!previousIp) {
		return 1;
	}

	if (ip == previousIp) {
		console.log("Ip hasn't changed. No action required")
	} else {
		console.log("Ip address has changed!")
		await setIpInDnsRecord(ip);
	}
	
	return 0;
}

main();
```

And that's about it for the script! You could make this simpler or more complex as you see fit.

Try it out by running the script

```bash
node script.js
```

## Run the script on a schedule
I am running this on a Raspberry Pi running the Raspberry Pi OS. This should be similar to any Linux environment (I think? 🤷🏻‍♂️).

I used a cron to schedule the node script. Open your cron schedules with
```bash
crontab -e
```

Add a new schedule for your script. Google cron syntax if you're unfamiliar. I set mine to trigger ever 10 minutes.

```crontab
*/10 * * * * node /home/YOURUSERNAME/path/to/script.js
```

Save that file with the new entry at the bottom.

Now just restart your machine and the cron should take effect.

As long as the device remains powered on and running inside your network, your DNS record will remain updated with your current IP address!

## Resources
*  [AWS Node SDK Docs](https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/getting-started-nodejs.html)
*  [AWS JavaScript Route53 SDK Docs](https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Route53.html)
* [AWS DDNS Example](https://github.com/awslabs/route53-dynamic-dns-with-lambda/blob/master/v1/dynamic_dns_lambda.py)
* [Crons on Raspberry Pi](https://bc-robotics.com/tutorials/setting-cron-job-raspberry-pi/)
* [DDNS Example from Anthony Heddings](https://gist.github.com/anthonyheddings/f22967967bbf524ed510c356678b2651)

#infrastructure

Shipping ECS Fargate logs to Datadog with FireLens (and what broke)

AWS Solutions Architect Associate Certification

Book Notes: DevOps Handbook, The

What I'm Learning in 2022

AWS Dynamic DNS

Get in touch