AWS Dynamic DNS

Battlesnake Programming Competition 2018

Book Notes: DevOps Handbook, The

Dark Mode React Toggle

Deno Tests & GitHub Actions CI

Deploy a Next.js Static Site to GitHub Pages

Migrating Frontend Assets During Grails 2 → 3 Upgrade

Lightshift Game

Building a Mechanical Keyboard from Scratch

My Introduction to Game Development

Next.js Tag Pages

Node CI/CD Pipeline using GitHub Actions & AWS Elastic Beanstalk

Resources I Found Valuable in 2022

Small Form Factor PC Build Part 1

Small Form Factor PC Part 2 - Water Cooling

Static Site Search Part 3 - Optimizing the Index

Static Site Search Part 1 - Preprocessing Articles

Static Site Search Part 2 - Search Component

How I use my Synology NAS as my personal cloud

What is a Synology NAS?

What I'm Learning in 2022

What is DevOps?


![](./static-site-search-optimizing-the-index.jpg)

## Static Site Search Series

This is Part 3 of a series on building a static site search from scratch.

[Part 1 - Preprocessing Articles](https://superflux.dev/blog/static-site-search-preprocessing-articles) walked through building a search index for all my markdown articles using their Title, Tags, and Excerpt as search spaces. The index is a map of all valid search terms and their corresponding article matches.

[Part 2 - Search Component](https://superflux.dev/blog/static-site-search-search-component) walked through creating a React component to use this search index. I used the Ant Design Select component as a base, then performed custom filtering using the search term and the search index. This gave me a simple and fast search experience for my blog. The main downside was the search index ended up being ~700kb for only my 20 articles.

In this part, I'm going to cover some optimizations I made to get that search index from ~700kb to ~22kb (for 20 articles).

## The Issue to solve

With my original implementation of the search index, I was storing the full post title and slug for each matching search term. This was convenient when performing the lookup because the data was right there. The issue is the amount of redundancy in the index. Each post title string and slug string was stored 100s of times.

With 20 posts my search index was ~700kb. That's a big chunk of data to download. And it will grow exponentially with the number of posts since each post adds 100s of entries.

So I had two thoughts to increase the efficiency of the search index. Using unique integer post IDs, and removing unneeded/unlikely search terms.

## Post IDs

Previously my posts didn't need IDs, beyond the unique post slugs. But now there's a nice use case.

First I added unique IDs to each articles metadata. Just simple consecutive & unique numbers counting up from 1.

```text
---
title: Static Site Search Part 1 - Preprocessing Articles
date: April 3, 2024
author: Tyrel Delaney
tags: static-site-generators nextjs javascript typesense search
hero: /images/posts/static-site-search-preprocessing-articles.jpg
excerpt: In this series of articles I discuss how I created a client-side search for my static website. After looking at a few options out there for static sites and Next.js projects, I decided the challenge to build my own search from scratch would be more enjoyable. This first article discusses how I preprocess my articles into a search index.
id: 20
---
```

In my `PostData`/`PostMetadata` DTO I added the `id` field

```typescript
export interface PostMetadata {
	title: string;
	slug: string;
	date: string;
	excerpt: string;
	hero: string;
	tags: string[];
	id: number;                 // <--
	readTimeInMinutes?: number;
}

export interface PostData extends PostMetadata {
	content: string
}
```

Then in my search index preprocessor script, I alter what I'm saving for each search term. Instead of a `{ title: string, slug: string }[]` for each search term I just save the post `id` in an array, a `number[]`.

In my _preprocessors/search-index/search-index.ts_, my `addToIndex` function looks like this

```typescript
const addToIndex = (searchIndex: Map<string, number[]>, key: string, post: PostData) => {
  if (!searchIndex.has(key)) {
    searchIndex.set(key, new Array());
  }
  let found = false;
  for (let item of searchIndex.get(key)) {
    if (item === post.id) {
      found = true;
    }
  }
  if (!found) {
    searchIndex.get(key).push(post.id);
  }
}
```

This change cut the search index size by more than 1/10. From ~700kb down to 62kb.

## Remove ngrams that don't start from the beginning of a token

In my original algorithm I was slicing all possible ngrams of length `n` from each token. For example, when `n = 3` and the token is `typescript`, the ngrams would be `typ`, `ype`, `pes`, `esc`, `scr`, `cri`, `rip`, and `ipt`. This covers a lot of bases for possible partial search terms, but it's very unlikely folks are typing `ipt` and intending to find matches for `typescript`. Its much more likely and common folks are typing the word from the beginning and simply expecting matches on partial words.

The previous implementation looked like this
```typescript
export const createNGrams = (token: string): Set<string> => {
	const n = 10;
	const nGrams: Set<string> = new Set();
	for (let nx = 1; nx <= n; nx++) {
		for (let i = 0; i < token.length - nx + 1; i++) {
			const nGram = token.slice(i, i + nx);
			nGrams.add(nGram);
		}
	};
	if (token.length > n) {
		nGrams.add(token);
	}
	return nGrams;
}
```

I changed it so I'm only saving substrings starting from the beginning of a token

```typescript
export const createNGrams = (token: string): Set<string> => {
	const n = 10;
	const nGrams: Set<string> = new Set();
	for (let nx = 1; nx <= n; nx++) {
		nGrams.add(token.slice(0, nx));      // <--
	};
	if (token.length > n) {
		nGrams.add(token);
	}
	return nGrams;
}
```

This reduced the number of index entries from 4044 to 1449, and further reduced the size of the final JSON from ~62kb to ~22kb.

Great! Now that my search index is a reasonable size, lets refactor my website to use this new index format.

## Importing search index during build time rather than lazy loading during runtime

Now that the search index is so small, I'd like to just fetch it in my build process and bundle it with the server-side rendered pages. Rather than loading it lazily.

This isn't necessary, the small file size gives a big performance boost. But doing it this way feels more in-line with Next static site generation.

### What I need for my search component

Since the search index now just contains the post IDs, I also need a list of posts to reference via their IDs.

So to make this refactor work I need pass the list of `PostMetadata` and the search index to my search component

### getStaticProps

In each of my pages (I have 3: *index.tsx*, *blog/\[slug\].tsx*, and *blog/tags/\[tag\].tsx*) within their `getStaticProps` I need to fetch and return the `PostMetadata[]` and the search index.

The first gotcha I ran into was I was loading the search index from disk and then parsing it back into it's `Map<string, number[]>` type. The issue with this is Next doesn't like returning this type from `getStaticProps` because `Map` isn't serializable (why not? 🤷🏻‍♂️). So instead what I do is just serialize it myself and return it as a `string`.

I created this helper function to fetch the search index from disk:

*utils/searchIndexFileUtils.ts*
```typescript
import fs from "fs";

export const getSearchIndex = (): Map<string, number[]> => {
	const searchIndexFilePath = "public/search-index.json";
	const searchIndexFile = fs.readFileSync(searchIndexFilePath, "utf-8");
	const searchIndexMap = new Map<string, number[]>();
	const searchIndexJson = JSON.parse(searchIndexFile);
	for (const key of Object.keys(searchIndexJson)) {
		searchIndexMap.set(key, searchIndexJson[key]);
	}
	return searchIndexMap;
}
```

I then also created these functions to convert this to and from JSON. I put these in a separate file from `getSearchIndex` because I wanted to be able to use them client-side and the `import fs from "fs"` would cause issues with that.

*utils/searchIndexUtils.ts*
```typescript
export const searchIndexToJson = (map: Map<string, number[]>): string => {
	const obj = Array.from(map.entries());
	return JSON.stringify(obj);
}

export const jsonToSearchIndex = (jsonStr: string): Map<string, number[]> => {
	const entries = JSON.parse(jsonStr) as [string, number[]][];
	return new Map(entries);
}
```

Then in each page, I refactored `getStaticProps` to fetch the `PostMetadata[]` and the search index. Here's the example from my *index.tsx* page:

```typescript
export const getStaticProps: GetStaticProps = async (
	context
): Promise<GetStaticPropsResult<{posts: PostMetadata[], searchIndexJson: string }>> => {
	const searchIndex = getSearchIndex();
	const searchIndexJson = searchIndexToJson(searchIndex);
	const postMetadataList: PostMetadata[] = getPostMetaData();
	return {
		props: {
			posts: postMetadataList.sort(sortPostsByDate),
			searchIndexJson
		},
	};
};
```

Make sure to also modify the corresponding props for the page component to handle the new props type. In this case it's `{ posts: PostMetadata[], searchIndexJson: string }`.

## Refactoring Search Component

Next I needed to add these values as props to my search component. Originally it was importing the search index itself within a `useEffect`. Now instead I want to pass the search index along with the posts metadata list in through props.

I add these values as props to my `SearchInput` component

*components/elements/SearchInput.tsx*
```tsx
export interface Props {
	searchIndexJson: string;
	postMetadataList: PostMetadata[];
}

export default function SearchInput(props: Props) {
	// ...
}
```

In the component's `useEffect`, I removed the logic I had to lazy load the search index from disk, and replaced it with some simple logic to deserialize the search index from json.

```tsx
useEffect(() => {
	if (!searchIndex.size) {
		setSearchIndex(jsonToSearchIndex(props.searchIndexJson));
	}

	// ...
})

```

Then in my `onChange` function, where I'm doing the work to convert the search term to an options list to display as the search results, instead of directly pulling the post title and slug from the search index I pull the post ID from the search index and then find that post's metadata in `props.postMetadataList`. Reference [my previous article on the Search Component](https://superflux.dev/blog/static-site-search-search-component) to see more details, but the relevant change is:

```tsx
const onChange = (searchTerm: string) => {

	// ...

	if (searchIndex.size && searchIndex.has(searchTerm)) {
		const newOptions = new Array<OptionValue>();
		for (const id of searchIndex.get(searchTerm)) {
			const postMetadata: PostMetadata = getPostMetadataById(id, props.postMetadataList);
			newOptions.push({
				value: postMetadata.slug,
				label: postMetadata.title
			});
		}
		setOptions(newOptions);
	} else {
		setOptions([]);
	}
	
}
```

That `getPostMetadataById` function is just doing a simple find right now. Perhaps in the future I'll refactor that `PostMetadata[]` into something more efficient so I can do a quick lookup by ID.

```typescript
export const getPostMetadataById = (id: number, postMetadataList: PostMetadata[]): PostMetadata => {
	return postMetadataList.find((post) => post.id === id);
}
```

## Conclusion

Ok great! I've cut the size of the search index down from ~700kb to ~22kb. I'm pulling my PostMetadata list into more places than before, but it's much smaller than the search index so say overall the data load per page is ~25kb. Huge improvement.

The search is maybe slightly slower than before, because I'm not just doing a simple Map lookup on the search term anymore. But we're talking like ~1ms or less, so it's not even noticeable. 

In the future, as my number of posts grow, I think I'd like to refactor the `PostMetadata[]` to be a Map on the post IDs instead. Something like `Map<number, PostMetadata>`. That would allow for quick lookup of both the search term results from the search index, and a quick lookup of the corresponding PostMetadata for each search result.

I'll leave it here for now. Thanks for reading, and I hope you found some value in this static site search series. Remember to visit [Part 1](https://superflux.dev/blog/static-site-search-preprocessing-articles) & [2](https://superflux.dev/blog/static-site-search-search-component) to see the my original implementation this refactor is building on.





![](./static-site-search-search-component.jpg)

## Static Site Search Series

In [my previous article on this topic](https://superflux.dev/blog/static-site-search-preprocessing-articles), I talked about preprocessing my articles and creating a search index. This search index allows for quick referencing the articles that match a particular search term.

In this article I'm going to talk about how I used this search index to perform searches.

## Desired Behaviour

I wanted a couple things from my search. I wanted it to display results in real-time as you searched, and I didn't want to implement any kind of search results page. I wanted it to just be intuitive and quick.

## Search Component

I use [Ant Design](https://ant.design) a bit on this site. It's just a library I'm familiar with.

They offer a component that they call [Select with Search](https://ant.design/components/select#select-demo-search). It's essentially a multi-select dropdown that lets you filter the options by typing into a text input.

By default, you provide it a list of options, and then the search will filter those options. You can customize the callbacks for both when an item is clicked or you press return, and the search callback that occurs whenever the input changes.

This was close enough to what I wanted and I'll describe how I customized it.

I started by making a wrapper around the Ant Design Select component so I can easily customize it and then stick it wherever I need to.

It started off something like this

*components/elements/SearchInput.tsx*
```tsx
import React from "react";
import { Select } from 'antd';

export default function SearchInput(props) {
	return (
		<Select
		    showSearch
		    placeholder="Select a person"
		    optionFilterProp="children"
		    options={[
			  {
				value: 'jack',
				label: 'Jack',
			  },
			  {
				value: 'lucy',
				label: 'Lucy',
			  }
		    ]}
		/>
	)
}
```

To more easily type the `options`, I created a little `OptionsValue` interface at the top of the file:

```tsx
interface OptionValue {
	label: string;
	value: string;
}
```

## Loading the Search Index

I'm doing this a very sketchy way, I think. I'm really not a frontend developer so I kinda just made this up as I went.

My plan is eventually to slim down the index enough that I can just build it in with the rest of my static site, but for now since it's 600+ KB I tried to lazy load it to not mess with the initial page load.

Essentially I'm just loading the file in a `useEffect` and setting it into the component state.

*components/elements/SearchInput.tsx*
```tsx
export default function SearchInput(props) {
	const [searchIndex, setSearchIndex] = useState<Map<string, object>>(new Map());

	useEffect(() => {
		if (!searchIndex) {
			import("./search-index.json")
				.then((data) => {
					const indexMap = new Map<string, object>();
					for (const key of Object.keys(data)) {
						indexMap.set(key, data[key]);
					}
					setSearchIndex(indexMap);
				})
				.catch((error) => {
					console.error("Error loading search index", error);
				})
		}
	}, []);

	return (
		<Select
		    showSearch
		    placeholder="Select a person"
		    optionFilterProp="children"
		    options={[
			  {
				value: 'jack',
				label: 'Jack',
			  },
			  {
				value: 'lucy',
				label: 'Lucy',
			  }
		    ]}
		/>
	)
}
```

## Options

So I could have just provided a list of the article titles to this component and called it a day. I wouldn't have had to bother with all the preprocessing and indexing. But that would only have let me search the titles, not the tags and the excerpts.

With my search index I get a lot more nuance. For example if you search "docker" you get 1 result which is my post titled "What I'm Learning in 2022", which just wouldn't work with a simple title search.

First I initialize an empty options array in component state

```tsx
const [options, setOptions] = useState<OptionValue[]>([]);
```

Then I pass this to the `Select` component

```tsx
<Select
	showSearch
	placeholder="Search"
	suffixIcon={<SearchOutlined />} // Custom Ant icon to show in the right of the input field
	options={options}
/>
```

Great! So now I have my search index available, and I can now implement my custom search functionality using the index.

## onSearch

The Select component from Ant Design has an overridable `onSearch` function. I'm going to override this function and use the current search term to populate the `options` using the values from the `searchIndex`.

A couple things I do in this code:

- I check to make sure I'm not repeating the same searches. Not sure how necessary this is or not, I don't remember. This uses a `useRef` on the component to track the search iterations and a `useState` to track the previous search term.
- I check for empty string search terms and simply set an empty list in the `options`, no need to check the search index.
- I was doing some stuff with lodash `debounce` and `useMemo` to delay the search, but I'm not sure if this is necessary. I'll show it afterwards as optional.

After those checks, I imply lowercase the search term and then pull the value from the `searchIndex`.

If the search term doesn't exist as a key in the `searchIndex` I simply set an empty array in `options`.

If the search term does exist, then I pull the values from the `searchIndex` for that key, I convert the `{ title: string, slug: string }` into a `{ value: string, label: string }` where `value` = `slug` and `label` = `title`. I build a new options array of all the values from that search term and then set that new array into `options` component state.

When that `options` component state gets set, the component then re-renders showing the new options.

Here's the code for this `onSearch` handler

```tsx
const [searchIndex, setSearchIndex] = useState<Map<string, object>>(new Map());
const [options, setOptions] = useState<OptionValue[]>([]);
const fetchRef = useRef(0);
const [previousSearchTerm, setPreviousSearchTerm] = useState<string>("");

const onChange = (searchTerm: string) => {
	fetchRef.current += 1
	const currentFetch = fetchRef.current;
	if (currentFetch !== fetchRef.current) {
		return;
	}
	if (searchTerm.length < 1) {
		setOptions([]);
		return;
	}
	searchTerm = searchTerm.toLowerCase();
	if (searchTerm === previousSearchTerm) {
		return;
	}
	setPreviousSearchTerm(searchTerm);

	if (searchIndex.entries() && searchIndex.has(searchTerm)) {
		const newOptions = new Array<OptionValue>();
		for (const item of searchIndex.get(searchTerm) as { title: string, slug: string}[]) {
			newOptions.push({
				value: item.slug,
				label: item.title
			})
		}
		setOptions(newOptions);
	}
}
```

Then on the `Select` component I pass a few more props.

- I set `filterOption={false}`. This essentially disables the built-in filtering using the search term and the labels. I don't need it to filter the results because I'm filtering the `options` myself.
- I pass a reference to my `onSearch` function in the `onSearch` prop.
- I pass a simple sort function in the `filterSort` prop to organize the options alphabetically.
- I pass `notFoundContent={null}` to get rid of the default empty state when no search results are found. This just hides the results pane when empty. Could maybe do some nicer customization here.

The component looks something like this:

```tsx
<Select
	showSearch
	filterOption={false}
	onSearch={onSearch}
	placeholder="Search"
	suffixIcon={<SearchOutlined />}
	filterSort={(a: OptionValue, b: OptionValue) => (a?.label ?? "").toLowerCase().localeCompare((b?.label ?? "").toLowerCase())}
	notFoundContent={null}
	options={options}
/>
```

![](./static-site-search-component-screenshot.png)

## Navigate to the selected page

Lastly, we want to actually navigate to the corresponding page if a value in the search is clicked.

We can pass a `onSelect` prop that handles this. In this function I simply use the `value` from the selected option which is the `slug` of the post, and Next `useRouter` to navigate to that page.

The function is as simple as this:

```tsx
const router = useRouter();

const navigateToArticle = (value: string) => {
	router.push(`/blog/${value}`);
}
```

And thats it! I won't touch on styling or how I placed this component on my website. You can see that for yourself! Also the styling is going to be unique for your project if you're looking to replicate this.

The final code for my customized component looks like this

*components/elements/SearchInput.tsx*
```tsx
import React, { useEffect, useState, useRef } from "react";
import { Select } from 'antd';
import { SearchOutlined } from '@ant-design/icons';
import { useRouter } from 'next/navigation'

interface OptionValue {
	label: string;
	value: string;
}

export default function SearchInput() {
	const router = useRouter();
	
	const [searchIndex, setSearchIndex] = useState<Map<string, object>>(new Map());
	const [options, setOptions] = useState<OptionValue[]>([]);
	const fetchRef = useRef(0);
	const [previousSearchTerm, setPreviousSearchTerm] = useState<string>("");
	
	const onChange = (searchTerm: string) => {
		fetchRef.current += 1
		const currentFetch = fetchRef.current;
		if (currentFetch !== fetchRef.current) {
			return;
		}
		if (searchTerm.length < 1) {
			setOptions([]);
			return;
		}
		searchTerm = searchTerm.toLowerCase();
		if (searchTerm === previousSearchTerm) {
			return;
		}
		setPreviousSearchTerm(searchTerm);
	
		if (searchIndex.entries() && searchIndex.has(searchTerm)) {
			const newOptions = new Array<OptionValue>();
			for (const item of searchIndex.get(searchTerm) as { title: string, slug: string}[]) {
				newOptions.push({
					value: item.slug,
					label: item.title
				})
			}
			setOptions(newOptions);
		}
	}

	const navigateToArticle = (value: string) => {
		router.push(`/blog/${value}`);
	}

	useEffect(() => {
		if (!searchIndex) {
			import("./search-index.json")
				.then((data) => {
					const indexMap = new Map<string, object>();
					for (const key of Object.keys(data)) {
						indexMap.set(key, data[key]);
					}
					setSearchIndex(indexMap);
				})
				.catch((error) => {
					console.error("Error loading search index", error);
				})
		}
	}, []);

	return (
		<Select
			showSearch
			filterOption={false}
			onSearch={onSearch}
			onSelect={navigateToArticle}
			placeholder="Search"
			suffixIcon={<SearchOutlined />}
			filterSort={(a: OptionValue, b: OptionValue) => (a?.label ?? "").toLowerCase().localeCompare((b?.label ?? "").toLowerCase())}
			notFoundContent={null}
			options={options}
		/>
	)
}
```

## Add delay to the search

If your running into some issues with your search firing multiple times, or you'd just like to add a delay for aesthetics, you can use `useMemo` and lodash `debounce`. Simply wrap your `onChange` function in the following

```tsx
const onChange = useMemo(() => {
	const getItems = (searchTerm: string) => {
		// your logic for filtering the options
	}
	return debounce(getItems, 50); // delay in ms
}, [searchIndex])
```

## Resources

- [Ant Design Select component](https://ant.design/components/select)

## Repository

This website in it's entirety is viewable on [GitHub](https://github.com/tyrelh/personal-site-nextjs).

The search logic doesn't have it's own repo yet. But I'll update that here if I ever split it out.

## Conclusion

I think this turned out really well. It works great and really only took a couple of days to figure out end-to-end. It was a pretty interesting and fun project too.

I was going to call this "Next.js Static Site Search", but in the end with this solution it really has nothing to do with Next. You should be able to incorporate this into any statically generated site, particularly those using Markdown and React, although you should be able to adapt it to other stacks as well.

This is working quite well for me in it's current state, but there are lots of optimizations to be made. If I get around to those I'll post a third part to this series explaining the optimizations I implemented.

Thanks for reading!


This article was written around July 2023.

If you'd like to learn how to deploy a static site using Next.js to GitHub Pages, [check out a previous article of mine on the topic](https://superflux.dev/blog/github-pages-using-nextjs).

This functionality for my website was heavily inspired by [Bionic Julia's post on the same topic](https://dev.to/bionicjulia/creating-dynamic-tag-pages-with-nextjs-nested-routes-nah).

![black next js logo on a plain white background](next-js-logo.png)

## Tl;dr
- [Goal](#goal): To make tag pages that filter my articles by a tag
- [Get tags from posts](#get-tags-from-posts): First step is to collate all the unique tags from my markdown article files.
- [Build the pages for tags](#build-the-pages-for-tags)
	- [getStaticPaths](#getstaticpaths): Get the unique list of tags to build the routes for.
	- [getStaticProps](#getstaticprops): For each tag, get the filtered list of posts.
	- [Tags page template](#tags-page-template): Render the filtered list of posts on a new page.
- [Link to tag pages](#link-to-tag-pages): I altered my hashtag component to link to the appropriate tag route.
- [Small gotcha](#small-gotcha): Need to strip some character from tags since they need to be URL safe now.
- [Conclusion](#conclusion)
- [Resources](#resources)

## Goal

My goal with this project was to make the tags I'm already displaying alongside articles to be clickable links that filter my posts by that tag.

My markdown posts already contain metadata that includes tags, so luckily I don't need to go back and tag all my posts.

## Get tags from posts

The first step is to get a list of the tags from our posts. This is so we can build the pages for each.

I built a function to read the tags from all my post markdown files. The tags are contained in their frontmatter. I use a `Set` to only get the unique tags with no duplicates.

```tsx
import { PostData } from "../dtos/PostData";

export const getTagsFromPostDataList = (postData: PostData[]): string[] => {
  const tags = new Set<string>(); // Set forces only unique tags
  postData.forEach(function (postDatum) {
    postDatum.tags.forEach(function (tag) {
      tags.add(tag);
    });
  }); 
  return Array.from(tags);
}
```

And I get the posts using an existing method that reads the posts from the files system, and parses the markdown and frontmatter from them into a list of custom `PostData` objects.

```tsx
import matter from "gray-matter";
import fs from "fs";
import path from "path";
import { PostData } from "../dtos/PostData";
import { calculateReadTimeOfText } from "./textUtils";

export const getPostData = (): PostData[] => {
  const files = fs.readdirSync(path.join("posts"));
  const postDataList: PostData[] = files.map((filename) => {
    const slug = filename.replace(".md", "");
    const markdownWithMeta = fs.readFileSync(
      path.join("posts", filename),
      "utf-8"
    );
    const { data: frontmatter, content } = matter(markdownWithMeta);
    const readTimeInMinutes = calculateReadTimeOfText(content);
    const postData: PostData = {
      title: frontmatter?.title,
      slug: slug,
      date: frontmatter?.date,
      hero: frontmatter?.hero,
      excerpt: frontmatter?.excerpt,
      tags: frontmatter?.tags ? frontmatter.tags.split(" ") : null,
      readTimeInMinutes: readTimeInMinutes,
      content: content
    };
    return postData;
  });
  return postDataList
}
```

Ok, so now I have a list of all the unique tags. Now what?

## Build the pages for tags

I wanted to have the tag pages follow the */blog/tags/\<tag\>* URL scheme. So I created a *tags* folder nested in */pages/blog* and created a new file named *\[tag\].tsx*. The square brackets mean this is a template file for the tag page. There will be a page generated for each unique tag.

[Next.js docs on Dynamic Routes](https://nextjs.org/docs/pages/building-your-application/routing/dynamic-routes)

### getStaticPaths

Within the tag template we need two methods. `getStaticPaths` will gather a list of all the paths that need to be generated. In this case it's a list of all the unique tags since they'll be used in the url path for each tag page in the format */blog/tags/\<tag\>*.

```tsx
export const getStaticPaths: GetStaticPaths = async (context): Promise<GetStaticPathsResult> => {
  const postData: PostData[] = getPostData();
  const tags = getTagsFromPostDataList(postData);
  const paths = tags.map((tag: string) => ({
    params: { tag }
  }));
  return {
    paths: paths,
    fallback: false
  };
};
```

I use the functions I defined above to pull the unique tags from my markdown articles and generate the paths.

[Next.js docs on *getStaticPaths*](https://nextjs.org/docs/pages/building-your-application/data-fetching/get-static-paths)

### getStaticProps

Next is `getStaticProps`. This function gathers the props that are passed to each tag page to render. In this case we want to pass a list of the articles that have a given tag which comes in through `params`.

I use the same function again to get all the post data, then I filter the posts to only the sunset that have the given tag.

```tsx
export async function getStaticProps({ params }: Params) {
  const postDataList: PostData[] = getPostData();
  const postsForTag: PostData[] = postDataList.filter((post: PostData) => 
    post.tags.includes(params.tag)
  );
  return {
    props: {
      tag: params.tag,
      postsForTag: postsForTag.sort(sortPostsByDate)
    },
  }
}
```

When returning the subset of posts I also make sure they're sorted by date so that I can render them in decending order on the tag page.

[Next.js docs on *getStaticProps*](https://nextjs.org/docs/pages/building-your-application/data-fetching/get-static-props)

### Tags page template

The last thing to do is build the template for the tag page itself. This was extremely simple as I was able to reuse components I already had.

I have a `ArticlePreviewList` component from the homepage that generates a list of previews for a given list of post data. Other than that there are some headings and misc.

```tsx
export default function TagPage({tag, postsForTag}) {
  return (
    <>
      <HeadW title="superflux" />
      <StickyHeader title={tag} path={["blog","tags"]}/>
      <SectionHeading>
        #{tag}
      </SectionHeading>
      <ArticlePreviewList articleMetadataList={postsForTag} />
      <SectionHeading>
        Get in touch
      </SectionHeading>
      <p>
        Feel free to contact me via <Anchor href="https://twitter.com/tyreldelaney">Twitter</Anchor>!
      </p>
    </>
  );
}
```

And that's it! With those few pieces Next.js will handle the rest for us. During the build step it will generate all the pages on all the routes for each tag.

![](tag-page-screenshot.png)

## Link to tag pages

One last piece is to make it easy to navigate to these new pages. I already had a visual tag component I was using within the article previews, just for informational purposes.

I created a new Hashtag component that wraps Ant Design's Tag component. So now I can apply a Next `Link` within that. I also have a custom `Anchor` component that obfuscates the Next `Link` away. Since it's a hashtag component I can hard code the route with just the tag prop itself being the final portion of the route.

```tsx
import { Tag } from "antd"
import Anchor from "./Anchor"

export interface Props {
  tag: string
}

export default function Hashtag(props: Props) {
  return (
    <Anchor key={props.tag} href={`/blog/tags/${props.tag}`}>
      <Tag key={props.tag}>#{props.tag}</Tag>
    </Anchor>
  )
}
```

## Small gotcha

One small issue I ran into is one tag I was using contained a `/` character, `ci/cd`. This wasn't a very forward thinking decision on my part.

I just needed to search through my articles and remove the `/` character from that tag, making it just `cicd`. Much simpler than trying to code for it.

## Conclusion

I've had tags on my posts for some time now, but this functionality makes them much more useful. Clicking a tag now brings you to a page that contains articles curated by that tag.

It's basic functionality, but it's still rewarding to build it yourself.

## Resources

- [Creating Dynamic Tag Pages With NextJS Nested Routes](https://bionicjulia.com/blog/creating-dynamic-tag-page-nextjs-nested-routes) 
- [Next.js docs on Dynamic Routes](https://nextjs.org/docs/pages/building-your-application/routing/dynamic-routes)
- [Next.js docs on *getStaticPaths*](https://nextjs.org/docs/pages/building-your-application/data-fetching/get-static-paths)
- [Next.js docs on *getStaticProps*](https://nextjs.org/docs/pages/building-your-application/data-fetching/get-static-props)


Let's say you've built an nice new static site using Next.js and you want to host it on GitHub Pages. Well there's a few steps to get it working, but it isn't too hard.

![next js logo and github logo](./NextJSGitHubPagesHero.png)

This article assumes you already have a website built locally and you're just looking to host it on [GitHub Pages](https://docs.github.com/en/pages/getting-started-with-github-pages). In the future I may write some how-tos for creating a blog using markdown and Next.js.

In this example I'm going to outline how I deploy to my main GitHub Pages repo for [superflux.dev](https://superflux.dev). It's hosted in my GitHub account's [main Pages repository](https://github.com/tyrelh/tyrelh.github.io). Everyone has access to this special repo. Just create a new repo with the title _your-github-username.github.io_. GitHub will serve the static content there on your own subdomain `https://your-github-username.github.io`. You can even connect a custom domain like I did, I'll talk about that optionally at the end.

You can also use this tutorial to push repo-specific pages. Google for that if your unfamiliar.

## Tl;dr

* [Static content](#static-content): A quick overview of how to build static assets for a Next.js site.
* [Install gh-pages package](#install-gh-pages-package): There's a lovely NPM package that makes it very simple to deploy static assets to GitHub Pages.
* [Branch setup](#branch-setup): I use two repositories on GitHub, one for my source code and one that is the Pages repository that gets hosted under your username.
* [Deploy script](#deploy-script): The deploy script uses the *gh-pages* package.
	* [Static content directory](#static-content-directory): Set the *out* directory in the deploy script that Next.js outputs static assets to when building and exporting.
	* [Remote & branch](#remote--branch): Set the remote and the branch that you want *gh-pages* to push your static assets to.
	* [Force push to remote](#force-push-to-remote): Allow the deploy script to force-push changes to your GitHub Pages repo.
	* [\_next hidden folder](#_next-hidden-folder): Some tweaks to make GitHub Pages play nicely with Next.js
* [\[Optional\] Configure a custom domain](#optional-configure-a-custom-domain): Allow a GitHub Pages site to be served over a custom domain name.
* [Resources](#resources)
* [Conclusion](#conclusion)

## Static content

GitHub Pages can only serve static content. You need to be able to export your Next.js project to static code. You can do this by running

```shell
next build && next export
```

`next export` will export the static code for your project to the _out/_ directory. This is the code we're going to want to host on GitHub Pages.

To simplify and make it easy to remember I added this as a `build` script in my *package.json*. Add/edit the `build` script:

```json
"build": "next build && next export"
```

## Install gh-pages package

I've used this package in the past and it works well for GitHub Pages websites. *[gh-pages](https://github.com/tschaub/gh-pages)* is a simple little tool that makes it easy to push projects to GitHub Pages repos or branches.

```shell
npm install gh-pages --save-dev
```

## Branch setup

I store my website source in a separate repository, in my case a repo called [personal-site-nextjs](https://github.com/tyrelh/personal-site-nextjs). By default *gh-pages* will deploy your static site to the `origin` remote. But we can change that. I like to keep the `origin` for my source repo pointing to the source remote.

* `personal-site-nextjs`: repo containing the source code for my website. I do development here
* `tyrelh.github.io`: my main GitHub Pages repo. This is were the compiled static assets are served from

You can also just use separate branches in the same GitHub Pages repository if you prefer. One for the source and one for the public assets. That feels messy to me so I prefer to keep them separate.

## Deploy script

Next we'll setup a deploy script in our _package.json_ to deploy our static content!

In your _package.json_ create a new script. Remember it should be on the same level as the other scripts, in the `"scripts"` block:

```json
"deploy": "gh-pages"
```

Now this by itself wont work yet, but I'm going to step you through each additional step to get the `gh-pages` command working. If you want to [skip to the end and just see the final script](#conclusion), go ahead!

### Static content directory

First you want to tell it what directory that your static assets are in. Remember this is the directory that `next export` in your `build` script outputs to. By default this is the _out/_ directory in your project. Make sure you run your build script first so that your static files are here.

Use the `-d` flag to set the directory.

```json
"deploy": "gh-pages -d out
```

### Remote & branch

We want to make sure `gh-pages` knows where it should push our static code to for hosting. It's good to set this up first so you don't accidentally push code somewhere you don't intend to.

For the `gh-pages` command, you can use the `-b` flag to tell it the remote branch you want to push to. Whatever the default branch of your _username.github.io_ repo is will work. `main` or `master` for example.

```json
"deploy": "gh-pages -d out -b main"
```

Next you'll want to set the remote. I like to hard code this in the script itself. Since it's going to be static for this project I don't see any reason not to. You can use the `-r` flag to set a remote URL. Copy the remote URL from your _username.github.io_ repo.

```json
"deploy": "gh-pages -d out -b main -r https://github.com/tyrelh/tyrelh.github.io.git"
```

### Force Push to Remote

When deploying new static content to your GitHub Pages repo, generally you're going to want to overwrite what is there. At least this is the simplest way to do it. Before setting this just double check that your branch and remote are set correctly. You don't want to overwrite anything you don't intend to!

Use the `-f` flag to force the push to the remote, overwriting files and changes that may exist on the remote.

```json
"deploy": "gh-pages -d out -b main -r https://github.com/tyrelh/tyrelh.github.io.git -f"
```

Ok! We're almost there! If this was a Jekyll project, or maybe a standard React project this would probably be all you need. But since we're using Next.js there is one more piece of config we need to do.

### \_next Hidden Folder

In your _out/_ directory of static assets, you should see a folder called _\_next/_. This is where Next.js puts your compiled JavaScript and CSS.

By default, GitHub pages actually ignores folders beginning with a \_. This is because GitHub Pages is setup to use Jekyll. Jekyll considers folders beginning with \_ to be hidden folders, so GitHub Pages just doesn't serve them.

You can deploy the code as is, and it will look like it worked since you can see the _\_next/_ directory in you _username.github.io_ repo. But any requests to those assets when you visit your website will result in a 404 not found error.

To fix this you need to add a file named _.nojekyll_ to the root of your _username.github.io_ repo. You can do this right on the GitHub website to test it out. Once that file is there and you give it a couple minutes to take effect, your website should work without 404 errors!

But now if you deploy your static code again that file will get erased, breaking your website. We need to incorporate this file into our deploy or build scripts. I put it at the beginning of my deploy script, but you could also put it at the end of your build script too if that makes more sense to you.

We want to create that `.nojekyll` file in our _out/_ directory after our code is built, but before we deploy it using `gh-pages`.

At the beginning of your deploy script, add an `echo` command to create the file. The file can contain anything, I just echo the string `true`. Use the `&&` between the `echo` and `gh-pages` commands so that they will run sequentially.

```json
"deploy": "echo 'true' > ./out/.nojekyll && gh-pages -d out -b main -r https://github.com/tyrelh/tyrelh.github.io.git -f"
```

This wont quite work, because by default `gh-pages` ignores hidden dotfiles! To tell `gh-pages` you want to include dotfiles (that's files beginning with a .), use the `-t` flag on the `gh-pages` command.

```json
"deploy": "echo 'true' > ./out/.nojekyll && gh-pages -d out -b main -r https://github.com/tyrelh/tyrelh.github.io.git -f -t"
```

And that's basically it!

## Configure your repo for GitHub Pages

You'll need to make sure your repo is configured for Pages. Go to your repo, click on the repo _Settings_, and click _Pages_ on the settings side bar. See the [GitHub Pages docs](https://docs.github.com/en/pages/getting-started-with-github-pages) for more general info about hosting static content.

## \[Optional\] Configure a custom domain

After you follow the [instructions here](https://docs.github.com/en/pages/configuring-a-custom-domain-for-your-github-pages-site/managing-a-custom-domain-for-your-github-pages-site) to setup your custom domain with your GitHub Pages website, you'll notice that you need a _CNAME_ file in the root of your repo for the domain to map correctly.

Since our deploy process overwrites all the files in our GitHub Pages repo on each deploy, we need to add this file to our deploy/build process so it will be included each time. This is similar to what we needed to do for the _.nojekyll_ file.

At the beginning of your deploy script, add another `echo` command. You want to output the root domain for your website to a file called _CNAME_. In my case my root domain is `superflux.dev`.

```json
"deploy": "echo 'superflux.dev' > ./out/CNAME && echo 'true' > ./out/.nojekyll && gh-pages -d out -b main -r https://github.com/tyrelh/tyrelh.github.io.git -f -t"
```

Make sure to include the `&&` between each command so they run sequentially!

## Resources

* Allow the _\_next_ folder to get picked up in repo: https://github.blog/2009-12-29-bypassing-jekyll-on-github-pages/
* `gh-pages` package to easily deploy to GitHub Pages repositories: https://github.com/tschaub/gh-pages
* GitHub Pages docs: https://docs.github.com/en/pages/getting-started-with-github-pages
* GitHub Pages custom domain docs: https://docs.github.com/en/pages/configuring-a-custom-domain-for-your-github-pages-site/managing-a-custom-domain-for-your-github-pages-site

## Conclusion

And now that's it! To summarize we built a deploy script in our _package.json_ that looks like this

```json
"deploy": "echo 'true' > ./out/.nojekyll && gh-pages -d out -b main -r https://github.com/tyrelh/tyrelh.github.io.git -f -t"
```

You'll want to make sure your `-b` branch is pointing to the branch being served on your GitHub Pages repo, and that the `-r` remote is the URL of your GitHub Pages repo.

After building your website locally just run `npm run deploy` to push your static code to your GitHub Pages site! Remember the `gh-pages` package uses _git_ to push code to the remotes, so you'll need to configure your username, email, and credentials to authenticate the push to your GitHub Pages repo.

Reach out to me on Twitter [@tyreldelaney](https://twitter.com/tyreldelaney) if you have any questions, comments, or suggestions!

*Note: this article was written in April 2022. It sat for a while in my queue of articles as we were gearing up to have our first child. Hopefully nothing has changed drastically with Next.js and GitHub Pages that renders this info unusable.*


Here are some things that I'd like to learn more about in 2022. This is kindof a reference for myself. So I can look back at this next year and reflect on what I learned and what I wanted to learn. [Let me know on twitter](https://twitter.com/tyreldelaney) if you are learning these same things or you have any tips or resources to share!

![Logo montage of all the technologies outlined in this article](what-im-learning-in-2022.jpg)

## Index

[Learning from scratch](#learning-from-scratch)

* [Docker](#docker)
* [Vim/NeoVim](#vimneovim)

[Learning more of](#learning-more-of)

* [Github Actions](#github-actions)
* [AWS DynamoDB](#aws-dynamodb)
* [AWS Client VPN, VPNs in general, and SSO](#aws-client-vpn-vpns-in-general-and-sso)
* [SSH](#ssh)
* [HTTPS/SSL](#httpsssl)
* [Raspberry Pi](#raspberry-pi)
* [Home Network](#home-network)

[Things I'm interested in but might not get to this year](#things-im-interested-in-but-might-not-get-to-this-year)

* [A better static site generator / CMS](#a-better-static-site-generator--cms)
* [Webhooks](#webhooks)

## Learning from scratch

### Docker
Docker really feels like one of those technologies that I've been conveniently ignoring for a little too long. I think what I'll do to start is simply try to containerize some smaller projects I'm working on this year to dip my toes in.

I acquired a handful of second-hand Raspberry Pis not long ago and I've got a few ideas for some small home automation & monitoring projects that might be perfect simple projects to dockerize.

A stretch goal would be to work up to containerizing our monolithic app at my day job.

### Vim/NeoVim
I've always just barely snuck by with my fluency in the terminal. The more I get into networking and Dev Ops the more this deficiency is showing. In general I'd just like to work in the terminal a bit more this year.

Specifically I'm going to try to develop a couple Node and Python projects I'm cooking up remotely on the Raspberry Pis I hope to run them on.

Really two main goals I'd like to accomplish with this practice:
1. Get proficient and comfortable using vim.
2. Learn more about making a portable and lightweight dev environment. Something that I can easily spin up on a Raspberry Pi or AWS EC2 instance over SSH and feel comfortable doing real work in.

## Learning more of

### Github Actions
Automation is a big focus for me this year. And really just because I rely on Github both personally and professionally, Github Actions is a natural choice to skill up in.

We're now looking into automating and outsourcing our test suite at work and since we use Github for source control and code reviews/pull requests it makes total sense to use Github Actions.

I've automated a few things using Github Actions already, namely [tests and deployment of my](https://github.com/tyrelh/battlesnake-typescript-node) [Battlesnake](https://play.battlesnake.com/) to AWS Elastic Beanstalk. Automating even just tests for our app at work will be a bit more of a challenge since it's a much older stack and has many more moving parts.

### AWS DynamoDB
I wasn't actually planning on learning DynamoDB at the start of the year. But a project idea came to mind and the possibility of using Dynamo to store data rather than something like S3 sounded intriguing.

Essentially what I'm planning is building out a few Raspberry Pi temperature sensors for my house. Maybe 4 or 5 in total monitoring a few rooms in the house as well as the outdoor temperature.

The idea would be to collect data from all these sensors using Raspberry Pis maybe every 5 mins or so, and ship that data off to Dynamo. Then building an infrastructure to rotate through tables. Basically make a new table each day with a high write provision, and rotate down the previous days tables read and write provisions. Maybe after some period archive the data in S3.

Might be over engineering for the task, but it's for learning.

[Best Practices for Handling Time Series Data in DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-time-series.html)

### AWS Client VPN, VPNs in general, and SSO
With my day job going fully remote this year, setting up secure remote tools has been high on our priority list. Part of that will be learning about AWS Client VPN which we may use to grant our devs access to some of our AWS infrastructure.

Part of this work is also moving to more zero trust architecture, and leaning into SSO for authentication.

Also I recently acquired a Synology NAS for my home (which we actually use to use in our office before we gave that space up). I'm looking to centralize alot of my personal cloud use to the Synology and move away from services like Google Drive (office and documents), iCloud (photos), and Notion (notes). Part of that will be to setup a VPN at home so that I can use those services hosted in my home from anywhere. As well as accessing some of my Raspberry Pis for remote development.

### SSH
Learning more about SSH and tunneling goes hand-in-hand with a few other learning priorities I have. Learning best practices for connecting securely and crafting secure infrastructure. Also using SSH to practice remote development and learning terminal based tools.

### HTTPS/SSL
Same as above. Learning more about best practices around secure remote communication. Getting more confident understanding certs and how they work.

### Raspberry Pi
I've owned a Raspberry Pi 3 for a handful of years now, but I've never really had a project for it. I experimented a bit with retro game emulation on it, but those projects are pretty trivial to setup.

Now I have a handful of Raspberry Pi 2s and 3s and I have some plans for some smaller single purpose projects for the house that will be perfect for them.

### Home network
This kinda just happened organically, but I have a lot of computers in my house now! What I'd like to do is centralize a bunch of things into a proper rack. I have a few computers that are mining cryptocurrency that I'd like to transfer to rack-mountable server cases. Along with my new Synology NAS and a few Raspberry Pis.

Wiring my house with Cat-6 probably won't happen any time soon, but centralizing a bunch of things in a rack will let me hard-wire them all to a small switch without needing to run cables through all my walls.

Like I mentioned above, also setting up network storage and remote access for my home network.

## Things I'm interested in but might not get to

### A better static site generator / CMS
My website currently is hand-built in Javascript and React with a few plugins that let me render my articles from markdown. There is still a fair amount of manual process to get each markdown file wired up to display as a page and route, but at least I don't need to reformat the article. I can just paste it in from my notes app and then wire it up.

Ideally I'd like to learn a better static site generator to simplify this process even more. I've been thinking of learning [Gatsby](https://www.gatsbyjs.com/) for a while now, but I've just found it a bit cumbersome and hard to get into.

Some alternatives I'm considering learning instead of Gatsby are [Jeckyll](https://jekyllrb.com/) or [Hexo](https://hexo.io/).

### Webhooks
Just something I'm interested in learning more about.

## End

[@ me on twitter](https://twitter.com/tyreldelaney) if you are learning these same things or you have any tips or resources to share!

#static-site-generators

Static Site Search Part 3 - Optimizing the Index

April 18, 2024 8 Minute Read

Static Site Search Part 2 - Search Component

April 10, 2024 7 Minute Read

Static Site Search Part 1 - Preprocessing Articles

April 3, 2024 15 Minute Read

Next.js Tag Pages

March 25, 2024 6 Minute Read

Deploy a Next.js Static Site to GitHub Pages

July 23, 2023 8 Minute Read

What I'm Learning in 2022

March 23, 2022 5 Minute Read

Get in touch