AWS Dynamic DNS

Battlesnake Programming Competition 2018

Book Notes: DevOps Handbook, The

Dark Mode React Toggle

Deno Tests & GitHub Actions CI

Deploy a Next.js Static Site to GitHub Pages

Migrating Frontend Assets During Grails 2 → 3 Upgrade

Lightshift Game

Building a Mechanical Keyboard from Scratch

My Introduction to Game Development

Next.js Tag Pages

Node CI/CD Pipeline using GitHub Actions & AWS Elastic Beanstalk

Resources I Found Valuable in 2022

Small Form Factor PC Build Part 1

Small Form Factor PC Part 2 - Water Cooling

Static Site Search Part 3 - Optimizing the Index

Static Site Search Part 1 - Preprocessing Articles

Static Site Search Part 2 - Search Component

How I use my Synology NAS as my personal cloud

What is a Synology NAS?

What I'm Learning in 2022

What is DevOps?


![](./static-site-search-optimizing-the-index.jpg)

## Static Site Search Series

This is Part 3 of a series on building a static site search from scratch.

[Part 1 - Preprocessing Articles](https://superflux.dev/blog/static-site-search-preprocessing-articles) walked through building a search index for all my markdown articles using their Title, Tags, and Excerpt as search spaces. The index is a map of all valid search terms and their corresponding article matches.

[Part 2 - Search Component](https://superflux.dev/blog/static-site-search-search-component) walked through creating a React component to use this search index. I used the Ant Design Select component as a base, then performed custom filtering using the search term and the search index. This gave me a simple and fast search experience for my blog. The main downside was the search index ended up being ~700kb for only my 20 articles.

In this part, I'm going to cover some optimizations I made to get that search index from ~700kb to ~22kb (for 20 articles).

## The Issue to solve

With my original implementation of the search index, I was storing the full post title and slug for each matching search term. This was convenient when performing the lookup because the data was right there. The issue is the amount of redundancy in the index. Each post title string and slug string was stored 100s of times.

With 20 posts my search index was ~700kb. That's a big chunk of data to download. And it will grow exponentially with the number of posts since each post adds 100s of entries.

So I had two thoughts to increase the efficiency of the search index. Using unique integer post IDs, and removing unneeded/unlikely search terms.

## Post IDs

Previously my posts didn't need IDs, beyond the unique post slugs. But now there's a nice use case.

First I added unique IDs to each articles metadata. Just simple consecutive & unique numbers counting up from 1.

```text
---
title: Static Site Search Part 1 - Preprocessing Articles
date: April 3, 2024
author: Tyrel Delaney
tags: static-site-generators nextjs javascript typesense search
hero: /images/posts/static-site-search-preprocessing-articles.jpg
excerpt: In this series of articles I discuss how I created a client-side search for my static website. After looking at a few options out there for static sites and Next.js projects, I decided the challenge to build my own search from scratch would be more enjoyable. This first article discusses how I preprocess my articles into a search index.
id: 20
---
```

In my `PostData`/`PostMetadata` DTO I added the `id` field

```typescript
export interface PostMetadata {
	title: string;
	slug: string;
	date: string;
	excerpt: string;
	hero: string;
	tags: string[];
	id: number;                 // <--
	readTimeInMinutes?: number;
}

export interface PostData extends PostMetadata {
	content: string
}
```

Then in my search index preprocessor script, I alter what I'm saving for each search term. Instead of a `{ title: string, slug: string }[]` for each search term I just save the post `id` in an array, a `number[]`.

In my _preprocessors/search-index/search-index.ts_, my `addToIndex` function looks like this

```typescript
const addToIndex = (searchIndex: Map<string, number[]>, key: string, post: PostData) => {
  if (!searchIndex.has(key)) {
    searchIndex.set(key, new Array());
  }
  let found = false;
  for (let item of searchIndex.get(key)) {
    if (item === post.id) {
      found = true;
    }
  }
  if (!found) {
    searchIndex.get(key).push(post.id);
  }
}
```

This change cut the search index size by more than 1/10. From ~700kb down to 62kb.

## Remove ngrams that don't start from the beginning of a token

In my original algorithm I was slicing all possible ngrams of length `n` from each token. For example, when `n = 3` and the token is `typescript`, the ngrams would be `typ`, `ype`, `pes`, `esc`, `scr`, `cri`, `rip`, and `ipt`. This covers a lot of bases for possible partial search terms, but it's very unlikely folks are typing `ipt` and intending to find matches for `typescript`. Its much more likely and common folks are typing the word from the beginning and simply expecting matches on partial words.

The previous implementation looked like this
```typescript
export const createNGrams = (token: string): Set<string> => {
	const n = 10;
	const nGrams: Set<string> = new Set();
	for (let nx = 1; nx <= n; nx++) {
		for (let i = 0; i < token.length - nx + 1; i++) {
			const nGram = token.slice(i, i + nx);
			nGrams.add(nGram);
		}
	};
	if (token.length > n) {
		nGrams.add(token);
	}
	return nGrams;
}
```

I changed it so I'm only saving substrings starting from the beginning of a token

```typescript
export const createNGrams = (token: string): Set<string> => {
	const n = 10;
	const nGrams: Set<string> = new Set();
	for (let nx = 1; nx <= n; nx++) {
		nGrams.add(token.slice(0, nx));      // <--
	};
	if (token.length > n) {
		nGrams.add(token);
	}
	return nGrams;
}
```

This reduced the number of index entries from 4044 to 1449, and further reduced the size of the final JSON from ~62kb to ~22kb.

Great! Now that my search index is a reasonable size, lets refactor my website to use this new index format.

## Importing search index during build time rather than lazy loading during runtime

Now that the search index is so small, I'd like to just fetch it in my build process and bundle it with the server-side rendered pages. Rather than loading it lazily.

This isn't necessary, the small file size gives a big performance boost. But doing it this way feels more in-line with Next static site generation.

### What I need for my search component

Since the search index now just contains the post IDs, I also need a list of posts to reference via their IDs.

So to make this refactor work I need pass the list of `PostMetadata` and the search index to my search component

### getStaticProps

In each of my pages (I have 3: *index.tsx*, *blog/\[slug\].tsx*, and *blog/tags/\[tag\].tsx*) within their `getStaticProps` I need to fetch and return the `PostMetadata[]` and the search index.

The first gotcha I ran into was I was loading the search index from disk and then parsing it back into it's `Map<string, number[]>` type. The issue with this is Next doesn't like returning this type from `getStaticProps` because `Map` isn't serializable (why not? 🤷🏻‍♂️). So instead what I do is just serialize it myself and return it as a `string`.

I created this helper function to fetch the search index from disk:

*utils/searchIndexFileUtils.ts*
```typescript
import fs from "fs";

export const getSearchIndex = (): Map<string, number[]> => {
	const searchIndexFilePath = "public/search-index.json";
	const searchIndexFile = fs.readFileSync(searchIndexFilePath, "utf-8");
	const searchIndexMap = new Map<string, number[]>();
	const searchIndexJson = JSON.parse(searchIndexFile);
	for (const key of Object.keys(searchIndexJson)) {
		searchIndexMap.set(key, searchIndexJson[key]);
	}
	return searchIndexMap;
}
```

I then also created these functions to convert this to and from JSON. I put these in a separate file from `getSearchIndex` because I wanted to be able to use them client-side and the `import fs from "fs"` would cause issues with that.

*utils/searchIndexUtils.ts*
```typescript
export const searchIndexToJson = (map: Map<string, number[]>): string => {
	const obj = Array.from(map.entries());
	return JSON.stringify(obj);
}

export const jsonToSearchIndex = (jsonStr: string): Map<string, number[]> => {
	const entries = JSON.parse(jsonStr) as [string, number[]][];
	return new Map(entries);
}
```

Then in each page, I refactored `getStaticProps` to fetch the `PostMetadata[]` and the search index. Here's the example from my *index.tsx* page:

```typescript
export const getStaticProps: GetStaticProps = async (
	context
): Promise<GetStaticPropsResult<{posts: PostMetadata[], searchIndexJson: string }>> => {
	const searchIndex = getSearchIndex();
	const searchIndexJson = searchIndexToJson(searchIndex);
	const postMetadataList: PostMetadata[] = getPostMetaData();
	return {
		props: {
			posts: postMetadataList.sort(sortPostsByDate),
			searchIndexJson
		},
	};
};
```

Make sure to also modify the corresponding props for the page component to handle the new props type. In this case it's `{ posts: PostMetadata[], searchIndexJson: string }`.

## Refactoring Search Component

Next I needed to add these values as props to my search component. Originally it was importing the search index itself within a `useEffect`. Now instead I want to pass the search index along with the posts metadata list in through props.

I add these values as props to my `SearchInput` component

*components/elements/SearchInput.tsx*
```tsx
export interface Props {
	searchIndexJson: string;
	postMetadataList: PostMetadata[];
}

export default function SearchInput(props: Props) {
	// ...
}
```

In the component's `useEffect`, I removed the logic I had to lazy load the search index from disk, and replaced it with some simple logic to deserialize the search index from json.

```tsx
useEffect(() => {
	if (!searchIndex.size) {
		setSearchIndex(jsonToSearchIndex(props.searchIndexJson));
	}

	// ...
})

```

Then in my `onChange` function, where I'm doing the work to convert the search term to an options list to display as the search results, instead of directly pulling the post title and slug from the search index I pull the post ID from the search index and then find that post's metadata in `props.postMetadataList`. Reference [my previous article on the Search Component](https://superflux.dev/blog/static-site-search-search-component) to see more details, but the relevant change is:

```tsx
const onChange = (searchTerm: string) => {

	// ...

	if (searchIndex.size && searchIndex.has(searchTerm)) {
		const newOptions = new Array<OptionValue>();
		for (const id of searchIndex.get(searchTerm)) {
			const postMetadata: PostMetadata = getPostMetadataById(id, props.postMetadataList);
			newOptions.push({
				value: postMetadata.slug,
				label: postMetadata.title
			});
		}
		setOptions(newOptions);
	} else {
		setOptions([]);
	}
	
}
```

That `getPostMetadataById` function is just doing a simple find right now. Perhaps in the future I'll refactor that `PostMetadata[]` into something more efficient so I can do a quick lookup by ID.

```typescript
export const getPostMetadataById = (id: number, postMetadataList: PostMetadata[]): PostMetadata => {
	return postMetadataList.find((post) => post.id === id);
}
```

## Conclusion

Ok great! I've cut the size of the search index down from ~700kb to ~22kb. I'm pulling my PostMetadata list into more places than before, but it's much smaller than the search index so say overall the data load per page is ~25kb. Huge improvement.

The search is maybe slightly slower than before, because I'm not just doing a simple Map lookup on the search term anymore. But we're talking like ~1ms or less, so it's not even noticeable. 

In the future, as my number of posts grow, I think I'd like to refactor the `PostMetadata[]` to be a Map on the post IDs instead. Something like `Map<number, PostMetadata>`. That would allow for quick lookup of both the search term results from the search index, and a quick lookup of the corresponding PostMetadata for each search result.

I'll leave it here for now. Thanks for reading, and I hope you found some value in this static site search series. Remember to visit [Part 1](https://superflux.dev/blog/static-site-search-preprocessing-articles) & [2](https://superflux.dev/blog/static-site-search-search-component) to see the my original implementation this refactor is building on.





![](./static-site-search-search-component.jpg)

## Static Site Search Series

In [my previous article on this topic](https://superflux.dev/blog/static-site-search-preprocessing-articles), I talked about preprocessing my articles and creating a search index. This search index allows for quick referencing the articles that match a particular search term.

In this article I'm going to talk about how I used this search index to perform searches.

## Desired Behaviour

I wanted a couple things from my search. I wanted it to display results in real-time as you searched, and I didn't want to implement any kind of search results page. I wanted it to just be intuitive and quick.

## Search Component

I use [Ant Design](https://ant.design) a bit on this site. It's just a library I'm familiar with.

They offer a component that they call [Select with Search](https://ant.design/components/select#select-demo-search). It's essentially a multi-select dropdown that lets you filter the options by typing into a text input.

By default, you provide it a list of options, and then the search will filter those options. You can customize the callbacks for both when an item is clicked or you press return, and the search callback that occurs whenever the input changes.

This was close enough to what I wanted and I'll describe how I customized it.

I started by making a wrapper around the Ant Design Select component so I can easily customize it and then stick it wherever I need to.

It started off something like this

*components/elements/SearchInput.tsx*
```tsx
import React from "react";
import { Select } from 'antd';

export default function SearchInput(props) {
	return (
		<Select
		    showSearch
		    placeholder="Select a person"
		    optionFilterProp="children"
		    options={[
			  {
				value: 'jack',
				label: 'Jack',
			  },
			  {
				value: 'lucy',
				label: 'Lucy',
			  }
		    ]}
		/>
	)
}
```

To more easily type the `options`, I created a little `OptionsValue` interface at the top of the file:

```tsx
interface OptionValue {
	label: string;
	value: string;
}
```

## Loading the Search Index

I'm doing this a very sketchy way, I think. I'm really not a frontend developer so I kinda just made this up as I went.

My plan is eventually to slim down the index enough that I can just build it in with the rest of my static site, but for now since it's 600+ KB I tried to lazy load it to not mess with the initial page load.

Essentially I'm just loading the file in a `useEffect` and setting it into the component state.

*components/elements/SearchInput.tsx*
```tsx
export default function SearchInput(props) {
	const [searchIndex, setSearchIndex] = useState<Map<string, object>>(new Map());

	useEffect(() => {
		if (!searchIndex) {
			import("./search-index.json")
				.then((data) => {
					const indexMap = new Map<string, object>();
					for (const key of Object.keys(data)) {
						indexMap.set(key, data[key]);
					}
					setSearchIndex(indexMap);
				})
				.catch((error) => {
					console.error("Error loading search index", error);
				})
		}
	}, []);

	return (
		<Select
		    showSearch
		    placeholder="Select a person"
		    optionFilterProp="children"
		    options={[
			  {
				value: 'jack',
				label: 'Jack',
			  },
			  {
				value: 'lucy',
				label: 'Lucy',
			  }
		    ]}
		/>
	)
}
```

## Options

So I could have just provided a list of the article titles to this component and called it a day. I wouldn't have had to bother with all the preprocessing and indexing. But that would only have let me search the titles, not the tags and the excerpts.

With my search index I get a lot more nuance. For example if you search "docker" you get 1 result which is my post titled "What I'm Learning in 2022", which just wouldn't work with a simple title search.

First I initialize an empty options array in component state

```tsx
const [options, setOptions] = useState<OptionValue[]>([]);
```

Then I pass this to the `Select` component

```tsx
<Select
	showSearch
	placeholder="Search"
	suffixIcon={<SearchOutlined />} // Custom Ant icon to show in the right of the input field
	options={options}
/>
```

Great! So now I have my search index available, and I can now implement my custom search functionality using the index.

## onSearch

The Select component from Ant Design has an overridable `onSearch` function. I'm going to override this function and use the current search term to populate the `options` using the values from the `searchIndex`.

A couple things I do in this code:

- I check to make sure I'm not repeating the same searches. Not sure how necessary this is or not, I don't remember. This uses a `useRef` on the component to track the search iterations and a `useState` to track the previous search term.
- I check for empty string search terms and simply set an empty list in the `options`, no need to check the search index.
- I was doing some stuff with lodash `debounce` and `useMemo` to delay the search, but I'm not sure if this is necessary. I'll show it afterwards as optional.

After those checks, I imply lowercase the search term and then pull the value from the `searchIndex`.

If the search term doesn't exist as a key in the `searchIndex` I simply set an empty array in `options`.

If the search term does exist, then I pull the values from the `searchIndex` for that key, I convert the `{ title: string, slug: string }` into a `{ value: string, label: string }` where `value` = `slug` and `label` = `title`. I build a new options array of all the values from that search term and then set that new array into `options` component state.

When that `options` component state gets set, the component then re-renders showing the new options.

Here's the code for this `onSearch` handler

```tsx
const [searchIndex, setSearchIndex] = useState<Map<string, object>>(new Map());
const [options, setOptions] = useState<OptionValue[]>([]);
const fetchRef = useRef(0);
const [previousSearchTerm, setPreviousSearchTerm] = useState<string>("");

const onChange = (searchTerm: string) => {
	fetchRef.current += 1
	const currentFetch = fetchRef.current;
	if (currentFetch !== fetchRef.current) {
		return;
	}
	if (searchTerm.length < 1) {
		setOptions([]);
		return;
	}
	searchTerm = searchTerm.toLowerCase();
	if (searchTerm === previousSearchTerm) {
		return;
	}
	setPreviousSearchTerm(searchTerm);

	if (searchIndex.entries() && searchIndex.has(searchTerm)) {
		const newOptions = new Array<OptionValue>();
		for (const item of searchIndex.get(searchTerm) as { title: string, slug: string}[]) {
			newOptions.push({
				value: item.slug,
				label: item.title
			})
		}
		setOptions(newOptions);
	}
}
```

Then on the `Select` component I pass a few more props.

- I set `filterOption={false}`. This essentially disables the built-in filtering using the search term and the labels. I don't need it to filter the results because I'm filtering the `options` myself.
- I pass a reference to my `onSearch` function in the `onSearch` prop.
- I pass a simple sort function in the `filterSort` prop to organize the options alphabetically.
- I pass `notFoundContent={null}` to get rid of the default empty state when no search results are found. This just hides the results pane when empty. Could maybe do some nicer customization here.

The component looks something like this:

```tsx
<Select
	showSearch
	filterOption={false}
	onSearch={onSearch}
	placeholder="Search"
	suffixIcon={<SearchOutlined />}
	filterSort={(a: OptionValue, b: OptionValue) => (a?.label ?? "").toLowerCase().localeCompare((b?.label ?? "").toLowerCase())}
	notFoundContent={null}
	options={options}
/>
```

![](./static-site-search-component-screenshot.png)

## Navigate to the selected page

Lastly, we want to actually navigate to the corresponding page if a value in the search is clicked.

We can pass a `onSelect` prop that handles this. In this function I simply use the `value` from the selected option which is the `slug` of the post, and Next `useRouter` to navigate to that page.

The function is as simple as this:

```tsx
const router = useRouter();

const navigateToArticle = (value: string) => {
	router.push(`/blog/${value}`);
}
```

And thats it! I won't touch on styling or how I placed this component on my website. You can see that for yourself! Also the styling is going to be unique for your project if you're looking to replicate this.

The final code for my customized component looks like this

*components/elements/SearchInput.tsx*
```tsx
import React, { useEffect, useState, useRef } from "react";
import { Select } from 'antd';
import { SearchOutlined } from '@ant-design/icons';
import { useRouter } from 'next/navigation'

interface OptionValue {
	label: string;
	value: string;
}

export default function SearchInput() {
	const router = useRouter();
	
	const [searchIndex, setSearchIndex] = useState<Map<string, object>>(new Map());
	const [options, setOptions] = useState<OptionValue[]>([]);
	const fetchRef = useRef(0);
	const [previousSearchTerm, setPreviousSearchTerm] = useState<string>("");
	
	const onChange = (searchTerm: string) => {
		fetchRef.current += 1
		const currentFetch = fetchRef.current;
		if (currentFetch !== fetchRef.current) {
			return;
		}
		if (searchTerm.length < 1) {
			setOptions([]);
			return;
		}
		searchTerm = searchTerm.toLowerCase();
		if (searchTerm === previousSearchTerm) {
			return;
		}
		setPreviousSearchTerm(searchTerm);
	
		if (searchIndex.entries() && searchIndex.has(searchTerm)) {
			const newOptions = new Array<OptionValue>();
			for (const item of searchIndex.get(searchTerm) as { title: string, slug: string}[]) {
				newOptions.push({
					value: item.slug,
					label: item.title
				})
			}
			setOptions(newOptions);
		}
	}

	const navigateToArticle = (value: string) => {
		router.push(`/blog/${value}`);
	}

	useEffect(() => {
		if (!searchIndex) {
			import("./search-index.json")
				.then((data) => {
					const indexMap = new Map<string, object>();
					for (const key of Object.keys(data)) {
						indexMap.set(key, data[key]);
					}
					setSearchIndex(indexMap);
				})
				.catch((error) => {
					console.error("Error loading search index", error);
				})
		}
	}, []);

	return (
		<Select
			showSearch
			filterOption={false}
			onSearch={onSearch}
			onSelect={navigateToArticle}
			placeholder="Search"
			suffixIcon={<SearchOutlined />}
			filterSort={(a: OptionValue, b: OptionValue) => (a?.label ?? "").toLowerCase().localeCompare((b?.label ?? "").toLowerCase())}
			notFoundContent={null}
			options={options}
		/>
	)
}
```

## Add delay to the search

If your running into some issues with your search firing multiple times, or you'd just like to add a delay for aesthetics, you can use `useMemo` and lodash `debounce`. Simply wrap your `onChange` function in the following

```tsx
const onChange = useMemo(() => {
	const getItems = (searchTerm: string) => {
		// your logic for filtering the options
	}
	return debounce(getItems, 50); // delay in ms
}, [searchIndex])
```

## Resources

- [Ant Design Select component](https://ant.design/components/select)

## Repository

This website in it's entirety is viewable on [GitHub](https://github.com/tyrelh/personal-site-nextjs).

The search logic doesn't have it's own repo yet. But I'll update that here if I ever split it out.

## Conclusion

I think this turned out really well. It works great and really only took a couple of days to figure out end-to-end. It was a pretty interesting and fun project too.

I was going to call this "Next.js Static Site Search", but in the end with this solution it really has nothing to do with Next. You should be able to incorporate this into any statically generated site, particularly those using Markdown and React, although you should be able to adapt it to other stacks as well.

This is working quite well for me in it's current state, but there are lots of optimizations to be made. If I get around to those I'll post a third part to this series explaining the optimizations I implemented.

Thanks for reading!


#search

Static Site Search Part 3 - Optimizing the Index

April 18, 2024 8 Minute Read

Static Site Search Part 2 - Search Component

April 10, 2024 7 Minute Read

Static Site Search Part 1 - Preprocessing Articles

April 3, 2024 15 Minute Read

Get in touch