Incremental Writes in Hugo
Lately I’ve been twiddling with a few sufficiently complex
Hugo themes and thought I’d share an interesting approach
for rendering small changes quickly. In absence of a better term —
“incremental writes” will do and this method works best when there’s a lot
happening inside a theme that renders in the 1,000 ~ 10,000+ and beyond
page range.
So what’s Hugo and why is it great? Well it’s the ultimate static site generator
(SSG) that just gets things done. The reason it’s great is due to its
The code discussed here isn’t necessarily intended to be used with
hugo server because it has its own
fast rendering mode that works well enough if a theme is adequately simple and
well written. This is more of an approach for rendering and previewing a large,
complex, and messy theme quickly with just the plain hugo command. My hugo
version is v0.108.0.
shell
$ hugo version
hugo v0.108.0+extended linux/amd64 BuildDate=unknownOverview of Site Rendering
Before going into the details of making hugo write files incrementally it’s
useful to have a simple idea of the minimum requirements of a theme and the way
a site is rendered.
shell
|-- config.json
|-- config.toml
|-- config.yaml
|-- content
| `-- markdown
| |-- first.md
| |-- second.md
| `-- third.md
`-- themes
`-- base
`-- layouts
`-- _default
`-- baseof.html
`-- index.htmlThe directory structure above along with one of the basic
configuration formats
are all that’s required to render a theme. Inside the content folder there’s the
Markdown source and inside the themes folder are
the layout files. Below is a minimal config.yaml that disables specific
outputs because for this explanation the home and page outputs are the
only concern.
yaml
---
baseURL:
theme: base
title: Base
languageCode: en-us
paginate: 10
summaryLength: 1
taxonomies:
tag: tags
disableKinds:
- sitemap
- term
- section
- taxonomy
outputs:
home:
- html
section:
- html
taxonomy:
- html
term:
- html
page:
- htmlDefault formats are output
“kinds”
that are split across home, section, taxonomy, term, and page. This separation
avoids thinking about redundant logic and common repetitive patterns. Home pages
are rendered by index.html, sections by section.html, taxonomies by
taxonomy.html, terms by term.html and pages by single.html.
| Kinds | Layouts |
|---|---|
home |
_default/index.html |
section |
_default/section.html |
taxonomy |
_default/taxonomy.html |
term |
_default/term.html |
page |
_default/single.html |
Base layouts are
the authoritative boilerplate that markup and
HTML (Hypertext Markup Language). The ideal scenario is to only extend base
layouts and keep visual interpolation and noise to a minimum.
shell
`-- themes
`-- base
`-- layouts
`-- _default
`-- baseof.html
`-- baseof.json
`-- baseof.xml
`-- baseof.txtFor HTML and other boilerplate if there’s confusion over what type or kind is
in use, then appending that information to the markup or data in a
non–obstructive way works wonders. A block opens up a space inside a base
template and a define elsewhere fills in or extends that space with the
desired content.
html
<!DOCTYPE html>
<html
lang="en-us"
data-kind="{{ .Page.Kind }}"
data-type="{{ .Page.Type }}"
>
<head>
<title>
{{- block "title" . -}}
{{ .Site.Title }}
{{- end -}}
</title>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
</head>
<body>
{{- block "main" . -}}
<p>
This is what you see if you don't
extend the base template's main block.
</p>
{{- end -}}
</body>
</html>Incremental Writes
Writing output incrementally exploits the way hugo evaluates base templates.
The whole site still has to be read but like a novel reading is faster than
writing — and the computer is not much different. So the speed up here makes
hugo mostly read things and not write them. It just so happens that if a base
template for a particular kind evaluates to nothing well then — hugo does
nothing.
Multiple conditions exist for choosing which files to process and write but the
most obvious one compares the modification dates of the source markdown with
the resulting index.html in the
$page variable and that’s
due to attempts at supporting both domain and sub–directory detection from the
base URL (Uniform Resource Locator).
true or false output.
go-html-template
{{- $input := . -}}
{{- $pageContext := $input -}}
{{- $markdown := print "content/" $pageContext.File -}}
{{- $markdownModTime := "" -}}
{{- $page := print "public/" (strings.TrimPrefix
$pageContext.Page.Site.BaseURL
$pageContext.Page.Permalink
) "index.html"
-}}
{{- $pageModTime := "" -}}
{{- if fileExists $markdown -}}
{{- $markdownModTime = (os.Stat $markdown).ModTime -}}
{{- end -}}
{{- if fileExists $page -}}
{{- $pageModTime = (os.Stat $page).ModTime -}}
{{- end -}}
{{- $modified := gt $markdownModTime $pageModTime -}}
{{- $output := or $modified (in (slice
"home"
"section"
"taxonomy"
"term"
) $pageContext.Page.Kind)
-}}
{{- return $output -}}One of the upsides to this particular function is that it also allows turning
off writes for particular parts of the site with an in slice check against
the current page’s kind.
html
{{- $modified := partial "function-page-modified.html" . -}}
{{- if $modified -}}
<!DOCTYPE html>
<html
lang="en-us"
data-kind="{{ .Page.Kind }}"
data-type="{{ .Page.Type }}"
>
<head>
<title>
{{- block "title" . -}}
{{ .Site.Title }}
{{- end -}}
</title>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
{{- $default := resources.Get "css/default.css" -}}
<link title ="Default" rel="stylesheet" href="{{ $default.Permalink }}" />
</head>
<body>
<main class="container">
{{- block "main" . -}}
<p>
This is what you see if you don't
extend the base template's main block.
</p>
{{- end -}}
</main>
</body>
</html>
{{- end -}}Absolutely no stray characters or
-) in the template logic removes
leading {{- and trailing -}} white–space. Repeat for every base template
type. The clip below shows a demonstration of this function with 20,000
pages.
index.html outputs are left untouched.See this repository for trying out different numbers and adjusting the complexity.
Applications and Utility
So how in the world is this useful? Mostly in the case of a theme that executes
lots of work. An upper bound of 20,000 pages for the device in the above clip
5 seconds hot plus write time for a single modified page change
even in the context of heavy image processing, bundling, and data
unmarshalling.
In other words no matter the complexity on previously rendered pages editing,
previewing, and building a new page will take at most ~5 seconds because
that’s how long it took my device to read all the source markdown.
It’s also important to know that the code paths for hugo and
hugo server are different and the
server will likely assume that the previously generated pages don’t exist even
if they are in the public folder. If this post makes any sense — you’re
probably using another web server anyway.
Finally be aware that care must be taken with shortcodes. Even shortcodes that are not in use are evaluated immediately at run time — this is an easy way to make builds slow. Keep them as simple as possible.
Conclusion
More could be said in regards to applying incremental switches especially to
pagination logic but my time was too limited to think that one through. The goal
here however is to avoid diving too far into incremental shenanigans and instead
find a good cut off point for archive rotation while still having fast previews
on pages when running the hugo command on a beefy theme. The
10,000 pages or even 100,000+ pages — more or less
depending on a theme’s template efficiency, disk I/O (Input/Output), and
device memory.
