> Discuss on [/r/git](https://www.reddit.com/r/git/comments/1lvbzo8/my_day_1_with_jj_jujutsu/) and [Bluesky](https://bsky.app/profile/iutensil.bsky.social/post/3ltnz23tp2k2z)

## TL;DR

I became productive with [jj](https://github.com/jj-vcs/jj) (Jujutsu, *dauntless* version control) on day 1. This is my story, along with my mindset changes, and delicious recipes. Scroll down to the end for a short list of when to use what command/alias.

## Why try `jj`?

Well, *dauntless* is not the official slogan, but I really need it to describe a liberating feeling much stronger than "fearless".

Git is an intimidating black box for some, but I don't fear it in my daily work, as I'm skilled enough to easily recover work if I messed up.

`jj`, on the other hand, gives me the ultimate power in the river of time:

> In `jj`, I can go anywhere in history, do anything, be done with it, and history will reshape itself accordingly.

I wouldn't know this when I first read [Steve's Jujutsu Tutorial](https://steveklabnik.github.io/jujutsu-tutorial/) months ago, it's well written but still a haystack to find the needle that would click with me.

[jujutsu on tangled](https://blog.tangled.sh/blog/stacking) intrigued me, by telling me "Jujutsu is built around structuring your work into meaningful commits", and showing how it helps contributors to iterate on changes with dependencies on other changes, individually reviewed, reworked, and merged. But what if I just use it on my personal repo, and don't care much about clean history?

[A Newbie's First Contribution to (Rust for) Linux: Git gud](https://blog.buenzli.dev/rust-for-linux-first-contrib/#git-gud) shows the author's Git workflow to contribute to Linux kernel with a lot of fast moving forks. The configuration for `rebase` and `rerere` seems really solving the roadblocks of contributing to a thriving project. But the author said: "Jujutsu is much better at this kind of workflow and what I was actually using most of the time. I first discovered this workflow while using Jujutsu and only later found out you could do something similar with Git." A refreshing remark! Maybe I do need native and friendly support for more advanced features, even just to alleviate my mind.

Eventually, pksunkara's [Git experts should try Jujutsu](https://pksunkara.com/thoughts/git-experts-should-try-jujutsu/), and his side-by-side workflow comparisons with Git, and his [`jj` configuration full of aliases](https://gist.github.com/pksunkara/622bc04242d402c4e43c7328234fd01c)  that gave me the first boost, so I finally decided to spend a day with  `jj`. I definitely no Git expert, but those aliases really look attempting, what if I can pick my few favorites?

I started the journey with a minimal goal. I just wanted to recover my Git workflow in `jj`, as `jj` coexists with a local Git repo, and I need `jj` to work with my GitHub (or any code forge I might migrate to in future). I learned much more, and they are much easier than I thought.

## Protect my main branch

First thing, I went to my GitHub settings and [protected my main branch](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches), to at least forbid force push, in case (my misuse of) `jj` messes up the remote. ([Here](https://ricky-lim.github.io/blog/protect-your-main-branch-in-github/) is how to do even better from CLI.)

## Install and configure `jj`

I use [just](https://github.com/casey/just) to write code snippets as tasks. Installing `jj` on Mac, and prepare the configurations is simply:

```make
prep-jj:
    which jj || brew install jj
    mkdir -p ~/.config/jj
    cp -f dotfiles/.config/jj/config.toml  ~/.config/jj/config.toml
    jj config set --user user.name "$(git config --get user.name)"
    jj config set --user user.email "$(git config --get user.email)"
    jj config list --no-pager
```

The [`dotfiles/.config/jj/config.toml`](https://github.com/utensil/forest/blob/main/dotfiles/.config/jj/config.toml) is an adapted version from pksunkara's, and all changed places are annotated with `[NOTE]` comments and inspiring links. By copying to the right path, it becomes the user-wide configuration.

`jj config set --user ` will write additional and more personal information into the same file (assuming git has already configured these information), so they came later. Finally, the configuration is listed for inspection.

Every time I've updated the configuration, or I'm on a new machine, I'll run `just prep-jj` again, so I'm all set again. Your way to manage code snippets might be different, but it all boils down to similar few lines.

You might want to change the line `editor = 'hx'` back to favorite editor, as mine is now [Helix](https://github.com/helix-editor/helix).

It also assumed that you have installed [delta](https://github.com/dandavison/delta), a syntax-highlighting pager for git, diff, grep, and blame output. It's eye candy, and makes the diff much more clear.

## Initialize the Git repo

`jj` can coexist, or officially, *colocate* with a local Git repo. I place all my local Git repos under `~/projects/` so my just task looks like this:

```make
init-jj PROJ:
    #!/usr/bin/env zsh
    cd ~/projects/{{PROJ}}
    jj status || jj git init --colocate
```

`jj status` will detect if it's already a `jj` repo, otherwise we will initialize `jj` to colocate with Git. I try to make all my tasks open to reentrance, so I can easily tweak and rerun them.

Actually it's already alias to `jj init` by pksunkara, but here we go with the origin command, to avoid locked-in. But later on, I'll use more aliases, and you may consult [the configuration](https://github.com/utensil/forest/blob/main/dotfiles/.config/jj/config.toml) for the full command.

## Thrown into limbo

Once `jj` is initialized, we are no longer on any branch of Git. If we go back to `git` or `lazygit`, we'll see that we are on a detached state.

I don't know about you, but my feeling was like floating in the air, with no solid ground under my feet.

I felt much better when I learned that what I commit as *change*s in `jj` will still be reflected in Git as commits, and commits can still find their parents.

We can go back and edit a *change* then commit again, it would be become a new Git commit, but its identity as the original *change* remains the same. So changes can still find their parent changes.

Actually, a branch in Git, is just a reference always point to the latest commit of the branch, so its history can be traced via parents.

`jj` has *bookmark* for pointing to a *change*. Unlike in Git, when you commit more changes on top of it, the *bookmark* doesn't move to the latest change automatically, however, *we* can move it.

With these 2 powerful concept, we are lifted off ground, start flying among changes, with bookmarks marking our path.

## The working copy

OK, we are not on any branch, so where are we?

`jj log` (or simple `jj l` or even `jj`) gave me something like these at the top:

```
@  rrptyzpt 0f23c873 (empty) (no description set)
◆  vmzsxwzu f800c4ed My last commit on the main branch
```

That's a lot to take in!

`f800c4ed` and  `My last commit on the main branch` can be recognized as my last Git commit hash and message.

`vmzsxwzu` is the *change id* assigned to it, along with a diamond ◆ indicating that it's immutable: It's written on the stars by Git, so we can't edit it.

But what's the commit-like stuff on top of it? I didn't like it, and even tried to `jj abandon` it, but it keeps coming back with a new *change id* and a new commit hash.

Finally, I accepted it as the "working copy", indicated by `@`. Unlike in Git, where I'll be on HEAD, along with some local changes to be staged, in `jj`, I'm already on the future commit as a working copy. That's why sometimes we need `@-` that points to the parent of `@`, the last committed *change*.

`(empty)` means I haven't made any edits to this *change*. `(no description set)` means I haven't described what I'm going to do for this change (via `jj desc`).

I'll feel much better if it's just `(no description)`, as the `set` makes me feel guilty for not providing a description ahead. And it turns out that I didn't need to! Even after I'm done with it, and moved on (`jj` will keep the change in its history, like automatically doing `git stash`). If I have trouble recognizing the change without a description, I could always use `jj d` to inspect the diff.

`jj s` (status) shows it more verbosely:

```
The working copy has no changes.                                                                                                                          │
│Working copy  (@) : rrptyzpt 0f23c873 (empty) (no description set)                                                                                        │
│Parent commit (@-): vmzsxwzu f800c4ed main | My last commit on the main branch
```

The change id of my working copy begins with `rr` (bold and colored, indicating we can refer to it uniquely with `rr`).

It's a bit annoying to enter and quit pager, and unable to see the latest changes, so I'v configured `jj` to use no pager and only occupy about 2/3 screen.

There's also alias `jj lp` that shows private (i.e. not pushed) changes, and `jj lg` that shows all changes, including elided changes that you just pushed.

## Commit

I worked on a repo that I'll also add my just tasks and configurations for `jj` into it. So I edit the justfile in my editor Helix, and saved it to disk.

Alias `jj d` will show the diff, that confirms that `jj` has seen our edits.

We can now commit the change. `jj commit` will commit everything, opening the configured editor for commit message (just save and quit to submit, quit without saving to cancel).

Aliases like `jj ci` will also fire up a built-in diff editor to interactively select which files, or even which parts (called sections) to commit. Arrow keys can fold/unfold files/sections, space to select, `c` to commit, and `q` to quit. (I couldn't find how to commit until I tried `c`, there is no visual hint at all, unless you use mouse to click on the menus, but I didn't realize that I could do it on day 1, I tried every possible key combination to trigger the menu, no luck).

I often find myself repeatly use `jj ci` to review, select and commit, to split my local edits into a few independent changes.

With alias `jj cim <message>`, we can directly supply the commit message from CLI without opening an editor, then select which parts to commit.

## New

After commit, we'll again on an empty change with no description, so we can continue making new edits.

But if after working for a while, I suddenly want to discard my (uncommitted) edits, I can do `jj n rr` to start over from the last change. The uncommitted edits will be still there in `jj` as a change, but local files will be reset to the state when we commited `rr`.

Here `rr` could be any other change id, or any bookmark, or any *revset* in general. We'll learn more about *revset* later, for now, we just need to know that a change id, a bookmark, `@` and `@-` that we know so far are all *revset* s.

`jj n` (new) effectively spins off a new history line from the parent change. We can go along the line for a few changes, and spin off from anywhere again, without having to create a named branch.

## Edit

In Git, I seldom amend a commit, usually I just use `reword` in lazygit to amend the commit message, or `reset --mixed` (`gm` in lazygit) to discard history after a commit, then stage and commit them in a better way again.

In `jj`, if I don't like a change, I could go back and edit it:

```
jj e <revset>
```

will auto stash any dirty edits, and reset local files to the state of that `<reveset>` (read: change), I can happily do any edits, or commit it with alternative selections of files or sections.

We don't need to explictly commit, when we use `jj n <revset>` or `jj e <revset>` to jump to anywhere else, the local edits will be automatically committed to the original change.

## Evolution of a change

Wait, what if I messed up when editing a commit, and left it, can I go back?

At that time, it's beyond what `jj undo` can easily do, especially we might be confused by what we did and didn't do.

The first thing we can do is to inspect how a change has evolved over time by

```
jj evolog -r <revset>
```

It looks like this:

```
❯ jj evolog -r uo
○  uoxoknux 92fb8d3a (no description set)-- operation a0693515ab31 (2025-07-09 11:54:16) snapshot working copy
○  uoxoknux hidden 1f73cdfd (no description set)-- operation 79813357f450 (2025-07-09 11:52:58) snapshot working copy
○  uoxoknux hidden c2bf8f4f (empty) (no description set)-- operation a22c21ce9fc1 (2025-07-09 11:52:32) new empty commit
```

We can inspect the diffs of each hidden commits with their commit hash via `jj dr <revset>` (alias for `diff -r`) where `<revset>` should be the commit hash to disambiguate.

The we can use either `jj n` or `jj e` on the commit hash of our choosing, to continue work.

Note that the change id would become ambiguous, so we need to `jj ad` one of the commit hash so we can continue to refer to the change id.

It looks like this for me:

<details>
<summary>click to expand</summary>
<pre>
❯ jj n 1f73
Working copy  (@) now at: unyrkpru 003d7750 (empty) (no description set)
Parent commit (@-)      : uoxoknux?? 1f73cdfd (no description set)
Added 0 files, modified 1 files, removed 0 files

❯ jj e 1f73
Working copy  (@) now at: uoxoknux?? 1f73cdfd (no description set)
Parent commit (@-)      : mxowtxvr 68b37035 (no description set)

❯ jj evolog -r uo
Error: Revset `uo` resolved to more than one revision
Hint: The revset `uo` resolved to these revisions:
  uoxoknux?? 92fb8d3a (no description set)
  uoxoknux?? 1f73cdfd (no description set)

Hint: Some of these commits have the same change id. Abandon the unneeded commits with `jj abandon <commit_id>`.

❯ jj ad 92f
Abandoned 1 commits:
  uoxoknux?? 92fb8d3a (no description set)
Rebased 1 descendant commits onto parents of abandoned commits
</pre>
</details>

We may also use `jj op diff --op <operationid>` to inspect meta changes of an operation (it has `--from` `--to` variant so it also works on a range of operations).

## PR with an ad hoc branch

> see also the official doc [Working with GitHub: Using a generated bookmark name](https://jj-vcs.github.io/jj/latest/github/#using-a-generated-bookmark-name)

If I like my changes so far on the current history line, I could open an unnamed PR by first commiting changes, then using alias

```
jj prw
```

where `prw` stands for PR working copy.

Per our config, it will push to a branch named `jj/<changeid>` on origin, then open a PR via `gh` (Github's official CLI to interact with issues, PRs, etc.), it will ask you for a title (default to the description of the first change), and a body, we could just press enter all the way to submit, and change anything later.

Personally I think such a tiny ad hoc PR is a good way to record the proposed changes, even for a personal repo. But if pusing to a branch is all you need, alias `jj pscw` (push changes of working copy) will do.

After merging the PR, we need

```
jj git fetch
```

to let both Git and `jj` to learn about the changes.

Here is how `jj l` looked like after I PR a change "Make a start on trying jj" as `#1`, merged it on GitHub, then fetched the changes:

```
◆  rzyozkll 929fe190 main Make a start on trying jj (#1)
│ 
◆  vmzsxwzu f800c4ed My last commit on the main branch
```

`gh pr view #1 -w` can open the PR in the browser.

If we have made edits to the change (but not new changes on top of it, because they would be pushed to their own `jj/<changeid>` branch), like `jj e` or `jj sqar` (introduced below), we can use

```
jj pscw
```

where `pscw` stands for pushing local changes of working copy.

The result looks like this:

```
Changes to push to origin:
  Move sideways bookmark jj/stwxmxoqkyym from 33c46b992efc to 4d3c2a60e0d2
remote: Resolving deltas: 100% (3/3), completed with 3 local objects.
```

If we have made other changes on top of the change, we can treat `jj/<changeid>` as a named branch like below (i.e. `jj bs jj/<changeid> -r <latest changeid>` and `jj psb jj/<changeid>`).

## PR with a named branch

This would best resemble a usual GitHub workflow.

First we create a local bookmark:

```
jj bs <bookmark> -r <revset>
```

`<revset>` could be just a change id, `@-` could come in handy to refer to the last committed change.

`<bookmark>` is the name of the bookmark, just like a normal branch name. And `bs` is the alias for `bookmark set`.

We can see if the bookmark is correctly placed from `jj l`. If we don't like it, we can either `jj undo` or `jj bd <bookmark>` (bookmark delete).

Then we can PR that bookmark:

```
jj prb <bookmark>
```

This would push to a branch named `<bookmark>` on Github, then use `gh` to PR it.

It looks like working on a branch in Git, but after we commit more changes, we need to call `jj bs -r <revset>` to set it to desired `<revset>`, then

```
jj psb <bookmark>
```

Note that the alias begins with `ps` instead of `pr`, meaning "push bookmark".

## Fetching

What if we committed some changes on GitHub, like accepting a review suggestion, we need to use `jj git fetch` to make `jj` know about the changes. We'll see something like

```
bookmark: our-name-of-bookmark@origin [updated] tracked
```

`origin` is the remote on GitHub. `[updated]` means `jj` noticed the changes on the remote, `tracked` means the local bookmark is also automatically updated.

If it's not tracked, we can use

```
jj bt our-name-of-bookmark@origin
```

to track it.

Here is how `jj l` looks like when I PR a few changes in this way then fetched it:

```
@  nozpulso fd589b2a (empty) (no description set)
○  kpxkrxvm a22ed20a configure-jj* Add frequently used jj command alias
○  tvvoymuu 6808370c Keybindings for builtin diff editor
○  ylnlzpvx b20ac2d1 Adapt jj config from pksunkara
○  rnkttumw f6296b0a Improve configs in `just init-jj`
◆  rzyozkll 929fe190 main Make a start on trying jj (#1)
◆  vmzsxwzu f800c4ed My last commit on the main branch
```

## Rebase

But I didn't get such a clean history from the start. I'd begun with a few fixup commits, and one of the commits is mixed with different changes. And initially, I pushed it with an ad hoc branch, accepted a few review suggestions on Github.

After opening the PR, I've noticed a weird non-empty commit `xm`

```
○  ...more changes...
◆  rzyozkll 929fe190 main Make a start on trying jj (#1)
│ ○  xmnykxym 4dcb41d9 (no description set)
├─┘
◆  vmzsxwzu f800c4ed My last commit on the main branch
```

After inspecting the diff, I believe its changes is aborbed into `rz` (`#1`), so I rebased `xm` on top of `rz` to see if there is any diff left.

```
❯ jj rb -s xm -d rz 
Rebased 1 commits to destination
```

where `rb` stands for `rebase`.

I just need to remember that I'll be rebasing source (`-s`) onto destination (`-d`).

Now I have

```
○  ...more changes...
│ ○  xmnykxym 2f622327 (empty) I think this is what absorbed into #1
├─┘
◆  rzyozkll 929fe190 main Make a start on trying jj (#1)
```

Nice, it's indeed empty, so I can be assured no work is lost.

It's an extremely simple use case, but I'm happy with being able to transplant a parallel change to be on top of another change, without much hassle.

## Squash

There are some other complications:

```
@  ...more changes...
│ ○  zquplvzx 2b572336 configure-jj Update dotfiles/.config/jj/config.toml
│ ○  nmuntxon 746cb300 Fix typo found by copilot
├─┘
○  ...some changes in between...
○  ylnlzpvx b20ac2d1 Adapt jj config from pksunkara
○  rnkttumw f6296b0a Improve configs in `just init-jj`
```

I want to absorb typo fixes `nm` and `zq` into `yl`, how do I do that?

It's as simple as correctly specifying `from` and `to`:

```
❯ jj sqa --from zq --to y
Rebased 3 descendant commits
Working copy  (@) now at: srnwoqzq c831d68c (empty) (no description set)
Parent commit (@-)      : kpxkrxvm 7d911838  jj/vzzzxqvnptov* | Add frequently used jj command alias
Added 0 files, modified 1 files, removed 0 files
```

where `sqa` is short for squash. Note from the output that any changes after them will be properly rebased.

There is also a shortcut if I just want to absorb a change into its parent: `jj sqar <revset>`.

I keep using the word "absorb" but `jj absorb` does a completely different thing (it splits a change, allowing some other changes to absorb it, by "moving each change to the closest mutable ancestor where the corresponding lines were modified last", that sounds like "desolve" to me).

## Summary

I could go on and try splitting a change, but the above is enough for day 1.

This doesn't demonstrate the full power of `jj`, but imagining how the same things could be done in Git, I'll definitely rather spend the time with `jj`.

I'll wrap up with a summary of the commands/aliases that would be enough for daily work:

### Getting started

| alias | when to use |
|-------|-------------|
| `jj init` | init `jj` in a local Git repo |

### Show

| alias | when to use |
|-------|-------------|
| `l`   | show log, `jj` works better |
| `lp`  | show private log |
| `lg`  | show all log, even elided changes |
| `s`   | show status |
| `w`   | show status and diff |
| `d` | show the diff of current change |
| `dr <revset>` | show the diff of revision `<revset>` |

### Edits

| alias | when to use |
|-------|-------------|
| `n <revset>` | create a new change based on `<revset>` |
| `cm <message>` | commit everything with `<message>` |
| `ci` | commit the change, choose interactively, write commit message in `$EDITOR` | 
| `e <revset>` | edit any `<revset>` after commit |

`<revset>` could be a short change id, a bookmark, and defaults to @ (working copy).

After edit, just go anywhere else (with e or n), the changes will be commited; or an explicit commit will do.

### Working with remote branch

| alias | when to use |
|-------|-------------|
| `git fetch` |  will update local bookmark from remote branch |
| `bt <branch>@origin` | track remote branch |
 
### PR with named branch

| alias | when to use |
|-------|-------------|
| `bs <bookmark> -r <revset>` | move bookmark `<bookmark>` to `<revset>` |
| `psb <bookmark>` | push bookmark to remote branch named `<bookmark>` |
| `prb <bookmark>` |  create a PR from `<bookmark>` to the default branch |

### PR with ad hoc branch

| alias | when to use |
|-------|-------------|
| `prw` | open (committed) working copy as a PR |
| `pscw` | push changes of working copy to remote branch named `jj/<changeid>` |

### History cleanup

| alias | when to use |
|-------|-------------|
| `undo` | undo last jj operation when messed up |
| `rb -s <src> -d <dst>` | rebase `<src>` onto `<dst>` |
| `sqar <revset>` | squash `<revset>` into its parent |
| `sqa --from <src> --to <dst>` | squash `<src>` into `<dst>` |
| `ad <revset>` | abandon a change |
| `evolog <revset>` | show how a change evolve in time |
| `op log` | show log of operations on jj |
| `bd <bookmark>` | delete a bookmark |

These aliases can be found in this [config file](https://github.com/utensil/forest/blob/main/dotfiles/.config/jj/config.toml) which is based on pksunkara's [gist](https://gist.github.com/pksunkara/622bc04242d402c4e43c7328234fd01c).

> TL;DR: I used `colima` to run an x86_64 Docker container (Ubuntu) on Apple silicon, to quickly test `zig build` with LLVM backend and with self-hosted backend, because it's both exciting and concerning (for missing all the goodies from LLVM) news.

After Self-Hosted x86 Backend is Now Default in Debug Mode dropped, I immediately wanted to try it out, but I only have an Apple Silicon machine (e.g. Mac Mini M4 Pro).

## Running an x86 container on Apple Silicon

According to Intel-on-ARM and ARM-on-Intel, I'm supposed to be able to run x86_64 Zig using `lima` with Apple's native virtualization framework and Rosetta. After some fiddling and searching, I've realized that I could just use `colima` to run an x86_64 container on an ARM64 VM, which is also backed by `lima`.

OK, let's get started:

First, install `colima` and prepare it properly:

```bash
# we need `docker` CLI as the client
which docker || (brew install docker; brew link docker)

# (optional) while we are at it, get `docker-compose` and `kubectl` too
which docker-compose || brew install docker-compose
which kubectl || brew install kubectl

# install colima
which colima || brew install colima

# this is to prevent othere docker daemons from interfering
docker context use default
```

Next, let's start colima with Apple's native virtualization framework and rosetta:

```bash
colima start --vm-type=vz --vz-rosetta
```

Because I have already used colima before, but without these flags, there is a warning saying that they are ignored, so I have to delete the existing colima VM and start over.

(Warning: the following command will also DELETE all existing images! So I commented out them to prevent accidental execution.)

```bash
# colima delete
```

Now, we can pull an x86_64 Docker container with Ubuntu:

```bash
# asssuming `docker login` has been done already
docker pull --platform linux/amd64 ubuntu:jammy
```

and start it (`--rm` means to remove the container after it exits, so we'll lose the changes made inside, remove this option if you want to keep the container):

```bash
docker run --rm -it --platform linux/amd64 ubuntu:jammy bash
```

Inside the container, let's confirm that we are indeed running x86_64:

```bash
uname -m
```

Cool, I'm seeing `x86_64`!

## Running `zig build`

Now, we can install Zig and try it out:

```bash
# we need a few basic utils
apt update
apt install -y wget xz-utils software-properties-common git

# Download the corresponding Zig version with self-hosted x86 backend
wget https://ziglang.org/builds/zig-x86_64-linux-0.15.0-dev.769+4d7980645.tar.xz
tar xvf zig-x86_64-linux-0.15.0-dev.769+4d7980645.tar.xz

# Make it available in PATH
export PATH=/zig-x86_64-linux-0.15.0-dev.769+4d7980645/:$PATH

# Verify its version and that it runs
zig version
# got: 0.15.0-dev.769+4d7980645
```
Let's create a simple Zig project to test building it:

```bash
mkdir projects
cd projects
mkdir zig_x86_64
cd zig_x86_64

zig init
zig build
```

Success!

`zig build run` gives

```
All your codebase are belong to us.
Run `zig build test` to run the tests.
```

and `zig build test --summary all` gives:

```
Build Summary: 5/5 steps succeeded; 3/3 tests passed
test success
├─ run test 1 passed 7ms MaxRSS:5M
│  └─ compile test Debug native cached 68ms MaxRSS:44M
└─ run test 2 passed 7ms MaxRSS:5M
   └─ compile test Debug native cached 67ms MaxRSS:44M
```

## Comparing with and without LLVM

But wait, how do I know it's actually using the self-hosted x86 backend?

Hopefully someone has a better way, I just took the longer way to force Zig to build with and without LLVM.

After reading the doc and some searching, I figured out that I could expose an extra option to `zig build` in my `build.zig` to set the corresponding flag for the executable, with only 2 edits:

```zig
    // EDIT 1
    const use_llvm = b.option(bool, "use_llvm", "Force use llvm or not") orelse false;

    const exe = b.addExecutable(.{
        .name = "zig_x86_64",
        .root_module = b.createModule(.{
            .root_source_file = b.path("src/main.zig"),
            .target = target,
            .optimize = optimize,
            .imports = &.{
                .{ .name = "zig_x86_64", .module = mod },
            },
        }),
        // EDIT 2
        .use_llvm = use_llvm,
    });
```

(Optional) I did the edits by running the following to install a helix editor so I can edit Zig code out-of-the-box:

```bash
# https://docs.helix-editor.com/package-managers.html#ubuntu-ppa
add-apt-repository -y ppa:maveonair/helix-editor
apt install helix
# then fire up `hx build.zig` and use it mostly like in Vim
# I also installed zls by
# cd /tmp
# wget https://builds.zigtools.org/zls-linux-x86_64-0.15.0-dev.197+48fb941e.tar.xz
# tar xvf zls-linux-x86_64-0.15.0-dev.197+48fb941e.tar.xz
# cp -f zls /usr/local/bin/
# and checked that it works by
# hx --health zig
# so I can use `gd` to go to definitions!
```

Cool, now let's try building with LLVM:

```bash
rm -rf .zig-cache && time zig build -Duse_llvm=true
```

```
real    0m3.068s
user    0m3.610s
sys     0m0.363s
```

Then without (which should be the x86 self-hosted backend):

```bash
rm -rf .zig-cache && time zig build -Duse_llvm=false
```

```
real    0m2.112s
user    0m2.812s
sys     0m0.361s
```

Wow, it's indeed faster without LLVM! I've tested this a few times and getting consistent results. I'll also try this on more complex projects later, but it's so exciting that I just wanted to write a note for this.

### UPDATE 2025-06-15

I further tried using poop to get more metrics.

First, get and install `poop`:

```bash
cd /projects && git clone https://github.com/andrewrk/poop && cd poop && zig build && cp -f zig-out/bin/poop /usr/local/bin/
```

Then let's run the cold start builds again:

```bash
cd /projects/zig_x86_64

rm -rf .zig-cache* && poop "zig build -Duse_llvm=true --cache-dir .zig-cache-llvm" "zig build -Duse_llvm=false --cache-dir .zig-cache-nollvm"
```

Well, that doesn't work due to permission denied. And `--cap-add PERFMON` or even `--cap-add SYS_ADMIN` didn't work. Not even `--privileged`. See this issue.

Let's try `hyperfine` instead:

```bash
apt install -y hyperfine
```

Taking comments by `mlugg0` on reddit into account, a few factors should be ruled out for a more fair comparison (with C):

1. rule out the build time for `build.zig`;
2. rule out the overhead of panic handler
3. squeeze a bit of performance at the cost of some safety by disabling C sanitization.

1 means to build `build.zig` before the benchmark and after cleaning the cache (observing that `zig build --help`  will build `build.zig` in order to get the options defined in the build script).

2 means to add the following to `main.zig`:

```zig
/// Don't print a fancy stack trace if there's a panic
pub const panic = std.debug.no_panic;
/// Don't print a fancy stack trace if there's a segfault
pub const std_options: std.Options = .{ .enable_segfault_handler = false };
```

3 means to pass `.sanitize_c = .off` to `root_module` in `build.zig`.

With

```bash
hyperfine --prepare "rm -rf .zig-cache* && zig build --help -Duse_llvm=true && zig build --help -Duse_llvm=false" "zig build -Duse_llvm=true" "zig build -Duse_llvm=false"
```

I got

```
Benchmark 1: zig build -Duse_llvm=true
  Time (mean ± σ):      1.392 s ±  0.052 s    [User: 1.287 s, System: 0.126 s]
  Range (min … max):    1.329 s …  1.473 s    10 runs

Benchmark 2: zig build -Duse_llvm=false
  Time (mean ± σ):     546.1 ms ±  13.6 ms    [User: 570.1 ms, System: 128.7 ms]
  Range (min … max):   532.9 ms … 575.9 ms    10 runs

Summary
  'zig build -Duse_llvm=false' ran
    2.55 ± 0.11 times faster than 'zig build -Duse_llvm=true'
```

which is indeed even faster!

discussions on /r/Zig, Bluesky

This is no way a professional prediction, just summarizing some ideas popped into my head in a recent discussion with friends. Read this with a grain of salt.

Reasoning, agent, and multimodality are top 3 research topics for LLMs, as far as I can tell. And for me, there is always the topic of efficiency or scalability. And of course, beyond that, world models, and intelligent robotics.

Verifiable reinforcement learning boosted the reasoning performance as explored by DeepSeek and others, but it requires much more output tokens, to a factor of almost 10. That would be 10x inference compute, and to solve the same problem in the same time, you would need another 10x compute to achieve that. Such a 100x compute demand, is not desirable.

I was wrong in 2023, about the inefficient inference of LLMs. I didn't like the quadratic scaling of the attention mechanism, but it turns out that there are more ways to work around the bottleneck than seeking linear attention such as RWKV and other alternatives. Inference time scaling, and MoE are both game changers.

So I would expect myself to be wrong again about the cost of reasoning. And there are preliminary researches showing that reasoning is already inside and can be further trained into the LLMs.

That is, reasoning should not be happening at the output layer, but intermediate layers. Writing down the reasoning process, should merely serve as a memory of explored paths, to keep things on track, or materialize the box to think out of.

The waste is not just on the input side, there are significant waste of input tokens by prompts that are not part of the instruction nor the context, but guidance. These guidances should not just be cached, but "compiled", in a more efficient and controllable manner, while remaining the interpretability.

Prompt engineering and reasoning by output tokens are two sides of the same coin: we are reaching better results in an interpretable manner, but much more expensive than it should be, and it would become obvious once we find the right principle. Of course, history relies on such detours to get back on track.

Agent is another layer of waste, on top of individual inferences. Agents fall into fruitless planning and exploration, failing to snap out of the infinite loop, by acquiring the extra information or triggering the extra side effects. Their interactions with tools (e.g., MCP), agents (e.g., A2A), and humans (e.g., A2H) are all inefficient due to using natural language as the intermediate representation.

Imagine an alien species with a far more efficient language, like the one depicted in *Arrival*. Such a language could express complex concepts like tool invocation with parameters using just a single token - or perhaps a single multimodal token combining multiple meanings.

There should be special dense tokens, to represent tools or even ways to think (e.g., the MCP for "Sequential Thinking"). And why should a token just be an embedding of a high dimensional vector? Why isn't it a curve, a hypersurface, or a manifold?

Take some time to think about this, while showing the visualization of the Biology of a Large Language Model on the screen.

I believe using modularity other than human perceptions, as the intermediate representation, is the key to the next breakthrough. They would also represent the world model better.

At the end of the day, it all comes down to human-level efficiency to reach human-level intelligence. And the efficiency will come from using the right principle, which will also result in better performance.

And all of reasoning, agent, multimodality, efficiency, world model, and robotics, are all covered by this one integrated solution.

And it's non-trivial to hunt their families down.

But yeah, that's pretty much my job description, that is, to set up mechanisms to eliminate similar problems from all systems in future.

They are from different kind of systems and varying technical stacks, so it would be interesting to pin these specimens down for a display.

How to hunt their families down are left as exercises for the readers.

## Legacy configuration removal

The system in question is a proxy server that routes requests to different backends that are responsible for communicating with different third-party APIs. Two keys in the request are used to determine the backend: `party` and `service`, which indicate *who* does *what*, respectively. For the same pair of these keys, even if there is only one type of backend, there could be multiple instances in different availability zones (AZs) and regions.

There is a subsystem (let's call it the "Switch") to route requests based on the health of these instances. It's vital for the switch to always work, so the requests won't go to a bad instance that is already unresponsive.

The Switch will reuse the *static* configurations of the proxy that lists combinations of `party` and `service`, along with *dynamic* instance information. The static and dynamic information as a whole is written to a cache, whenever an arbiter detects unhealthy instances. The proxy will use the cache to change its routing dynamically.

There was a special party (let's call it "Bob") that requires 2 steps/services (namely "Pre" and "Main") to complete what other parties can do in 1 step ("Main"). These 2 steps are stateful in the sense that if an intance can't do "Main", requests should not be routed to "Pre" either.

So when the Switch reads the blob from the cache, a special check is in place to ensure "Bob" & "Pre" are in the combination list, if they are not, the switch will reject to read, and report an error as a monitoring metric which didn't set up a proper alerting threshold for sudden increase in error rate, because it's an error that never happened before, and is missed when SREs setting up the thresholds for important metrics.

10 years passed by, and "Bob" can also do it in one step, "Main". There are also many legacy combinations that are not used anymore. One day it becomes necessary to remove unused combinations, the includes the combination "Bob" & "Pre".

Bang! The Switch rejects all new writes to the cache for 6 days and no one noticed because no major outage happened, the error rate due to unhealthy instances was low enough to be ignored. The issue is only discovered when the same person is observing the Switch for a different deployment, and it took almost a day to figure out the root cause.

The interesting part? The same person who removed the legacy configuration placed those special checks 10 years ago, the reviewer of the removal also reviewed the check 10 years ago. The check was just an unthoughtful precaution, recalling the check and predicting its ramifications on config removal didn't cross the minds of both of them, even during the initial stage of debugging.

## Too many concurrent slow queries

There is a batch processing system (let's call it the "Sink") that loads different kinds of data from asynchronous nodes of production database clusters, according the configured rules about data source, conditions, how to process etc. There are many rate limits in place to prevent the system from overwhelming the database clusters, such as limiting the concurrent number of tasks, and the threshold of the number of concurrent queries on the same database node. It's also an assumption that the asynchronous nodes are not used for user-facing online queries but other batch processing systems.

The rate limits gradually converged to a balance that maximizes the throughput without overwhelming the nodes even during stress tests of high data volume.

There is also a separate monitor metric that tracks slow queries over a certain threshold, and the threshold was also converging, so SREs could focus on the top issues.

One day, many different databases nodes were reporting high load, only for a few seconds each, but continuously affecting an online service that happens to be using asynchronous nodes, as the service is considered less time-sensitive thus tolerable to the delay that asynchronous nodes might have. SREs and DBAs are confused for quite a while, until a technical lead who guessed it might be caused by the batch processing system Sink, which turned out to be true.

The developers of Sink were also confused, as the concurrent queries on the same node, and the slow queries were both within the limits.

The truth did come out in the end. There was 2 tasks configured in Sink, which will be broken down into many sub-tasks for different time spans, and subsequently, queries for different shards of the same table across different asynchronous nodes. And there is a bug in the task configuration that caused slow queries, and the bug was copy-pasted to affect both tasks. Both tasks were running slow slow, but the database nodes could handle the load along with online queries, as the concurrent slow queries are well below the CPU cores that the nodes could use.

One that day, both the tasks were requested to be rerun for 2 time spans by the same person who introduced the bug, which caused concurrent slow queries on the same node to exceed the core count, blocking the online queries. But as the queries are rate-limited so they hit different nodes in a rotating manner, each node was only affected for a few seconds, the overall impact could only be observed by SREs monitoring the online services, and DBAs considered the phenomenon as a normal load spike.

The lesson learned? Each piece of measures were working individually, but they could not account for combinations of different issues, namely too many *concurrent* (and) *slow* queries, and only the combination could cause an impactful issue. And the information about different parts of the system (the online service, database nodes, and Sink) were not shared across teams.

## OOM when retiring old worker processes

This story is about an SOA framework written in C++ (let's call it the "Framework") that will spawn many worker processes to handle requests for a service, and when a new version of the service is deployed, it will gracefully retire the old worker processes, and spawn new ones of the new version to handle requests. The framework also integrates a few plugins to extend its functionality, and the plugins have their own lifecycle management, e.g. `init`, and `fini`.

One of these plugins (let's call it "Charlie") will starts an asynchronous thread for each worker process, and signals the termination of the thread in `fini`. It recommends an explicit call to `fini`, but `fini` was considered trivial enough to be also called in the destructor of the plugin. It's a bad practice in the eyes of any C++ expert, but it worked well for a long time, there was also no issue calling `fini` twice, once explicitly and once in the destructor.

Every few ten thousand deployments, there is an OOM ("Out of memory") when retiring the old worker processes, but it will be killed per cgroup thresholds on memory consumptions, so only few requests are impacted overtime, eventually SREs concluded that the OOMs are not isolated events.

It was a nightmare to reproduce the issue without know the cause upfront, and code looked innocent enough in review. Finally enough data were collected on rare reproductions, and the clues pointed to `fini` in the destructor of Charlie, but there is only the signaling in `fini`. Further investigation revealed that the asynchronous thread was sending a heartbeat request which caused the OOM.

During sending the request, there was a call to a method using local const static `std::array` holding some `std::string` initialized by literals. That static variable had been destroyed when the asynchronous thread is still alive, which would be signaled to terminate later in `fini` in the destructor of Charlie. The destructor was doing its thing not too non-trivial, but too late, due to the undefined execution order of destructors.

Why would this cause an OOM? After destruction, the heap memory involved were still valid, as they were not released but reused by the allocator, thus there is no SEGFAULT, but the memory for `size()` of the strings contains garbage value, which happens to be a large number, and the strings were appending to a `std::stringstream` to form the heartbeat request, causing endless memory allocation until the OOM.

The issue became more reproducible after using an allocator (`tcmalloc`) that would retain the heap memory longer after destruction. It would have been caught by various sanitizers if we have integrated them in CI.

## Silently hanging tasks

It's again about the batch processing system Sink. Internally it uses an MQ to queue tasks, so it can retry failed tasks. To prevent aggressive retries, there is a backoff mechanism that will exponentially increase the delay between retries, and the maximum number of retries is also limited.

One day, an incident caused all available zones of the MQ to drop a certain percentage of network packets, causing a stable failure rate of task dispatching. If the retries are performed in a regular rate, the aggregate delay of tasks would not be too long.

Unfortunately, the backoff mechanism backfired, along with other rate-limiting mechanisms, eventually causing all task to hang indefinitely. Again, existing metrics were not able to pinpoint the issue, and the developers took some time to figure out the root cause. But as the incident causing all SREs to be in a all-hands-on-deck mode, the hung were noticed quickly from less sensitive monitoring metrics, resulted in a relatively timely fix.

A more sensitive monitoring metric was deployed to detect hanging tasks, and it caught another issue a few days later, and the issue was handled swiftly before causing actual delay.

The issue was that there was a silly bug that runs `remove()` for every element in a huge array, and overall it would take `O(n^2)` time to finish, it's almost forever for such huge arrays. The removal would only be triggered when certain conditions are met, and the bug of removal was copy-pasted from some other utility package.

The bug should only hang one processes that handled the task, unfortunately, the dispatcher would detect timeout tasks, considered the process as dead, reassign the task to other processes when the task was still occupying the old process which was burying its head deep in the `remove()` calls, missing its own checkpoint for timeout. The re-assignment happened over and over again for each timeout, eventually causing all processes to be busy with the same task, and the `remove()` calls would almost never finish. The world literally stopped for that task.

With early detection of hanging tasks, developers were able to try fixes quickly. It should not be easy to find the root cause, but the developers noticed long pauses between two log entries, and the removal loop was the only code there when they wanted to add some logging between the log entries, thus the obvious bug was immediately identified.

It's really lucky that the first issue that was not too serious helped to deploy the more sensitive monitoring metric, so the second issue, which could be much fatal, was detected and fixed promptly, on the morning of the first day of a national holiday. Task re-assignment were then limited to prevent poluting more processes with a bad task.

## LLM's takeaways

Disclaimer: The above stories are slightly modified from the actual events, and they are mannually written by me, not generated by LLM.

I asked DeepSeek to write some takeaways as I was a bit tired after telling these stories. This is the version I'm happy with:

1. **Legacy safeguards breed hidden traps**
   Undocumented checks (Bug 1) and copy-pasted code (Bugs 2, 4) linger as systemic debt. Always audit dependencies before removal or reuse.

2. **Test failure *combinations*, not thresholds**
   Rate limits + slow queries (Bug 2), retries + packet loss (Bug 4): isolated safeguards fail when risks collide. Model interactions, not silos.

3. **Own lifecycle control explicitly**
   Destructor chaos (Bug 3) and hanging tasks (Bug 4) stem from assuming cleanup. Govern threads, memory, and resources with strict ownership.

4. **Instrument for why, not what**
   Missing alerts for config errors (Bug 1), slow queries (Bug 2), or hung processes (Bug 4) delay fixes. Metrics must expose root causes, not just symptoms.

**Core**: Systems fail in layers. Anticipate overlap, govern lifecycles ruthlessly, and monitor for causality—not just correlation.

Well, at least I hope so, for 2025.

I started keeping a Learning diary on math and tech since September 2024.

Before that, I was working on a series of math notes, in a Zen mode, for months. I annotated PDFs in Margin Note 4, and forged definitions, diagrams into Forester trees which are bi-directionally linked notes that can be rendered into beautiful web pages and LaTeX PDFs, until I'm satisfied. During that time, I didn't read news, unrelated books, and solely focused on the books and papers needed for polishing the notes. I held my breath until one note reached a certain stage, so I might move on to a next note.

It was a mental paradise.

Then came the learning diary idea. Since September, constant context switch had dominated me at work. My attention for learning, had inevitably become fragmented. I started to check out interesting stuff from social media, still about math and tech, they are indeed potential elements of what I might want to work on. So I wished to log these readings or just findings daily, so they are not lost in time.

The habit was well-kept, to the end of 2024, even when the load at work had become unbearably distracting and exhausting. The breadcrumbs would be a gold mine for future me.

But it was a disaster to the mental state of my mind.

During that time, I managed to go back to a few open-source projects that I was working on. Most of them have almost no users, but I still feel the obligation to keep them up to date. Usually I just had little time for each of them, so after some chores, I ran out of time and energy for new features, and moved on to chores for a next project. This partially reflected my mental state caused by my fragmented learning.

The world had become an endless curiosity for me, the potential to work on a new project inspired by an element just learned about, was dazzling.

When I stood at the beginning of 2025, I realized that I'm almost fruitless in 2024. I didn't finish the notes before September, although they are nice in their partially completed form. I didn't create anything in a whole after September. I enjoyed the reward of the journey, but came to the end empty-handed.

I no longer log my learning daily in 2025 due to various reasons. My work and life still consume a lot of my time at the beginning of a year, like always. I still log them, but almost just weekly.

April is coming, I would hope to reach a similar Zen mode like the better half of last year. Hence, this post is poured out of me.

The following are written in Markdown, and submitted as ~jonsterling/forester#87.

## TL;DR

This is the proposal I mentioned in #82. It's an informal RFC about what I wish to do about embedding both the code and the resource (primarily images) generated in forester output.

The code is there so that the resource generated is fully reproducible, the resource is there so that if the consumer of the forester output (e.g. browser or LaTeX) could gracefully downgrade to use the resource instead. And the pipeline handles regeneration when the source code or dependencies (e.g. imported external files, used assets) change.

## Feasibility

The proposal is already feasible with current forester implementation with some external add-ons, so it might not bring too much complexity to immediate forester development. It can be viewed as not a feature request, but a way to make use of existing features whose equivalents may or may not be included in future forester development, e.g. XML/XSLT etc. That said, it would be nice to have some degree of native support from forester to smooth things out. Actually, this proposal is inspired by how forester handles LaTeX diagrams embedding, with visions on more diagram tools and beyond.

Now that let's assume the feasibility is not a problem, and take a closer look at the approach.

## The customizable, hybrid Approach

The "customizable" part in the approach means the user can choose the kind of tools and pipelines (e.g. parameter customzation or post-processing) to generated the resource, as well as the format(s) of the resource. Currently, the only natively support tool is `latex`, with fixed parameters (e.g. no `--shell-escape`), and a pipeline to generate DVI then post-process it with `divsvgm`, and the format is SVG, with a naming convention of hashing the source code to guarantee browser-cache invalidation of the image whenever the source dode is changed. The pipeline also watch the source change and trigger the regeneration and copying related assets to the output, to an extent that it would be triggered by changes of upstream macro (as it would impact the source code) but not including some `.tex`/`.sty` files that's "imported" in the source code, but forester has no way of detecting that in the current implementation.

Imagine all these can be customized for a second, it would be a great productivity boost for the users, but would this bring a ton of complexity into forester? Not neccessarily. If we view forester as a kernel, then it could just provide a minimal infrastructure to allow all these to happen, instead of handling all tools, pipelins, and formats by itself.

The "hybrid" part in the approach, at least in the web context, means the code can be executed both in browser (browser-side render, BSR for short, which might result in a more interactive experience, e.g. WebGL animations), and in server-side generation (SSR). The technical possibility thanks to that JS libraries that can be executed both in browser and in a JS runtime on server-side (e.g. node.js, deno), and that native libraries can be executed both on server, and in browser as WASM, that includes programs written in C, Rust, Go and more.

In the LaTeX context (or equivalently Typst or other authoring tools), the hybrid approach means whether to execute the code inside the document environment to regenerate, e.g. to preserve the same font and style settings, or to just include the externally generated diagrams. The current forester-to-latex takes the former approach, and usually results in a more desired, style-consistent result, while the latter may have the advantage of stabibility and efficiency, as the diagram could have unexpected errors or deformations during regeneration, and it could be more time consuming to regenerate too many diagrams when you are just authoring some text, and expect to preview the result in real-time.

Now it become clear that this hybrid approach facilitates both the reproducibliity and the graceful downgrade, as long as there is a customizable pipeline to handle it.

## A teaser: authoring tool candidates

Let's get a bit more concrete about the tools that could be used in the customizable pipeline, and see how it opens to endless possibilities. Here are some candidates that I have experimented with:

- Rich text (including math, diagrams)
  - Markdown: a easy-to-write, source-code-readable format
    - both BSR and SSR capable
    - it plays well with forester markups, as fr XML nodes are transformed to HTML tags to be rendered as is by the markdown renderer
    - it can downgrade to plain text (but preserving rendered forester markups) when the markdown renderer is not loaded
    - experiment: test markdownit (via markdown-it )
    - it's at least very convenient for writing paragraph/link/list/footnote-heavy notes compared to forester markups
    - plus, users could write the same text for forester, or for external communication, such as this very proposal which is written in Markdown, submitted to forester ticket system, but also rendered in my forest
  - Typst
    - both BSR and SSR capable
    - it can be made to plays with forester markups, to support links/cross-references/citations etc.
    - it renders to text-selctable SVG, indistinguishable from ordinary text
    - besides text and math, it can be used as a modern diagram authoring tool with decent scripting support
    - experiment: test typst (via typst.ts )
  - Syntax-highlighted code blocks
    - both BSR and SSR capable
    - experiment: test syntax highlighting (via shiki )
- Static diagrams
  - Penrose: creating diagrams with a declarative language with separation of domain knowlege, layout constraints&styling, and concrete substances
    - both BSR and SSR capable
    - experiment: test penrose
  - Pikchr: an open-source text-to-diagram tool, the language is simple yet has great support for relative positioning and flexible path guiding
    - experiment: test pikchr
    - SSR first, BSR-capable thanks to WASM
  - D2: a modern diagram scripting language, preferred over PlantUML, Mermaid, and Graphviz, for its great layout support for complex diagrams and its declarative design.
    - no BSR experiments yet, but I have used its SSR with other static site generators
- Animated, interactive diagrams
  - WebGL
    - experiments:  , test interactions between ray-marching and 3D objects
    - still trying to work out a SSR pipeline, but combining `xvfb-run`, headless-gl, three.js, should make it possible, e.g. see this gist.
    - a less preferred alternative is to use the SVGRenderer addon from three.js, but its render result is visually worse than WebGLRenderer, per my experiment in test three.js to SVG.

For SSR for other output format, users could use XSLT to transform the XML output to make the downstream tools (e.g. LaTeX) to consume the source code, or the resource produced by the SSR pipeline. If neither is consumable, the same content could be excluded, using a macro like `webonly` to surround the forester markup.

Note that these experiments are mostly done for only BSR. SSR experiments are not done for forester, as I have learned the feasibility from my experiment with other static site generators. Without SSR gracefully downgrade, it will take a bit time to load on first visit.

## Elements of an implementation external to forester

Currently, forester can:

- consume `.tree` files, and output `.xml` files, from which the final HTML are produced by XSLT on browser side, or by LaTeX on server side
- allowing user to define macros, which could emitted HTML tags in the XML output (this could also be done as XML tags)
- forester 4.3 uses `fr:resource`, `fr:img`, `fr:resource-source` (with type and part attributes) for rendered SVG image, and the LaTeX source code, which is general enough to be used for other `type` of code that renders to images

To make it available to BSR, the users could:

- use XSLT to transform `fr:resource` tags to HTML tags with certain `class` attribute
- use CSS to render them to show the source code, or a loading animation, or a fallback image rendered by the SSR pipeline described below
- some JS can be loaded on demand to replace the the HTML tag with a rendered, potentially interactive HTML element

To make a SSR pipeline external to forester, the users could:

- define macros accepting the source code to emit `fr:resource`, `fr:img`, `fr:resource-source` following the forester practice on LaTeX diagrams, but with a filename specified by the user, instead of generated from the source code hash
- use a file change watcher (e.g. watchexec) to check the change of XML, then detect `fr:resource-source` presence, then render the source using external scripts to call the corresponding tools by the `type` attribute, and generate an image file with the specified filename
- use a browser cache bustering mechanism (e.g. use XSLT to generate a random number to be appended to the image URL) for reloading the latest image whenever the source code changes if the user refreshes the page

So far, BSR are implemented for the experiments mentioned above, and SSR could wait for the standardization of `fr:resource`, and the resolution of this proposal, or just use any other XML tags in a similar spirit.

## Concluding remarks

The text above is organized so that a reader might gradually see the full vision of the proposal, then the feasibility, and many possible path, downgraded or even upgraded.

To reiterate, this is already feasible with current forester implementation with some external add-ons, and will continue to work if the used features or their equivalents are available in future forester.

It might coincide with the direction of forester development (e.g. the on-going pipeline refactor might just provide the minimal native support needed for this approach to work more smoothly), or it might exceed what forester was originally designed for (thus should not pose a maintenance burden to forester). Hence this informal RFC.

I'm experimenting using Jon Sterling's Forester for building Zettelkasten-style forests of evergreen notes, like his Notes on duploid theory (source), CAIMEO's The Rabbit Hole (source) or Trebor’s forest (source).

I wish to use it to organize many definitions, theorems and discussions about the same mathematical concepts/topics.

My hope is to find these notes easier to reuse and refer to in blogs, papers, blueprints etc. Preferably, I would like to be able to author them in Forester markup, LaTeX, Markdown, Typst, and various diagram tools, freely and in a reproducible manner (see my post on this, and relevant experiments).