Four ways to count to five

2026-10-16T00:00:00+00:00
In February I wanted to see how small a Solana program could get without a framework. solana-program</code> had been split into small crates, so I wrote solana-fib-cpi</a> with six of them: solana-account-info</code>, solana-cpi</code>, solana-instruction</code>, solana-msg</code>, solana-program-entrypoint</code> and solana-pubkey</code>. No Anchor, no solana-program</code>.</p>
The program computes Fibonacci numbers in the most wasteful way I could think of. One instruction creates a PDA at ["fib", payer]</code> with 25 bytes, a</code>, b</code>, n</code> and a bump. Then the program calls itself through CPI, and every call advances (a, b, n)</code> to (b, a + b, n - 1)</code> until n</code> reaches zero. Solana allows direct self-recursion and refuses indirect reentrancy, and the CPI docs</a> put the stack depth at 5, so n = 4</code> is as far as one transaction goes. Four steps from a = 0, b = 1</code> ends at b = 5</code>. Hence the title.</p>
The tests passed. I meant to compare it with the new frameworks. The first thing I found was that it could not run on mainnet.</p>
It only ran in my tests
#</a>
</h2>
My tests registered the program with processor!</code>, which runs it as native Rust inside the test. For this post I built every version with cargo build-sbf</code> and ran the real .so</code> in LiteSVM</a> with the mainnet feature set. The February program failed on the first self-CPI:</p>
Program 3yDL675dKEnNZfvDz6dznB3n3qpXpiZJK4pSYkSAd1Xr invoke [1]
</span>Program 11111111111111111111111111111111 invoke [2]
</span>Program 11111111111111111111111111111111 success
</span>Program log: init: a=0 b=1 n=4
</span>Unknown program 3yDL675dKEnNZfvDz6dznB3n3qpXpiZJK4pSYkSAd1Xr
</span>Program 3yDL675dKEnNZfvDz6dznB3n3qpXpiZJK4pSYkSAd1Xr failed: An account required by the instruction is missing
</span></code></pre>
On mainnet, a program can only invoke a program that is one of its own instruction accounts, and calling yourself is no exception. My instruction passed the PDA, the payer and the System Program, never the program itself.</p>
Switching my tests from the native processor to the .so</code> did not catch it either. All six still passed. Agave 2.x has a feature gate, lift_cpi_caller_restriction</code>, that looks the callee up in the whole transaction instead, and solana-program-test</code> turns on every feature by default. That one was never activated on mainnet: its feature account does not exist, and newer Agave releases have dropped it. With the feature turned off in the test, three of the six fail with the same Unknown program</code>.</p>
The fix is a fourth readonly account, the program, forwarded on every self-CPI. It is in the repo</a> now, with tests that load the .so</code> and turn that feature off, and every version below has it. I wrote about the same trap in the post on my quant book</a>, from the other side: a test only proves what it actually runs, and here that includes the runtime's feature set.</p>
What changed since February #</a>
</h2>
Before measuring anything I checked what my February code was standing on.</p>

solana-program</code> now lives in anza-xyz/solana-sdk</a> and is a facade: version 5.1.0 is mostly re-exports of the small crates. The ones I used at 2</code> are now at 3.x and 4.x.</li>
Pubkey</code> is now Address</code>. SDK v3 introduced Address</code>, and Pubkey</code> is a type alias of it.</li>
Rent got cheaper. I had hardcoded (128 + len) * 3480 * 2</code>. SIMD-0194</a> replaced that pair with a single lamports_per_byte</code> of 6,960, and SIMD-0437 has been lowering it in steps since. Mainnet charges 5,080 per byte today, so my program overpaid by about 287,000 lamports per PDA, and the gap grows with each step down. Read Rent::get()</code>.</li>
The stack depth is still 5, so four nested CPIs. SIMD-0268 raises that to 8 nested CPIs and is accepted, but its feature is not active on mainnet yet.</li>
</ul>
What dropping solana-program</code> buys #</a>
</h2>
The standard entrypoint in solana-program-entrypoint</code> deserializes the input buffer into a Vec</code> of AccountInfo</code>, each one holding its lamports and data behind Rc<RefCell<...>></code>. That is a heap allocation and a set of reference counted cells before your code reads a single byte.</p>
Pinocchio</a> does the opposite. Its README describes it as a library with "no external dependencies" whose entrypoint works "with no copies or allocations". An AccountView</code> points into the input buffer the runtime already wrote, and the crate is no_std</code>, so a program can declare no_allocator!()</code> and have no heap at all. The account views now come from the SDK itself, solana-account-view</code> and solana-instruction-view</code>, and Pinocchio is a thin layer over them.</p>
The new frameworks build on that layer. Anchor v2 is the one most people will meet: its crates depend on Pinocchio 0.11, pinocchio-system</code> and a handful of split crates, and not on solana-program</code> at all. Its README</a> calls it "a drop-in speedup for Anchor v1" that is "up to 94% smaller and 3× faster or more per instruction". Those are its own numbers, so I measured mine.</p>
One program, four ways #</a>
</h2>
Same logic, same accounts, same 25 bytes of state (plus the 8 byte discriminator in both Anchor versions), four builds:</p>

February, the split crates at 2.x, with the account fix and nothing else.</li>
Anchor 1.2.1, the current stable release, on the split crates and their standard entrypoint.</li>
Pinocchio 0.11.2, with pinocchio-system</code> for the account creation.</li>
Anchor 2.0.0-rc.1, the first release candidate on crates.io.</li>
</ol>
All four were built with cargo build-sbf</code> 3.1.10 and platform-tools v1.52, and run in LiteSVM with the mainnet feature set. Each transaction runs init with n = 4</code>, five levels deep, and every version ends at a = 3, b = 5</code>. Logging costs a lot and every library logs differently, so the main builds have no logs. Every number held across three runs. The code and the benchmark are in examples/four-ways-to-count-to-five</a>, and just bench</code> reproduces the table.</p>
The Pinocchio version reads close to the February one, with views instead of infos:</p>
let</span> [</span>pda,</span> payer,</span> system,</span> program,</span> ..</span>]</span></span> =</span> accounts else</span> {</span>
</span></span>    return</span> Err</span>(</span>ProgramError::</span></span>NotEnoughAccountKeys</span>)</span></span>;</span>
</span></span></span>}</span></span>;</span>
</span>
</span>if</span> pda.</span>is_data_empty</span>(</span></span>)</span></span> {</span>
</span></span>    //</span> ... check the signer, the System Program and the PDA address
</span></span></span>    CreateAccount::</span></span>with_minimum_balance(</span>payer,</span> pda,</span> DATA_LEN</span>,</span> program_id,</span> None</span></span>)</span></span>?</span>
</span></span>        .</span>invoke_signed</span>(</span>&</span>[</span>Signer::</span></span>from(</span>&</span>seeds</span>)</span></span>]</span></span></span>)</span></span>?</span>;</span>
</span></span>    //</span> ... write a = 0, b = 1, n and the bump straight into the account
</span></span></span></span>}</span></span>
</span></code></pre>
Source: pinocchio/src/lib.rs</code></a></p>
Anchor v2 looks like Anchor, minus the lifetimes, with state that is read in place:</p>
#</span>[</span>derive</span>(</span></span></span>Accounts</span></span>)</span></span></span>]</span></span>
</span>pub</span> struct</span> </span>Fib</span> </span>{</span>
</span></span></span>    #</span>[</span>account</span>(</span></span></span>
</span></span></span></span></span></span>        init_if_needed,</span>
</span></span></span></span></span></span>        payer =</span> payer,</span>
</span></span></span></span></span></span>        seeds =</span> [b</span>"</span>fib"</span></span>,</span> payer.address</span>(</span></span></span>)</span></span></span>.as_ref</span>(</span></span></span>)</span></span></span>],</span>
</span></span></span></span></span></span>        bump =</span> pda.bump,</span>
</span></span></span></span></span></span>    </span></span>)</span></span></span>]</span></span>
</span></span></span>    pub</span> pda</span>:</span> Account<</span>FibState></span></span>,
</span></span></span>    #</span>[</span>account</span>(</span></span></span>mut</span></span>)</span></span></span>]</span></span>
</span></span></span>    pub</span> payer</span>:</span> Signer,
</span></span></span>    pub</span> system_program</span>:</span> Program<</span>System></span></span>,
</span></span></span>    pub</span> fib_program</span>:</span> Program<</span>FibProgram></span></span>,
</span></span></span></span>}</span></span></span>
</span>
</span>#</span>[</span>account</span>]</span></span>
</span>pub</span> struct</span> </span>FibState</span> </span>{</span>
</span></span></span>    pub</span> a</span>:</span> PodU64,
</span></span></span>    pub</span> b</span>:</span> PodU64,
</span></span></span>    pub</span> n</span>:</span> PodU64,
</span></span></span>    pub</span> bump</span>:</span> u8</span>,
</span></span></span></span>}</span></span></span>
</span></code></pre>
Source: anchor-v2/src/lib.rs</code></a></p>
The results:</p>
Version</th> .so</code> bytes</th> Init, n = 4 (CU)</th> Deepest level alone (CU)</th></tr></thead>

Anchor 1.2.1</td> 147,368</td> 33,686</td> 4,637</td></tr>
February, split crates 2.x</td> 41,616</td> 14,517</td> 878</td></tr>
Anchor 2.0.0-rc.1</td> 42,480</td> 17,505</td> 766</td></tr>
Pinocchio 0.11.2</td> 13,792</td> 7,595</td> 90</td></tr>
</tbody></table>
The deepest level is the last step, which does the Fibonacci arithmetic and returns without another CPI, so it is close to the fixed cost of entering each program.</p>
Reading the numbers #</a>
</h2>
Anchor v1 is the expensive one. It costs 4.4 times what Pinocchio costs for the whole transaction, and its binary is more than ten times larger. Part of that is my program shape: with init_if_needed</code> and a plain bump</code>, Anchor v1 runs find_program_address</code> and reads rent on every level, even when the account exists, because it cannot trust a stored bump on an account it might be creating. It also needs exit()</code> before the self-CPI and reload()</code> after, or the outer level writes its stale copy over the inner ones. A v1 program split into separate init and step instructions would pay less. It would also be a different program.</p>
Anchor v2 halves it: 17,505 CU against 33,686, and a binary three and a half times smaller. It reads the bump from the account (bump = pda.bump</code>), works on the account bytes in place, and needs no reload after the CPI. Its optional guardrails cost 88 CU, so turning them off is not where the difference is.</p>
The levels make the comparison cleaner. After init, every recursive step adds a constant amount in each version, and the CPI itself is the same runtime work for all four:</p>
Version</th> CU per recursive step</th></tr></thead>

Anchor 1.2.1</td> 6,687</td></tr>
Anchor 2.0.0-rc.1</td> 3,893</td></tr>
February, split crates 2.x</td> 2,606</td></tr>
Pinocchio 0.11.2</td> 1,160</td></tr>
</tbody></table>
Per step, v2 is 1.7 times cheaper than v1 for this program, not the "3× faster" in its README. The README benchmarks other programs, so both can be true. Pinocchio does the whole step for a third of what v2 spends, so most of v2's remaining cost is the framework.</p>
The hand-written split crates sit between them. The February version, without logs, is cheaper than Anchor v2 for the whole transaction and slightly more expensive at the deepest level. It still pays for the standard entrypoint, and it has none of Anchor's checks.</p>
Pinocchio is the floor, 90 CU for a level that reads 24 bytes, adds two numbers and writes them back. Most of what is left in its 7,595 is the CPIs themselves. The same direction shows up in SIMD-0266</a>, where the Token program rewritten on Pinocchio, p-token, takes Approve</code> from 2,904 CU to 124.</p>
Logs are not free either. The February program as I wrote it, with its msg!</code> calls, costs 18,316 CU. Removing them saves 3,799, about a fifth of the transaction.</p>
What Anchor v2 changes #</a>
</h2>

No <'info></code> lifetimes, and handlers take &mut Context<T></code>.</li>
Pubkey</code> becomes Address</code>, following the SDK.</li>
Account<T></code> is zero-copy by default, for T: Pod</code>. The Borsh behaviour of v1 moves to BorshAccount<T></code>.</li>
PDA bumps are computed at compile time when every seed is a literal.</li>
The crate is no_std</code>, with alloc</code> as a default feature.</li>
</ul>
It is still a release candidate. 2.0.0-rc.1</code> was published on 2026-08-12 and 2.0.0-rc.2</code> on 2026-10-12, and the v2 README still says "Alpha. Not audited". Building this example hit three rough edges: the #[program]</code> derive needed wincode</code> as a direct dependency, there was no generated program type to use in Program<T></code>, and the self-CPI went through a hidden __cpi_accounts_fib</code> module because the public cpi</code> module disables the entrypoint.</p>
What it costs #</a>
</h2>
Pinocchio's 90 CU come with no checks I did not write: the // ...</code> in the snippet above is where the signer, the System Program and the PDA address get checked by hand, and the February version is the same. Anchor v2 brings the checks back as a release candidate with the rough edges above. Anchor v1 is the stable release and the most expensive one in every column.</p>
Testing the real .so</code> costs a second harness: every program built with cargo build-sbf</code> and loaded into LiteSVM with the mainnet feature set, next to the processor!</code> tests, which passed for a program that never ran on mainnet.</p>
What I would do now #</a>
</h2>
For a program where compute matters, Pinocchio, with the account views from the SDK. For a program a team has to read and change for years, Anchor v2 once it is released, because it keeps the Anchor shape and drops most of the cost.</p>
The title has an expiry date. SIMD-0268 raises nesting to 8 CPIs once its feature is active on mainnet, and then the same program counts to 34 in one transaction. Which of the four pays most for the extra four levels is the next thing to measure.</p>


Solana without the fork
2026-10-13T00:00:00+00:00
Last week (Oct 9) a tweet from Claire Fan</a> showed up on my timeline: "For the first time in Solana history, we can generate Solana programs directly from upstream rustc and LLVM!" Starting with sbpf-linker</a> 0.2.3, a no_std</code> program builds with cargo +nightly build</code>. Claire also wrote: "Anything that doesn't work should be considered a bug, and I'll fix it!"</p>
I had to try it.</p>
Every program goes through a fork #</a>
</h2>
Every Solana program you have deployed was compiled by a fork. Solana runs programs in a virtual machine that executes sBPF, its own flavor of eBPF, and the compiler that produces it is platform-tools</a>: Anza's fork of rustc</a> and of LLVM</a>. cargo build-sbf</code> downloads it and builds with it.</p>
Keeping a fork of a compiler current is a lot of work, and it shows in the version numbers. Platform-tools v1.57, the release I tested with, ships rustc 1.95.0-dev from August 17. The nightly I used is rustc 1.101.0. Every Rust feature from those six versions reaches Solana programs late.</p>
sbpf-linker takes another route. Rust already has a bpfel-unknown-none</code> target, and upstream LLVM compiles to it. The linker takes the eBPF object that rustc produces and turns it into an sBPF v3 program. Everything Solana specific lives in a .cargo/config.toml</code>: no stack unwinding, a 4 KiB stack frame, a few instruction forms LLVM must not emit, and --cfg=target_os="solana"</code> so existing crates take their Solana code paths.</p>
The timing works because of sBPF v3. It moves syscalls to link time: a program calls a syscall by the murmur3 hash of its name, so the linker does not need to know anything about how the validator loads programs. I checked the feature account</a> on mainnet, and v3 deployment has been active since slot 428,976,000, on June 26. A program built this way can be deployed today. With Agave 4.4, expected in November, v3 becomes the only format</a> mainnet accepts for new deploys and upgrades.</p>
Reading it first #</a>
</h2>
I read code before I run it, and a linker runs on every build. This one is a pleasant read. sbpf-linker is a handful of Rust files, and it does nothing a linker should not: no network calls, no other processes, and it writes only the .so</code> it was asked for. The crates.io packages match their git tags file for file, every dependency comes from crates.io, and the release workflow pins its actions by commit and publishes with crates.io trusted publishing. The syscall hashes in the template and the builtins crate all match murmur3 of their names.</p>
Everything below runs in Docker, with sbpf-linker built from its crates.io source, apart from the local validator at the end.</p>
A counter #</a>
</h2>
The template</a> prints "Hello, world!", builds into a 240 byte .so</code>, and its test passes after one change I get to below. I wanted something closer to a real program: accounts, an owner check, data I write to, and errors.</p>
So I wrote a counter with pinocchio</a> 0.11.2. One instruction adds the u64 in the instruction data to the u64 stored in an account the program owns.</p>
use</span> pinocchio::</span></span>{</span>error::</span></span>ProgramError,</span> AccountView,</span> Address,</span> ProgramResult</span>}</span></span>;</span>
</span>
</span>pinocchio::</span></span>program_entrypoint!</span>(</span>process_instruction</span>)</span></span>;</span>
</span>pinocchio::</span></span>no_allocator!</span>(</span></span>)</span></span>;</span>
</span>pinocchio::</span></span>nostd_panic_handler!</span>(</span></span>)</span></span>;</span>
</span>
</span>pub</span> fn</span> </span>process_instruction</span></span>(</span>program_id</span>:</span> &</span>Address, accounts</span>:</span> &</span>mut</span> [AccountView], data</span>:</span> &</span>[u8</span>]</span>)</span></span></span></span> -></span> ProgramResult</span> </span>{</span>
</span></span></span>    let</span> [</span>counter]</span></span> =</span> accounts else</span> {</span>
</span></span></span></span>        return</span> Err</span>(</span>ProgramError::</span></span>NotEnoughAccountKeys</span>)</span></span>;</span>
</span></span></span></span>    </span>}</span></span>;</span>
</span></span></span>
</span></span></span>    if</span> !</span>counter.</span>owned_by</span>(</span>program_id</span>)</span></span> {</span>
</span></span></span></span>        return</span> Err</span>(</span>ProgramError::</span></span>IncorrectProgramId</span>)</span></span>;</span>
</span></span></span></span>    </span>}</span></span>
</span></span></span>
</span></span></span>    //</span> ... writable check
</span></span></span></span>
</span></span></span>    let</span> by:</span> [</span>u8</span>;</span> 8</span>]</span></span> =</span> data.</span>try_into</span>(</span></span>)</span></span>.</span>map_err</span>(</span>|</span></span></span>_|</span></span> </span>ProgramError::</span></span>InvalidInstructionData</span></span>)</span></span>?</span>;</span>
</span></span></span>    let</span> mut</span> bytes =</span> counter.</span>try_borrow_mut</span>(</span></span>)</span></span>?</span>;</span>
</span></span></span>    let</span> stored:</span> &</span>mut</span> [</span>u8</span>;</span> 8</span>]</span></span> =</span> (</span>&</span>mut</span> bytes[</span>..</span>]</span></span></span>)</span></span>.</span>try_into</span>(</span></span>)</span></span>.</span>map_err</span>(</span>|</span></span></span>_|</span></span> </span>ProgramError::</span></span>InvalidAccountData</span></span>)</span></span>?</span>;</span>
</span></span></span>    let</span> next =</span> u64</span>::</span></span>from_le_bytes(</span>*</span>stored</span>)</span></span>
</span></span></span>        .</span>checked_add</span>(</span>u64</span>::</span></span>from_le_bytes(</span>by</span>)</span></span></span>)</span></span>
</span></span></span>        .</span>ok_or</span>(</span>ProgramError::</span></span>ArithmeticOverflow</span>)</span></span>?</span>;</span>
</span></span></span>    *</span>stored =</span> next.</span>to_le_bytes</span>(</span></span>)</span></span>;</span>
</span></span></span>    Ok</span>(</span>(</span></span>)</span></span></span>)</span></span>
</span></span></span></span>}</span></span></span>
</span></code></pre>
This is a plain pinocchio program. Nothing in it knows which compiler built it. I added pinocchio, ran cargo +nightly build-bpf</code>, and it built on the first try.</p>
That is less luck than it looks. pinocchio already pulls in its syscalls for target_arch = "bpf"</code>, the upstream target, next to target_os = "solana"</code>, and solana-define-syscall</code> computes the murmur3 hashes at compile time when static-syscalls</code> is set.</p>
Five tests, two compilers #</a>
</h2>
Five Mollusk</a> tests cover the add and each way it should fail: an overflow, an account owned by another program, a read only account, and instruction data that is not 8 bytes. I built the same crate through platform-tools v1.57 for sBPF v3 too, and ran the same five tests against both .so</code> files.</p>
</th> Upstream and sbpf-linker</th> platform-tools v1.57</th></tr></thead>

Tests</td> 5 of 5</td> 5 of 5</td></tr>
.so</code> size</td> 3,824 bytes</td> 7,680 bytes</td></tr>
Add 5 to 37</td> 56 CU</td> 54 CU</td></tr>
</tbody></table>
Half the size, and two compute units apart. I ran the set three times and got the same numbers each time. Mollusk counts compute units, it does not time anything.</p>
On Solana, deploying a program means paying rent for every byte of it, so half the size is half the rent.</p>
On a real validator #</a>
</h2>
Mollusk runs the program, it does not deploy it. So I deployed the upstream .so</code> to solana-test-validator</code> from Agave 4.3.0, the version mainnet runs, started with --clone-feature-set</code> so it has exactly the features active on mainnet. The deploy went through, and three real transactions did what they should:</p>
Transaction</th> Result</th></tr></thead>

Create the counter and add 37</td> counter is 37</td></tr>
Add 5</td> counter is 42, in 56 CU, the same as in Mollusk</td></tr>
Add u64::MAX</code></td> rejected, "Program arithmetic overflowed"</td></tr>
</tbody></table>
I first tried to run the validator in Docker, like everything else here. Agave only ships Linux binaries for x86_64, so on my Mac it ran under emulation, and it would not start: Agave 4 needs io_uring, and the emulated container does not have it ("Function not implemented"). Lifting Docker's seccomp filter did not help. Agave could be built for arm64 inside Docker, but it is a large Rust build, so just validator</code> downloads Agave's own binary for your machine and runs it from there.</p>
One thing I changed #</a>
</h2>
It is cargo test</code>. The template sets build-std</code> under [unstable]</code>, which applies to the host build too, and the host build then fails with E0152, duplicate lang items. The builtins repo already has the fix in its own tests: put -Z build-std</code> in the alias, so only the BPF build uses it.</p>
[</span>alias</span></span>]</span>
</span>build-bpf</span></span> =</span> "</span>build --release --target bpfel-unknown-none -Z build-std=core,alloc"</span></span>
</span></code></pre>
It is a few lines, and a PR I plan to send.</p>
Where I would like to help #</a>
</h2>
The libcalls are the young part. LLVM turns some operations into calls to helper functions, and solana-compiler-builtins</code> provides them for this target. Multiply works and is cheap. u128 division does not link yet: the linker asks for __udivti3</code>, and after exporting it, for __bpf_trap</code>. An f64 divide works but costs 736 CU against 192 on platform-tools. My counter never divides, so it never hit either problem, but any program that computes a price will.</p>
What it costs today #</a>
</h2>
A pinned nightly.</strong> The target is Tier 3 in Rust, so core</code> is built from source along with your program. The linker repo pins a nightly and a bot updates it, so you need to pin yours too.</p>
no_std</code> only.</strong> pinocchio fits. I did not try Anchor or solana-program</code> on this path.</p>
Young.</strong> Version 0.2.3. I would test a program here for a while before it holds anyone's money.</p>
Try it #</a>
</h2>
The counter, a Dockerfile with both toolchains and a Justfile</code> are in examples/solana-without-the-fork</a>. just test</code> builds both .so</code> files and runs the five tests against each, and just validator</code> deploys the upstream one to a local validator and sends it real transactions.</p>
Thanks to Claire</a> and the Blueshift</a> team for doing the compiler and assembly work so the rest of us can build on stock Rust.</p>


To queue or not to queue
2026-10-12T00:00:00+00:00
Ten years ago my company ran on Beanstalkd</a>. Jobs went into tubes, each with a priority, an optional delay and a time to run. A worker reserved a job, and if it did not finish within that time the server released the job for someone else. A job that kept failing got buried, set aside where the server would never touch it again, until I kicked it back. At our peak that setup moved more than 500 million background jobs a month.</p>
I rarely see Beanstalkd mentioned now. The project is mature, which is a kind way to say its last commits were in March 2025, and the conversation moved to managed queues, Redis job libraries and Kafka. The ideas stayed. A time to run is a visibility timeout. A buried job is a dead letter. Kicking is a redrive. Every queue I have used since has the same few parts, with different names.</p>
This post is the follow up to Waiting and working in Rust</a>. That one was about running work concurrently inside one process. This one is about the step after: when work should leave the process and go through a queue, when it should not, and what you have to get right either way. The examples are in Rust and TypeScript, and they all run: examples/to-queue-or-not-to-queue</a>.</p>
What a queue buys you #</a>
</h2>
A queue sits between the code that decides work should happen and the code that does it. The producer does not wait, so a request can return in milliseconds while the slow part, an email, a webhook, a settlement, happens later. A burst of a thousand events turns into a steady stream the workers can keep up with, instead of a thousand things running at once. When a worker fails, the message is still there, and a durable queue keeps it across crashes and deploys. And one event can feed several consumers that know nothing about each other.</p>
Each of these has a price, and the price is the same everywhere: the work is no longer synchronous, so the caller cannot see its result, and the delivery guarantee you get is almost always at least once. SQS</a> says it plainly for standard queues: every message is delivered at least once, "but in some cases, a message may be delivered more than once". Most brokers work the same way by default. So the question to design for is how many times a job ran.</p>
When not to queue #</a>
</h2>
Most work does not need a queue.</p>
If the caller needs the result, a queue only adds a second round trip and a place for the answer to get lost. If the work finishes in the request, run it in the request. If all you want is to stop a thousand tasks from running at once, you want a limit on concurrency, which is a semaphore:</p>
let</span> limit =</span> Arc::</span></span>new(</span>Semaphore::</span></span>new(</span>2</span></span>)</span></span></span>)</span></span>;</span>
</span>let</span> handles:</span> Vec</span><</span>_</span>></span></span> =</span> (</span>0</span>..</span>30</span>u64</span></span>)</span></span>
</span>    .</span>map</span>(</span>|</span></span></span>i</span>|</span></span> </span>{</span>
</span></span></span></span>        let</span> limit =</span> Arc::</span></span>clone(</span>&</span>limit</span>)</span></span>;</span>
</span></span></span></span>        tokio::</span></span>spawn(</span>async move</span> {</span>
</span></span></span></span></span></span>            let</span> _permit =</span> limit.</span>acquire_owned</span>(</span></span>)</span></span>.</span>await.</span>unwrap</span>(</span></span>)</span></span>;</span>
</span></span></span></span></span></span>            settle</span>(</span>&</span>Transfer {</span> id:</span> format!</span>(</span></span>"</span>tx{i}</span>:0"</span></span></span>)</span></span>,</span> amount:</span> i </span>}</span></span></span>)</span></span>.</span>await;</span>
</span></span></span></span></span></span>        </span>}</span></span></span>)</span></span>
</span></span></span></span>    </span>}</span></span></span></span>)</span></span>
</span>    .</span>collect</span>(</span></span>)</span></span>;</span>
</span></code></pre>
Source: rust/src/bin/queue.rs</code></a></p>
Thirty settlements, two at a time, 330 ms on my machine, and nothing to deploy.</p>
Two of my own projects make the same choice for different reasons. The broker in passbox</a> serves one connection at a time with no runtime and no queue, because a biometric prompt blocks anyway. orch</a> does have work waiting, but its queue is a table: tasks live in SQLite and GitHub issues, and a tick loop picks up whatever is ready every ten seconds. When the work already lives in a database, a loop that reads it can be the whole queue.</p>
Postgres can serve a queue from an ordinary table with SELECT ... FOR UPDATE SKIP LOCKED</code>, which the docs</a> describe as a way "to avoid lock contention with multiple consumers accessing a queue-like table". If the database is already there, the queue can live in the same transaction as the data it is about.</p>
Queues in a database had a bad reputation for a long time, and it was earned. Workers polling one table all waited on the same row locks, so a busy queue turned into lock contention. SKIP LOCKED</code> fixed that part, but it still has costs. Every claim and every completion is a write, and in Postgres the old row versions stay until vacuum removes them, so a busy queue table needs vacuum to keep up. Workers poll unless something wakes them. The queue shares the database with everything else, so a backlog slows the application. And there is no fan out or replay. For jobs tied to data in the same database, at a volume it handles comfortably, it is a good queue. When the queue starts to compete with the app, or several services need the same events, it is time for a broker.</p>
An event bus is not a queue #</a>
</h2>
In TypeScript the first thing to reach for is an event emitter, which is not a queue. Node's EventEmitter</code></a> "calls all listeners synchronously in the order in which they were registered". There is no buffer and no retry, and nothing survives a crash. An async listener makes it worse, because emit</code> does not wait for it:</p>
const</span> bus</span></span> </span>=</span> new</span> EventEmitter</span>(</span>{</span> captureRejections</span></span>:</span></span> true</span> </span>}</span></span>)</span></span></span>
</span>
</span>bus</span>.</span>on</span></span>(</span>'</span>transfer'</span></span>,</span> (</span>id</span>:</span> string</span></span>)</span></span> </span>=></span> </span>console</span>.</span>log</span></span>(</span>`</span>saved ${</span></span>id</span></span>}</span></span>`</span></span>)</span>)</span>
</span>bus</span>.</span>on</span></span>(</span>'</span>transfer'</span></span>,</span> async</span> (</span>id</span>:</span> string</span></span>)</span></span> </span>=></span> {</span>
</span></span></span>  await</span> new</span> Promise</span>(</span>r</span> </span>=></span> </span>setTimeout</span></span>(</span>r</span>,</span> 10</span>)</span>)</span></span>
</span></span></span>  throw</span> new</span> Error</span>(</span>`</span>notify failed for ${</span></span>id</span></span>}</span></span>`</span></span>)</span></span>
</span></span></span>}</span></span></span>)</span>
</span>bus</span>.</span>on</span></span>(</span>'</span>error'</span></span>,</span> (</span>e</span>:</span> Error</span></span>)</span></span> </span>=></span> </span>console</span>.</span>log</span></span>(</span>`</span>error event: ${</span></span>e</span>.</span>message</span></span>}</span></span>`</span></span>)</span>)</span>
</span>
</span>console</span>.</span>log</span></span>(</span>'</span>before emit'</span></span>)</span>
</span>const</span> hadListeners</span></span> </span>=</span> bus</span>.</span>emit</span></span>(</span>'</span>transfer'</span></span>,</span> '</span>tx1:0'</span></span>)</span></span>
</span>console</span>.</span>log</span></span>(</span>`</span>after emit, returned ${</span></span>hadListeners</span></span>}</span></span>, nothing was awaited`</span></span>)</span>
</span></code></pre>
Source: typescript/emitter.ts</code></a></p>
before emit
</span>saved tx1:0
</span>after emit, returned true, nothing was awaited
</span>error event: notify failed for tx1:0
</span></code></pre>
The sync listener ran inside emit</code>. The async one failed later, and the error only reached anyone because of captureRejections</code>. Without it, the Node docs warn, an async handler "can lead to an unhandled rejection". An event bus decouples code inside one process, and anything on it is gone when the process dies.</p>
A queue inside the process #</a>
</h2>
Between "no queue" and "a broker" there is a queue in memory, and this is where Rust and TypeScript differ.</p>
Tokio gives you the queue and its backpressure. A bounded mpsc</code></a> channel "provides backpressure": send</code> waits until there is room. The unbounded version is documented as having "the ability of causing the process to run out of memory". Two workers sharing one receiver, with a set of ids they have already handled:</p>
let</span> (</span>tx,</span> rx</span>)</span></span> =</span> mpsc::</span></span>channel::</span></span><</span>Transfer></span></span>(</span>4</span></span>)</span></span>;</span>
</span>let</span> rx =</span> Arc::</span></span>new(</span>Mutex::</span></span>new(</span>rx</span>)</span></span></span>)</span></span>;</span>
</span>let</span> seen =</span> Arc::</span></span>new(</span>Mutex::</span></span>new(</span>HashSet::</span></span>new(</span></span>)</span></span></span>)</span></span></span>)</span></span>;</span>
</span>
</span>for</span> _</span> in</span> 0</span>..</span>2</span> {</span>
</span></span>    let</span> rx =</span> Arc::</span></span>clone(</span>&</span>rx</span>)</span></span>;</span>
</span></span>    let</span> seen =</span> Arc::</span></span>clone(</span>&</span>seen</span>)</span></span>;</span>
</span></span>    tokio::</span></span>spawn(</span>async move</span> {</span>
</span></span></span></span>        loop</span> {</span>
</span></span></span></span></span>            let</span> Some</span>(</span>t</span>)</span></span> =</span> rx.</span>lock</span>(</span></span>)</span></span>.</span>await.</span>recv</span>(</span></span>)</span></span>.</span>await else</span> {</span> break</span> </span>}</span></span>;</span>
</span></span></span></span></span>            if</span> !</span>seen.</span>lock</span>(</span></span>)</span></span>.</span>await.</span>insert</span>(</span>t.</span>id.</span>clone</span>(</span></span>)</span></span></span>)</span></span> {</span>
</span></span></span></span></span></span>                continue</span>;</span> //</span> delivered twice, already settled
</span></span></span></span></span></span></span>            </span>}</span></span>
</span></span></span></span></span>            settle</span>(</span>&</span>t</span>)</span></span>.</span>await;</span>
</span></span></span></span></span>        </span>}</span></span>
</span></span></span></span>    </span>}</span></span></span>)</span></span>;</span>
</span></span></span>}</span></span>
</span></code></pre>
Node has no bounded channel in the standard library, so the TypeScript version writes one: a put</code> that waits while the queue is full and a take</code> that waits while it is empty. It is about forty lines in typescript/queue.ts</code></a>. Both versions push 40 messages, ten of them duplicates, through a queue of four with two workers:</p>
rust:        settled 30 of 40 messages (30 unique) in 324ms, slowest send waited 22ms
</span>typescript:  settled 30 of 40 messages (30 unique) in 315ms, slowest put waited 21ms
</span></code></pre>
The numbers match because the work is waiting, and both runtimes wait well. The differences are elsewhere. Tokio can run those two workers on two threads, and the compiler makes the shared set prove it is safe to share, with Arc</code> and a Mutex</code>. Node runs both workers on one thread, so the set needs no lock, and nothing in the language stops you from forgetting the bound. Tokio's channel is bounded unless you pick the unbounded one on purpose. In TypeScript a bound exists only if you write it.</p>
The duplicate check is the important line in both. A worker that does the work twice for a duplicate message has a bug, because the queue only ever promised at least once. The set in memory is the demo. In production the idempotency key goes into the database with the result, in the same write.</p>
Events in, a queue out: an indexer #</a>
</h2>
The design I like most for this came from an indexer I built in TypeScript for a Solana program. It used both kinds of events, each for what it is good at.</p>
The producer side received transactions by webhook. An RPC provider posted every transaction that touched the program to an authenticated endpoint, the indexer parsed and validated each one, and the parser emitted typed events on an in-process event bus: one per program event, one per parsed instruction and one per parsed transaction. Listeners were small classes with an optional when</code> guard and a handle</code>. One saved the parsed instructions to the database, a couple sent notifications, and one republished every program event to a Pub/Sub topic, with the event type as a message attribute.</p>
Webhooks were the choice on purpose. A websocket subscription such as logsSubscribe</code></a> works too, but it only delivers while the socket is connected, and a dropped connection is a silent gap. A webhook from a provider that retries until it gets a success response is more reliable, because the retry is the provider's job: the endpoint answers 202</code> when it has the batch and 500</code> when it does not.</p>
That last listener is the boundary. Everything before it is fast, local and in one process. Everything after it is durable and can fan out to any number of services that never import the indexer.</p>
The consumer side was a worker on a Pub/Sub subscription. It dispatched each message to a handler by its type attribute, and each handler could set retries, a delay, a unique lock so the same job never ran twice at once, and whether to ack on start or on finish. It acked on success, and on failure waited and nacked so Pub/Sub would deliver it again. When it was at its limit of 30 running jobs, it nacked new messages instead of taking them.</p>
The shape holds up. Reading it again next to the docs, these are the parts I would change.</p>


A backfill for when the provider gives up. Webhook retries do not last forever. A stored cursor, the last signature processed, and a backfill with getSignaturesForAddress</code></a>, which pages through history with before</code> and until</code>, up to 1,000 signatures per call, would close any gap the provider leaves.</p>
</li>

A retried batch re-emits events. The parser emitted program events while it parsed. If one transaction in a batch failed after earlier ones had already emitted, the endpoint answered 500</code>, the provider sent the whole batch again, and those events fired twice, including the publish to Pub/Sub. Each event needs a key that does not change between attempts, such as the transaction signature plus the instruction index, and every consumer has to treat a second copy as already done.</p>
</li>

The 202</code> went out before the listeners finished. The endpoint emitted the events and answered right away, and as the event bus section showed, emit</code> does not wait for async listeners. A listener that failed after that, the Pub/Sub publish included, never caused a 500</code>, so the provider never retried it. Awaiting the listeners that matter before answering, or writing to an outbox first, puts that failure back where the retry can see it.</p>
</li>

Let the subscription do the retries. Nacking to wait brings the message straight back. A retry policy</a> on the subscription backs off instead, and the client's flow control caps the messages in flight without nacking. The attempt counter the worker read was also always 0</a> without a dead letter topic, so retry limits based on it never fired.</p>
</li>

Publish from the database, not next to it. Saving to the database and publishing to Pub/Sub were two separate steps, so a crash between them loses one. The transactional outbox</a> writes the message into the database in the same transaction as the data, and a separate process publishes it. It can publish twice, which is fine, because the consumers are idempotent anyway.</p>
</li>
</ul>
Events all the way down #</a>
</h2>
The biggest version of this I built was the automation engine at Bulldesk, running on the same servers and queues as those 500 million jobs a month. A customer drew a campaign as a flow of steps, for example: wait two days, send an email, if they opened it add a tag, otherwise send a text. Every step was a job, and almost everything between steps was an event.</p>
It started with domain events. A lead was created, visited a page, had an order change status, moved stage in a CRM. Each of those fired an event, and a queued listener picked it up on its own listeners</code> queue, 15 seconds later. The listener found the active campaigns whose trigger matched that event and dispatched the first step of each one as a job. I wrote up a small piece of the same idea in 2018, in Refactoring to Events</a>: a Stripe webhook controller that maps each webhook type to its own Laravel event, with listeners doing the work.</p>
Every step job began with one more event, ActionStart</code>, dispatched so that the first listener to answer false</code> stopped the job. Eight listeners voted in order: is this step already running, was it deleted, is sending disabled, is it inside the campaign's schedule, is the lead still in the campaign, was this step already applied to this lead, is it a decision, do the segments still match. Each one was a few lines and checked one thing. A job that got a false</code> was recorded as skipped.</p>
When a step finished, its outcome was an event too: applied, not applied or skipped. Their listeners released the lock, wrote the log, scheduled the yes or no branch of a decision, and queued the next step, which is how a campaign moved forward without anything holding the whole flow in memory.</p>
Order mattered inside one lead's campaign, and almost nowhere else, so that is the only place the engine kept it. A lead's steps ran one at a time even though each one was queued: a step dispatched the next only after it finished, delayed by the next step's own wait plus ten seconds, on its own automation</code> queue. Across leads, everything ran in parallel. A bulk send to a segment walked its leads and dispatched one job per lead to one of five sending queues, picked at random, with a random delay of up to 90 seconds, so a send to a large list spread across workers and across time instead of landing all at once. It also stored its position as it went and stopped when paused, so a paused or crashed send picked up where it left off. The queued listeners on one event were separate jobs and ran side by side, while the guards on ActionStart</code> ran inline and in order, because each one had to answer before the next could decide.</p>
The chain was the second design. The first one queued every action of a campaign the moment a lead entered it, each delayed to the time it should run: three weekly emails were three jobs sitting in the queue for weeks. That made a lot of small problems. When the one in the middle failed, the rest were already queued behind it with nothing tying them to it. Editing the third email was fine, because a queued job held a reference to the email, its class and id, and loaded the content when it ran. Changing the step was not. If the customer swapped that email for another one, the queued job pointed at a deleted email and did nothing. If they replaced it with a different action, the job for the old action was still waiting in the queue. And if they added actions to the campaign, leads already in it never got them, because their whole campaign had been queued the day they entered. So I changed it to queue only the first action, and have each action, when it finished, check for a next one and queue that, until the campaign ended. With the chain, a step was queued at most one wait ahead, and the guards on ActionStart</code> checked the campaign again when it ran.</p>
That was the point of splitting a campaign into small actions, each its own step. Every action did one thing, so it finished fast, and when something was wrong it failed fast, at one step, without touching the steps already done. A step that failed went to a failed jobs table in MySQL and its error to an error tracker, and from there I could retry one job or all of them. A retried step that succeeded queued the next one, so an exception in one action no longer broke the campaign for that lead, it carried on from where it stopped. Every piece was small and could change on its own. The trade is that the flow of a campaign lives in the event map rather than in any one file, so following one lead through it means reading the logs.</p>
Those logs had a cost of their own. Every step wrote to the log every time it ran, for every lead in every campaign, all into the same MySQL database the app ran on. The same table was also the duplicate check. Before a step ran, the guard for "already applied" looked in it for that action, lead and campaign, and skipped the step if it found a row, unless the campaign was set to repeat or the lead was added by hand. So every step read the table before it wrote to it. Writes never stopped, and in the end the database could not keep up with them. When we exited, I was looking for a different kind of database for the automation logs, one built to take a lot of writes at once.</p>
Locks: one message, one worker #</a>
</h2>
With one worker, taking a message is enough. With several, each message needs a lock, usually called a lease, so only one worker has it at a time. Without it, two workers read the same message and both run it.</p>
The brokers give you the lease. Beanstalkd's time to run, SQS's visibility timeout</a> and Pub/Sub's ack deadline</a> all hide a message from other workers while one has it.</p>
A table that workers poll has none of this unless you build it. In Postgres it is one query: SKIP LOCKED</code> passes over the rows another worker is claiming right now, and a locked_until</code> column is the lease after that:</p>
UPDATE</span> jobs
</span>SET</span> locked_until =</span> now(</span>)</span></span> +</span> interval '</span>30 seconds'</span></span>,
</span>    attempts     =</span> attempts +</span> 1</span>
</span>WHERE</span> id =</span> (
</span>  SELECT</span> id FROM</span> jobs
</span>  WHERE</span> run_at <=</span> now(</span>)</span></span>
</span>    AND</span> (locked_until IS</span> NULL</span> OR</span> locked_until <</span> now(</span>)</span></span>)
</span>  ORDER BY</span> id
</span>  LIMIT</span> 1</span>
</span>  FOR UPDATE</span> SKIP LOCKED
</span>)
</span>RETURNING id, payload, attempts;
</span></code></pre>
Source: postgres/claim.sql</code></a></p>
I ran it with workers claiming at the same time against three jobs, using postgres/demo.sh</code></a>:</p>
worker A holds job 1 in an open transaction
</span>worker B: claimed 2 attempt 1      (skipped the row A was claiming)
</span>worker C: claimed 3 attempt 1
</span>worker D: nothing                  (job 1 is locked, 2 and 3 hold a lease)
</span>job 1's lease expires
</span>worker E: claimed 1 attempt 2      (the work outlived its lease)
</span></code></pre>
The last line is the catch. A lease is a timer, and if the worker only acknowledges at the end, any job that runs longer than the lease goes to a second worker while the first is still on it. Now it runs twice, at the same time. There are two ways out. Acknowledge at the start, and a crash loses the job, which is at most once. Or acknowledge at the end and extend the lease while you work. SQS and the Pub/Sub clients both let a worker extend its lease while it works. The indexer's worker made this a choice per handler, with an option to ack on start.</p>
A lease covers one message. Two different messages about the same thing, two deposits for the same account, need a lock on the thing itself. The indexer had that too, a unique lock per job key, but it lived in the memory of one worker process. With two instances, each had its own set of locks, and both could run the same job. A lock that several workers share has to live somewhere they all see: a row lock in the database the work writes to, or a Redis key set with SET key token NX EX seconds</code></a>, released by a script that deletes it only if it still holds your token.</p>
Bulldesk's engine used the simple version, and it shows the catch from the other side. It ran on several servers with several queues, and its cache was Redis, so a lock in the cache was shared by every worker on every machine. The first guard on every step took one with the cache's add</code>, keyed on the step, the lead and the sending, for 60 seconds. Laravel's add</code> was a single Lua script, check that the key does not exist and set it with an expiry, and Redis runs a script atomically, so two workers could never both get it.</p>
The timing left a gap, even if it rarely mattered. Actions were fast, and when something was wrong they failed fast, but the step jobs were allowed to run for 120 seconds, twice the lock. A step still running after a minute would have lost its lock, and a second worker could have started the same step for the same lead. The release had the matching gap: it was a plain delete, so a step that finished after its lock expired could delete the lock another worker had just taken, which is exactly the case the Redis docs' token check exists for. Fast work hides both. The rule for work that can be slow is that the lease outlasts the longest the work is allowed to take, and only the holder can release it.</p>
Retries and backoff #</a>
</h2>
Every queue retries. The part you design is when to stop and how long to wait.</p>
Errors come in two kinds. A timeout, a 429 or a dropped connection may succeed next time. A bad payload or an unknown account will fail forever, and retrying it only burns time and hides it. Retry the first kind, send the second straight to a dead letter queue, and give up on the first kind after a limit.</p>
The wait between tries should grow, and it should be random. Exponential backoff alone makes every worker that failed together retry together. Full jitter waits a random time between zero and the exponential ceiling, which spreads them out:</p>
//</span> Full jitter: a random wait between zero and the exponential ceiling
</span></span>fn</span> </span>backoff</span></span>(</span>attempt</span>:</span> u32</span></span>)</span></span></span></span> -></span> Duration</span> </span>{</span>
</span></span></span>    let</span> ceiling =</span> BASE</span>.</span>saturating_mul</span>(</span>1</span> <</span><</span> attempt.</span>min</span>(</span>16</span></span>)</span></span></span>)</span></span>.</span>min</span>(</span>CAP</span></span>)</span></span>;</span>
</span></span></span>    let</span> random =</span> RandomState::</span></span>new(</span></span>)</span></span>.</span>build_hasher</span>(</span></span>)</span></span>.</span>finish</span>(</span></span>)</span></span>;</span>
</span></span></span>    Duration::</span></span>from_millis(</span>random %</span> (</span>ceiling.</span>as_millis</span>(</span></span>)</span></span> as</span> u64</span> +</span> 1</span></span>)</span></span></span>)</span></span>
</span></span></span></span>}</span></span></span>
</span></code></pre>
//</span></span> Full jitter: a random wait between zero and the exponential ceiling</span>
</span>const</span> backoff</span></span></span> </span>=</span> (</span>attempt</span>:</span> number</span></span>)</span></span> </span>=></span> </span>Math</span>.</span>random</span></span>(</span>)</span> *</span> Math</span>.</span>min</span></span>(</span>CAP_MS</span>,</span> BASE_MS</span> *</span> 2</span> *</span>*</span> attempt</span>)</span></span>
</span></code></pre>
Source: rust/src/bin/retry.rs</code></a>, typescript/retry.ts</code></a></p>
Both versions run the same four messages, with a base of 100 ms, a cap of 2 seconds and five attempts:</p>
ok: settled on attempt 1
</span>flaky: settled on attempt 3
</span>dead letters: bad (unknown account), down (gave up after 5: 503)
</span></code></pre>
In Rust the two kinds of error are an enum, and the match</code> will not compile if a new kind is added and not handled. In TypeScript they are two classes and an instanceof</code> check, and an error from somewhere else falls through to "permanent" unless you remember to classify it.</p>
Exactly once is always scoped #</a>
</h2>
Every broker that offers exactly once means something narrower than the words.</p>

SQS FIFO</strong> deduplicates by deduplication id, but only within a 5 minute window</a>.</li>
Pub/Sub</strong> exactly-once delivery only applies</a> "when subscribers connect to the service in the same region", and push subscriptions do not support it.</li>
Kafka</strong> has an idempotent producer and transactions that write to several partitions atomically. Its design docs</a> are clear that exactly once "for other destination systems generally requires cooperation with such systems".</li>
</ul>
That other system is your consumer. The database write, the webhook or the token transfer at the end of the pipeline is outside every one of those guarantees. Make that step idempotent, and at least once is enough.</p>
Choosing one #</a>
</h2>
</th> Delivery</th> Ordering</th> Replay</th> Fits</th></tr></thead>

SQS</a></td> at least once, FIFO dedup for 5 min</td> best effort, or per message group with FIFO</td> no, retention up to 14 days</td> jobs on AWS, little to operate</td></tr>
Pub/Sub</a></td> at least once, exactly once in region</td> per ordering key</td> retention up to 31 days</td> fan out on GCP</td></tr>
Kafka</a></td> at least once, transactions inside Kafka</td> per partition</td> yes, consumers rewind their offset</td> event logs many systems read</td></tr>
Redis Streams</a></td> at least once with consumer groups</td> per stream</td> while the stream is kept</td> already running Redis</td></tr>
Postgres SKIP LOCKED</code></a></td> what your transaction gives you</td> whatever you order by</td> it is a table</td> work tied to data in the same database</td></tr>
Beanstalkd</a></td> at least once, time to run</td> priority, then arrival</td> no</td> fast, simple job queues on your own servers, with no replication</td></tr>
</tbody></table>
Kafka is the odd one. A queue deletes a message once it is acked, and in Kafka events "are not deleted after consumption"</a>, so a new consumer can read the whole history. That makes it an event log more than a queue, and it is the right tool when the events themselves are the record. It is moving toward queues too: Kafka 4.2 made share groups</a> production ready, where consumers share a partition and acknowledge each message on its own, instead of one consumer per partition committing an offset.</p>
What it costs #</a>
</h2>
Duplicates are the default.</strong> Every option above delivers at least once in its normal mode. A consumer that is not idempotent breaks on its first retry.</p>
Ordering costs throughput.</strong> Ordering keys, message groups and partitions all limit parallelism to one consumer per key. Ask for ordering only where the domain needs it, usually per account or per user, never globally.</p>
A queue hides failure.</strong> A request that fails shows an error. A job that fails sits in a dead letter queue that nobody reads. A dead letter queue needs an alert and an owner.</p>
Another system to run.</strong> A broker is a service with its own outages, limits and bill. The table you already have, or a semaphore, costs nothing.</p>
The rule I use #</a>
</h2>
Run it in the request if the caller needs the answer. Use a semaphore if the problem is too much at once. Use an in-process queue when one process does the work and losing it on a crash is acceptable. Use a broker when the work must survive the process, or when more than one service needs to hear about it. Whichever you pick, give every message a key, make every consumer idempotent, retry only what can succeed, and put an alert on the dead letters.</p>


My finances in plain text
2026-10-09T00:00:00+00:00
Since 2022 every real I spend goes into a text file. Bank accounts, credit cards, investments and my consulting company across many Brazilian banks, plus DeFi positions. About 13,400 transactions so far, and most of them I never typed.</p>
The file is a Beancount</a> ledger. The part that took years is everything around it: getting the statements, turning them into transactions, putting each one in the right account, and doing that every day without me.</p>
Today about 95% of it runs on its own. My part is small: a few statements only exist behind a bank login, so I download those and email them to myself, and now and then I fix a category the system got wrong. Downloading, importing, categorizing, checking balances, removing duplicates and archiving all happen without me.</p>
Why plain text #</a>
</h2>
I tried the usual options first.</p>
Finance apps</strong> are nice for a few months. Then they change the pricing, drop a bank, or shut down, and the history goes with them. You also cannot ask them a question they did not plan for.</p>
A spreadsheet</strong> holds everything and checks nothing. A typo in a formula moves money between categories and nobody notices for a year.</p>
Open Finance aggregators</strong> exist in Brazil, but they are built for companies, they do not cover all my accounts, and they put a third party between me and my bank data.</p>
Beancount is double entry accounting in a text file. Every transaction has to balance, and I can assert that an account holds an exact amount on a given day. If the numbers do not match the bank, bean-check</code> fails. It lives in git, so every change has a diff. And because it is text, I can query it with SQL, read it with Fava</a>, and hand it to an agent.</p>
The catch is that Beancount gives you the format and nothing else. Brazilian banks do not give individuals an API, they give you a PDF by email.</p>
Getting the statements #</a>
</h2>
Every account has an entry in a config file that says where its statements come from:</p>
bank_a_credit_card</span></span>:</span>
</span>  download_method</span></span>:</span> email</span>      #</span> Proton Bridge over IMAP
</span></span>bank_b_credit_card</span></span>:</span>
</span>  download_method</span></span>:</span> mail_app</span>   #</span> AppleScript against Mail.app
</span></span>defi</span></span>:</span>
</span>  download_method</span></span>:</span> api</span>        #</span> on-chain events
</span></span></code></pre>
Most banks send the monthly bill to my email, so the downloader searches the mailbox, grabs the attachment, and drops it in a folder per bank. Some accounts arrive through Proton Bridge, others through Mail.app because they go to a different address. DeFi has no statement at all, so that importer reads the contract events straight from the chain.</p>
A few statements never reach my inbox. They only exist behind a bank login with 2FA. For those I download the file and email it to myself with a fixed subject. From there the pipeline treats it like any other bill. This is the main manual step left, a few minutes a month, and I stopped fighting it.</p>
The importers #</a>
</h2>
An importer reads one kind of file and returns Beancount entries. I wrote fourteen of them on top of beangulp</a>: PDF credit card bills, bank statements as PDF, spreadsheet and CSV, brokerage notes, my company's invoices and DeFi events.</p>
Each importer has a test with a real statement next to it and the expected output in a .beancount</code> file. When a bank changes its PDF layout, the test fails before the ledger gets wrong numbers.</p>
None of these rules were new to me. I have worked with credit cards and finance for years, and I used to reconcile these statements by hand. The work was turning what I already knew into code:</p>
Credit card dates are not transaction dates.</strong> A bill closes on some day of the month, and the balance on the bill only makes sense after that day. So the importer reads the close date from the PDF, moves older purchases to the day after it, and writes the balance assertion two days after close. Beancount checks balances at the start of the day, so that order matters.</p>
Installments repeat across bills.</strong> "Amazon (5/10)" shows up ten months in a row. The importer tags them #recurring</code> so I can see how much of next month is already spent.</p>
One bank sends image PDFs.</strong> There is no text to extract, so that importer renders each page at 300 DPI and runs Tesseract in Portuguese. Tesseract reads the "C" for credit as "€" often enough that the code replaces it.</p>
The importer owns the balance assertion.</strong> Every statement ends with the balance the bank says I have. The importer writes that as a balance</code> directive, so the moment I import, Beancount tells me if something is missing. I never let it add pad</code> directives on its own. A pad hides a difference, and the difference is the thing I want to see.</p>
Categorization #</a>
</h2>
A transaction from a bill says "PG *LOJA CENTRO 03/06" and nothing else. Someone has to decide that it is household shopping.</p>
The first pass is rules. A categorizer wraps every importer and matches the payee against a list of regular expressions, one list per account:</p>
"</span></span></span>Expenses:Shopping:Household"</span></span></span>:</span> [</span>
</span></span>    "</span></span></span>^(Mercado Livre|Leroy|Amazon|Havan|...)"</span></span></span>,</span>
</span></span>]</span></span>,
</span>"</span></span></span>Income:Cashback"</span></span></span>:</span> [</span>
</span></span>    "</span></span></span>^(Cashback|Benefício do cartão)"</span></span></span>,</span>
</span></span>]</span></span>,
</span></code></pre>
Today that is 65 accounts and around a thousand merchant patterns, collected one annoying transaction at a time. Specific rules go first, so a broad pattern does not catch a merchant that belongs somewhere else. A category that starts with !</code> flags the transaction for review instead of trusting the match.</p>
What the rules miss goes to smart_importer</a>, which trains on my own history and guesses the payee and the account. It is right most of the time on merchants it has seen before and useless on new ones, which is why the rules come first.</p>
When both get it wrong, I fix the category by hand. That is the other thing I still do. It happens a few times a month, mostly with a new merchant, and the fix often becomes a new pattern so the same merchant does not come back.</p>
Duplicates #</a>
</h2>
This took longer than everything else.</p>
The same money shows up in more than one place. A PIX between two of my banks appears in both statements. A bill gets downloaded twice. beangulp has a flag that compares new entries with the ledger and comments out duplicates, but by default it only checks that amounts are within 5% and dates within two days. Two purchases of the same price on the same day look like one.</p>
So every card importer overrides the comparison and also requires the description to match. Transfers between my own accounts needed their own rule, restricted to the same pair of accounts, because a loan payment of R$ 3,500.00 should not match a PIX of R$ 3,500.00 just because the numbers are equal.</p>
Running it every day #</a>
</h2>
All of this used to be a monthly evening with coffee. Now orch</a>, the tool I built to run coding agents on a schedule, does it every morning at 8.</p>
The job is a markdown prompt. It tells an agent to read the config, download new statements for every automated account, run the close pipeline, run bean-check</code>, archive the files, and send me a short Telegram message with what came in and what is behind. The agent follows a skill file that holds everything I learned about the importers, so it does not have to rediscover the date rules or the dedup bugs each time.</p>
The agent is not doing the accounting. The importers and Beancount do that, and they are deterministic. The agent is there for the parts that break: Proton Bridge is not running, a bank changed the file name, a statement came in a format the importer has not seen. It reports those instead of guessing.</p>
On a normal day I read the Telegram message and do nothing else.</p>
Prices and everything else #</a>
</h2>
Once the ledger is complete, it can answer more than "how much did I spend". Price sources pull SELIC, CDI and IPCA from the Central Bank, stock prices from brapi, US macro data from FRED, and a construction cost index from Sinduscon. The ledger has more than 54,000 price lines.</p>
I also track sleep, HRV and steps as commodities in the same file. It sounds strange, but Beancount already knows how to store a number per day and plot it, so I did not need another tool.</p>
What it costs #</a>
</h2>
Time.</strong> Each importer is a weekend at first and a few hours every time the bank changes the PDF. Fourteen of them is a lot of weekends.</p>
Mail access on macOS.</strong> An agent launched by a background service does not inherit the terminal's permissions, so the Mail.app accounts fail with an AppleScript authorization error until I grant access to the right process. The iCloud mailbox search also times out at 120 seconds on a big inbox. Some mornings the report just says those accounts were skipped.</p>
The last 5%.</strong> A few statements still need a manual download, and the categorizer still misses new merchants. I do not think the rest is worth automating.</p>
It is only mine.</strong> The importers are written for my banks and my categories, and right now nobody else could run them without reading the code.</p>
Why I keep doing it #</a>
</h2>
I can ask the ledger anything. How much a loan really costs after fees and insurance. What I spend when I count the outflows no app counts as spending, like loan principal. Whether a month was unusual or just felt like it. The answers come from a file I own, checked against the bank, and nothing about it depends on a company staying in business.</p>
What comes next #</a>
</h2>
Get it off my Mac.</strong> Everything runs on my laptop today. If the lid is closed at 8, the job does not run. I want the daily job on a small server at home. The Mac stays in the picture for two things that only exist there: Mail.app and the key that holds my secrets.</p>
Teach passbox to fetch mail.</strong> I wrote passbox</a> so agents can use a secret without seeing it, and it already works across machines: a Linux box asks my Mac over Tailscale, and a fingerprint releases the secret. The price API keys already come from passbox, and the rest of the pipeline's secrets, like the passwords that open the bank PDFs, are moving there now. The server will also need my statements, so passbox needs a Mail tool. The job asks the Mac to search a mailbox, and the Mac sends back the PDF attachments and nothing else from the inbox. The search has to be asynchronous, because the iCloud search already takes more than 120 seconds. The job starts it, gets a handle, and checks back later. This tool cannot ask for a fingerprint, because nobody is awake to give one when the job runs, so it is limited by what it returns instead.</p>
Two other passbox changes come with the move. Today the broker cannot tell which machine is asking, so it cannot have rules like "the server can read the price API keys and nothing else". Fixing that means running Tailscale inside the broker so it sees the real caller. And the server should get one approval in the morning that covers the secrets the day's jobs need, instead of a prompt for each read.</p>
Dockerize the whole pipeline, as two services.</strong> Fava already runs in a container. The downloaders, importers and price sources still run in my local Python environment. I want to split them into two images. The first one does the work: it downloads statements, imports them, fetches prices, commits the ledger and pushes. It is the only one that talks to passbox. The second one only shows the ledger: it runs Fava at home and does git pull</code> on a loop. Since git is the only link between the two, they can run on different machines, and moving either one is starting a container somewhere else.</p>
Open source some importers.</strong> The bank importers have nothing personal in them, because the categories live in a separate file. Few people use Beancount in Brazil, and each of them probably wrote their own parser for the same banks. The work before publishing is the test fixtures, which are my real statements today and need to become fake ones.</p>
If you keep your finances in plain text, especially with Brazilian banks, I would like to know how you handle the statements. And if you would use these importers, tell me, because that decides which ones I publish first.</p>

Corrections, 2026-10-12.</strong></p>

The 5% tolerance in beangulp's duplicate check applies to amounts. Dates match within two days.</li>
</ul>


Is Ed25519 quantum resistant? Checking Solana's seed claim
2026-10-08T00:00:00+00:00
I was scrolling Twitter today and found this post:</p>


    Unlike many networks, Solana users don't need to go into "bunker mode" ahead of an accelerated quantum timeline or classical attack provided by AI.</p>
    Jacob Creech (@jacobvcreech) October 8, 2026</a>
</blockquote>
Gabriel Koerich

Four ways to count to five

Solana without the fork

To queue or not to queue

My finances in plain text

Is Ed25519 quantum resistant? Checking Solana's seed claim
