{
  "version": "https://jsonfeed.org/version/1.1",
  "title": "joshday.io",
  "home_page_url": "https://joshday.io/",
  "feed_url": "https://joshday.io/feed.json",
  "description": "Field notes on the bugs whose cause is more interesting than the bug. Each note names the shape of a defect, the tell that gives it away, and the fix.",
  "language": "en",
  "items": [
    {
      "id": "https://joshday.io/notes/a-cap-before-the-filter-starves-the-rows-you-wanted/",
      "url": "https://joshday.io/notes/a-cap-before-the-filter-starves-the-rows-you-wanted/",
      "title": "A cap applied before the filter starves the rows you wanted",
      "summary": "A result cap fills with the majority before it reaches your rows, and the page still looks full. Put the filter inside the query and test a busy range.",
      "content_html": "<p>The query returned a full page of 100 results, and 7 of the 11 journal entries I wanted were not in it.</p>\n<p>A timeline view in a personal notes tool of mine asked for one month with a cap of 100 and no type filter. That month had 366 dated pages in the index, most of them another kind of page, and 11 actual journal entries. The untyped call returned 4 of the 11.</p>\n<p>Nothing in the response looked broken. A hundred rows is a hundred rows. The fix was to pass the type into the query, so the cap applies after the filter.</p>\n<h2>The shape</h2>\n<p>A query has a result cap (k, LIMIT, top N) whose ordering has nothing to do with the caller’s intent. Usually it is recency or insertion order. While the population sharing that ordering is small, the cap never bites.</p>\n<p>Then a large, unrelated population starts sharing the same key space, and the wanted rows become a minority inside it. The cap fills with the majority before reaching them. The query still returns a full page of results, so nothing about the response looks broken.</p>\n<h2>The tell</h2>\n<p>Counts disagree with the source of truth only in busy ranges. A quiet month returns every row and a busy month returns a handful.</p>\n<p>The missing rows all trace back to one type, never a random scatter. Sort the returned rows by whatever the cap ordered on, and the wanted type is clustered just past the cutoff.</p>\n<h2>The fix</h2>\n<p>Apply the selective filter inside the query, before the cap, whenever the caller has a specific type in mind. Filtering after the query returns does not help, because the rows are already gone.</p>\n<p>Then add a regression test that uses a busy range where the wanted rows are a deliberate minority. A quiet range never triggers the bug, so a test on it proves nothing.</p>\n<h2>Try it</h2>\n<p>The example is new, written for this post, using <code>node:sqlite</code> (built into Node, in memory) with made-up rows. Days are plain integers. The quiet range, days 1 to 30, has 40 pages and 11 journal entries. The busy range, days 31 to 60, has 366 pages and the same 11 journal entries. I sized it like the real case, but my entries sit in different places, so my run returns 3 of 11 where the real call returned 4.</p>\n<p>Node prints a one-line experimental notice for <code>node:sqlite</code>. The script silences it, so the output below is everything that printed.</p>\n<pre><code class=\"language-js\">// Uses node:sqlite (built in, in-memory). Days are plain integers; the data is made up.\nimport assert from \"node:assert/strict\";\n\n// node prints a one-line \"experimental\" notice for node:sqlite; keep the output to what the script prints.\nprocess.removeAllListeners(\"warning\");\nconst { DatabaseSync } = await import(\"node:sqlite\");\n\nconst db = new DatabaseSync(\":memory:\");\ndb.exec(\"CREATE TABLE pages (id INTEGER PRIMARY KEY, day INTEGER, type TEXT)\");\nconst insert = db.prepare(\"INSERT INTO pages (day, type) VALUES (?, ?)\");\n\n// Days 1..30 are the quiet range: 40 pages, 11 of them journal entries.\n// Days 31..60 are the busy range: 366 pages, the same 11 journal entries among them.\nfunction fill(firstDay, lastDay, total, journalDays) {\n  const days = lastDay - firstDay + 1;\n  let placed = 0;\n  for (let d = firstDay; d &lt;= lastDay; d++) {\n    const others = Math.floor((total - journalDays.length) / days) + (d - firstDay &lt; (total - journalDays.length) % days ? 1 : 0);\n    for (let i = 0; i &lt; others; i++, placed++) insert.run(d, \"note\");\n    if (journalDays.includes(d)) { insert.run(d, \"journal\"); placed++; }\n  }\n  return placed;\n}\nfill(1, 30, 40, [2, 5, 8, 11, 14, 17, 20, 23, 26, 28, 30]);\nfill(31, 60, 366, [32, 35, 38, 41, 44, 47, 50, 53, 56, 58, 60]);\n\nconst CAP = 100;\n\n// Wrong: ask for everything in the range, cap it, then keep the journal entries.\nfunction capThenFilter(from, to) {\n  const rows = db.prepare(\"SELECT day, type FROM pages WHERE day BETWEEN ? AND ? ORDER BY day, id LIMIT ?\")\n    .all(from, to, CAP);\n  return { page: rows, wanted: rows.filter((r) =&gt; r.type === \"journal\") };\n}\n\n// Right: the type is part of the query, so the cap applies after it.\nfunction filterThenCap(from, to) {\n  const rows = db.prepare(\"SELECT day, type FROM pages WHERE day BETWEEN ? AND ? AND type = ? ORDER BY day, id LIMIT ?\")\n    .all(from, to, \"journal\", CAP);\n  return { page: rows, wanted: rows };\n}\n\nconst ranges = [\n  { name: \"quiet\", from: 1, to: 30 },\n  { name: \"busy\", from: 31, to: 60 },\n];\n\nfor (const { name, from, to } of ranges) {\n  const truth = db.prepare(\"SELECT COUNT(*) AS n FROM pages WHERE day BETWEEN ? AND ? AND type = 'journal'\").get(from, to).n;\n  const all = db.prepare(\"SELECT COUNT(*) AS n FROM pages WHERE day BETWEEN ? AND ?\").get(from, to).n;\n  const a = capThenFilter(from, to);\n  const b = filterThenCap(from, to);\n  console.log(`${name} range: ${all} pages, ${truth} journal entries`);\n  console.log(`  cap, then filter: ${a.page.length} rows came back, ${a.wanted.length} of ${truth} journal entries   (last day reached: ${a.page.at(-1).day})`);\n  console.log(`  filter, then cap: ${b.page.length} rows came back, ${b.wanted.length} of ${truth} journal entries`);\n}\n\n// The regression test. A quiet range cannot fail; the busy one can.\nconsole.log(\"test: every journal entry in the range comes back\");\nconst queries = [\n  { label: \"cap, then filter\", query: capThenFilter },\n  { label: \"filter, then cap\", query: filterThenCap },\n];\nfor (const { label, query } of queries) {\n  for (const { name, from, to } of ranges) {\n    try {\n      assert.equal(query(from, to).wanted.length, 11);\n      console.log(`  ${label}, ${name} range: pass`);\n    } catch {\n      console.log(`  ${label}, ${name} range: FAIL`);\n    }\n  }\n}\n</code></pre>\n<pre><code>quiet range: 40 pages, 11 journal entries\n  cap, then filter: 40 rows came back, 11 of 11 journal entries   (last day reached: 30)\n  filter, then cap: 11 rows came back, 11 of 11 journal entries\nbusy range: 366 pages, 11 journal entries\n  cap, then filter: 100 rows came back, 3 of 11 journal entries   (last day reached: 39)\n  filter, then cap: 11 rows came back, 11 of 11 journal entries\ntest: every journal entry in the range comes back\n  cap, then filter, quiet range: pass\n  cap, then filter, busy range: FAIL\n  filter, then cap, quiet range: pass\n  filter, then cap, busy range: pass\n</code></pre>\n<p>Both queries hand back something in the busy range: 100 rows from the broken one and 11 from the fixed one. Only one of them is complete. The tell is in <code>last day reached: 39</code>. The cap ordered by day, ran out of room on day 39, and the journal entries on later days were never in the result.</p>\n<p>The test block at the bottom is the regression test the fix needs. The quiet range passes with the broken query, which is the trap: a test that only used the quiet range would have certified it.</p>",
      "image": "https://joshday.io/og/a-cap-before-the-filter-starves-the-rows-you-wanted.png",
      "date_published": "2026-11-03T00:00:00.000Z",
      "tags": [
        "Retrieval and knowledge store",
        "sql",
        "search",
        "testing"
      ],
      "language": "en"
    },
    {
      "id": "https://joshday.io/notes/a-freshness-fallback-reads-deletion-as-health/",
      "url": "https://joshday.io/notes/a-freshness-fallback-reads-deletion-as-health/",
      "title": "A freshness check with a fallback reads deletion as health",
      "summary": "A liveness check that falls back to a proxy goes green on the one failure it exists to catch: the thing it watches being gone.",
      "content_html": "<p>A weekly job on one of my own automation setups got removed from the scheduler, and the daily health check that exists to notice exactly that stayed quiet for about a month.</p>\n<p>The check never errored. It had a fallback, and the fallback did its job a little too well.</p>\n<p>Here is the mechanism. It is short, and I suspect it lives in more places than mine.</p>\n<h2>The shape</h2>\n<p>A freshness check asks “did job X run on schedule?” and answers by reading X’s own record. Reading a file can fail for dull reasons, so I gave the check a plan B: when the record is unreadable, look at the newest modified time across a folder of files. That was meant to cover a transient read failure.</p>\n<p>The trouble is the most likely reason a record becomes unreadable: X was deleted. And in my case that folder gets a file written every night by a different job, so the proxy always read under a day old.</p>\n<p>So the moment X disappears, the check quietly stops measuring X and starts measuring its neighbors. The status stays green. The one event the check exists to catch is the one event it can no longer see. Dang…so that happened.</p>\n<p>Two smaller things hid the same problem in the same setup. A backup job that reran the primary stamped completion under the primary’s name, so in a week when the primary hung, the record still said healthy. And when a cleanup job finally killed a stalled run after about 20 hours, the history showed exit code 1 with last activity 18 seconds after start. That reads as a crash at startup. It was really a run waiting on a prompt nobody was there to answer.</p>\n<p>Three different symptoms, one habit: a record that says something true about a neighbor gets read as a statement about the subject.</p>\n<h2>The tell</h2>\n<p>The check’s detail string changed method. It literally printed the word FALLBACK while the status stayed OK. And nobody goes reading the detail on a green row.</p>\n<p>A method change on a liveness check is itself the alarm. If a check can switch from “ask the subject” to “ask something nearby”, that switch deserves at least as much noise as a failure, because from then on the check is answering a different question than the one on its label.</p>\n<p>The other two had tells too. A completion stamped under the wrong name looks like a primary with a suspiciously good record. A run that took 20 hours and shows 18 seconds of activity was never a crash at startup. It was waiting.</p>\n<h2>The fix</h2>\n<ul>\n<li>When a liveness check cannot find its subject, it returns FAIL with “subject missing”. Not a proxy reading.</li>\n<li>The proxy stays, as a separate informational signal that can never turn the check green. It is still handy in the failure message: “record gone, folder fresh” tells you the box is up and the job is not.</li>\n<li>A fallback lane records completion under its own name. Otherwise a primary that fails every week looks healthy whenever the fallback succeeds.</li>\n<li>A reaper that kills a stalled process records why, so run history keeps the difference between a stall and a crash.</li>\n</ul>\n<p>None of this is hard. It is just the opposite of what feels responsible while you are writing the fallback.</p>\n<h2>Try it</h2>\n<p>The example below is new, written for this post, not lifted from the real setup. It simulates a weekly job’s record plus a neighbor file that is 3 hours old, and runs both versions of the check through three situations.</p>\n<pre><code class=\"language-js\">const fs = require(\"fs\");\nconst os = require(\"os\");\nconst path = require(\"path\");\n\nconst HOUR = 60 * 60 * 1000;\nconst MAX_AGE = 8 * 24 * HOUR; // weekly job plus a day of slack\n\nfunction readLastRun(recordPath) {\n  try {\n    return Date.parse(JSON.parse(fs.readFileSync(recordPath, \"utf8\")).lastRun);\n  } catch {\n    return null; // missing, unreadable, or not JSON\n  }\n}\n\nfunction newestMtime(dir) {\n  return Math.max(...fs.readdirSync(dir).map((f) =&gt; fs.statSync(path.join(dir, f)).mtimeMs));\n}\n\nconst hours = (ms) =&gt; Math.round(ms / HOUR);\n\n// The version I shipped: if the record is unreadable, ask the neighbors.\nfunction checkFreshnessWithFallback(recordPath, neighborDir, now) {\n  const lastRun = readLastRun(recordPath);\n  if (lastRun !== null) {\n    const age = now - lastRun;\n    return { status: age &lt;= MAX_AGE ? \"OK\" : \"FAIL\", detail: `record says last run ${hours(age)}h ago` };\n  }\n  const age = now - newestMtime(neighborDir);\n  return { status: age &lt;= MAX_AGE ? \"OK\" : \"FAIL\", detail: `FALLBACK newest file in folder is ${hours(age)}h old` };\n}\n\n// The fixed version: no record means no job. The proxy is reported, never graded.\nfunction checkFreshness(recordPath, neighborDir, now) {\n  const lastRun = readLastRun(recordPath);\n  const proxy = `(info only) newest file in folder is ${hours(now - newestMtime(neighborDir))}h old`;\n  if (lastRun === null) {\n    return { status: \"FAIL\", detail: `subject missing: no readable record. ${proxy}` };\n  }\n  const age = now - lastRun;\n  return { status: age &lt;= MAX_AGE ? \"OK\" : \"FAIL\", detail: `record says last run ${hours(age)}h ago` };\n}\n\n// --- simulation ---------------------------------------------------------\nconst root = fs.mkdtempSync(path.join(os.tmpdir(), \"fresh-\"));\nconst recordPath = path.join(root, \"weekly-report.json\");\nconst neighborDir = path.join(root, \"outputs\");\nfs.mkdirSync(neighborDir);\n\nconst now = Date.now();\n\n// A different nightly job wrote this file 3 hours ago.\nconst neighbor = path.join(neighborDir, \"nightly.log\");\nfs.writeFileSync(neighbor, \"ok\\n\");\nfs.utimesSync(neighbor, new Date(now - 3 * HOUR), new Date(now - 3 * HOUR));\n\nconst scenarios = [\n  [\"healthy\", () =&gt; fs.writeFileSync(recordPath, JSON.stringify({ lastRun: new Date(now - 72 * HOUR).toISOString() }))],\n  [\"stale\", () =&gt; fs.writeFileSync(recordPath, JSON.stringify({ lastRun: new Date(now - 30 * 24 * HOUR).toISOString() }))],\n  [\"job deleted\", () =&gt; fs.rmSync(recordPath, { force: true })],\n];\n\nconst versions = [\n  [\"fallback\", checkFreshnessWithFallback],\n  [\"fixed\", checkFreshness],\n];\n\nconsole.log(\"scenario\".padEnd(13) + \"version\".padEnd(10) + \"status\".padEnd(8) + \"detail\");\nfor (const [name, setup] of scenarios) {\n  setup();\n  for (const [label, fn] of versions) {\n    const r = fn(recordPath, neighborDir, now);\n    console.log(name.padEnd(13) + label.padEnd(10) + r.status.padEnd(8) + r.detail);\n  }\n}\n\nfs.rmSync(root, { recursive: true, force: true });\n</code></pre>\n<p>Output, as printed:</p>\n<pre><code>scenario     version   status  detail\nhealthy      fallback  OK      record says last run 72h ago\nhealthy      fixed     OK      record says last run 72h ago\nstale        fallback  FAIL    record says last run 720h ago\nstale        fixed     FAIL    record says last run 720h ago\njob deleted  fallback  OK      FALLBACK newest file in folder is 3h old\njob deleted  fixed     FAIL    subject missing: no readable record. (info only) newest file in folder is 3h old\n</code></pre>\n<p>Read the first four rows twice. With a record present, the shipped version behaves perfectly, including failing a stale job. Any test I would have written for it created the record first, so any test I would have written passed. Only the last pair differs, and the difference is the whole story: OK on one side, FAIL on the other, for the same deleted job and the same fresh neighbor.</p>\n<p>The shipped version even tells you what it did. It just says it in the detail column, where nobody looks.</p>",
      "image": "https://joshday.io/og/a-freshness-fallback-reads-deletion-as-health.png",
      "date_published": "2026-11-03T00:00:00.000Z",
      "tags": [
        "Monitors that measure the wrong thing",
        "monitoring",
        "reliability",
        "automation"
      ],
      "language": "en"
    }
  ]
}