Building Repo2Txt: engineering notes on a 100% client-side tool

Repo2Txt looks simple on the surface — pick a folder, get a text file. Under the hood, there's a surprising amount of engineering to make it fast, correct, and private. This post covers the key technical decisions.

1. Reading files without a server

The first challenge: how do you read a user's local files in a web browser? There are two APIs:

Option A: <input type="file" webkitdirectory>

This is the widely-supported, fallback approach. The user picks a folder, and the browser gives you a FileList of all files inside. It works in every modern browser but doesn't give you a directory handle — you can't "remember" the folder for next time.

Option B: File System Access API (showDirectoryPicker)

This is the modern API (Chrome, Edge, Opera). It gives you a FileSystemDirectoryHandle that you can traverse recursively, read files from, and even persist across sessions (with permission). Repo2Txt uses this when available, with Option A as a fallback.

// Simplified — actual code handles permissions, errors, and recursion
const dirHandle = await window.showDirectoryPicker({ mode: 'read' })
for await (const entry of dirHandle.values()) {
  if (entry.kind === 'file') {
    const file = await entry.getFile()
    // ... process file
  }
}

2. Binary detection: the git heuristic

Not all files are text. If you try to decode a PNG as UTF-8, you get garbage. We need to detect binaries and skip them.

We use the same heuristic Git uses: if a NUL byte appears in the first 2KB, treat the file as binary. This catches 99.9% of binaries (images, videos, compiled code) without false-positiving on UTF-8 text with high-byte characters.

function isBinaryBytes(bytes: Uint8Array): boolean {
  const checkLen = Math.min(bytes.length, 2048)
  if (checkLen === 0) return false
  // Native vectorized search — extremely fast
  if (bytes.subarray(0, checkLen).includes(0)) return true
  // For small files, NUL check is enough
  if (checkLen < 512) return false
  // For larger files, check non-text byte ratio
  let nonText = 0
  for (let i = 0; i < checkLen; i++) {
    const b = bytes[i]
    if (b === 9 || b === 10 || b === 13) continue
    if (b >= 32 && b <= 126) continue
    if (b >= 128) continue
    nonText++
  }
  return nonText / checkLen > 0.3
}

3. Gitignore parsing: more complex than you'd think

The .gitignore format looks simple but has subtle rules:

• Lines starting with ! are negations (re-include previously excluded files).
• Trailing / means directory-only.
• Leading / anchors to the repo root.
• Internal / (not at start or end) also anchors.
• * matches anything except /.
• ? matches a single character except /.
• ** matches zero or more path segments.
• Patterns without / match at any level.

Repo2Txt parses .gitignore files at every directory level (not just the root) and applies them hierarchically. A node_modules/ in a sub-package of a monorepo only excludes that sub-package's node_modules, not the entire repo's.

4. Token estimation: a useful approximation

Exact tokenization requires a model-specific tokenizer (BPE for GPT, SentencePiece for Gemini, etc.). We don't have access to those, and we don't want to ship 5MB of tokenizer data.

Instead, we use a simple heuristic: ~4 characters per token for English text. This is configurable in Settings. It's not exact — it overestimates for code (which is denser) and underestimates for prose — but it's close enough to tell you whether your repo will fit in a 200K context window.

5. Performance: keeping the UI responsive

The biggest performance challenge is processing thousands of files without freezing the UI. Our approach:

• Pre-filter before reading: check exclusion rules (gitignore, hardcoded, custom globs) before calling file.arrayBuffer(). Skipped files never touch the I/O system.
• Concurrency limit of 32: modern browsers handle 32+ concurrent file reads, but more causes memory pressure.
• Yield to the event loop every 60 files: prevents the UI thread from blocking for more than ~50ms at a time.
• Batched progress updates every 25 files: instead of updating React state on every file (which causes 1000+ re-renders), we batch updates.
• Skip binary detection for known text extensions: ~80% of files have text extensions (.ts, .js, .py, .md). Running binary detection on all of them is wasted CPU.

The result: a 1000-file repo converts in under 2 seconds on a modern laptop, with the progress bar updating smoothly the whole time.

6. Output formatting: four formats, one source of truth

Repo2Txt supports four output formats: Plain Text, Markdown, JSON, and LLM Chat. Each is generated from the same RepoFile[] array, so switching formats is instant (no re-processing).

The Markdown format is the default — it works well with GitHub, Gemini, ChatGPT, and most chat UIs that render markdown. JSON is for programmatic pipelines. Plain Text is for terminals. LLM Chat wraps everything in a natural-language narrative optimized for Claude/GPT/Gemini chat windows.

Conclusion

Building a fast, correct, privacy-first tool is harder than it looks — but the constraints are also what make it good. Client-side architecture forced us to be efficient with CPU and memory. Privacy constraints forced us to avoid analytics and third-party services. The result is a tool that's faster, cheaper, and more trustworthy than the server-side alternatives.

If you're building a developer tool, consider going client-side. Your users will thank you — and so will your server bill.