net|grep

Getting started

netgrep is grep over HTTP, in a browser or on a server. Point it at a URL and a pattern: it streams the response through ripgrep's real regex engine — the grep-matcher, grep-regex and grep-searcher crates, unmodified from crates.io — and answers the moment a matching line arrives, without waiting for the last byte and without holding the file in memory. There is no index to build and no backend to run.

Requirements

Install

# Using pnpm
pnpm add @netgrep/netgrep

# Using npm
npm install @netgrep/netgrep

No bundler configuration is required. netgrep loads its WebAssembly through a standard new URL('…', import.meta.url) reference, which Vite, webpack 5, Rollup, esbuild, Parcel, Bun and — for a Worker — wrangler all understand. The binary is loaded as soon as the module is imported — fetched in the background in a browser, read from disk in Node, inlined into the script by a Workers bundler — and the first search waits for it.

import { matches } from '@netgrep/netgrep';

const found = await matches(
  'https://netgrep.diegopasquali.com/logs/apache.txt',
  'jk2_init',
);

For a plain does-it-occur question that boolean is the whole answer. Searching covers grep, which gives you every matching line instead, and The matching line covers what a hit contains.

The URL is absolute, and in three of the four runtimes it has to be: only a browser has a document for a relative path to be resolved against. That and the rest of what differs per runtime — Node's version floor, Deno's two permission flags, the Worker bundle's size — are in Runtimes.

When to use it

netgrep applies to a file you can address but cannot preprocess: a document someone else publishes, an artefact or build log served openly or behind a signed URL, a file in a bucket you do not own. An index would answer faster, but building one means owning the build, and it cannot answer about a file that appeared a minute ago. A file behind a login is reachable only if you can hand netgrep the credential yourself, through the request options — it sends none of its own.

It also fits files you do own, when standing up a search backend is not worth it: a real-time search over a blog's raw post files needs nothing deployed. One runs on my blog, and its source is public.

Use an index instead when there are many files, when they are large, when you can preprocess them, or when results have to be ranked. netgrep reads the whole file unless you stop it early, and it reports matches in the order they occur rather than ordering them by relevance; Pagefind, Lunr and FlexSearch do those things well. The limitations page is specific about where the line falls.

Searching

There are two functions, and which one you want is decided by what you do with the answer.

Does this file contain it?

matches answers with a boolean and reads no more of the file than it must.

import { matches } from '@netgrep/netgrep';

const found = await matches('/logs/app.log', 'ECONNREFUSED');

The first hit ends the transfer, so a match near the head of a 240 MB file costs a few chunks rather than the file. A file with no match is read to the end, because proving an absence is what that takes. Nothing crosses out of WebAssembly but the boolean — no line is copied and no terminator is counted, which is why this is cheaper than grep rather than merely narrower.

Which lines contain it?

grep yields every matching line, in file order, as it is found.

import { grep } from '@netgrep/netgrep';

for await (const hit of grep('/logs/app.log', 'ECONNREFUSED')) {
  console.log(hit.lineNumber, hit.line);
}

Hits arrive while the file is still downloading. Memory stays flat however large it is: one network chunk and the incomplete line at its end are all that is ever held, and each hit is built at the moment it is yielded. The matching line covers what a hit contains.

Iteration drives everything

Nothing is fetched until the first next() — which the for await above issues — so a pattern that will not compile throws from the loop rather than from the grep call.

Leaving the loop terminates the transfer. break, return and a throw all do it, so taking the first hit and stopping is how you get the cheapest possible answer that also tells you what matched:

for await (const hit of grep(url, pattern)) {
  console.log(hit.line);
  break; // the rest of the file is never downloaded
}

An error can follow hits you already have

A connection that drops at 180 MB gives you every hit up to that point and then throws. Those hits are correct and complete for the bytes that were read — they are not provisional, and there is nothing to roll back. What you do not know is what the rest of the file contained.

const hits = [];

try {
  for await (const hit of grep(url, pattern)) hits.push(hit);
} catch (cause) {
  // `hits` is what the file said before it stopped arriving.
}

matches has no such halfway state: it rejects, and there is no partial boolean.

Request options

Both functions take a fetch object handed to the request unchanged — an Authorization header, an API key, credentials: 'include', and the AbortSignal that Cancelling is about.

await matches(url, pattern, {
  fetch: { headers: { Authorization: `Bearer ${token}` } },
});

netgrep owns the request because it needs the response body to stream, so this is the only way in. It is passed through whole, so method and body come with it and are neither honoured specially nor rejected: netgrep searches whatever body comes back.

onProgress

Both functions also take onProgress, called after each network chunk with the cumulative bytes read:

await matches(url, pattern, {
  onProgress: (bytesRead) => setRead(bytesRead),
});

It fires whether or not anything has matched, which makes it the only sign of life during a long hitless stretch — and the place to call controller.abort() from when you decide the search has run long enough (see Cancelling).

These are decompressed bytes delivered to the page, not bytes on the wire: a gzipped response moves far fewer. No total comes with them, deliberately — Content-Length on a compressed response is the compressed size, so comparing the two would drive a progress bar that finishes at a few per cent. Show the number climbing, not a fraction of a file.

The matching line

Every hit grep yields is the same shape:

type NetgrepHit = {
  line: string;                            // terminator stripped
  ranges: Array<{ start: number; end: number }>; // UTF-16 offsets into `line`
  lineNumber: number;                      // 1-based, counted from the file
};

There is nothing to opt into. A streamed hit with no line would be meaningless, so the line, its ranges and its number are unconditional — matches is what you call when you want none of them.

line

The whole line, not the matched fragment, with its terminator stripped — \n and a \r\n alike.

A match on an empty line is a hit with line: "". Branch on whether a hit was yielded at all, never on line being truthy or on ranges.length.

Bytes that are not valid UTF-8 are decoded lossily rather than rejected, so a line from a mixed-encoding log arrives with replacement characters rather than throwing.

ranges

Where the pattern matched within line, in order, as UTF-16 code-unit offsets — JavaScript's own string indexing, so line.slice(start, end) is the matched text with no conversion. They are not byte offsets, and they are relative to the returned line, never to the file.

They come from the engine rather than from a second pass in JavaScript, which is the only way they can be right: a JS re-match cannot reproduce smart case or the regex crate's syntax, so it would disagree with the verdict it was meant to explain.

ranges can be empty on a real hit — see the cap below.

lineNumber

The line's 1-based position in the file, not in the network chunk it arrived in, and it counts non-matching lines too. Exact until a single line outgrows 64 KB, past which it gains a line each time the window slides (Limitations).

maxLineBytes

Lines are truncated to 4096 bytes by default. Pass maxLineBytes to change it:

grep(url, pattern, { maxLineBytes: 512 });

The cut happens inside WebAssembly, before the copy, so a minified bundle or a one-line data dump cannot move megabytes per file into JavaScript. It is taken on a UTF-8 character boundary, and it applies to the line's content — the terminator is stripped first.

The pattern is matched against the full line and then the ranges are cut to fit: one straddling the cut is clamped, and one starting past it is dropped. So a hit whose every match sits past the cut arrives with ranges: []. It is still a hit — the line matched — and the string simply cannot show a position it does not hold. Matching the truncated slice instead would let $ match at the cut and report a match the real line does not contain.

What is absent, and by choice

No byte offsets into the file, no match counts, no ranking. Each is refused for its own stated reason rather than by blanket policy — decision 0020, 0022 and 0027 record them.

Surrounding context lines are the one thing deferred rather than refused. 0027 records the design and nothing here forecloses it.

Patterns

A pattern is anything the Rust regex crate understands, the same crate ripgrep itself uses. Smart case is hardcoded on:

Pattern Behaviour
sherlock (all lowercase) case-insensitive, matches Sherlock
Sherlock (contains an uppercase character) case-sensitive, does not match sherlock

This is not configurable. Lowercase your pattern to search case-insensitively.

An invalid pattern (a stray (, or a literal newline from a pasted two-line string) is an ordinary failure: matches rejects and grep throws from the first turn of the loop, both carrying the regex crate's own diagnostic and both before the connection opens — a typo costs no request at all. Nothing needs escaping in advance, and one bad keystroke in a search box does not affect the searches after it.

Cancelling

A search in flight can be stopped two ways, and which one you have depends on which function you called.

Leave the loop

grep's generator terminates the transfer on any exit — break, return, a throw. That is the idiomatic path and it needs no extra machinery:

for await (const hit of grep(url, pattern)) {
  render(hit);
  if (enough(hit)) break; // the rest of the file is never downloaded
}

But a loop that is finding nothing has no body to break from. Across a hitless stretch of a 240 MB file, grep yields nothing to react to, so there is nothing to cancel from. For that, and for matches, which exposes no loop at all, you need a signal — and onProgress, which fires per chunk whether anything matched or not, is where the decision to abort a search that is finding nothing gets made.

Pass a signal

AbortSignal goes in the fetch options, where it is already a standard key:

const controller = new AbortController();

const pending = matches(url, pattern, { fetch: { signal: controller.signal } });

// A keystroke later:
controller.abort();

try {
  const found = await pending;
} catch (cause) {
  // The abort lands here. Nothing was answered.
}

An aborted request rejects — grep throws from the iteration, matches rejects its promise — and stops the transfer rather than merely abandoning it, so a fast typist does not queue up hundreds of megabytes of superseded reads.

There is no top-level signal option. It lives in fetch so there is never a precedence rule to remember between two of them.

Caching

netgrep keeps nothing. Every search streams the file from the network, holding one chunk and the incomplete line at its end — so searching a 500 MB file costs the same memory as searching a 5 KB one, and searching the same URL twice costs two requests.

What a repeat actually costs is the runtime's decision, not netgrep's. In a browser the request goes through the HTTP cache like any other fetch, so it is your response headers that decide whether the second search re-downloads the file, revalidates it for a 304, or is answered from disk without touching the network at all:

cache-control: public, max-age=600
etag: "..."

A warm HTTP hit is still delivered as a stream, so a search answered from the browser's cache still delivers its first hit without waiting for the whole file.

Off the browser, assume there is no cache at all. Node keeps no persistent HTTP cache by default, so two searches of one URL from a script are two downloads however generous the response headers are. Cloudflare Workers does have a cache of its own, reachable through its Cache API — but putting a repeat behind it is the Worker's decision and its code, not something netgrep does or can be asked to do.

There is no configuration for any of this. The library used to keep downloaded bytes in memory, on by default, behind an enableMemoryCache flag; that was removed because the platform does the same job better where it does it at all — in a browser it has eviction, it persists across page loads, and it is shared with everything else the page fetches. Where the platform does not do it, netgrep still will not: holding whole files for the lifetime of a process is the cost that keeping nothing exists to avoid.

Two searches of one URL that overlap will each download it; see the limitation on concurrent searches.

Limitations

What netgrep gets wrong, and what it deliberately does not do. Each defect is pinned by a test, so it cannot change unnoticed.

Defects

Inside a line longer than 64 KB, results are approximate

Inside a single line longer than 64 KB, netgrep searches a sliding window rather than the whole line, and three things follow. The window is all the engine is given and neither function can be told otherwise, so the first two hit both functions, in both directions.

A match longer than the window, or one spanning the seam between two of them, is missed outright: matches answers false and grep never yields the line. And ^ and $ can match at a window's edge, which is not where the line begins or ends: matches answers true when no line in the file really begins that way, and the line grep yields for it does not begin where it claims to.

The third is grep's alone: a hit inside such a line is reported more than once, with a line number that gains one at every window slide, so numbers after an over-long line are approximate rather than exact. Two lesser effects: the line a hit carries begins at the window's edge, so it is a mid-line fragment; and newline-free input is answered more slowly, because nothing can be searched until the window fills or the download ends.

Ordinary log lines, source files and prose are nowhere near this; a minified bundle or a single-line data dump is.

^/$ also anchor to a bare \r, not just \r\n

Fixing $ on CRLF input meant enabling the regex engine's CRLF-aware anchors, and they treat a lone \r as a line boundary too, not only a \r\n pair. So foo$ and ^bar both match "foo\rbar\n", on either side of the bare \r — matches that did not happen before. The line splitter disagrees: it still only ever breaks on \n, so the line grep yields for that match is the whole unsplit text, not "foo" or "bar" alone. A caller cannot predict which boundary applies from the documented behaviour of either.

By design

These are not bugs and will not be fixed.

netgrep does not detect binary files

netgrep searches every byte it is given as text. It does not sniff for binary content and does not decline to search it, so pointing it at an image, an archive or an executable produces matches wherever the pattern's bytes occur, and a yielded line is whatever the surrounding bytes decode to — lossily, so anything that is not valid UTF-8 becomes U+FFFD. This is deliberate: the alternative ripgrep offers is to abandon the whole block of lines on the first NUL byte, which discarded matches that came before it and made a file's answer depend on a byte the caller never looked for. Deciding what a url points at is the caller's job, and it is the one thing the caller can actually do.

Concurrent searches of one URL each download it

netgrep retains nothing between searches, so there is no buffer for a second caller to be handed. Two searches of one URL that overlap therefore both download it. The answers are correct; the second request is wasted. Sharing the download would mean either keeping the whole file in memory — the cost that retaining nothing exists to avoid — or teeing the response stream, which would give the second caller the first one's cancellation as well, turning a wasted request into a wrong answer. In a browser the HTTP cache still applies, so what a repeat costs is whatever the host's response headers say; in Node there is no such cache by default, and the second request is a second download.

Matches are reported in the order they occur, not ranked

netgrep answers whether a pattern occurs in a file, or yields every line it occurs on as that line is found, with each match highlighted within it. It does not rank, and not because ranking was left for later: ranking needs a scoring model, and netgrep has no term statistics, no document frequencies and no index to build one from. There is nothing to rank with. Match counts are refused for a different reason again: a consumer holding the hits already has the count, and a second source of truth could only disagree with it. So hits come back in the order they occur in the file, with no counts and no relevance ordering. If you need results sorted by how well they match, that is what a prebuilt index is for: Pagefind, Lunr and FlexSearch all do it, and netgrep is not trying to.

Runtimes

netgrep runs in a browser, in Node, in Deno and in Cloudflare Workers. The two functions are identical in all four — same arguments, same results, same one-chunk memory — because only the loading of the WebAssembly differs, and the package picks that per runtime through a conditional import. There is nothing to configure and no runtime to name: import { grep, matches } from '@netgrep/netgrep' is the whole of it everywhere.

What is left is a handful of per-runtime facts a caller still has to know, one per section below. They are short, and each of them is the thing that goes wrong first.

Browser

Nothing to do. The browser is the default, Getting started is written against it, and the demo is it running. The binary is fetched in the background the moment the module is imported, and the first search waits for it.

Node

Node 18.19+ or 20.6+ — the package declares ^18.19.0 || >=20.6.0 in engines — and ESM only: there is no require entry point. The floor is not fetch's, which arrived in 18.0: Node reads the binary off disk rather than fetching it, and resolves its path with a synchronous, unflagged import.meta.resolve, which is exactly the pair of versions where that became available. Below the floor the failure is loud and early — the boot throws while the module is evaluating, so the import fails rather than the first search.

The URL must be absolute, and this is the one that catches browser code moved to a script. fetch in a browser resolves a relative path against the document; Node has no document to resolve against, so '/logs/app.log' is not a URL at all and fetch throws before a byte moves. The same string that works in a page cannot work here.

// grep-log.mjs — run with: node grep-log.mjs
import { grep } from '@netgrep/netgrep';

// Absolute. A leading slash is a path, and there is no page to resolve it against.
const url = 'https://netgrep.diegopasquali.com/logs/apache.txt';

for await (const hit of grep(url, 'jk2_init')) {
  console.log(`${hit.lineNumber}\t${hit.line}`);
}

That is the demo's 8.7 MB Apache log, and the lines start printing while it is still downloading. Stopping early is break, as everywhere else — see Cancelling.

Bun resolves the same node condition and would probably work unchanged. It has not been run, so it is not claimed.

Deno

Deno needs no boot of its own: its fetch reads the file: URL the default loader builds for the binary, so it takes byte for byte the path the browser takes. The script is the Node one above, unchanged — same file, same import, same output. Only the command differs, and it needs two permissions rather than one:

deno run --allow-net --allow-read grep-log.mjs

--allow-net is the search. --allow-read is the boot — nothing in your code opens a file, but Deno resolving that file: URL is a read and is checked as one. Loading the engine is a filesystem operation wearing a fetch's clothes.

Without --allow-read the failure contains none of your code. The import succeeds, because this boot fails into a rejected promise rather than throwing during evaluation; then an uncaught NotCapable: requires read access arrives from inside ext:deno_fetch, over a stack that ends in netgrep's own files under node_modules and holds no frame you wrote. It reads like a fault in the runtime. It is a missing flag.

Cloudflare Workers

Nothing to configure. Wrangler resolves the .wasm import from inside the package and inlines the binary into the deployed script; no compatibility flag is involved, nodejs_compat included, because this boot touches no node: API.

// src/index.ts — wrangler deploy
import { matches } from '@netgrep/netgrep';

export default {
  async fetch(request: Request): Promise<Response> {
    const log = new URL(request.url).searchParams.get('log');

    if (log === null) return new Response('missing ?log', { status: 400 });

    return Response.json({ found: await matches(log, 'ECONNREFUSED') });
  },
};

The log can be 200 MB. The isolate holds one chunk of it at a time and answers as soon as the first match arrives rather than after the last byte, and no request pays for the WebAssembly: it is compiled when the isolate starts, not fetched per call.

That inlining is the cost, and it is charged at deploy. A Worker importing netgrep bundles to about 1156 KiB, 493 KiB gzipped — measured with wrangler deploy --dry-run, and almost all of it the regex engine's Unicode tables. It counts against the Worker script-size limit, so check both figures against the one your plan allows before adopting: a handler that was a few KiB becomes most of a megabyte.

Cross-origin permission is a browser rule

Needing Access-Control-Allow-Origin from the file's host is the browser's rule, and off the browser it does not exist — Node, Deno and a Worker will read a URL whose host sends no such header at all. Whatever authorization the host demands still applies and still has to be passed in. Getting started covers both halves; this is only the note that one of the costs it lists stops applying here.