Node.js File System, Streams and Buffers: Read Big Files Safely

A utility company drops a text file of electricity-meter readings on your server every night. In testing it has 50 lines and readFile handles it instantly. In production it has millions, the process memory climbs while the whole file is held as one string, and the job is killed. Nothing was wrong with the code's logic; it just treated a flow of data as a single object. Streams are Node's answer, and buffers are the unit they move.

Bytes first: what a Buffer is

Files and sockets deal in bytes, not characters. A Buffer is Node's fixed-size chunk of raw bytes (a subclass of Uint8Array). Text only appears when you choose an encoding to interpret those bytes, and the choice matters because one character can take more than one byte.

const label = "₹250 café";
const buf = Buffer.from(label, "utf8");

console.log("characters:", label.length, "| bytes:", buf.length);
console.log(buf);
console.log(buf.toString("hex"));
console.log("first 3 bytes decode to:", buf.subarray(0, 3).toString());
console.log("cut mid-character:", buf.subarray(0, 2).toString());
console.log(Buffer.from("meter-0042").toString("base64"));

const header = Buffer.alloc(8);          // zero-filled
header.writeUInt16BE(0xcafe, 0);         // magic number
header.writeUInt32BE(1_250_000, 2);      // reading in watt-hours
console.log(header, header.readUInt32BE(2));

characters: 9 | bytes: 12
<Buffer e2 82 b9 32 35 30 20 63 61 66 c3 a9>
e282b932353020636166c3a9
first 3 bytes decode to: ₹
cut mid-character: �
bWV0ZXItMDA0Mg==
<Buffer ca fe 00 13 12 d0 00 00> 1250000

The label has 9 characters but 12 bytes: the rupee sign takes three bytes (e2 82 b9) and the accented letter takes two. Cutting the buffer after two bytes produces the replacement character, because the bytes no longer form a valid sequence. This is the root of a real bug you will meet shortly with streams. The last lines show the other typical use of buffers: writing a binary header with explicit sizes and byte order (BE means big-endian).

The fs/promises toolbox

Node's file API comes in three flavours. Use the promise version in application code, the synchronous one in start-up scripts, and the callback one only when maintaining old code.

The script below creates a private temporary folder, writes a small readings file, appends to it, parses it, asks the file system for its size and listing, and then deliberately opens a file that does not exist. Cleanup sits in finally so the folder is removed even when something throws.

import { mkdtemp, writeFile, appendFile, readFile, readdir, stat, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";

const dir = await mkdtemp(path.join(tmpdir(), "meters-"));
const file = path.join(dir, "readings.csv");

try {
  await writeFile(file, "meter,kwh\nM-17,310.5\nM-18,128\n");
  await appendFile(file, "M-19,402.25\n");

  const text = await readFile(file, "utf8");
  const rows = text.trim().split("\n").slice(1).map((line) => {
    const [meter, kwh] = line.split(",");
    return { meter, kwh: Number(kwh) };
  });
  const total = rows.reduce((sum, r) => sum + r.kwh, 0);
  console.log(rows.length, "readings, total", total, "kWh");

  const info = await stat(file);
  console.log("size in bytes:", info.size, "| is file:", info.isFile());
  console.log(await readdir(dir));

  await readFile(path.join(dir, "missing.csv"), "utf8");
} catch (err) {
  console.log(err.code, "-", err.syscall);
} finally {
  await rm(dir, { recursive: true, force: true });
}

3 readings, total 840.75 kWh
size in bytes: 42 | is file: true
[ 'readings.csv' ]
ENOENT - open

Failures from the file system carry a stable code such as ENOENT (no such file), EACCES (permission denied), EEXIST or EISDIR. Branch on err.code, never on the message text, which differs between operating systems. Also note that readFile without an encoding returns a Buffer: printing it gives <Buffer 6d 65 74 65 72> rather than your text.

Streams: process data while it is still arriving

A stream delivers data in chunks. A readable file stream emits a Buffer of up to 64 KiB at a time by default (the highWaterMark option controls that). You can see the chunking directly by reading a 200,000-byte file with two settings.

import { createReadStream, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import path from "node:path";

const file = path.join(tmpdir(), "chunk-demo.bin");
writeFileSync(file, Buffer.alloc(200_000, 1)); // 200,000 bytes of 0x01

async function chunkSizes(options) {
  const sizes = [];
  for await (const chunk of createReadStream(file, options)) sizes.push(chunk.length);
  return sizes;
}

console.log(await chunkSizes());
console.log((await chunkSizes({ highWaterMark: 16 * 1024 })).length, "chunks of up to 16 KiB");

[ 65536, 65536, 65536, 3392 ]
13 chunks of up to 16 KiB

Three full 64 KiB chunks and a smaller final one. Memory use is bounded by the chunk size no matter how big the file is. Streams also implement backpressure: if the consumer (say, a slow network connection) cannot keep up, the writable side signals the readable side to pause, so a fast producer never floods memory. The pipeline helper wires that up for you.

A full pipeline: generate, read line by line, compress

The next program does three jobs on a 100,000-line log: it writes the file from a generator (so the lines never exist all at once), reads it back with readline to total the readings, and finally pipes it through an uppercase transform and gzip into a new file.

import { createReadStream, createWriteStream } from "node:fs";
import { mkdtemp, stat, rm } from "node:fs/promises";
import { createInterface } from "node:readline";
import { pipeline } from "node:stream/promises";
import { Readable, Transform } from "node:stream";
import { createGzip } from "node:zlib";
import { tmpdir } from "node:os";
import path from "node:path";

const dir = await mkdtemp(path.join(tmpdir(), "stream-"));
const raw = path.join(dir, "readings.log");

// 1. Produce 100,000 lines lazily; only a few are in memory at any moment.
function* fakeReadings() {
  for (let i = 0; i < 100_000; i++) {
    const kwh = (i % 97) + 0.5;
    yield `M-${i % 500},${kwh}\n`;
  }
}
await pipeline(Readable.from(fakeReadings()), createWriteStream(raw));
console.log("log size:", (await stat(raw)).size, "bytes");

// 2. Consume it line by line without loading the file.
let lines = 0;
let total = 0;
const rl = createInterface({ input: createReadStream(raw), crlfDelay: Infinity });
for await (const line of rl) {
  lines++;
  total += Number(line.split(",")[1]);
}
console.log("lines:", lines, "| total kWh:", total);

// 3. Transform + compress in one pipeline.
const shout = new Transform({
  transform(chunk, _enc, done) {
    done(null, chunk.toString().toUpperCase());
  },
});
const gz = path.join(dir, "readings.log.gz");
await pipeline(createReadStream(raw), shout, createGzip(), createWriteStream(gz));
console.log("gzip smaller than original:", (await stat(gz)).size < (await stat(raw)).size);

await rm(dir, { recursive: true, force: true });

log size: 1067690 bytes
lines: 100000 | total kWh: 4849685
gzip smaller than original: true

pipeline from node:stream/promises connects the streams, applies backpressure, destroys every stream if one fails, and returns a promise you can await in a try/catch. That last behaviour is why it beats the older a.pipe(b).pipe(c) chain, where an error in the middle leaves the other streams open and unhandled.

Common mistakes and the symptoms they cause

• Calling `readFile` on a huge file. Symptom: memory grows to the file's size and the process may die with ERR_FS_FILE_TOO_LARGE or an out-of-memory crash. Fix: stream it.
• Using `readFileSync` inside a request handler. Symptom: every other request stalls while the disk is read. Fix: promise API or a stream (see the event loop lesson).
• Using `.pipe()` without error handlers. Symptom: an error event on any stream with no listener crashes the process, and if you attach a listener to only one stream the others stay open, leaking file handles and leaving half-written output. Fix: await pipeline(...) inside try/catch; a missing source file simply rejects with ENOENT and every stream in the chain is destroyed.
• Relative paths that depend on where you launched Node. Symptom: ENOENT when started from another folder. Fix: build paths from import.meta.url or process.cwd() explicitly.
• Forgetting the encoding. Symptom: logs show <Buffer ...> instead of text. Fix: pass "utf8" or call .toString("utf8").
• Comparing `string.length` with byte sizes. Symptom: a content-length header or a size limit that is wrong for non-English text. Fix: Buffer.byteLength(text).

How this shows up in real projects and interviews

File and stream work appears in CSV imports, log shipping, report exports, image uploads and proxying large downloads. An interview question such as "how would you process a 10 GB file in Node?" is really asking whether you know streams, chunk boundaries and backpressure; mentioning pipeline and a line-oriented reader shows practical experience. Streams are also what an HTTP request and response are, which is exactly where the next lesson picks up: Build a Node.js HTTP Server with the Built-in http Module.