Streams allow Node.js to process data piece by piece instead of loading the entire data into memory at once. They are especially useful for working with large files, network requests, and responses.
A stream is an interface for reading or writing data continuously
in smaller pieces called chunks. Node.js provides stream support
through the built-in stream module.
Streams are commonly used for:
Imagine a file containing several gigabytes of data. Reading the entire file at once may require a large amount of memory. A stream can read and process smaller chunks.
const fs = require("fs");
const stream = fs.createReadStream("large.txt");
stream.on("data", function(chunk) {
console.log("Received a chunk");
});
stream.on("end", function() {
console.log("File reading completed");
});The data event is emitted whenever a chunk is available while the readable stream is flowing.
Node.js provides four main types of streams.
| Stream Type | Purpose | Example |
|---|---|---|
| Readable | Reads data. | fs.createReadStream() |
| Writable | Writes data. | fs.createWriteStream() |
| Duplex | Reads and writes data. | TCP socket |
| Transform | Reads data, transforms it, and writes the result. | Compression stream |
A readable stream supplies data that your program can consume.
The fs.createReadStream() method creates a readable
stream for a file.
Create a file named student.txt containing:
Welcome to Node.js Streams!Now create a JavaScript file:
const fs = require("fs");
const readStream = fs.createReadStream("student.txt", {
encoding: "utf8"
});
readStream.on("data", function(chunk) {
console.log("Data:", chunk);
});
readStream.on("end", function() {
console.log("Reading completed");
});
readStream.on("error", function(error) {
console.log("Error:", error.message);
});Data: Welcome to Node.js Streams!
Reading completed
A writable stream allows your program to write data to a destination,
such as a file. The fs.createWriteStream() method
creates a writable file stream.
const fs = require("fs");
const writeStream = fs.createWriteStream("output.txt");
writeStream.write("Hello Student!\n");
writeStream.write("Learning Node.js Streams.\n");
writeStream.end();
writeStream.on("finish", function() {
console.log("Writing completed");
});
writeStream.on("error", function(error) {
console.log("Error:", error.message);
});Writing completedContent of output.txt:
Hello Student!
Learning Node.js Streams.The finish event occurs after end() has been called and all data has been processed by the writable stream.
The pipe() method sends data from a readable stream
to a writable stream. It also helps manage the flow of data when
the destination cannot accept it as quickly as the source produces it.
const fs = require("fs");
const readStream = fs.createReadStream("student.txt");
const writeStream = fs.createWriteStream("copy.txt");
readStream.pipe(writeStream);
writeStream.on("finish", function() {
console.log("File copied successfully");
});
readStream.on("error", function(error) {
console.log("Read error:", error.message);
});
writeStream.on("error", function(error) {
console.log("Write error:", error.message);
});File copied successfullyThis example copies the contents of student.txt into copy.txt. The source and destination files must be accessible.
A transform stream is a special type of duplex stream. It receives input, modifies or processes it, and produces output.
const { Transform } = require("stream");
const upperCaseStream = new Transform({
transform(chunk, encoding, callback) {
callback(null, chunk.toString().toUpperCase());
}
});
upperCaseStream.on("data", function(data) {
console.log(data.toString());
});
upperCaseStream.write("hello student");
upperCaseStream.end();HELLO STUDENTThe transform() method receives a chunk and passes the transformed result to its callback.
Streams emit events that indicate when data is available, when processing finishes, or when an error occurs.
| Event | Meaning |
|---|---|
data |
A readable stream provides a chunk of data. |
end |
A readable stream has no more data to provide. |
finish |
A writable stream has processed all data after end(). |
error |
An error occurs in the stream. |
close |
The stream or its underlying resource has closed. |
drain |
A writable stream can accept more data after backpressure. |
Backpressure occurs when data is produced faster than a writable stream can process it. Node.js provides mechanisms to avoid overwhelming the destination with too much buffered data.
const fs = require("fs");
const writeStream = fs.createWriteStream("output.txt");
let i = 0;
function writeData() {
let canContinue = true;
while (i < 100000 && canContinue) {
i++;
canContinue = writeStream.write(
"This is line " + i + "\n"
);
}
if (i < 100000) {
writeStream.once("drain", writeData);
} else {
writeStream.end();
}
}
writeStream.on("finish", function() {
console.log("All lines written");
});
writeStream.on("error", function(error) {
console.log("Error:", error.message);
});
writeData();All lines writtenWhen write() returns false, the program waits for drain before writing more data. This helps control memory usage.
Node.js provides pipeline() to connect streams and
handle errors across the pipeline. It is often a safer choice
than manually coordinating several streams.
const fs = require("fs");
const { pipeline } = require("stream");
pipeline(
fs.createReadStream("student.txt"),
fs.createWriteStream("pipeline-copy.txt"),
function(error) {
if (error) {
console.log("Pipeline failed:", error.message);
} else {
console.log("Pipeline completed successfully");
}
}
);Pipeline completed successfullyThe callback receives an error if the pipeline fails. Otherwise, it indicates that the pipeline completed.
Select the correct answer for each question.