How to do Piping/Streaming of JavaScript objects in Node.js ?

This article gives brief information about the piping/stream implementation in node.js and explains the working of various parts as buffers, streams and how to implement piping in node.js

Buffers: These are the temporary storage spot for a chunk of data that is being transferred from one place to another, the buffer is filled with data then passed along. Buffer transfers the small chunks of data at a time. Buffers are very useful as we don’t have to wait for the complete transfer process because we fill up the buffer which is the temporary storage spot for the data which can then be passed altogether.

Streams: They are the data handling method that is used for the transfer of the data. It increases the performance as the data is processed even before the complete transfer of the data. It collects the small chunks of buffer together and when the buffer is filled then it passes the data down the stream to be processed and sent to the client. They are a data-handling method and are used to read or write input into output following a proper sequence. Buffer and streams together while playing an online video you don’t need to wait for the complete video to be processed rather is small sections transferred with the help of buffers and streams bit by bit.

Readable Stream: Readable streams are used to extract information from any source that they are used to read the data that is present on any source, and then we can use that data to transfer to any source or to store in our system. Readable streams emit an event in the form of chunks (received from the buffers) that is a small part of the data that needs to be processed.

Example: Create a read.txt file in the current project directory with some sample text, in this case, we have the following text.

THIS IS READ.TXT FILE

Filename: index.js

Javascript

// Requiring file system module
var fs = require('fs');

// Creating a read stream
var myReadStream = fs.createReadStream('read.txt', 'utf8');

myReadStream.on('data', function (chunk) {
  console.log("new chunk received");

  // Printing the chunk
  console.log(chunk);
});

Output:

new chunk received 
THIS IS READ.TXT FILE

Writable Streams: Writable stream lets you send the data to any destined source that can rather be any file or to stream online. These streams also emit the events to ensure proper functioning at regular intervals of time. 

Example: Create an index.js file with the following code.

Filename: index.js

Javascript

// Requiring file system module
var fs = require('fs');

// Creating a write stream
// It will create a txt file in provided location
var myWriteStream =
    fs.createWriteStream(__dirname + '/write.txt');

// Writing on stream
myWriteStream.write("Greetings from GeeksforGeeks");

Output: A new file named write.txt will be generated in the current directory which will contain the following data.

Greetings from GeeksforGeeks

Piping using pipeline():

Piping is a method of transferring data from one stream to other directly with the help of buffers so that the data transfer process can start as soon as it fills up a buffer. In other words, piping is used to process streamed data in various steps. This is a module method to pipe between streams forwarding errors and properly cleaning up and provide a callback when the pipeline is complete.

Example: Create a read.txt file in the current project directory with some sample text, in this case, we have the following text.

THIS IS READ.TXT FILE

Filename: index.js

Javascript

var fs = require('fs');
const zlib = require('zlib');
const { pipeline } = require('stream');

// Constructing promisify
const { promisify } = require('util');
const pipelineAsync = promisify(pipeline);

// Creating a read stream
var myReadStream = fs.createReadStream(
      __dirname + '/read.txt', 'utf8');

// Creating a write stream
var myWriteStream = fs.createWriteStream(
      __dirname + '/write.txt');

// Creating transform stream
const transform = zlib.createGzip();

(async function run() {
   try {

      // Pipelining three streams
      await pipelineAsync(

         // Reading from read stream
         myReadStream,

         // Transforming the file
         transform,

         // Writing on write stream
         myWriteStream
      );
      console.log("Pipeline accomplished");
   }
   catch (err) {
      console.error('pipeline failed with error:', err);
   }
})();

Output: We will see the following output and the write.txt file will be generated in the current directory.

Pipeline accomplished

Note: The pipeline should be used instead of a pipe, as the pipe is unsafe, but implementation on the pipe is given below:

Filename: index.js

Javascript

// Requiring file system module
var fs = require('fs');

// Creating a read stream
var myReadStream = fs.createReadStream(
        __dirname + '/read.txt','utf8');

// Creating a write stream
var myWriteStream = fs.createWriteStream(
        __dirname + '/write.txt');

// Piping using pipe
myReadStream.pipe(myWriteStream);
console.log("Pipeing accomplished");

Output: We will see the following output and the write.txt file will be generated in the current directory.

Pipeing accomplished