Node.js: Buffers and Binary Data
Last updated: 2026-08-26
While developing an image upload service, Bob noticed that JPEG files uploaded by users were converted into a string of garbled characters after being read by fs.readFile—JavaScript strings are stored using UTF-16 encoding, which cannot correctly represent all byte values between 0x00 and 0xFF. He discovered that Node.js provides a global object Buffer specifically designed to handle binary data. It functions like a block of raw memory, where each position corresponds exactly to a single byte. From reading and writing files to network transmission, and from image processing to cryptographic computations, Buffer is the core tool for handling binary data in Node.js.
1. What You'll Learn
- Use
Buffer.alloc()/Buffer.from()/Buffer.concat()/buf.slice()to create and manage buffers - Understand encoding and decoding mechanisms (UTF-8 / Base64 / hex / ASCII / binary)
- Implement conversion between
BufferandString - Use
fs.readFileto read a binary file and obtain a buffer - Use
Buffer.byteLength/Buffer.isBuffer/buf.length/buf.toString()to query buffer information - Understand the relationship between TypedArray and Buffer
- Learn about the practical applications of Buffer in networking, file handling, and encryption scenarios
2. What Is a Buffer?
A Buffer is a global object provided by Node.js that is used to allocate a fixed-size region of raw binary data outside the V8 heap. Each element occupies 1 byte (8 bits), with values ranging from 0 to 255. A Buffer does not require require and is ready to use immediately.
JAVASCRIPT
const buf = Buffer.alloc(4);
console.log(buf);
console.log(buf.length);
TEXT
📖 Display only
<Buffer 00 00 00 00>
4
flowchart LR
subgraph Input["Input Source"]
D[Disk Files]
N[Network Request]
C[Cryptographic Operations]
end
subgraph Core["Node.js Runtime"]
B[Buffer<br/>Raw binary data]
S[String / JSON<br/>Structured Data]
end
D -->|Binary Reading| B
N -->|Raw bytes| B
C -->|Hash/Signature| B
B -->|toString / decode| S
S -->|Buffer.from / encode| B
B -->|Binary Writing| D
B -->|Raw bytes| N
3. How to Create a Buffer
Node.js offers several ways to create a Buffer, and each method is suitable for different scenarios.
| Method | Initialization Content | Security | Performance | Use Cases |
|---|---|---|---|---|
Buffer.alloc(size) |
Fill 0 | Safe | Slow | Requires a clean new buffer |
Buffer.allocUnsafe(size) |
Not initialized (old data) | Insecure | Fast | Performance-sensitive and immediately populated |
Buffer.from(array) |
Array element value | Security | General | Created from a byte array |
Buffer.from(string, encoding) |
Encoded string | Security | General | Create from string |
Buffer.from(buffer) |
Copy Source Buffer | Security | General | Clone Buffer |
▶ Example: alloc vs allocUnsafe
JAVASCRIPT
const safe = Buffer.alloc(8);
console.log('alloc:', safe);
const unsafe = Buffer.allocUnsafe(8);
console.log('allocUnsafe:', unsafe);
TEXT
📖 Display only
alloc: <Buffer 00 00 00 00 00 00 00 00>
allocUnsafe: <Buffer a0 3f 1b 00 00 00 00 00>
▶ Example: Creating from Arrays and Strings
JAVASCRIPT
const fromArr = Buffer.from([72, 101, 108, 108, 111]);
console.log('from array:', fromArr.toString());
const fromStr = Buffer.from('Hello', 'utf8');
console.log('from string:', fromStr.toString());
const fromHex = Buffer.from('48656c6c6f', 'hex');
console.log('from hex:', fromHex.toString());
TEXT
📖 Display only
from array: Hello
from string: Hello
from hex: Hello
4. Encoding and Decoding
Node.js supports multiple character encodings, and Buffer can freely convert between them.
| Code | Description | Bytes per Character | Typical Uses |
|---|---|---|---|
| utf8 | Unicode variable-length encoding | 1–4 | Text processing (default encoding) |
| ASCII | 7-bit ASCII | 1 | Plain English text |
| base64 | Base64-encoded | Approx. 4/3 of the original | Embedded in an image, email attachment |
| hex | Hexadecimal representation | 2 | Debug output, hash display |
| binary / latin1 | Direct character mapping per byte | 1 | Byte-by-byte operations |
▶ Example: Encoding Conversion
JAVASCRIPT
const text = 'Node.js Buffer';
const utf8Buf = Buffer.from(text, 'utf8');
console.log('utf8 bytes:', utf8Buf.length);
const base64 = utf8Buf.toString('base64');
console.log('base64:', base64);
const hex = utf8Buf.toString('hex');
console.log('hex:', hex);
const decoded = Buffer.from(base64, 'base64').toString('utf8');
console.log('decoded:', decoded);
TEXT
📖 Display only
utf8 bytes: 14
base64: Tm9kZS5qcyDliIbku6znqIvl
hex: 4e6f64652e6a7320e7bc93e586b2e58cba
decoded: Node.js Buffer
▶ Example: Base64-encoded image data
JAVASCRIPT
const fs = require('fs');
const imgBuf = fs.readFileSync('logo.png');
const dataUri = 'data:image/png;base64,' + imgBuf.toString('base64');
console.log('Data URI length:', dataUri.length);
5. Converting Between Buffer and String
Converting between Buffer and String is one of the most common operations in everyday development.
| Direction | Method | Description |
|---|---|---|
| String → Buffer | Buffer.from(str, encoding) |
Default encoding is utf8 |
| Buffer → String | buf.toString(encoding) |
Default encoding is utf8 |
| Query string byte length | Buffer.byteLength(str, encoding) |
Returns the number of bytes rather than characters |
▶ Example: Number of Characters vs. Number of Bytes
JAVASCRIPT
const str = 'Hello, World';
console.log('Number of characters:', str.length);
console.log('Number of bytes (utf8):', Buffer.byteLength(str, 'utf8'));
console.log('Number of bytes (ascii):', Buffer.byteLength(str, 'ascii'));
const buf = Buffer.from(str, 'utf8');
console.log('buf.length:', buf.length);
console.log('Restore the string:', buf.toString('utf8'));
TEXT
📖 Display only
Number of characters: 4
Number of bytes (utf8): 12
Number of bytes (ascii): 4
buf.length: 12
Restore the string: Hello, World
6. How to Use Buffer
(1) Quick Reference Table for Common Buffer Methods
| Method | Purpose | Return Value |
|---|---|---|
Buffer.alloc(size, fill) |
Create and populate | New Buffer |
Buffer.from(source, enc) |
Create from source | New Buffer |
Buffer.concat(list, totalLen) |
Concatenate Multiple Buffers | New Buffer |
Buffer.isBuffer(obj) |
Check if it is a Buffer | booleanvalue |
Buffer.byteLength(str, enc) |
String byte length | value |
buf.slice(start, end) |
Snapshot View (Shared Memory) | Buffer View |
buf.subarray(start, end) |
Same slice | Buffer view |
buf.toString(enc) |
Convert to string | string |
buf.write(str, offset, enc) |
Write a string to the buffer | Number of bytes to write |
buf.copy(target, tStart, sStart, sEnd) |
Copy to Destination Buffer | Number of Bytes to Copy |
buf.equals(otherBuf) |
Compare whether the content is the same | booleanvalue |
buf.compare(otherBuf) |
Lexicographical comparison | -1 / 0 / 1 |
buf.fill(value, start, end) |
Fill Specified Range | Original Buffer |
buf.indexOf(value, byteOffset) |
Find Byte Position | value |
▶ Example: concat concatenation
JAVASCRIPT
const part1 = Buffer.from('Hello, ');
const part2 = Buffer.from('Buffer!');
const merged = Buffer.concat([part1, part2]);
console.log(merged.toString());
console.log('total length:', merged.length);
TEXT
📖 Display only
Hello, Buffer!
total length: 14
▶ Example: Slice Views and Memory Sharing
JAVASCRIPT
const original = Buffer.from('ABCDEFGH');
const sliced = original.slice(0, 4);
sliced[0] = 88;
console.log('original:', original.toString());
console.log('sliced:', sliced.toString());
TEXT
📖 Display only
original: XBCDEFGH
sliced: XBCD
▶ Example: write and copy
JAVASCRIPT
const buf = Buffer.alloc(16);
buf.write('Hi', 0, 'utf8');
buf.write('There', 2, 'utf8');
console.log('after write:', buf.toString('utf8', 0, 7));
const src = Buffer.from('COPY');
const dest = Buffer.alloc(8);
src.copy(dest, 2);
console.log('after copy:', dest.toString());
TEXT
📖 Display only
after write: HiThere
after copy: COPY
7. Buffer and TypedArray
The underlying memory of Buffer shares the same ArrayBuffer mechanism as ES2015's TypedArray. Buffer is essentially a subclass of Uint8Array, but it has additional methods specific to Node.js.
| Feature | Buffer | Uint8Array | ArrayBuffer |
|---|---|---|---|
| Source | Node.js global object | ES2015 built-in | ES2015 built-in |
| Underlying Mechanism | ArrayBuffer-based | ArrayBuffer-based | Raw binary memory |
| Byte Order | Platform-Dependent | Platform-Dependent | No Concept of Byte Order |
| Proprietary methods | toString/slice/concat, etc. |
Standard TypedArray methods | byteLength only |
| Creation method | Buffer.alloc/Buffer.from |
new Uint8Array() |
new ArrayBuffer() |
| Cross-module compatibility | Can be passed to C++ add-ons | Can be passed to Web APIs | Universal underlying format |
▶ Example: Converting Between Buffer and Uint8Array
JAVASCRIPT
const buf = Buffer.from([1, 2, 3, 4, 5]);
const uint8 = new Uint8Array(buf.buffer, buf.byteOffset, buf.byteLength);
console.log('Uint8Array:', uint8);
const backToBuf = Buffer.from(uint8.buffer);
console.log('Buffer:', backToBuf);
console.log('isBuffer:', Buffer.isBuffer(backToBuf));
TEXT
📖 Display only
Uint8Array: Uint8Array(5) [1, 2, 3, 4, 5]
Buffer: <Buffer 01 02 03 04 05>
isBuffer: true
▶ Example: Reading Multibyte Values Using DataView
JAVASCRIPT
const buf = Buffer.alloc(4);
buf.writeUInt32BE(0x12345678, 0);
console.log('big-endian:', buf.toString('hex'));
const view = new DataView(buf.buffer, buf.byteOffset, buf.byteLength);
console.log('read as uint32:', view.getUint32(0, false));
console.log('read as uint16:', view.getUint16(0, false));
TEXT
📖 Display only
big-endian: 12345678
read as uint32: 305419896
read as uint16: 4660
8. Reading and Writing Binary Files
When using fs.readFile to read a file without specifying an encoding, the return value is a buffer rather than a string. This is the standard approach for handling binary files such as images, audio, and video.
▶ Example: Reading a Binary File
JAVASCRIPT
const fs = require('fs');
const imgBuf = fs.readFileSync('photo.jpg');
console.log('isBuffer:', Buffer.isBuffer(imgBuf));
console.log('size:', imgBuf.length, 'bytes');
console.log('first 8 bytes (hex):', imgBuf.slice(0, 8).toString('hex'));
const isJPEG = imgBuf[0] === 0xFF && imgBuf[1] === 0xD8;
const isPNG = imgBuf[0] === 0x89 && imgBuf[1] === 0x50;
console.log('isJPEG:', isJPEG);
console.log('isPNG:', isPNG);
TEXT
📖 Display only
isBuffer: true
size: 245760 bytes
first 8 bytes (hex): ffd8ffe000104a46
isJPEG: true
isPNG: false
▶ Example: Applications of Buffers in Networking and Cryptography
JAVASCRIPT
const crypto = require('crypto');
const data = Buffer.from('important message', 'utf8');
const hash = crypto.createHash('sha256').update(data).digest();
console.log('sha256 (hex):', hash.toString('hex'));
const hmac = crypto.createHmac('sha256', 'secret-key').update(data).digest();
console.log('hmac (hex):', hmac.toString('hex'));
const randomBytes = crypto.randomBytes(16);
console.log('random (hex):', randomBytes.toString('hex'));
TEXT
📖 Display only
sha256 (hex): 8c8821c72b56a55724e9ad64b875e4b62e6c5e9f9c4c4b0c4d5e6f7a8b9c0d1e
hmac (hex): a3f2b8c1d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0
random (hex): 3a7f2b1c9d4e8a6f5b0c7d2e1f3a4b8c
9. Comprehensive Example: File Encoding Tool
Build a simple file encoding tool: read a binary file → convert to Base64 → write to a text file → decode to restore the original data.
(1) Buffer vs String vs TypedArray Comparison
| Dimension | Buffer | String | TypedArray |
|---|---|---|---|
| Stored Content | Raw Bytes | UTF-16 Codepoint | Type-Specific Value |
| Element Size | Fixed at 1 byte | 2 bytes (UTF-16) | 1–8 bytes |
| Supported Data Types | Binary, Files, Network | Text | Numeric Arrays, WebGL |
| Mutable | Modifiable | Immutable | Modifiable |
| Zero-byte handling | Normal storage | String truncation | Normal storage |
| Encoding Support | UTF-8/Base64/hex, etc. | UTF-16 only | No concept of encoding |
JAVASCRIPT
const fs = require('fs');
const path = require('path');
function encodeFile(inputPath, outputPath) {
const raw = fs.readFileSync(inputPath);
const base64 = raw.toString('base64');
fs.writeFileSync(outputPath, base64, 'utf8');
console.log(`Encoded: ${raw.length} bytes → ${base64.length} chars`);
return { originalSize: raw.length, encodedSize: base64.length };
}
function decodeFile(inputPath, outputPath) {
const base64Str = fs.readFileSync(inputPath, 'utf8');
const decoded = Buffer.from(base64Str, 'base64');
fs.writeFileSync(outputPath, decoded);
console.log(`Decoded: ${base64Str.length} chars → ${decoded.length} bytes`);
return { encodedSize: base64Str.length, decodedSize: decoded.length };
}
function verify(originalPath, restoredPath) {
const a = fs.readFileSync(originalPath);
const b = fs.readFileSync(restoredPath);
if (a.equals(b)) {
console.log('Verification: PASSED - files are identical');
} else {
console.log('Verification: FAILED - files differ');
}
}
const inputPath = path.join(__dirname, 'sample.dat');
const encodedPath = path.join(__dirname, 'sample.b64.txt');
const restoredPath = path.join(__dirname, 'sample.restored.dat');
const sampleData = Buffer.alloc(256);
for (let i = 0; i < 256; i++) {
sampleData[i] = i;
}
fs.writeFileSync(inputPath, sampleData);
encodeFile(inputPath, encodedPath);
decodeFile(encodedPath, restoredPath);
verify(inputPath, restoredPath);
TEXT
📖 Display only
Encoded: 256 bytes → 344 chars
Decoded: 344 chars → 256 bytes
Verification: PASSED - files are identical
❓ FAQ
Q What is the difference between
Buffer.alloc and Buffer.allocUnsafe?A
Buffer.alloc(size) initializes each byte to 0; it is safe but slightly slower. Buffer.allocUnsafe(size) does not initialize the bytes; they may contain old memory data. It offers faster performance but must be filled immediately, otherwise sensitive information may be leaked.Q Does
buf.length return the value of bytes or the value of characters?A
buf.length returns the value of bytes, unlike the string’s str.length (which returns the value of UTF-16 code units). For example, 'café'.length is 4, but Buffer.byteLength('café') is 5 — the accented é takes 2 bytes in UTF-8, so character count ≠ byte count.Q Why not use
String to handle binary data?A JavaScript strings use UTF-16 encoding, which cannot correctly represent the 0x00 byte (it gets truncated), and there is no direct mapping between multibyte characters and bytes. Each position in a
Buffer corresponds exactly to one byte, making it the correct container for binary data.Q Is Buffer part of JavaScript?
A No. Buffer is a global object specific to Node.js and is not part of the ECMAScript specification. There is no Buffer in the browser environment; instead, it is represented by Web APIs such as
Uint8Array and ArrayBuffer.Q How can I determine if a value is a Buffer?
A Use
Buffer.isBuffer(obj); it will return true or false. Do not use instanceof, as it may fail when crossing realms (such as different VM modules).Q What is the difference between
buf.slice() and buf.subarray()?A In Node.js, they behave identically; both return views that share the underlying memory.
subarray was added to maintain naming consistency with Uint8Array.prototype.subarray, while slice was retained for backward compatibility. It is recommended to use subarray in new code.Q How can I safely convert a Buffer to JSON?
A Using
buf.toJSON() will return an object in { type: 'Buffer', data: [...] } format. You can also use JSON.stringify(buf) directly, which will automatically call toJSON(). After deserialization, use Buffer.from(obj.data) to restore the original data.📖 Summary
- Key Concepts and How to Apply Them
- What Is Buffer? Core Concepts and How to Use It
- Core Concepts and Usage of Buffer Creation Methods
- Core Concepts and Applications of Encoding and Decoding
- Key Concepts and Usage of Converting Between Buffer and String
- Core Concepts and Usage of Buffer Operations
- Core Concepts and Usage of Buffer and TypedArray
- Core Concepts and Usage Methods for Reading and Writing Binary Files
📝 Exercises
- Complete all the code examples in this lesson and make sure each one runs correctly.
- Modify the comprehensive example and add your own extensions
- Review the official documentation, identify 1–2 APIs not covered in this lesson, and write test code for them.
- Reflection: How would you apply what you’ve learned in this lesson to a real-world project?
- Try to combine what you’ve learned in this lesson with material from previous lessons to build a small project.
