Go: Go File I/O and JSON Processing
Last updated: 2026-08-26
File I/O and JSON are two fundamental skills in Go واجهة خلفية development—the
ospackage provides file operations at the system call level, while theencoding/jsonpackage enables seamless communication between Go and the JavaScript world.
File operations and JSON processing in the Go standard library are designed with a clever, unified approach: they are all based on the io.Reader and io.Writer interfaces. In this lesson, you will master all the core skills for file I/O and JSON processing.
1. You will learn
- OS file operations (create/open/read/write)
os.ReadFile/os.WriteFile: Read and write in a single operation- bufio buffered read/write (line-by-line/large files)
encoding/jsonMarshal / Unmarshal- JSON Encoder/Decoder Streaming Processing
- JSON tags control serialization behavior
- Build a قاعدة بيانات export tool using file I/O and JSON
2. The True Story of a Full-Stack Engineer
(1) Pain Point: Manually concatenating JSON strings
Charlie is a full-stack engineer who needs to export user data from the قاعدة بيانات into a JSON file for use by the front end:
"I didn't want to use a third-party library, so I manually constructed the JSON سلسلة. As it turned out, there was a
\"character in the username field, which completely broke the JSON format. The خطأ didn't show up until I was halfway through exporting 1 million rows of data—and it took another two hours to roll back the changes."
He opened the code he had written:
// Bad code: manually concatenating JSON strings
func exportUserJSON(users []User) string {
result := "["
for i, u := range users {
if i > 0 {
result += ","
}
// Manual concatenation, double quotes not escaped
result += "{\"name\":\"" + u.Name + "\",\"age\":" + string(u.Age) + "}"
}
result += "]"
return result
}
If u.Name contains " or \, the generated JSON will be corrupted. Furthermore, string(u.Age) converts numbers to ASCII—28 becomes \x1c.
(2) Go Solution: encoding/json + File I/O
// json_exporter.go
package main
import (
"encoding/json"
"fmt"
"os"
)
type User struct {
Name سلسلة `json:"name"`
Age int `json:"age"`
City سلسلة `json:"city"`
}
func main() {
users := []User{
{Name: "Alice", Age: 28, City: "Shanghai"},
{Name: `Bob "The Builder"`, Age: 32, City: "Beijing"},
{Name: "Charlie", Age: 25, City: "Guangzhou"},
}
// JSON serialization to file (no manual concatenation needed)
file, _ := os.Create("users.json")
defer file.Close()
encoder := json.NewEncoder(file)
encoder.SetIndent("", " ")
encoder.Encode(users)
fmt.Println("Export successful!")
// Verification: read it back
data, _ := os.ReadFile("users.json")
fmt.Println(سلسلة(data))
}
Output:
Export successful!
[
{
"name": "Alice",
"age": 28,
"city": "Shanghai"
},
{
"name": "Bob \"The Builder\"",
"age": 32,
"city": "Beijing"
},
{
"name": "Charlie",
"age": 25,
"city": "Guangzhou"
}
]
(3) Benefits: Comparison of JSON Processing
| Method | Double Quote Escaping | Special Characters | Large Files | Code Volume |
|---|---|---|---|---|
| Manual concatenation | ❌ Manual escaping | ❌ Prone to errors | ❌ OOM | ~50 lines |
json.Marshal |
✅ Automatic | ✅ Automatic | ❌ Full memory | ~5 lines |
json.Encoder |
✅ Automatic | ✅ Automatic | ✅ Stream-based | ~5 lines |
encoding/json package. It automatically handles escaping, encoding, and indentation, and correctly handles Go's UTF-8 characters.
3. OS File Operations
(1) Opening and Creating Files
package main
import (
"fmt"
"os"
)
func main() {
// Create (or truncate) file
f, _ := os.Create("test.txt")
defer f.Close()
// Write string
f.WriteString("Hello, Go Files!\n")
f.Write([]byte("Second line\n"))
// Open in append mode
f2, _ := os.OpenFile("test.txt", os.O_APPEND|os.O_WRONLY, 0644)
defer f2.Close()
f2.WriteString("Appended line\n")
// Read file
data, _ := os.ReadFile("test.txt")
fmt.Print(string(data))
}
Output:
Hello, Go Files!
Second line
Appended line
(2) os.ReadFile / os.WriteFile (one-time read/write)
package main
import (
"fmt"
"os"
)
func main() {
// Write (one-time)
content := []byte("line1\nline2\nline3\n")
err := os.WriteFile("data.txt", content, 0644)
if err != nil {
fmt.Printf("Write خطأ: %v\n", err)
return
}
// Read (one-time)
data, err := os.ReadFile("data.txt")
if err != nil {
fmt.Printf("Read خطأ: %v\n", err)
return
}
fmt.Printf("Read %d bytes:\n%s", len(data), data)
}
▶ Example: Copying a File
package main
import (
"fmt"
"io"
"os"
)
func copyFile(src, dst string) (int64, error) {
sourceFile, err := os.Open(src)
if err != nil {
return 0, err
}
defer sourceFile.Close()
destFile, err := os.Create(dst)
if err != nil {
return 0, err
}
defer destFile.Close()
// io.Copy uses a default 32KB buffer
return io.Copy(destFile, sourceFile)
}
func main() {
// First write the source file
os.WriteFile("source.txt", []byte("Hello World!\n"), 0644)
n, err := copyFile("source.txt", "dest.txt")
if err != nil {
fmt.Printf("Copy error: %v\n", err)
return
}
fmt.Printf("Copied %d bytes\n", n)
data, _ := os.ReadFile("dest.txt")
fmt.Printf("Content: %s", data)
}
Output:
Copied 13 bytes
Content: Hello World!
os.ReadFile reads the entire file into memory—which is suitable for small files (< 100MB). Large files (such as logs 1GB+) must be processed using bufio or io.Copy for streaming.
4. bufio Buffered I/O
(1) bufio reads line by line
package main
import (
"bufio"
"fmt"
"os"
)
func main() {
// First write a test file
content := ""
for i := 1; i <= 100; i++ {
content += fmt.Sprintf("Line %d\n", i)
}
os.WriteFile("large.txt", []byte(content), 0644)
// Read line by line
file, _ := os.Open("large.txt")
defer file.Close()
scanner := bufio.NewScanner(file)
lineCount := 0
for scanner.Scan() {
lineCount++
if lineCount <= 3 {
fmt.Println(scanner.Text())
}
}
fmt.Printf("...total %d lines\n", lineCount)
}
Output:
Line 1
Line 2
Line 3
...total 100 lines
▶ Example: bufio buffer write
package main
import (
"bufio"
"fmt"
"os"
)
func main() {
file, _ := os.Create("buffered.txt")
defer file.Close()
writer := bufio.NewWriter(file)
writer.WriteString("First line\n")
writer.WriteString("Second line\n")
writer.WriteString("Third line\n")
fmt.Printf("Buffer size: %d\n", writer.Buffered())
// Important: flush the buffer to disk
writer.Flush()
data, _ := os.ReadFile("buffered.txt")
fmt.Printf("Content:\n%s", data)
}
(3) os vs bufio
| Feature | Direct OS read/write | Bufio buffering |
|---|---|---|
| Internal Mechanism | System Call (one syscall per read/write operation) | User-Mode Buffer (reduces the number of system calls) |
| Suitable Scenarios | Small files, random access | Large files, sequential read/write |
| Read line by line | Not supported | bufio.Scanner |
| Performance (large files) | Slow (high number of syscalls) | Fast (90% reduction in syscalls) |
| Default Buffer | N/A | 4 KB (configurable) |
5. encoding/json serialization
(1) json.Marshal / json.Unmarshal
package main
import (
"encoding/json"
"fmt"
)
type Product struct {
Name string `json:"name"`
Price float64 `json:"price"`
InStock bool `json:"in_stock"`
Tags []string `json:"tags,omitempty"`
Internal string `json:"-"` // not serialized
}
func main() {
p := Product{
Name: "Go Mug",
Price: 19.99,
InStock: true,
Tags: []string{"gift", "office"},
Internal: "secret123",
}
// Serialize
data, _ := json.Marshal(p)
fmt.Printf("Marshaled: %s\n", data)
// With indentation
pretty, _ := json.MarshalIndent(p, "", " ")
fmt.Printf("Pretty:\n%s\n", pretty)
// Deserialize
var p2 Product
json.Unmarshal([]byte(`{"name":"Go T-Shirt","price":29.99,"in_stock":false}`), &p2)
fmt.Printf("Unmarshaled: %+v\n", p2)
}
Output:
Marshaled: {"name":"Go Mug","price":19.99,"in_stock":true,"tags":["gift","office"]}
Pretty:
{
"name": "Go Mug",
"price": 19.99,
"in_stock": true,
"tags": [
"gift",
"office"
]
}
Unmarshaled: {Name:Go T-Shirt Price:29.99 InStock:false Tags:[] Internal:}
▶ Example: Dynamic JSON Parsing (interface{})
package main
import (
"encoding/json"
"fmt"
)
func main() {
data := `{"name":"Alice","age":28,"address":{"city":"Shanghai","zip":"200000"}}`
// Parse into map (no need to pre-define a struct)
var result map[سلسلة]interface{}
json.Unmarshal([]byte(data), &result)
fmt.Printf("name: %v\n", result["name"])
fmt.Printf("age: %v (type=%T)\n", result["age"], result["age"])
// Nested map requires type assertion (comma-ok safe pattern)
if addr, ok := result["address"].(map[سلسلة]interface{}); ok {
fmt.Printf("city: %v\n", addr["city"])
}
}
Output:
name: Alice
age: 28 (type=float64)
city: Shanghai
float64 by default—so age is displayed as 28 but its type is float64. If you want to use int, either specify the type using a struct, or use json.Decoder with UseNumber().
6. JSON Encoder/Decoder Streaming Processing
▶ Example: Encoder writes to a file in real time
package main
import (
"encoding/json"
"fmt"
"os"
)
type User struct {
Name string `json:"name"`
Age int `json:"age"`
}
func main() {
users := []User{
{Name: "Alice", Age: 28},
{Name: "Bob", Age: 32},
{Name: "Charlie", Age: 25},
}
// Stream write to file (no complete []byte constructed)
file, _ := os.Create("users.json")
defer file.Close()
encoder := json.NewEncoder(file)
encoder.SetIndent("", " ")
for _, u := range users {
encoder.Encode(u) // write one by one
}
data, _ := os.ReadFile("users.json")
fmt.Println(string(data))
}
(2) Decoder: Stream-based reading (line-by-line JSON)
package main
import (
"encoding/json"
"fmt"
"strings"
)
type User struct {
Name سلسلة `json:"name"`
Age int `json:"age"`
}
func main() {
// Simulate JSON Lines format (one JSON كائن per line)
input := `{"name":"Alice","age":28}
{"name":"Bob","age":32}
{"name":"Charlie","age":25}`
// Stream decode
decoder := json.NewDecoder(strings.NewReader(input))
for {
var u User
err := decoder.Decode(&u)
if err != nil {
break // EOF
}
fmt.Printf("Decoded: %s (%d)\n", u.Name, u.Age)
}
}
Output:
Decoded: Alice (28)
Decoded: Bob (32)
Decoded: Charlie (25)
(3) Marshal vs Encoder
| Property | json.Marshal | json.Encoder |
|---|---|---|
| Output Destination | []byte (memory) |
io.Writer (any destination) |
| Memory Usage | Build Complete []byte | Stream Write Buffer |
| Large Files | ❌ OOM Risk | ✅ Security |
| Indentation Control | MarshalIndent |
SetIndent() |
| Typical Scenarios | API Responses / Small Data | File Writing / Network Streams |
7. The Complete Guide to JSON Tags
(1) علامة option
type Config struct {
Name string `json:"name"` // field name mapping
Omit string `json:"omit,omitempty"` // omit on zero value
Skip string `json:"-"` // skip this field
String int `json:"string"` // number converted to string
}
(2) All options
| Option | Syntax | Effect |
|---|---|---|
| Rename | json:"new_name" |
Change the JSON field name to "new_name" |
| omitempty | json:"name,omitempty" |
Omit this field if the value is zero |
| Skip | json:"-" |
Do not serialize/deserialize |
| string | json:"id,string" |
Convert numbers to strings (JavaScript-compatible) |
| Nesting | json:"-" intermediate struct |
Controlling nested serialization |
▶ Example: omitempty + string in practice
package main
import (
"encoding/json"
"fmt"
)
type APIResponse struct {
Code int `json:"code"`
Message سلسلة `json:"message,omitempty"` // omitted when empty
Data any `json:"data,omitempty"` // omitted when nil
ID int64 `json:"id,سلسلة"` // int64 → سلسلة
}
func main() {
resp := APIResponse{
Code: 200,
Data: nil, // will be omitted
ID: 9876543210123,
}
data, _ := json.MarshalIndent(resp, "", " ")
fmt.Println(سلسلة(data))
}
Output:
{
"code": 200,
"id": "9876543210123"
}
8. Complete Example: Database JSON Export Tool
// db_exporter.go
package main
import (
"encoding/json"
"fmt"
"os"
"time"
)
// ---------- Data Model ----------
type Product struct {
ID int `json:"id"`
Name string `json:"name"`
Price float64 `json:"price"`
Stock int `json:"stock"`
Category string `json:"category,omitempty"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at,omitempty"`
}
type ExportConfig struct {
FilePath string `json:"file_path"`
PrettyPrint bool `json:"pretty_print"`
IncludeMeta bool `json:"include_meta"`
}
// ---------- Simulated Database ----------
type Database struct {
products []Product
}
func NewDatabase() *Database {
return &Database{
products: []Product{
{ID: 1, Name: "Go Mug", Price: 19.99, Stock: 100, Category: "accessories",
CreatedAt: time.Now()},
{ID: 2, Name: `Go "Gopher" T-Shirt`, Price: 29.99, Stock: 50, Category: "clothing",
CreatedAt: time.Now()},
{ID: 3, Name: "Go Programming Book", Price: 49.99, Stock: 200, Category: "books",
CreatedAt: time.Now()},
},
}
}
func (db *Database) QueryAll() []Product {
return db.products
}
// ---------- Exporter ----------
type Exporter struct {
config ExportConfig
}
func NewExporter(config ExportConfig) *Exporter {
return &Exporter{config: config}
}
func (e *Exporter) ExportToFile(products []Product) error {
file, err := os.Create(e.config.FilePath)
if err != nil {
return fmt.Errorf("create file: %w", err)
}
defer file.Close()
encoder := json.NewEncoder(file)
if e.config.PrettyPrint {
encoder.SetIndent("", " ")
}
if e.config.IncludeMeta {
wrapper := map[string]interface{}{
"exported_at": time.Now().Format(time.RFC3339),
"total": len(products),
"products": products,
}
return encoder.Encode(wrapper)
}
return encoder.Encode(products)
}
func (e *Exporter) ExportToMemory(products []Product) ([]byte, error) {
if e.config.PrettyPrint {
return json.MarshalIndent(products, "", " ")
}
return json.Marshal(products)
}
// ---------- Importer (reads JSON file) ----------
type Importer struct{}
func NewImporter() *Importer {
return &Importer{}
}
func (imp *Importer) ImportFromFile(path string) ([]Product, error) {
file, err := os.Open(path)
if err != nil {
return nil, fmt.Errorf("open file: %w", err)
}
defer file.Close()
var products []Product
decoder := json.NewDecoder(file)
if err := decoder.Decode(&products); err != nil {
// Try the wrapped format: must rebuild decoder after Seek
file.Seek(0, 0)
decoder = json.NewDecoder(file)
var wrapped struct {
Products []Product `json:"products"`
}
if err2 := decoder.Decode(&wrapped); err2 != nil {
return nil, fmt.Errorf("decode: %w", err)
}
products = wrapped.Products
}
return products, nil
}
func main() {
db := NewDatabase()
products := db.QueryAll()
// 1. Stream JSON write to file
exporter := NewExporter(ExportConfig{
FilePath: "export.json",
PrettyPrint: true,
IncludeMeta: true,
})
if err := exporter.ExportToFile(products); err != nil {
fmt.Printf("Export error: %v\n", err)
return
}
fmt.Println("Exported to export.json")
// 2. Read back to verify
importer := NewImporter()
imported, err := importer.ImportFromFile("export.json")
if err != nil {
fmt.Printf("Import error: %v\n", err)
return
}
fmt.Printf("Imported %d products\n", len(imported))
for _, p := range imported {
fmt.Printf(" %d. %s ($%.2f)\n", p.ID, p.Name, p.Price)
}
// 3. Display file content
data, _ := os.ReadFile("export.json")
fmt.Printf("\nFile content:\n%s\n", data)
}
Expected Output:
Exported to export.json
Imported 3 products
1. Go Mug ($19.99)
2. Go "Gopher" T-Shirt ($29.99)
3. Go Programming Book ($49.99)
File content:
{
"exported_at": "2026-07-08T10:00:00Z",
"total": 3,
"products": [
{
"id": 1,
"name": "Go Mug",
"price": 19.99,
"stock": 100,
"category": "accessories",
"created_at": "2026-07-08T10:00:00Z"
},
...
]
}
flowchart LR
A[Go struct] --> B[json.Marshal]
A --> C[json.Encoder]
B --> D["[]byte (memory)"]
D --> E[os.WriteFile / HTTP Response]
C --> F[File / net.Conn / bytes.Buffer]
F --> G[Stream write]
H[JSON file/stream] --> I[json.Decoder]
I --> J[Go struct]
H --> K[json.Unmarshal]
K --> J
style A fill:#e1f5fe
style J fill:#e1f5fe
omitempty option during JSON export does not work for zero-value times (time.Time{})—time.Time{} is not nil and will not be omitted. If you need to omit zero-value times, use the pointer type *time.Time.
❓ FAQ
os.ReadFile and ioutil.ReadFile?os.ReadFile is a replacement introduced in Go 1.16; it works the same way. ioutil.ReadFile has been deprecated starting with Go 1.19. You should use os.ReadFile / os.WriteFile.os.File.Read call always triggers a system call (with an overhead of about 1 µs), whereas bufio reads 4 KB at a time, reducing system calls by 99.9%.json.Marshal and json.Encoder?Marshal; for writing to files/network streams or large files → Encoder. The advantage of Encoder is that it operates in a streaming manner and does not consume a lot of memory.struct tag control JSON fields?json:"name" renames the field; json:"name,omitempty" omits zero values; json:"-" skips the field; json:"id,string" converts numbers to strings. Separate multiple options with commas.encoding/json outputs UTF-8 by default—non-ASCII characters are output as-is. If you need ASCII-only output (as required by some legacy systems), use json.MarshalIndent or SetEscapeHTML(false).json.Decoder for streaming decoding (decoder.Decode(&v) to decode one object at a time), or the JSON Lines format (one JSON object per line). Do not use json.Unmarshal to read the entire file.map[string]interface{} or json.RawMessage (deferred parsing). map[string]interface{} is flexible but requires type assertions; json.RawMessage preserves the raw JSON for later parsing.os.Create and os.OpenFile?os.Create(f) is equivalent to os.OpenFile(f, os.O_RDWR|os.O_CREATE|os.O_TRUNC, 0666)—it truncates (overwrites) the file if it already exists. os.OpenFile supports more modes (append, read-only, create a new file, etc.).📖 Summary
os.ReadFile/os.WriteFile: Reading from and writing to small files in a single operationbufio.Scannerprocesses large files line by line, andbufio.Writerbuffers writes- The
encoding/jsonpackage automatically handles serialization and deserialization, eliminating the need to manually construct strings json.Marshalprocesses data in memory, whilejson.Encoderwrites data to a streamjson.Decoderreads data in a stream, making it suitable for large files- JSON tag:
omitempty/-/stringcontrols serialization behavior - File I/O and JSON are unified through the
io.Reader/io.Writerinterfaces
📝 Exercises
-
Basic Problem (Difficulty ⭐): Write a program that uses
os.ReadFileto read a text file, counts the number of lines, words, and characters, and writes the results to another file. You must usestrings.Fields,bufio.Scanner, andos.WriteFile. -
Advanced Problem (Difficulty ⭐⭐): Implement a
TodoListmanager that supports adding, listing, completing, and deleting items, and usesencoding/jsonto serialize data to a file for persistence. Requirements: Use JSON tags to control field names, and useomitemptyto handle optional fields. -
Challenge Problem (Difficulty ⭐⭐⭐): Implement a multi-format exporter that exports data from the
[]Productdata source simultaneously in three formats: JSON, JSON Lines (one JSON object per line), and CSV. You must useos.Create+json.Encoder(for JSON) /fmt.Fprintf(for CSV) and a sharedio.Writerinterface.