Go: Go File I/O and JSON Processing

Last updated: 2026-08-26

File I/O and JSON are two fundamental skills in Go واجهة خلفية development—the os package provides file operations at the system call level, while the encoding/json package enables seamless communication between Go and the JavaScript world.

File operations and JSON processing in the Go standard library are designed with a clever, unified approach: they are all based on the io.Reader and io.Writer interfaces. In this lesson, you will master all the core skills for file I/O and JSON processing.

1. You will learn



2. The True Story of a Full-Stack Engineer

(1) Pain Point: Manually concatenating JSON strings

Charlie is a full-stack engineer who needs to export user data from the قاعدة بيانات into a JSON file for use by the front end:

"I didn't want to use a third-party library, so I manually constructed the JSON سلسلة. As it turned out, there was a \" character in the username field, which completely broke the JSON format. The خطأ didn't show up until I was halfway through exporting 1 million rows of data—and it took another two hours to roll back the changes."

He opened the code he had written:

GO
// Bad code: manually concatenating JSON strings
func exportUserJSON(users []User) string {
    result := "["
    for i, u := range users {
        if i > 0 {
            result += ","
        }
        // Manual concatenation, double quotes not escaped
        result += "{\"name\":\"" + u.Name + "\",\"age\":" + string(u.Age) + "}"
    }
    result += "]"
    return result
}

If u.Name contains " or \, the generated JSON will be corrupted. Furthermore, string(u.Age) converts numbers to ASCII—28 becomes \x1c.

(2) Go Solution: encoding/json + File I/O

GO
// json_exporter.go
package main

import (
    "encoding/json"
    "fmt"
    "os"
)

type User struct {
    Name سلسلة `json:"name"`
    Age  int    `json:"age"`
    City سلسلة `json:"city"`
}

func main() {
    users := []User{
        {Name: "Alice", Age: 28, City: "Shanghai"},
        {Name: `Bob "The Builder"`, Age: 32, City: "Beijing"},
        {Name: "Charlie", Age: 25, City: "Guangzhou"},
    }

    // JSON serialization to file (no manual concatenation needed)
    file, _ := os.Create("users.json")
    defer file.Close()

    encoder := json.NewEncoder(file)
    encoder.SetIndent("", "  ")
    encoder.Encode(users)

    fmt.Println("Export successful!")

    // Verification: read it back
    data, _ := os.ReadFile("users.json")
    fmt.Println(سلسلة(data))
}

Output:

TEXT 📖 Display only
Export successful!
[
  {
    "name": "Alice",
    "age": 28,
    "city": "Shanghai"
  },
  {
    "name": "Bob \"The Builder\"",
    "age": 32,
    "city": "Beijing"
  },
  {
    "name": "Charlie",
    "age": 25,
    "city": "Guangzhou"
  }
]

(3) Benefits: Comparison of JSON Processing

Method Double Quote Escaping Special Characters Large Files Code Volume
Manual concatenation ❌ Manual escaping ❌ Prone to errors ❌ OOM ~50 lines
json.Marshal ✅ Automatic ✅ Automatic ❌ Full memory ~5 lines
json.Encoder ✅ Automatic ✅ Automatic ✅ Stream-based ~5 lines
💡 Tip: Never manually construct JSON—use the encoding/json package. It automatically handles escaping, encoding, and indentation, and correctly handles Go's UTF-8 characters.



3. OS File Operations

(1) Opening and Creating Files

GO
package main

import (
    "fmt"
    "os"
)

func main() {
    // Create (or truncate) file
    f, _ := os.Create("test.txt")
    defer f.Close()

    // Write string
    f.WriteString("Hello, Go Files!\n")
    f.Write([]byte("Second line\n"))

    // Open in append mode
    f2, _ := os.OpenFile("test.txt", os.O_APPEND|os.O_WRONLY, 0644)
    defer f2.Close()
    f2.WriteString("Appended line\n")

    // Read file
    data, _ := os.ReadFile("test.txt")
    fmt.Print(string(data))
}

Output:

TEXT 📖 Display only
Hello, Go Files!
Second line
Appended line

(2) os.ReadFile / os.WriteFile (one-time read/write)

GO
package main

import (
    "fmt"
    "os"
)

func main() {
    // Write (one-time)
    content := []byte("line1\nline2\nline3\n")
    err := os.WriteFile("data.txt", content, 0644)
    if err != nil {
        fmt.Printf("Write خطأ: %v\n", err)
        return
    }

    // Read (one-time)
    data, err := os.ReadFile("data.txt")
    if err != nil {
        fmt.Printf("Read خطأ: %v\n", err)
        return
    }
    fmt.Printf("Read %d bytes:\n%s", len(data), data)
}

▶ Example: Copying a File

GO
package main

import (
    "fmt"
    "io"
    "os"
)

func copyFile(src, dst string) (int64, error) {
    sourceFile, err := os.Open(src)
    if err != nil {
        return 0, err
    }
    defer sourceFile.Close()

    destFile, err := os.Create(dst)
    if err != nil {
        return 0, err
    }
    defer destFile.Close()

    // io.Copy uses a default 32KB buffer
    return io.Copy(destFile, sourceFile)
}

func main() {
    // First write the source file
    os.WriteFile("source.txt", []byte("Hello World!\n"), 0644)

    n, err := copyFile("source.txt", "dest.txt")
    if err != nil {
        fmt.Printf("Copy error: %v\n", err)
        return
    }
    fmt.Printf("Copied %d bytes\n", n)

    data, _ := os.ReadFile("dest.txt")
    fmt.Printf("Content: %s", data)
}
▶ Try it Yourself

Output:

TEXT 📖 Display only
Copied 13 bytes
Content: Hello World!
🔥 Common Mistake: os.ReadFile reads the entire file into memory—which is suitable for small files (< 100MB). Large files (such as logs 1GB+) must be processed using bufio or io.Copy for streaming.



4. bufio Buffered I/O

(1) bufio reads line by line

GO
package main

import (
    "bufio"
    "fmt"
    "os"
)

func main() {
    // First write a test file
    content := ""
    for i := 1; i <= 100; i++ {
        content += fmt.Sprintf("Line %d\n", i)
    }
    os.WriteFile("large.txt", []byte(content), 0644)

    // Read line by line
    file, _ := os.Open("large.txt")
    defer file.Close()

    scanner := bufio.NewScanner(file)
    lineCount := 0
    for scanner.Scan() {
        lineCount++
        if lineCount <= 3 {
            fmt.Println(scanner.Text())
        }
    }
    fmt.Printf("...total %d lines\n", lineCount)
}

Output:

TEXT 📖 Display only
Line 1
Line 2
Line 3
...total 100 lines

▶ Example: bufio buffer write

GO
package main

import (
    "bufio"
    "fmt"
    "os"
)

func main() {
    file, _ := os.Create("buffered.txt")
    defer file.Close()

    writer := bufio.NewWriter(file)
    writer.WriteString("First line\n")
    writer.WriteString("Second line\n")
    writer.WriteString("Third line\n")

    fmt.Printf("Buffer size: %d\n", writer.Buffered())

    // Important: flush the buffer to disk
    writer.Flush()

    data, _ := os.ReadFile("buffered.txt")
    fmt.Printf("Content:\n%s", data)
}
▶ Try it Yourself

(3) os vs bufio

Feature Direct OS read/write Bufio buffering
Internal Mechanism System Call (one syscall per read/write operation) User-Mode Buffer (reduces the number of system calls)
Suitable Scenarios Small files, random access Large files, sequential read/write
Read line by line Not supported bufio.Scanner
Performance (large files) Slow (high number of syscalls) Fast (90% reduction in syscalls)
Default Buffer N/A 4 KB (configurable)


5. encoding/json serialization

(1) json.Marshal / json.Unmarshal

GO
package main

import (
    "encoding/json"
    "fmt"
)

type Product struct {
    Name     string   `json:"name"`
    Price    float64  `json:"price"`
    InStock  bool     `json:"in_stock"`
    Tags     []string `json:"tags,omitempty"`
    Internal string   `json:"-"`  // not serialized
}

func main() {
    p := Product{
        Name:     "Go Mug",
        Price:    19.99,
        InStock:  true,
        Tags:     []string{"gift", "office"},
        Internal: "secret123",
    }

    // Serialize
    data, _ := json.Marshal(p)
    fmt.Printf("Marshaled: %s\n", data)

    // With indentation
    pretty, _ := json.MarshalIndent(p, "", "  ")
    fmt.Printf("Pretty:\n%s\n", pretty)

    // Deserialize
    var p2 Product
    json.Unmarshal([]byte(`{"name":"Go T-Shirt","price":29.99,"in_stock":false}`), &p2)
    fmt.Printf("Unmarshaled: %+v\n", p2)
}

Output:

TEXT 📖 Display only
Marshaled: {"name":"Go Mug","price":19.99,"in_stock":true,"tags":["gift","office"]}
Pretty:
{
  "name": "Go Mug",
  "price": 19.99,
  "in_stock": true,
  "tags": [
    "gift",
    "office"
  ]
}
Unmarshaled: {Name:Go T-Shirt Price:29.99 InStock:false Tags:[] Internal:}

▶ Example: Dynamic JSON Parsing (interface{})

GO
package main

import (
    "encoding/json"
    "fmt"
)

func main() {
    data := `{"name":"Alice","age":28,"address":{"city":"Shanghai","zip":"200000"}}`

    // Parse into map (no need to pre-define a struct)
    var result map[سلسلة]interface{}
    json.Unmarshal([]byte(data), &result)

    fmt.Printf("name: %v\n", result["name"])
    fmt.Printf("age: %v (type=%T)\n", result["age"], result["age"])

    // Nested map requires type assertion (comma-ok safe pattern)
    if addr, ok := result["address"].(map[سلسلة]interface{}); ok {
        fmt.Printf("city: %v\n", addr["city"])
    }
}
▶ Try it Yourself

Output:

TEXT 📖 Display only
name: Alice
age: 28 (type=float64)
city: Shanghai
💡 Tip: Numbers in JSON are parsed as float64 by default—so age is displayed as 28 but its type is float64. If you want to use int, either specify the type using a struct, or use json.Decoder with UseNumber().



6. JSON Encoder/Decoder Streaming Processing

▶ Example: Encoder writes to a file in real time

GO
package main

import (
    "encoding/json"
    "fmt"
    "os"
)

type User struct {
    Name string `json:"name"`
    Age  int    `json:"age"`
}

func main() {
    users := []User{
        {Name: "Alice", Age: 28},
        {Name: "Bob", Age: 32},
        {Name: "Charlie", Age: 25},
    }

    // Stream write to file (no complete []byte constructed)
    file, _ := os.Create("users.json")
    defer file.Close()

    encoder := json.NewEncoder(file)
    encoder.SetIndent("", "  ")

    for _, u := range users {
        encoder.Encode(u)  // write one by one
    }

    data, _ := os.ReadFile("users.json")
    fmt.Println(string(data))
}
▶ Try it Yourself

(2) Decoder: Stream-based reading (line-by-line JSON)

GO
package main

import (
    "encoding/json"
    "fmt"
    "strings"
)

type User struct {
    Name سلسلة `json:"name"`
    Age  int    `json:"age"`
}

func main() {
    // Simulate JSON Lines format (one JSON كائن per line)
    input := `{"name":"Alice","age":28}
{"name":"Bob","age":32}
{"name":"Charlie","age":25}`

    // Stream decode
    decoder := json.NewDecoder(strings.NewReader(input))
    for {
        var u User
        err := decoder.Decode(&u)
        if err != nil {
            break  // EOF
        }
        fmt.Printf("Decoded: %s (%d)\n", u.Name, u.Age)
    }
}

Output:

TEXT 📖 Display only
Decoded: Alice (28)
Decoded: Bob (32)
Decoded: Charlie (25)

(3) Marshal vs Encoder

Property json.Marshal json.Encoder
Output Destination []byte (memory) io.Writer (any destination)
Memory Usage Build Complete []byte Stream Write Buffer
Large Files ❌ OOM Risk ✅ Security
Indentation Control MarshalIndent SetIndent()
Typical Scenarios API Responses / Small Data File Writing / Network Streams


7. The Complete Guide to JSON Tags

(1) علامة option

GO
type Config struct {
    Name     string   `json:"name"`               // field name mapping
    Omit     string   `json:"omit,omitempty"`      // omit on zero value
    Skip     string   `json:"-"`                   // skip this field
    String   int      `json:"string"`              // number converted to string
}

(2) All options

Option Syntax Effect
Rename json:"new_name" Change the JSON field name to "new_name"
omitempty json:"name,omitempty" Omit this field if the value is zero
Skip json:"-" Do not serialize/deserialize
string json:"id,string" Convert numbers to strings (JavaScript-compatible)
Nesting json:"-" intermediate struct Controlling nested serialization

▶ Example: omitempty + string in practice

GO
package main

import (
    "encoding/json"
    "fmt"
)

type APIResponse struct {
    Code    int    `json:"code"`
    Message سلسلة `json:"message,omitempty"`  // omitted when empty
    Data    any    `json:"data,omitempty"`      // omitted when nil
    ID      int64  `json:"id,سلسلة"`           // int64 → سلسلة
}

func main() {
    resp := APIResponse{
        Code: 200,
        Data: nil,     // will be omitted
        ID:   9876543210123,
    }

    data, _ := json.MarshalIndent(resp, "", "  ")
    fmt.Println(سلسلة(data))
}
▶ Try it Yourself

Output:

TEXT 📖 Display only
{
  "code": 200,
  "id": "9876543210123"
}


8. Complete Example: Database JSON Export Tool

GO
// db_exporter.go
package main

import (
    "encoding/json"
    "fmt"
    "os"
    "time"
)

// ---------- Data Model ----------

type Product struct {
    ID        int       `json:"id"`
    Name      string    `json:"name"`
    Price     float64   `json:"price"`
    Stock     int       `json:"stock"`
    Category  string    `json:"category,omitempty"`
    CreatedAt time.Time `json:"created_at"`
    UpdatedAt time.Time `json:"updated_at,omitempty"`
}

type ExportConfig struct {
    FilePath     string `json:"file_path"`
    PrettyPrint  bool   `json:"pretty_print"`
    IncludeMeta  bool   `json:"include_meta"`
}

// ---------- Simulated Database ----------

type Database struct {
    products []Product
}

func NewDatabase() *Database {
    return &Database{
        products: []Product{
            {ID: 1, Name: "Go Mug", Price: 19.99, Stock: 100, Category: "accessories",
                CreatedAt: time.Now()},
            {ID: 2, Name: `Go "Gopher" T-Shirt`, Price: 29.99, Stock: 50, Category: "clothing",
                CreatedAt: time.Now()},
            {ID: 3, Name: "Go Programming Book", Price: 49.99, Stock: 200, Category: "books",
                CreatedAt: time.Now()},
        },
    }
}

func (db *Database) QueryAll() []Product {
    return db.products
}

// ---------- Exporter ----------

type Exporter struct {
    config ExportConfig
}

func NewExporter(config ExportConfig) *Exporter {
    return &Exporter{config: config}
}

func (e *Exporter) ExportToFile(products []Product) error {
    file, err := os.Create(e.config.FilePath)
    if err != nil {
        return fmt.Errorf("create file: %w", err)
    }
    defer file.Close()

    encoder := json.NewEncoder(file)
    if e.config.PrettyPrint {
        encoder.SetIndent("", "  ")
    }

    if e.config.IncludeMeta {
        wrapper := map[string]interface{}{
            "exported_at": time.Now().Format(time.RFC3339),
            "total":       len(products),
            "products":    products,
        }
        return encoder.Encode(wrapper)
    }

    return encoder.Encode(products)
}

func (e *Exporter) ExportToMemory(products []Product) ([]byte, error) {
    if e.config.PrettyPrint {
        return json.MarshalIndent(products, "", "  ")
    }
    return json.Marshal(products)
}

// ---------- Importer (reads JSON file) ----------

type Importer struct{}

func NewImporter() *Importer {
    return &Importer{}
}

func (imp *Importer) ImportFromFile(path string) ([]Product, error) {
    file, err := os.Open(path)
    if err != nil {
        return nil, fmt.Errorf("open file: %w", err)
    }
    defer file.Close()

    var products []Product
    decoder := json.NewDecoder(file)
    if err := decoder.Decode(&products); err != nil {
        // Try the wrapped format: must rebuild decoder after Seek
        file.Seek(0, 0)
        decoder = json.NewDecoder(file)
        var wrapped struct {
            Products []Product `json:"products"`
        }
        if err2 := decoder.Decode(&wrapped); err2 != nil {
            return nil, fmt.Errorf("decode: %w", err)
        }
        products = wrapped.Products
    }

    return products, nil
}

func main() {
    db := NewDatabase()
    products := db.QueryAll()

    // 1. Stream JSON write to file
    exporter := NewExporter(ExportConfig{
        FilePath:    "export.json",
        PrettyPrint: true,
        IncludeMeta: true,
    })
    if err := exporter.ExportToFile(products); err != nil {
        fmt.Printf("Export error: %v\n", err)
        return
    }
    fmt.Println("Exported to export.json")

    // 2. Read back to verify
    importer := NewImporter()
    imported, err := importer.ImportFromFile("export.json")
    if err != nil {
        fmt.Printf("Import error: %v\n", err)
        return
    }
    fmt.Printf("Imported %d products\n", len(imported))
    for _, p := range imported {
        fmt.Printf("  %d. %s ($%.2f)\n", p.ID, p.Name, p.Price)
    }

    // 3. Display file content
    data, _ := os.ReadFile("export.json")
    fmt.Printf("\nFile content:\n%s\n", data)
}

Expected Output:

TEXT 📖 Display only
Exported to export.json
Imported 3 products
  1. Go Mug ($19.99)
  2. Go "Gopher" T-Shirt ($29.99)
  3. Go Programming Book ($49.99)

File content:
{
  "exported_at": "2026-07-08T10:00:00Z",
  "total": 3,
  "products": [
    {
      "id": 1,
      "name": "Go Mug",
      "price": 19.99,
      "stock": 100,
      "category": "accessories",
      "created_at": "2026-07-08T10:00:00Z"
    },
    ...
  ]
}
100%
flowchart LR
    A[Go struct] --> B[json.Marshal]
    A --> C[json.Encoder]
    B --> D["[]byte (memory)"]
    D --> E[os.WriteFile / HTTP Response]
    C --> F[File / net.Conn / bytes.Buffer]
    F --> G[Stream write]
    H[JSON file/stream] --> I[json.Decoder]
    I --> J[Go struct]
    H --> K[json.Unmarshal]
    K --> J
    style A fill:#e1f5fe
    style J fill:#e1f5fe
🔥 Common Mistake: The omitempty option during JSON export does not work for zero-value times (time.Time{})—time.Time{} is not nil and will not be omitted. If you need to omit zero-value times, use the pointer type *time.Time.


❓ FAQ

Q What is the difference between os.ReadFile and ioutil.ReadFile?
A os.ReadFile is a replacement introduced in Go 1.16; it works the same way. ioutil.ReadFile has been deprecated starting with Go 1.19. You should use os.ReadFile / os.WriteFile.
Q How is bufio faster than direct reads and writes using os?
A bufio maintains a buffer (4KB by default) in user space, which significantly reduces the number of system calls. A direct os.File.Read call always triggers a system call (with an overhead of about 1 µs), whereas bufio reads 4 KB at a time, reducing system calls by 99.9%.
Q How do I choose between json.Marshal and json.Encoder?
A For data sizes < 100KB or when in-memory processing is required → Marshal; for writing to files/network streams or large files → Encoder. The advantage of Encoder is that it operates in a streaming manner and does not consume a lot of memory.
Q How does the struct tag control JSON fields?
A json:"name" renames the field; json:"name,omitempty" omits zero values; json:"-" skips the field; json:"id,string" converts numbers to strings. Separate multiple options with commas.
Q How is non-ASCII encoding handled in JSON?
A Go's encoding/json outputs UTF-8 by default—non-ASCII characters are output as-is. If you need ASCII-only output (as required by some legacy systems), use json.MarshalIndent or SetEscapeHTML(false).
Q How do I handle large JSON files?
A Use json.Decoder for streaming decoding (decoder.Decode(&v) to decode one object at a time), or the JSON Lines format (one JSON object per line). Do not use json.Unmarshal to read the entire file.
Q How do I parse JSON with an unknown structure?
A Use map[string]interface{} or json.RawMessage (deferred parsing). map[string]interface{} is flexible but requires type assertions; json.RawMessage preserves the raw JSON for later parsing.
Q What is the difference between os.Create and os.OpenFile?
A os.Create(f) is equivalent to os.OpenFile(f, os.O_RDWR|os.O_CREATE|os.O_TRUNC, 0666)—it truncates (overwrites) the file if it already exists. os.OpenFile supports more modes (append, read-only, create a new file, etc.).

📖 Summary


📝 Exercises

  1. Basic Problem (Difficulty ⭐): Write a program that uses os.ReadFile to read a text file, counts the number of lines, words, and characters, and writes the results to another file. You must use strings.Fields, bufio.Scanner, and os.WriteFile.

  2. Advanced Problem (Difficulty ⭐⭐): Implement a TodoList manager that supports adding, listing, completing, and deleting items, and uses encoding/json to serialize data to a file for persistence. Requirements: Use JSON tags to control field names, and use omitempty to handle optional fields.

  3. Challenge Problem (Difficulty ⭐⭐⭐): Implement a multi-format exporter that exports data from the []Product data source simultaneously in three formats: JSON, JSON Lines (one JSON object per line), and CSV. You must use os.Create + json.Encoder (for JSON) / fmt.Fprintf (for CSV) and a shared io.Writer interface.

Web-Tutorial.com

Web-Tutorial Tech Team

A team of developers maintaining programming tutorials. Each tutorial is written and reviewed by developers with expertise in that field. We work to keep our content accurate and reliable — if you spot an issue, please let us know.

100%

🙏 帮我们做得更好

我们是刚上线的编程教程站,几个人的小团队,精力有限。页面虽经检查,难免还有疏漏——链接失效、排版错乱、内容有误、语言生硬……

如果您发现了,麻烦告诉我们,我们会在收到反馈后第一时间进行修复,再次感谢您的光临 🙏