File I/O

Lesson 20: File I/O

Life Analogy

Imagine you are a librarian:

Open a file = Take a book from the shelf and open it to read
Create a file = Take a blank notebook and start writing content
Read a file = Flip through the pages and read the text
Write a file = Use a pen to write new content on paper
Close a file = Close the book and put it back on the shelf
Delete a file = Permanently remove the book from the shelf
Traverse a directory = Tour all the shelves and sections of the library

Just as libraries have strict borrowing rules, operating systems have permission controls for files. You need a "library card" (permissions) to read and write files, and you must "return" (close) them promptly when done, otherwise other readers won't be able to use them.

Core Concepts

Go provides rich file I/O capabilities through its standard library, primarily involving the following packages:

Package	Purpose
`os`	Low-level file operations: open, create, delete, rename
`io`	Generic I/O interfaces: `Reader`, `Writer`
`bufio`	Buffered reading and writing: `Scanner`, `Reader`, `Writer`
`io/ioutil` (deprecated)	Functionality moved to `os` package after Go 1.16
`filepath`	Cross-platform path handling: join, split, match

Basic File Operation Flow

TEXT

Open file → Read/Write operations → Close file
   │                       │
   └── defer f.Close() ────┘   // Ensure closing when function exits

Comparison of Three Reading Methods

Method	Characteristics	Use Case
`os.ReadFile`	Loads entire file into memory at once	Small files (< 100MB)
`bufio.Scanner`	Line-by-line scanning, memory-friendly	Large files, line-by-line processing
`io.ReadAll`	Reads everything into `[]byte`	Network responses and other streaming data

Basic Syntax and Usage

1. Opening and Creating Files

package main

import (
    "fmt"
    "os"
)

func main() {
    // Open file in read-only mode (file must exist)
    f, err := os.Open("data.txt")
    if err != nil {
        fmt.Println("Open failed:", err)
        return
    }
    defer f.Close() // 💡 Always use defer to close files

    // Open file in read-write mode (file must exist)
    f2, err := os.OpenFile("data.txt", os.O_RDWR, 0644)
    if err != nil {
        fmt.Println("Open failed:", err)
        return
    }
    defer f2.Close()

    // Create a new file (truncate if it exists)
    f3, err := os.Create("newfile.txt")
    if err != nil {
        fmt.Println("Create failed:", err)
        return
    }
    defer f3.Close()
}

💡 Tip: os.Open is equivalent to os.OpenFile(name, os.O_RDONLY, 0) — read-only, no writing.

2. OpenFile Flag Bits

// Common flag bit combinations
os.O_RDONLY  // Read-only
os.O_WRONLY  // Write-only
os.O_RDWR    // Read-write
os.O_APPEND  // Append mode
os.O_CREATE  // Create if file doesn't exist
os.O_TRUNC   // Truncate on open (clear contents)

// Append writing
f, err := os.OpenFile("log.txt",
    os.O_WRONLY|os.O_APPEND|os.O_CREATE, 0644)

💡 Tip: Permission 0644 means the owner can read and write, others can only read (Unix systems).

3. Reading Files

package main

import (
    "bufio"
    "fmt"
    "os"
)

func main() {
    // ===== Method 1: Read the entire file at once =====
    data, err := os.ReadFile("config.txt")
    if err != nil {
        fmt.Println("Read failed:", err)
        return
    }
    fmt.Println(string(data))

    // ===== Method 2: Read line by line (recommended for large files) =====
    file, err := os.Open("access.log")
    if err != nil {
        fmt.Println("Open failed:", err)
        return
    }
    defer file.Close()

    scanner := bufio.NewScanner(file)
    lineNum := 0
    for scanner.Scan() { // Scan line by line
        lineNum++
        fmt.Printf("Line %d: %s\n", lineNum, scanner.Text())
    }
    if err := scanner.Err(); err != nil {
        fmt.Println("Scan error:", err)
    }
}

💡 Tip: bufio.Scanner has a default maximum token length of 64KB. For extra-long lines, call scanner.Buffer() to increase the buffer size.

4. Writing Files

package main

import (
    "bufio"
    "fmt"
    "os"
)

func main() {
    // ===== Method 1: Write all at once =====
    err := os.WriteFile("output.txt", []byte("Hello, Go!\n"), 0644)
    if err != nil {
        fmt.Println("Write failed:", err)
        return
    }

    // ===== Method 2: Use bufio.Writer (buffered) =====
    f, err := os.Create("buffered.txt")
    if err != nil {
        fmt.Println("Create failed:", err)
        return
    }
    defer f.Close()

    writer := bufio.NewWriter(f)
    for i := 1; i <= 5; i++ {
        fmt.Fprintf(writer, "Line %d content\n", i)
    }
    writer.Flush() // 💡 Must Flush, otherwise data stays in buffer and won't be written to disk

    // ===== Method 3: Direct write =====
    f2, _ := os.Create("direct.txt")
    defer f2.Close()
    f2.WriteString("Write string directly\n")
    f2.Write([]byte("Write byte slice directly\n"))
}

💡 Tip: bufio.Writer significantly improves performance for frequent small writes because it reduces the number of system calls. But you must always call Flush() at the end.

5. Deleting and Renaming

// Delete a file
err := os.Remove("temp.txt")
if err != nil {
    fmt.Println("Delete failed:", err)
}

// Delete a directory and all its contents
err = os.RemoveAll("temp_dir/")

// Rename / move a file
err = os.Rename("old.txt", "new.txt")

6. Path Handling (filepath Package)

package main

import (
    "fmt"
    "path/filepath"
)

func main() {
    // Join paths (cross-platform safe)
    p := filepath.Join("data", "logs", "app.log")
    fmt.Println(p) // data\logs\app.log (Windows) or data/logs/app.log (Linux)

    // Split path
    dir, file := filepath.Split("/home/user/doc.txt")
    fmt.Println("Directory:", dir)   // /home/user/
    fmt.Println("File:", file)       // doc.txt

    // Get extension
    ext := filepath.Ext("report.pdf")
    fmt.Println(ext) // .pdf

    // Get filename without extension
    name := filepath.Base("report.pdf")
    fmt.Println(name) // report.pdf

    // Absolute path
    abs, _ := filepath.Abs("relative/path")
    fmt.Println(abs)

    // Path matching (glob pattern)
    matched, _ := filepath.Match("*.go", "main.go")
    fmt.Println(matched) // true
}

💡 Tip: Always use filepath.Join to concatenate paths — never manually concatenate / or \, otherwise your code will have issues across platforms.

7. Directory Operations and Traversal

package main

import (
    "fmt"
    "os"
    "path/filepath"
)

func main() {
    // Create directories
    os.Mkdir("logs", 0755)
    os.MkdirAll("logs/2024/01", 0755) // Recursive creation

    // Read directory contents
    entries, err := os.ReadDir(".")
    if err != nil {
        fmt.Println("Failed to read directory:", err)
        return
    }
    for _, entry := range entries {
        info, _ := entry.Info()
        fmt.Printf("%-10s %8d bytes  %s\n",
            entry.Name(), info.Size(), info.Mode())
    }

    // Recursively traverse directory tree
    fmt.Println("\n=== Recursive Traversal ===")
    filepath.Walk(".", func(path string, info os.FileInfo, err error) error {
        if err != nil {
            return err
        }
        prefix := "📄"
        if info.IsDir() {
            prefix = "📁"
        }
        fmt.Printf("%s %s\n", prefix, path)
        return nil
    })
}

💡 Tip: filepath.Walk visits every file. For very large directory trees, use filepath.WalkDir (Go 1.16+) for better performance, as it reduces stat system calls.

Examples

Example: File Copy Tool (Difficulty ⭐)

Implement a simple file copy function that supports files of any size.

package main

import (
    "fmt"
    "io"
    "os"
)

// copyFile copies the source file to the destination path
func copyFile(src, dst string) (int64, error) {
    // Open source file
    sourceFile, err := os.Open(src)
    if err != nil {
        return 0, fmt.Errorf("failed to open source file: %w", err)
    }
    defer sourceFile.Close()

    // Get source file info (for setting permissions)
    sourceInfo, err := sourceFile.Stat()
    if err != nil {
        return 0, fmt.Errorf("failed to get file info: %w", err)
    }

    // Create destination file (inherits source file permissions)
    destFile, err := os.OpenFile(dst,
        os.O_WRONLY|os.O_CREATE|os.O_TRUNC,
        sourceInfo.Mode())
    if err != nil {
        return 0, fmt.Errorf("failed to create destination file: %w", err)
    }
    defer destFile.Close()

    // Use io.Copy for streaming copy (automatically handles buffer)
    bytesWritten, err := io.Copy(destFile, sourceFile)
    if err != nil {
        return 0, fmt.Errorf("failed to copy data: %w", err)
    }

    return bytesWritten, nil
}

func main() {
    // Create test file
    os.WriteFile("source.txt", []byte("This is the source file content.\nUsed to test the copy function.\n"), 0644)

    // Execute copy
    n, err := copyFile("source.txt", "copy.txt")
    if err != nil {
        fmt.Println("Error:", err)
        return
    }
    fmt.Printf("Copy complete, %d bytes total\n", n)

    // Verify result
    data, _ := os.ReadFile("copy.txt")
    fmt.Println("Copied content:", string(data))

    // Cleanup
    os.Remove("source.txt")
    os.Remove("copy.txt")
}

▶ Try it Yourself

TEXT

Copy complete, 45 bytes total
Copied content: This is the source file content.
Used to test the copy function.

Example: Log File Analyzer (Difficulty ⭐⭐)

Read a log file, count by level, and output a summary.

package main

import (
    "bufio"
    "fmt"
    "os"
    "strings"
)

// LogStats holds log statistics
type LogStats struct {
    Total   int
    Error   int
    Warning int
    Info    int
    Debug   int
}

// analyzeLog analyzes the log file and returns statistics
func analyzeLog(filename string) (*LogStats, error) {
    file, err := os.Open(filename)
    if err != nil {
        return nil, fmt.Errorf("failed to open log file: %w", err)
    }
    defer file.Close()

    stats := &LogStats{}
    scanner := bufio.NewScanner(file)

    // Increase buffer size for extra-long lines
    scanner.Buffer(make([]byte, 1024*1024), 1024*1024)

    for scanner.Scan() {
        line := scanner.Text()
        stats.Total++

        switch {
        case strings.Contains(line, "[ERROR]"):
            stats.Error++
        case strings.Contains(line, "[WARNING]"):
            stats.Warning++
        case strings.Contains(line, "[INFO]"):
            stats.Info++
        case strings.Contains(line, "[DEBUG]"):
            stats.Debug++
        }
    }

    if err := scanner.Err(); err != nil {
        return nil, fmt.Errorf("error reading log: %w", err)
    }

    return stats, nil
}

func main() {
    // Create simulated log file
    logContent := `2024-01-15 08:00:01 [INFO] Service started successfully
2024-01-15 08:00:05 [DEBUG] Loading configuration file
2024-01-15 08:01:10 [WARNING] Disk space below 80%
2024-01-15 08:02:30 [ERROR] Database connection timeout
2024-01-15 08:02:35 [INFO] Retrying database connection
2024-01-15 08:03:00 [ERROR] Authentication failed: user admin
2024-01-15 08:03:01 [INFO] Request processing complete
2024-01-15 08:04:00 [DEBUG] Cache hit rate 95%
2024-01-15 08:05:00 [WARNING] API response time exceeded 2s
`
    os.WriteFile("app.log", []byte(logContent), 0644)

    // Analyze logs
    stats, err := analyzeLog("app.log")
    if err != nil {
        fmt.Println("Error:", err)
        return
    }

    // Output report
    fmt.Println("========== Log Analysis Report ==========")
    fmt.Printf("Total lines: %d\n", stats.Total)
    fmt.Printf("ERROR:   %d (%.1f%%)\n", stats.Error,
        float64(stats.Error)/float64(stats.Total)*100)
    fmt.Printf("WARNING: %d (%.1f%%)\n", stats.Warning,
        float64(stats.Warning)/float64(stats.Total)*100)
    fmt.Printf("INFO:    %d (%.1f%%)\n", stats.Info,
        float64(stats.Info)/float64(stats.Total)*100)
    fmt.Printf("DEBUG:   %d (%.1f%%)\n", stats.Debug,
        float64(stats.Debug)/float64(stats.Total)*100)
    fmt.Println("===========================================")

    // Cleanup
    os.Remove("app.log")
}

▶ Try it Yourself

TEXT

========== Log Analysis Report ==========
Total lines: 9
ERROR:   2 (22.2%)
WARNING: 2 (22.2%)
INFO:    3 (33.3%)
DEBUG:   2 (22.2%)
===========================================

Example: Directory Synchronization Tool (Difficulty ⭐⭐⭐)

Implement a simple directory synchronization function: scan the source directory and copy new or modified files to the destination directory.

package main

import (
    "fmt"
    "io"
    "os"
    "path/filepath"
    "time"
)

// FileInfo cached file information
type FileInfo struct {
    Path    string
    ModTime time.Time
    Size    int64
}

// scanDir scans a directory and returns a file information map
func scanDir(dir string) (map[string]FileInfo, error) {
    files := make(map[string]FileInfo)

    err := filepath.Walk(dir, func(path string, info os.FileInfo, err error) error {
        if err != nil {
            return err
        }
        if info.IsDir() {
            return nil
        }

        // Calculate relative path as key
        relPath, err := filepath.Rel(dir, path)
        if err != nil {
            return err
        }

        files[relPath] = FileInfo{
            Path:    path,
            ModTime: info.ModTime(),
            Size:    info.Size(),
        }
        return nil
    })

    return files, err
}

// copyFileData copies a single file
func copyFileData(src, dst string) error {
    srcFile, err := os.Open(src)
    if err != nil {
        return err
    }
    defer srcFile.Close()

    // Ensure destination directory exists
    dstDir := filepath.Dir(dst)
    if err := os.MkdirAll(dstDir, 0755); err != nil {
        return err
    }

    dstFile, err := os.Create(dst)
    if err != nil {
        return err
    }
    defer dstFile.Close()

    _, err = io.Copy(dstFile, srcFile)
    return err
}

// syncDir synchronizes the source directory to the destination directory
func syncDir(src, dst string) error {
    fmt.Printf("Sync: %s → %s\n\n", src, dst)

    // Scan both directories
    srcFiles, err := scanDir(src)
    if err != nil {
        return fmt.Errorf("failed to scan source directory: %w", err)
    }

    dstFiles, err := scanDir(dst)
    if err != nil && !os.IsNotExist(err) {
        return fmt.Errorf("failed to scan destination directory: %w", err)
    }

    copied, updated, skipped := 0, 0, 0

    // Iterate over source directory files
    for relPath, srcInfo := range srcFiles {
        dstPath := filepath.Join(dst, relPath)

        dstInfo, exists := dstFiles[relPath]

        switch {
        case !exists:
            // New file, copy it
            fmt.Printf("  [New] %s\n", relPath)
            if err := copyFileData(srcInfo.Path, dstPath); err != nil {
                return fmt.Errorf("failed to copy %s: %w", relPath, err)
            }
            copied++

        case srcInfo.ModTime.After(dstInfo.ModTime) || srcInfo.Size != dstInfo.Size:
            // File modified, update it
            fmt.Printf("  [Updated] %s\n", relPath)
            if err := copyFileData(srcInfo.Path, dstPath); err != nil {
                return fmt.Errorf("failed to update %s: %w", relPath, err)
            }
            updated++

        default:
            // File unchanged, skip
            skipped++
        }
    }

    fmt.Printf("\nSync complete: %d new, %d updated, %d skipped\n",
        copied, updated, skipped)
    return nil
}

func main() {
    // Create test directory structure
    os.MkdirAll("src_dir/subdir", 0755)
    os.WriteFile("src_dir/main.go", []byte("package main\n"), 0644)
    os.WriteFile("src_dir/readme.txt", []byte("README\n"), 0644)
    os.WriteFile("src_dir/subdir/util.go", []byte("package util\n"), 0644)

    // Create destination directory (with some files)
    os.MkdirAll("dst_dir", 0755)
    os.WriteFile("dst_dir/readme.txt", []byte("Old README\n"), 0644)
    os.WriteFile("dst_dir/old.txt", []byte("This file is not in the source directory\n"), 0644)

    // Execute sync
    err := syncDir("src_dir", "dst_dir")
    if err != nil {
        fmt.Println("Sync error:", err)
        return
    }

    // Verify result
    fmt.Println("\n=== Destination Directory Contents ===")
    filepath.Walk("dst_dir", func(path string, info os.FileInfo, err error) error {
        if err != nil {
            return err
        }
        relPath, _ := filepath.Rel("dst_dir", path)
        prefix := "📁"
        if !info.IsDir() {
            prefix = "📄"
        }
        fmt.Printf("  %s %s\n", prefix, relPath)
        return nil
    })

    // Cleanup
    os.RemoveAll("src_dir")
    os.RemoveAll("dst_dir")
}

▶ Try it Yourself

TEXT

Sync: src_dir → dst_dir

  [New] main.go
  [Updated] readme.txt
  [New] subdir\util.go

Sync complete: 2 new, 1 updated, 0 skipped

=== Destination Directory Contents ===
  📁 .
  📁 subdir
  📄 subdir\util.go
  📄 main.go
  📄 old.txt
  📄 readme.txt

Practical Application Scenarios

Scenario 1: Configuration File Hot-Reload

Monitor configuration file changes and automatically reload.

package main

import (
    "encoding/json"
    "fmt"
    "os"
    "time"
)

// Config application configuration
type Config struct {
    Server   ServerConfig   `json:"server"`
    Database DatabaseConfig `json:"database"`
}

type ServerConfig struct {
    Port int    `json:"port"`
    Host string `json:"host"`
}

type DatabaseConfig struct {
    DSN         string `json:"dsn"`
    MaxOpenConn int    `json:"max_open_conn"`
}

// loadConfig loads configuration from a JSON file
func loadConfig(filename string) (*Config, error) {
    data, err := os.ReadFile(filename)
    if err != nil {
        return nil, fmt.Errorf("failed to read config file: %w", err)
    }

    var config Config
    if err := json.Unmarshal(data, &config); err != nil {
        return nil, fmt.Errorf("failed to parse config: %w", err)
    }

    return &config, nil
}

// watchConfig monitors config file changes and reloads
func watchConfig(filename string, interval time.Duration, callback func(*Config)) {
    var lastModTime time.Time

    // Initial load
    info, err := os.Stat(filename)
    if err == nil {
        lastModTime = info.ModTime()
        if config, err := loadConfig(filename); err == nil {
            callback(config)
        }
    }

    ticker := time.NewTicker(interval)
    defer ticker.Stop()

    for range ticker.C {
        info, err := os.Stat(filename)
        if err != nil {
            fmt.Printf("Failed to check config file: %v\n", err)
            continue
        }

        if info.ModTime().After(lastModTime) {
            fmt.Printf("[%s] Config file change detected, reloading...\n",
                time.Now().Format("15:04:05"))

            config, err := loadConfig(filename)
            if err != nil {
                fmt.Printf("Reload failed: %v\n", err)
                continue
            }

            lastModTime = info.ModTime()
            callback(config)
        }
    }
}

func main() {
    // Create initial configuration
    configJSON := `{
    "server": {
        "port": 8080,
        "host": "localhost"
    },
    "database": {
        "dsn": "user:pass@tcp(localhost:3306)/mydb",
        "max_open_conn": 25
    }
}`
    os.WriteFile("config.json", []byte(configJSON), 0644)
    defer os.Remove("config.json")

    // Start config monitoring
    go watchConfig("config.json", 2*time.Second, func(config *Config) {
        fmt.Printf("  Server: %s:%d\n", config.Server.Host, config.Server.Port)
        fmt.Printf("  Database: %s (Max connections: %d)\n",
            config.Database.DSN, config.Database.MaxOpenConn)
    })

    // Simulate running
    time.Sleep(3 * time.Second)

    // Simulate config update
    fmt.Println("\n>> Updating config file...")
    updatedJSON := `{
    "server": {
        "port": 9090,
        "host": "0.0.0.0"
    },
    "database": {
        "dsn": "user:pass@tcp(db-host:3306)/mydb",
        "max_open_conn": 50
    }
}`
    os.WriteFile("config.json", []byte(updatedJSON), 0644)

    // Wait for change detection
    time.Sleep(5 * time.Second)
}

TEXT

  Server: localhost:8080
  Database: user:pass@tcp(localhost:3306)/mydb (Max connections: 25)

>> Updating config file...
[14:30:02] Config file change detected, reloading...
  Server: 0.0.0.0:9090
  Database: user:pass@tcp(db-host:3306)/mydb (Max connections: 50)

Scenario 2: Batch File Rename Tool

Rename files in a directory according to rules.

package main

import (
    "fmt"
    "os"
    "path/filepath"
    "strings"
)

// RenameRule rename rule
type RenameRule struct {
    Find    string // String to find
    Replace string // Replacement string
}

// batchRename renames files in batch
func batchRename(dir string, rule RenameRule) ([]string, error) {
    entries, err := os.ReadDir(dir)
    if err != nil {
        return nil, fmt.Errorf("failed to read directory: %w", err)
    }

    var renamed []string

    for _, entry := range entries {
        if entry.IsDir() {
            continue
        }

        oldName := entry.Name()
        newName := strings.ReplaceAll(oldName, rule.Find, rule.Replace)

        if oldName == newName {
            continue // No rename needed
        }

        oldPath := filepath.Join(dir, oldName)
        newPath := filepath.Join(dir, newName)

        // Check if target file already exists
        if _, err := os.Stat(newPath); err == nil {
            fmt.Printf("  [Skipped] %s → %s (target already exists)\n", oldName, newName)
            continue
        }

        if err := os.Rename(oldPath, newPath); err != nil {
            fmt.Printf("  [Error] %s: %v\n", oldName, err)
            continue
        }

        fmt.Printf("  [Renamed] %s → %s\n", oldName, newName)
        renamed = append(renamed, newName)
    }

    return renamed, nil
}

func main() {
    // Create test files
    os.MkdirAll("photos", 0755)
    testFiles := []string{
        "IMG_20240115_001.jpg",
        "IMG_20240115_002.jpg",
        "IMG_20240115_003.jpg",
        "IMG_20240116_001.jpg",
        "IMG_20240116_002.jpg",
        "notes.txt",
    }
    for _, name := range testFiles {
        os.WriteFile(filepath.Join("photos", name), []byte(""), 0644)
    }

    // Rule 1: Replace prefix
    fmt.Println("=== Rule 1: IMG → Photo ===")
    rule1 := RenameRule{Find: "IMG_", Replace: "Photo_"}
    renamed, _ := batchRename("photos", rule1)
    fmt.Printf("Renamed %d files total\n\n", len(renamed))

    // Rule 2: Add date prefix
    fmt.Println("=== Rule 2: Photo_ → 2024_Vacation_ ===")
    rule2 := RenameRule{Find: "Photo_", Replace: "2024_Vacation_"}
    renamed, _ = batchRename("photos", rule2)
    fmt.Printf("Renamed %d files total\n", len(renamed))

    // Show final results
    fmt.Println("\n=== Final File List ===")
    entries, _ := os.ReadDir("photos")
    for _, entry := range entries {
        fmt.Printf("  %s\n", entry.Name())
    }

    // Cleanup
    os.RemoveAll("photos")
}

TEXT

=== Rule 1: IMG → Photo ===
  [Renamed] IMG_20240115_001.jpg → Photo_20240115_001.jpg
  [Renamed] IMG_20240115_002.jpg → Photo_20240115_002.jpg
  [Renamed] IMG_20240115_003.jpg → Photo_20240115_003.jpg
  [Renamed] IMG_20240116_001.jpg → Photo_20240116_001.jpg
  [Renamed] IMG_20240116_002.jpg → Photo_20240116_002.jpg
Renamed 5 files total

=== Rule 2: Photo_ → 2024_Vacation_ ===
  [Renamed] Photo_20240115_001.jpg → 2024_Vacation_20240115_001.jpg
  [Renamed] Photo_20240115_002.jpg → 2024_Vacation_20240115_002.jpg
  [Renamed] Photo_20240115_003.jpg → 2024_Vacation_20240115_003.jpg
  [Renamed] Photo_20240116_001.jpg → 2024_Vacation_20240116_001.jpg
  [Renamed] Photo_20240116_002.jpg → 2024_Vacation_20240116_002.jpg
Renamed 5 files total

=== Final File List ===
  2024_Vacation_20240115_001.jpg
  2024_Vacation_20240115_002.jpg
  2024_Vacation_20240115_003.jpg
  2024_Vacation_20240116_001.jpg
  2024_Vacation_20240116_002.jpg
  notes.txt

❓ FAQ

Q1: Why does memory usage spike when reading large files?

// ❌ Wrong: entire file loaded into memory
data, _ := os.ReadFile("huge.log") // 10GB file → memory explosion

// ✅ Correct: read line by line
file, _ := os.Open("huge.log")
defer file.Close()

scanner := bufio.NewScanner(file)
for scanner.Scan() {
    line := scanner.Text()
    // Process one line at a time, minimal memory usage
    processLine(line)
}

Key Point: os.ReadFile is suitable for small files (< 100MB). For large files, always use bufio.Scanner to process line by line / in chunks.

Q2: What happens if `defer f.Close()` is placed before the `err != nil` check?

// ❌ May cause nil pointer panic
f, err := os.Open("file.txt")
defer f.Close() // If Open fails, f is nil, Close will panic
if err != nil {
    return err
}

// ✅ Correct: check error first
f, err := os.Open("file.txt")
if err != nil {
    return err
}
defer f.Close() // Ensures f is not nil

Q3: How to handle path differences between Windows and Linux?

// ❌ Hardcoded path separators
path := "data" + "/" + "file.txt"     // Works on Linux, risky on Windows
path := "data" + "\\" + "file.txt"    // Works on Windows, fails on Linux

// ✅ Use filepath.Join
path := filepath.Join("data", "file.txt")

// ✅ Use raw strings (Windows paths)
path := `C:\Users\admin\file.txt`

// ✅ Get relative path from absolute path
rel, _ := filepath.Rel("/home/user", "/home/user/docs/file.txt")
// rel = "docs/file.txt"

Q4: What to do when data is lost during file writes?

// Cause: used bufio.Writer but forgot to Flush
writer := bufio.NewWriter(f)
writer.WriteString("important data")
// Program crashes or exits → data still in buffer, not written to disk

// ✅ Solution 1: Always defer Flush
defer writer.Flush()

// ✅ Solution 2: Write critical data directly
f.WriteString("important data") // Direct write, bypasses buffer

// ✅ Solution 3: Sync to disk immediately after writing
f.Sync() // Calls fsync system call

📖 Summary

This lesson covered the core knowledge of Go file I/O:

Knowledge Point	Key Takeaway
`os.Open/Create/Remove`	Low-level file operations, requires manual `defer Close()`
`os.ReadFile/WriteFile`	Recommended one-time read/write method for Go 1.16+
`bufio.Scanner`	Best choice for reading large files line by line
`bufio.Writer`	Performance optimization for frequent small writes, don't forget `Flush()`
`filepath.Join`	The correct way to do cross-platform path concatenation
`filepath.Walk/WalkDir`	Recursively traverse directory trees
`io.Copy`	Streaming copy, automatically manages buffer

Core Principles:

Always defer f.Close() — after the error check
Small files use os.ReadFile, large files use bufio.Scanner
Path concatenation uses filepath.Join — don't manually concatenate separators
bufio.Writer must Flush() — otherwise data may be lost
Error handling cannot be skipped — file operation errors are especially important

📝 Exercises

Exercise 1: Word Counter

Write a program that reads a text file, counts the number of words, lines, and characters, and outputs the results.

// Hints:
// - Use bufio.Scanner to read line by line
// - Use strings.Fields() to split each line into words
// - Count all three metrics and format the output

Exercise 2: CSV to JSON

Write a program that reads a CSV file (first row is headers), converts it to JSON array format, and writes it to a new file.

// Hints:
// - Use bufio.Scanner to read CSV line by line
// - First row serves as keys for JSON objects
// - Subsequent rows are values, split with strings.Split
// - Use encoding/json to serialize the output

Exercise 3: Directory Size Calculator

Write a program that recursively calculates the total size of a specified directory and groups statistics by file type (extension).

// Hints:
// - Use filepath.WalkDir to traverse the directory
// - Use map[string]int64 to accumulate size by extension
// - Use humanize for formatted output (e.g., 1.5MB, 230KB)
// - Handle files without extensions (categorize as "no extension")

Next Lesson

In the next lesson, we will learn about JSON Processing — how to parse and generate JSON data in Go, which is a fundamental skill for building APIs and handling configuration files.

👉 Lesson 21: JSON Processing

Web-Tutorial Tech Team

A team of developers maintaining programming tutorials. Each tutorial is written and reviewed by developers with expertise in that field. We work to keep our content accurate and reliable — if you spot an issue, please let us know.

About Us

File I/O

Lesson 20: File I/O

Life Analogy

Core Concepts

Basic File Operation Flow

Comparison of Three Reading Methods

Basic Syntax and Usage

1. Opening and Creating Files

2. OpenFile Flag Bits

3. Reading Files

4. Writing Files

5. Deleting and Renaming

6. Path Handling (filepath Package)

7. Directory Operations and Traversal

Examples

Example: File Copy Tool (Difficulty ⭐)

Example: Log File Analyzer (Difficulty ⭐⭐)

Example: Directory Synchronization Tool (Difficulty ⭐⭐⭐)

Practical Application Scenarios

Scenario 1: Configuration File Hot-Reload

Scenario 2: Batch File Rename Tool

❓ FAQ

Q1: Why does memory usage spike when reading large files?

Q2: What happens if defer f.Close() is placed before the err != nil check?

Q3: How to handle path differences between Windows and Linux?

Q4: What to do when data is lost during file writes?

📖 Summary

📝 Exercises

Exercise 1: Word Counter

Exercise 2: CSV to JSON

Exercise 3: Directory Size Calculator

Next Lesson

🙏 帮我们做得更好

Q2: What happens if `defer f.Close()` is placed before the `err != nil` check?