How can I efficiently process large JSON arrays without loading them entirely into memory in Go?

Decoding Large Stream JSON

When dealing with massive JSON arrays stored in files, it's crucial to avoid loading the entire array into memory, as this can lead to out-of-memory errors. Instead, consider streaming the JSON data element by element.

One approach to achieve this is demonstrated in the encoding/json documentation:

package main

import (
    "encoding/json"
    "fmt"
    "log"
    "strings"
)

func main() {
    const jsonStream = `
                [
                    {"Name": "Ed", "Text": "Knock knock."},
                    {"Name": "Sam", "Text": "Who's there?"},
                    {"Name": "Ed", "Text": "Go fmt."},
                    {"Name": "Sam", "Text": "Go fmt who?"},
                    {"Name": "Ed", "Text": "Go fmt yourself!"}
                ]
            `
    type Message struct {
        Name, Text string
    }
    dec := json.NewDecoder(strings.NewReader(jsonStream))

    // read open bracket
    t, err := dec.Token()
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf("%T: %v\n", t, t)

    // while the array contains values
    for dec.More() {
        var m Message
        // decode an array value (Message)
        err := dec.Decode(&m)
        if err != nil {
            log.Fatal(err)
        }

        fmt.Printf("%v: %v\n", m.Name, m.Text)
    }

    // read closing bracket
    t, err = dec.Token()
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf("%T: %v\n", t, t)

}

In this sample code:

• A decoder object (dec) is created to read JSON data from a string.
• The decoder reads tokens one by one, starting with the opening bracket of the array.
• A for loop iterates over array elements, parsing each into a Message struct.
• The names and text of each message are printed.
• The loop continues until no more elements are available in the array, indicated by the dec.More() function.
• Finally, the decoder reads and prints the closing bracket of the array.

This approach allows you to stream a large JSON array without loading the entire data structure into memory, optimizing resource usage and enabling efficient processing.

The above is the detailed content of How can I efficiently process large JSON arrays without loading them entirely into memory in Go?. For more information, please follow other related articles on the PHP Chinese website!