How Can I Profile Memory Usage in Python?

Profiling Memory Usage in Python

Background:

Optimizing code for memory efficiency becomes crucial when exploring algorithms and their performance. To achieve this, monitoring memory usage is essential.

Python Memory Analysis:

Python provides the timeit function for runtime profiling. However, for memory analysis, Python 3.4 introduces the tracemalloc module.

Using tracemalloc:

To profile memory usage with tracemalloc:

import tracemalloc

# Start collecting memory usage data
tracemalloc.start()

# Execute code to analyze memory usage
# ...

# Take a snapshot of the memory usage data
snapshot = tracemalloc.take_snapshot()

# Display the top lines with memory consumption
display_top(snapshot)

Other approaches:

1. Background Memory Monitor Thread:

This approach creates a separate thread that periodically monitors memory usage while the main thread executes code:

import resource
import queue
from threading import Thread

def memory_monitor(command_queue, poll_interval=1):
    while True:
        try:
            command_queue.get(timeout=poll_interval)
            # Pause the code execution and record the memory usage
        except Empty:
            max_rss = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
            print('max RSS', max_rss)

# Start the memory monitor thread
queue = queue.Queue()
poll_interval = 0.1
monitor_thread = Thread(target=memory_monitor, args=(queue, poll_interval))
monitor_thread.start()

2. Using /proc/self/statm (Linux Only):

On Linux, the /proc/self/statm file provides detailed memory usage statistics, including:

Size    Total program size in pages
Resident    Resident set size in pages
Shared    Shared pages
Text    Text (code) pages
Lib    Shared library pages
Data    Data/stack pages

The above is the detailed content of How Can I Profile Memory Usage in Python?. For more information, please follow other related articles on the PHP Chinese website!