How to gracefully handle 'pipe closed' error in Python multi-process Pipe communication?

Python multi-process Pipe communication: elegant solution to "pipe closed" error

You may encounter a headache "pipe closed" error when using Pipe method in Python's multiprocessing module for interprocess communication. This article will analyze this problem in depth and provide a more robust solution.

Problems usually occur in parent-child process communication scenarios. For example, after a child process performs a time-consuming task, it sends data to the parent process through a pipeline and waits for the parent process's signal to end itself. If the parent process ends early and the child process tries to receive data from a closed pipeline, a "pipeline closed" error is raised.

While adding time.sleep() and explicitly closing child processes in parent processes can solve the problem, this is not an ideal solution because it relies on active waiting and control of the parent process.

A more elegant solution is to catch the EOFError exception in the child process . This exception is raised when the child process tries to receive data from the pipeline when the parent process ends early. By catching this exception, the child process can exit gracefully, avoiding program crashes.

The following shows the improved service.py and single.py code:

Improved service.py :

 import os
from multiprocessing import Process, Pipe

def start_child_process(child_conn):
    # Simulate time-consuming tasks# run_server_for_long_time()
    child_conn.send({"port": 123, "ret": 1, "pid": os.getpid()})

    try:
        signal = child_conn.recv() # Wait for parent process signal if signal:
            child_conn.close()
    except EOFError as e:
        print(f"EOFError: {e}") # Elegant handling of exceptions# You can add the necessary cleanup work here class Server:
    def __init__(self):
        self.child_conn = None
        self.child = None
        self.parent_conn, self.child_conn = Pipe()

    def run(self):
        self.child = Process(target=start_child_process, name="my_child_process", args=(self.child_conn,))
        self.child.start()

        data = self.parent_conn.recv()
        result = {
            "endpoints": {
                "http": f"http://127.0.0.1:{data['port']}/cmd",
                "ws": f"ws://127.0.0.1:{data['port']}/api",
            }
        }
        return result

    def stop(self):
        self.parent_conn.send(True)
        self.child.join()
        self.child = None

if __name__ == "__main__":
    server = Server()
    r = server.run()
    print("r:", r)

Improved single.py :

 from service import Server

def main():
    server = Server()
    result = server.run()
    print("r:", result)
    # server.stop() # Optional: explicitly stop the child process if __name__ == "__main__":
    main()

Catching EOFError by adding try...except block in the child process, the child process ends gracefully even if the parent process exits unexpectedly, avoiding the "pipeline closed" error. This approach improves the robustness of the program without relying on explicit shutdown operations of the parent process.

The above is the detailed content of How to gracefully handle 'pipe closed' error in Python multi-process Pipe communication?. For more information, please follow other related articles on the PHP Chinese website!