Python进程创建方式

在 Python multiprocessing 中，子进程是如何被创建的，由 start method 决定multiprocessing。get_start_method()

方式	操作系统
fork	Linux / Unix（默认）
spawn	Windows（默认）、macOS(>=3.8)
forkserver	Linux

fork拷贝几乎所有资源

调用操作系统 fork()子进程复制父进程内存,初始状态完全一致（写时复制）。

父进程
 └── fork()
      └── 子进程（内存几乎零成本复制）

fork创建子进程的特点是：创建速度快、内存利用率高、代码简单；继承父进程状态、多线程程序fork极其危险、数据库连接、锁、Socket会被复制。应用的场景主要是：离线计算、批处理，大规模计算，不适应Web服务，已创建线程的进程，含连接池的服务。

• fork完全复制进程

  import multiprocessing
    
  def task():
      name.append("child")
      print(
          f"Current Process id is {multiprocessing.current_process().pid},name is :{name}"
      )
    
    
  if __name__ == "__main__":
      name = []
      multiprocessing.set_start_method("fork")
      process = multiprocessing.Process(target=task)
      process.start()
      process.join()
    
      name.append("main")
      print(
          f"Current Process id is {multiprocessing.current_process().pid},name is :{name}"
      )

  output:

  Current Process id is 17567,name is :['child']
  Current Process id is 17565,name is :['main']

• fork复制进程锁

  import multiprocessing
  
  
  def task():
      lock.acquire()
      print("child use lock")
      name.append("child")
      lock.release()
      print(
          f"Child Process id is {multiprocessing.current_process().pid},name is :{name}"
      )
  
  
  if __name__ == "__main__":
      name = ["main"]
      multiprocessing.set_start_method("fork")
      lock = multiprocessing.RLock()
      process = multiprocessing.Process(target=task)
      process.start()
      process.join()
      lock.acquire()
      print("main use lock")
      name.append("submain")
      lock.release()
      print(f"Main Process id is {multiprocessing.current_process().pid},name is :{name}")

  output:

  child use lock
  Child Process id is 19459,name is :['main', 'child']
  main use lock
  Main Process id is 19458,name is :['main', 'submain']

spawn参数传必备资源

启动一个全新的Python解释器，重新import主模块，通过pickle^1^传递参数。

pickle:把 Python 对象打包成字节，方便在不同进程之间传输，然后在子进程里还原。

全新进程
 └── import main.py
      └── 执行 target

spawn创建进程的特点是：安全、不继承父进程资源、线程安全；启动慢、占用内存大、需要可pickle的对象。适应的场景是Web服务、AI推理服务、金融系统。

import multiprocessing


def task(data):
    data.append("child")
    print(
        f"Child Process id is {multiprocessing.current_process().pid},args is :{data}"
    )


if __name__ == "__main__":
    name = []
    multiprocessing.set_start_method("spawn")
    process = multiprocessing.Process(target=task, args=(name,))
    process.start()
    process.join()
    name.append("main")
    print(f"Main Process id is {multiprocessing.current_process().pid},name is :{name}")

output:

Child Process id is 18830,args is :['child']
Main Process id is 18827,name is :['main']

forkserver参数传必备资源

先启动一个forkserver，子进程都从server fork，避免直接fork主进程。

主进程 ──> forkserver
              ├── 子进程1
              ├── 子进程2

forkserver的特点是：避免fork多线程主进程、性能优于spawn、状态更干净；只支持Linux、配置复杂。适应的场景是：大型后端服务、高可靠系统。

import multiprocessing


def task(data):
    data.append("child")
    print(
        f"Child Process id is {multiprocessing.current_process().pid},args is :{data}"
    )


if __name__ == "__main__":
    name = []
    multiprocessing.set_start_method("forkserver")
    process = multiprocessing.Process(target=task, args=(name,))
    process.start()
    process.join()
    name.append("main")
    print(f"Main Process id is {multiprocessing.current_process().pid},name is :{name}")

output:

Child Process id is 19015,args is :['child']
Main Process id is 19011,name is :['main']