Multiprocessing
The multiprocessing module spawns separate OS processes, each with their own Python interpreter and memory space — bypassing the GIL entirely.
Multiprocessing = opening separate restaurant branches. Each branch (process) has its own kitchen, staff, and inventory. No shared stove fights, but shipping ingredients between branches (pickling) is expensive.
ProcessPoolExecutor (from concurrent.futures) is the high-level API. Processes communicate via Queue, Pipe, or shared memory (Value, Array). Data passed between processes must be picklable. Process startup is expensive (~50–100ms per process) — use pools, not one-off processes. multiprocessing.Pool.map() distributes work across a pool of workers.
The default start method is 'spawn' on Windows and macOS (fork-safe issues), 'fork' on Linux. Fork is fast but inherits parent memory including file descriptors and lock state — can cause deadlocks if locks are held at fork time. 'spawn' starts a fresh interpreter (safe, slower). 'forkserver' is a compromise. Shared memory (multiprocessing.shared_memory in Python 3.8+) avoids pickling overhead for large arrays. For ML/data workloads, use ProcessPoolExecutor with chunksize tuning. Interprocess communication via Queue uses pickle — large objects are expensive to transfer.
Multiprocessing bypasses the GIL by running separate Python processes — each process has its own interpreter, memory space, and GIL. This gives true CPU parallelism across cores. Use ProcessPoolExecutor for the cleanest API. The costs: process startup is ~100ms, and passing data between processes requires serialization (pickling). For CPU-bound number crunching, the parallelism benefit far outweighs these costs. For data-heavy workloads, use shared_memory to avoid pickling overhead.
Lambdas and locally-defined functions can't be pickled and thus can't be passed to ProcessPoolExecutor on non-Linux platforms. Define workers at module level. Also, if you modify a shared object in a subprocess, changes are NOT reflected in the parent — processes have independent memory.