.. only:: html
.. figure:: /auto_examples/images/thumb/sphx_glr_reuse_executor_thumb.png
:ref:`sphx_glr_auto_examples_reuse_executor.py`
.. raw:: html
.. toctree::
:hidden:
/auto_examples/reuse_executor
.. raw:: html
.. container:: sphx-glr-footer
.. container:: sphx-glr-download
:download:`Download all examples in Python source code: auto_examples_python.zip `
.. container:: sphx-glr-download
:download:`Download all examples in Jupyter notebooks: auto_examples_jupyter.zip `
.. rst-class:: sphx-glr-signature
`Generated by Sphinx-Gallery `_
joblib-loky-51247f4/docs/index.rst 0000664 0000000 0000000 00000000436 15256713413 0017010 0 ustar 00root root 0000000 0000000 .. loky documentation master file, created by
sphinx-quickstart on Wed Aug 2 16:50:13 2017.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
.. include:: ../README.rst
.. toctree::
auto_examples/index
API
joblib-loky-51247f4/docs/requirements.txt 0000664 0000000 0000000 00000000033 15256713413 0020424 0 ustar 00root root 0000000 0000000 sphinx-gallery
matplotlib
joblib-loky-51247f4/examples/ 0000775 0000000 0000000 00000000000 15256713413 0016032 5 ustar 00root root 0000000 0000000 joblib-loky-51247f4/examples/README.txt 0000664 0000000 0000000 00000000051 15256713413 0017524 0 ustar 00root root 0000000 0000000 Gallery
=======
Basic usage of ``loky``
joblib-loky-51247f4/examples/cloudpickle_wrapper.py 0000664 0000000 0000000 00000010216 15256713413 0022442 0 ustar 00root root 0000000 0000000 # -*- coding: utf-8 -*-
"""
Serialization of un-picklable objects
=====================================
This example highlights the options for tempering with loky serialization
process.
"""
# Code source: Thomas Moreau
# License: BSD 3 clause
import sys
import time
import traceback
from loky import set_loky_pickler
from loky import get_reusable_executor
from loky import wrap_non_picklable_objects
###############################################################################
# First, define functions which cannot be pickled with the standard ``pickle``
# protocol. They cannot be serialized with ``pickle`` because they are defined
# in the ``__main__`` module. They can however be serialized with
# ``cloudpickle``.
#
def func_async(i, *args):
return 2 * i
###############################################################################
# With the default behavior, ``loky`` is to use ``cloudpickle`` to serialize
# the objects that are sent to the workers.
#
executor = get_reusable_executor(max_workers=1)
print(executor.submit(func_async, 21).result())
###############################################################################
# For most use-cases, using ``cloudpickle``` is efficient enough. However, this
# solution can be very slow to serialize large python objects, such as dict or
# list, compared to the standard ``pickle`` serialization.
#
# We have to pass an extra argument with a large list (or another large python
# object).
large_list = list(range(1000000))
t_start = time.time()
executor = get_reusable_executor(max_workers=1)
executor.submit(func_async, 21, large_list).result()
print(f"With cloudpickle serialization: {time.time() - t_start:.3f}s")
###############################################################################
# To mitigate this, it is possible to fully rely on ``pickle`` to serialize
# all communications between the main process and the workers. This can be done
# with an environment variable ``LOKY_PICKLER=pickle`` set before the
# script is launched, or with the switch ``set_loky_pickler`` provided in the
# ``loky`` API.
#
# Now set the `loky_pickler` to use the pickle serialization from stdlib. Here,
# we do not pass the desired function ``call_function`` as it is not picklable
# but it is replaced by ``id`` for demonstration purposes.
set_loky_pickler("pickle")
t_start = time.time()
executor = get_reusable_executor(max_workers=1)
executor.submit(id, large_list).result()
print(f"With pickle serialization: {time.time() - t_start:.3f}s")
###############################################################################
# However, the function and objects defined in ``__main__`` are not
# serializable anymore using ``pickle`` and it is not possible to call
# ``func_async`` using this pickler.
#
try:
executor = get_reusable_executor(max_workers=1)
executor.submit(func_async, 21, large_list).result()
except Exception:
traceback.print_exc(file=sys.stdout)
###############################################################################
# ``loky`` provides a wrapper function
# :func:`wrap_non_picklable_objects` to wrap the non-picklable function and
# indicate to the serialization process that this specific function should be
# serialized using ``cloudpickle``. This changes the serialization behavior
# only for this function and keeps using ``pickle`` for all other objects. The
# drawback of this solution is that it modifies the object. This should not
# cause many issues with functions but can have side effects with object
# instances.
#
@wrap_non_picklable_objects
def func_async_wrapped(i, *args):
return 2 * i
t_start = time.time()
executor = get_reusable_executor(max_workers=1)
executor.submit(func_async_wrapped, 21, large_list).result()
print(f"With default and wrapper: {time.time() - t_start:.3f}s")
###############################################################################
# The same wrapper can also be used for non-picklable classes. Note that the
# side effects of :func:`wrap_non_picklable_objects` on objects can break magic
# methods such as ``__add__`` and can mess up the ``isinstance`` and
# ``issubclass`` functions. Some improvements will be considered if use-cases
# are reported.
#
joblib-loky-51247f4/examples/reuse_executor.py 0000664 0000000 0000000 00000002233 15256713413 0021445 0 ustar 00root root 0000000 0000000 """
Reuse Executor
==============
This example highlights the ``loky`` API to reuse a ``ReusableProcessPool``.
The factory ``get_reusable_executor`` provides an executor. As long as this
executor is not in a broken state, it is reused for all the computation.
"""
import os
from loky import get_reusable_executor
def func_async(i):
pid = os.getpid()
return (2 * i, pid)
def test_1():
executor = get_reusable_executor(max_workers=1)
return executor.submit(func_async, 1)
def test_2():
executor = get_reusable_executor(max_workers=1)
return executor.submit(func_async, 2)
def test_3():
executor = get_reusable_executor(max_workers=1)
return executor.submit(func_async, 3)
f1 = test_1()
f2 = test_2()
f3 = test_3()
main_pid = os.getpid()
results = [f1.result(), f2.result(), f3.result()]
pids = [pid for _, pid in results]
for i, (val, pid) in enumerate(results):
assert val == 2 * (i + 1)
assert pid != main_pid
print("All the jobs were run in a process different from main process")
assert len(set(pids)) == 1
print(
"All the computation where run in a single `ProcessPoolExecutor` with "
f"worker pid={pids[0]}"
)
joblib-loky-51247f4/examples/reuseable_executor.py 0000664 0000000 0000000 00000002657 15256713413 0022303 0 ustar 00root root 0000000 0000000 """
Advanced Executor Setup
=======================
It is possible to reuse an executor even if it has a complicated setup. When
using the parameter `reuse=True`, the executor is resized if needed but the
arguments stay the same.
"""
from time import sleep
import multiprocessing as mp
import loky
from loky import get_reusable_executor
# Store the initialization status in a global variable of a module.
loky._INITIALIZER_STATUS = "uninitialized"
def initializer(x):
print(f"[{mp.current_process().name}] init")
loky._INITIALIZER_STATUS = x
def return_initializer_status(delay=0):
sleep(delay)
return getattr(loky, "_INITIALIZER_STATUS", "uninitialized")
executor = get_reusable_executor(
max_workers=2,
initializer=initializer,
initargs=("initialized",),
context="loky",
timeout=1000,
)
assert loky._INITIALIZER_STATUS == "uninitialized"
executor.submit(return_initializer_status).result()
assert executor.submit(return_initializer_status).result() == "initialized"
# With reuse=True, the executor use the same initializer
executor = get_reusable_executor(max_workers=4, reuse=True)
for x in executor.map(return_initializer_status, [0.5] * 4):
assert x == "initialized"
# With reuse='auto', the initializer is not used anymore as a new executor
# is created.
executor = get_reusable_executor(max_workers=4)
for x in executor.map(return_initializer_status, [0.1] * 4):
assert x == "uninitialized"
joblib-loky-51247f4/loky/ 0000775 0000000 0000000 00000000000 15256713413 0015172 5 ustar 00root root 0000000 0000000 joblib-loky-51247f4/loky/__init__.py 0000664 0000000 0000000 00000002121 15256713413 0017277 0 ustar 00root root 0000000 0000000 r"""The :mod:`loky` module manages a pool of worker that can be re-used across time.
It provides a robust and dynamic implementation os the
:class:`ProcessPoolExecutor` and a function :func:`get_reusable_executor` which
hide the pool management under the hood.
"""
from concurrent.futures import (
ALL_COMPLETED,
FIRST_COMPLETED,
FIRST_EXCEPTION,
CancelledError,
Executor,
TimeoutError,
as_completed,
wait,
)
from ._base import Future
from .backend.context import cpu_count
from .backend.reduction import set_loky_pickler
from .reusable_executor import get_reusable_executor
from .cloudpickle_wrapper import wrap_non_picklable_objects
from .process_executor import BrokenProcessPool, ProcessPoolExecutor
__all__ = [
"get_reusable_executor",
"cpu_count",
"wait",
"as_completed",
"Future",
"Executor",
"ProcessPoolExecutor",
"BrokenProcessPool",
"CancelledError",
"TimeoutError",
"FIRST_COMPLETED",
"FIRST_EXCEPTION",
"ALL_COMPLETED",
"wrap_non_picklable_objects",
"set_loky_pickler",
]
__version__ = "3.7.0"
joblib-loky-51247f4/loky/_base.py 0000664 0000000 0000000 00000002041 15256713413 0016612 0 ustar 00root root 0000000 0000000 ###############################################################################
# Modification of concurrent.futures.Future
#
# author: Thomas Moreau and Olivier Grisel
#
# adapted from concurrent/futures/_base.py (17/02/2017)
# * Do not use yield from
# * Use old super syntax
#
# Copyright 2009 Brian Quinlan. All Rights Reserved.
# Licensed to PSF under a Contributor Agreement.
from concurrent.futures import Future as _BaseFuture
from concurrent.futures._base import LOGGER
# To make loky._base.Future instances awaitable by concurrent.futures.wait,
# derive our custom Future class from _BaseFuture. _invoke_callback is the only
# modification made to this class in loky.
# TODO investigate why using `concurrent.futures.Future` directly does not
# always work in our test suite.
class Future(_BaseFuture):
def _invoke_callbacks(self):
for callback in self._done_callbacks:
try:
callback(self)
except BaseException:
LOGGER.exception(f"exception calling callback for {self!r}")
joblib-loky-51247f4/loky/backend/ 0000775 0000000 0000000 00000000000 15256713413 0016561 5 ustar 00root root 0000000 0000000 joblib-loky-51247f4/loky/backend/__init__.py 0000664 0000000 0000000 00000000470 15256713413 0020673 0 ustar 00root root 0000000 0000000 import os
from multiprocessing import synchronize
from .context import get_context
def _make_name():
return f"/loky-{os.getpid()}-{next(synchronize.SemLock._rand)}"
# monkey patch the name creation for multiprocessing
synchronize.SemLock._make_name = staticmethod(_make_name)
__all__ = ["get_context"]
joblib-loky-51247f4/loky/backend/_posix_reduction.py 0000664 0000000 0000000 00000003360 15256713413 0022512 0 ustar 00root root 0000000 0000000 ###############################################################################
# Extra reducers for Unix based system and connections objects
#
# author: Thomas Moreau and Olivier Grisel
#
# adapted from multiprocessing/reduction.py (17/02/2017)
# * Add adapted reduction for LokyProcesses and socket/Connection
#
import os
import socket
import _socket
from multiprocessing.connection import Connection
from multiprocessing.context import get_spawning_popen
from .reduction import register
HAVE_SEND_HANDLE = (
hasattr(socket, "CMSG_LEN")
and hasattr(socket, "SCM_RIGHTS")
and hasattr(socket.socket, "sendmsg")
)
def _mk_inheritable(fd):
os.set_inheritable(fd, True)
return fd
def DupFd(fd):
"""Return a wrapper for an fd."""
popen_obj = get_spawning_popen()
if popen_obj is not None:
return popen_obj.DupFd(popen_obj.duplicate_for_child(fd))
elif HAVE_SEND_HANDLE:
from multiprocessing import resource_sharer
return resource_sharer.DupFd(fd)
else:
raise TypeError(
"Cannot pickle connection object. This object can only be "
"passed when spawning a new process"
)
def _reduce_socket(s):
df = DupFd(s.fileno())
return _rebuild_socket, (df, s.family, s.type, s.proto)
def _rebuild_socket(df, family, type, proto):
fd = df.detach()
return socket.fromfd(fd, family, type, proto)
def rebuild_connection(df, readable, writable):
fd = df.detach()
return Connection(fd, readable, writable)
def reduce_connection(conn):
df = DupFd(conn.fileno())
return rebuild_connection, (df, conn.readable, conn.writable)
register(socket.socket, _reduce_socket)
register(_socket.socket, _reduce_socket)
register(Connection, reduce_connection)
joblib-loky-51247f4/loky/backend/_win_reduction.py 0000664 0000000 0000000 00000001253 15256713413 0022144 0 ustar 00root root 0000000 0000000 ###############################################################################
# Extra reducers for Windows system and connections objects
#
# author: Thomas Moreau and Olivier Grisel
#
# adapted from multiprocessing/reduction.py (17/02/2017)
# * Add adapted reduction for LokyProcesses and socket/PipeConnection
#
import socket
from multiprocessing import connection
from multiprocessing.reduction import _reduce_socket
from .reduction import register
# register reduction for win32 communication objects
register(socket.socket, _reduce_socket)
register(connection.Connection, connection.reduce_connection)
register(connection.PipeConnection, connection.reduce_pipe_connection)
joblib-loky-51247f4/loky/backend/context.py 0000664 0000000 0000000 00000050300 15256713413 0020615 0 ustar 00root root 0000000 0000000 ###############################################################################
# Basic context management with LokyContext
#
# author: Thomas Moreau and Olivier Grisel
#
# adapted from multiprocessing/context.py
# * Create a context ensuring loky uses only objects that are compatible
# * Add LokyContext to the list of context of multiprocessing so loky can be
# used with multiprocessing.set_start_method
# * Implement a CFS-aware amd physical-core aware cpu_count function.
#
import ctypes
import math
import multiprocessing as mp
import os
import subprocess
import sys
import traceback
import warnings
from ctypes import wintypes
from multiprocessing import get_context as mp_get_context
from multiprocessing.context import BaseContext
from concurrent.futures.process import _MAX_WINDOWS_WORKERS
from .process import LokyProcess, LokyInitMainProcess
# Apparently, on older Python versions, loky cannot work 61 workers on Windows
# but instead 60: Β―\_(γ)_/Β―
if sys.version_info < (3, 10):
_MAX_WINDOWS_WORKERS = _MAX_WINDOWS_WORKERS - 1
START_METHODS = ["loky", "loky_init_main", "spawn"]
if sys.platform != "win32":
START_METHODS += ["fork", "forkserver"]
_DEFAULT_START_METHOD = None
# Cache for the number of physical cores, to avoid repeating subprocess
# calls or re-parsing /proc/cpuinfo. Keyed by None for the whole machine, or
# by a frozenset of logical CPU ids when restricted to a CPU affinity mask,
# see `_count_physical_cores`.
physical_cores_cache = {}
def get_context(method=None):
# Try to overload the default context
method = method or _DEFAULT_START_METHOD or "loky"
if method == "fork":
# If 'fork' is explicitly requested, warn user about potential issues.
warnings.warn(
"`fork` start method should not be used with "
"`loky` as it does not respect POSIX. Try using "
"`spawn` or `loky` instead.",
UserWarning,
)
try:
return mp_get_context(method)
except ValueError:
raise ValueError(
f"Unknown context '{method}'. Value should be in "
f"{START_METHODS}."
)
def set_start_method(method, force=False):
global _DEFAULT_START_METHOD
if _DEFAULT_START_METHOD is not None and not force:
raise RuntimeError("context has already been set")
assert method is None or method in START_METHODS, (
f"'{method}' is not a valid start_method. It should be in "
f"{START_METHODS}"
)
_DEFAULT_START_METHOD = method
def get_start_method():
return _DEFAULT_START_METHOD
def cpu_count(only_physical_cores=False):
"""Return the number of CPUs the current process can use.
The returned number of CPUs accounts for:
* the number of CPUs in the system, as given by
``multiprocessing.cpu_count``;
* the CPU affinity settings of the current process
(available on some Unix systems);
* Cgroup CPU bandwidth limit (available on Linux only, typically
set by docker and similar container orchestration systems);
* the value of the LOKY_MAX_CPU_COUNT environment variable if defined.
and is given as the minimum of these constraints.
If ``only_physical_cores`` is True, return the number of physical cores
instead of the number of logical cores (hyperthreading / SMT): this is
given as the minimum of the same constraints as above, together with the
number of physical cores.
On Linux, the computed number of physical cores is restrained to the
physical cores reachable through the CPU affinity mask.
On other platforms, it's the number of physical cores on the whole
machine, since the affinity mask cannot be taken into account there.
Note that on Windows, the returned number of CPUs cannot exceed 61 (or 60 for
Python < 3.10), see:
https://bugs.python.org/issue26903.
It is also always larger or equal to 1.
"""
# Note: os.cpu_count() is allowed to return None in its docstring
os_cpu_count = os.cpu_count() or 1
if sys.platform == "win32":
# On Windows, attempting to use more than 61 CPUs would result in a
# OS-level error. See https://bugs.python.org/issue26903. According to
# https://learn.microsoft.com/en-us/windows/win32/procthread/processor-groups
# it might be possible to go beyond with a lot of extra work but this
# does not look easy.
os_cpu_count = min(os_cpu_count, _MAX_WINDOWS_WORKERS)
cpu_affinity_set = _cpu_count_affinity_set()
cpu_count_affinity = (
os_cpu_count if cpu_affinity_set is None else len(cpu_affinity_set)
)
cpu_count_cgroup = _cpu_count_cgroup(os_cpu_count)
cpu_count_loky = int(os.environ.get("LOKY_MAX_CPU_COUNT", os_cpu_count))
if not only_physical_cores:
return max(
1,
min(
os_cpu_count,
cpu_count_affinity,
cpu_count_cgroup,
cpu_count_loky,
),
)
cpu_count_physical, exception = _count_physical_cores(
# On Linux, try to collapse SMT/hyper-threading sibling logical CPUs
# reachable through the current CPU affinity mask that share the same
# physical core. See https://github.com/joblib/loky/issues/639.
# On other platforms we lack easy access to CPU topology info to refine
# the count, so just pass None for `cpu_affinity_set`.
cpu_affinity_set
if sys.platform == "linux"
else None
)
if cpu_count_physical == "not found":
cpu_count_physical = os_cpu_count
_warn_physical_cores_not_found(exception)
return max(
1,
min(
os_cpu_count,
cpu_count_physical,
cpu_count_affinity,
cpu_count_cgroup,
cpu_count_loky,
),
)
def _warn_physical_cores_not_found(exception):
if exception is not None:
# warns only the first time
warnings.warn(
"Could not find the number of physical cores for the "
f"following reason:\n{exception}\n"
"Returning the number of logical cores instead. You can "
"silence this warning by setting LOKY_MAX_CPU_COUNT to "
"the number of cores you want to use."
)
traceback.print_tb(exception.__traceback__)
def _cpu_count_cgroup(os_cpu_count):
# Cgroup CPU bandwidth limit available in Linux since 2.6 kernel
cpu_max_fname = "/sys/fs/cgroup/cpu.max"
cfs_quota_fname = "/sys/fs/cgroup/cpu/cpu.cfs_quota_us"
cfs_period_fname = "/sys/fs/cgroup/cpu/cpu.cfs_period_us"
cpu_quota_us = None
cpu_period_us = None
if os.path.exists(cpu_max_fname):
# cgroup v2
# https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html
with open(cpu_max_fname) as fh:
# Parse the quota and period values
parts = fh.read().strip().split()
if len(parts) == 2:
cpu_quota_us, cpu_period_us = parts
# If len(parts) != 2, leave as None and fall back to v1
# If we didn't get values from cgroup v2, try cgroup v1
if cpu_quota_us is None or cpu_period_us is None:
if os.path.exists(cfs_quota_fname) and os.path.exists(
cfs_period_fname
):
# cgroup v1
# https://www.kernel.org/doc/html/latest/scheduler/sched-bwc.html#management
with open(cfs_quota_fname) as fh:
cpu_quota_us = fh.read().strip()
with open(cfs_period_fname) as fh:
cpu_period_us = fh.read().strip()
else:
# No Cgroup CPU bandwidth limit (e.g. non-Linux platform)
cpu_quota_us = "max"
if cpu_quota_us == "max":
# No active Cgroup quota on a Cgroup-capable platform
return os_cpu_count
else:
cpu_quota_us = int(cpu_quota_us)
cpu_period_us = int(cpu_period_us)
if cpu_quota_us > 0 and cpu_period_us > 0:
return math.ceil(cpu_quota_us / cpu_period_us)
else: # pragma: no cover
# Setting a negative cpu_quota_us value is a valid way to disable
# cgroup CPU bandwidth limits
return os_cpu_count
def _cpu_count_affinity_set():
"""Return the current CPU affinity mask as a set of logical CPU ids.
Return None if the affinity mask cannot be determined on this platform,
for instance because neither `os.sched_getaffinity` nor `psutil` are
available.
"""
if hasattr(os, "sched_getaffinity"):
try:
return os.sched_getaffinity(0)
except NotImplementedError:
pass
# On some platforms, os.sched_getaffinity does not exist or raises
# NotImplementedError, let's try with the psutil if installed.
try:
import psutil
p = psutil.Process()
if hasattr(p, "cpu_affinity"):
return set(p.cpu_affinity())
except ImportError: # pragma: no cover
if (
sys.platform == "linux"
and os.environ.get("LOKY_MAX_CPU_COUNT") is None
):
# Some platforms don't implement os.sched_getaffinity on Linux which
# can cause severe oversubscription problems. Better warn the
# user in this particularly pathological case which can wreck
# havoc, typically on CI workers.
warnings.warn(
"Failed to inspect CPU affinity constraints on this system. "
"Please install psutil or explictly set LOKY_MAX_CPU_COUNT."
)
return None
def _count_physical_cores(cpu_set=None):
"""Return a tuple (number of physical cores, exception)
If the number of physical cores is found, exception is set to None.
If it has not been found, return ("not found", exception).
If `cpu_set` is not None, only the logical CPUs it contains are
considered when counting physical cores, which also collapses
SMT/hyper-threading sibling logical CPUs that share the same physical
core. Only implemented for Linux, where per-CPU topology information is
readily available; callers are expected to only pass a non-None
`cpu_set` on Linux.
The number of physical cores is cached (per distinct `cpu_set` when one
is passed) to avoid repeating subprocess calls or re-parsing
/proc/cpuinfo.
"""
exception = None
# First check if the value is cached
global physical_cores_cache
cache_key = None if cpu_set is None else frozenset(cpu_set)
if cache_key in physical_cores_cache:
return physical_cores_cache[cache_key], None
# Not cached yet, find it
try:
if sys.platform == "linux":
cpu_count_physical = _count_physical_cores_linux(cpu_set)
elif cpu_set is not None:
raise NotImplementedError(
"affinity-aware physical core counting is only "
"implemented on Linux"
)
elif sys.platform == "win32":
cpu_count_physical = _count_physical_cores_win32()
elif sys.platform == "darwin":
cpu_count_physical = _count_physical_cores_darwin()
elif sys.platform.startswith("freebsd"):
cpu_count_physical = _count_physical_cores_freebsd()
else:
raise NotImplementedError(f"unsupported platform: {sys.platform}")
# if cpu_count_physical < 1, we did not find a valid value
if cpu_count_physical < 1:
raise ValueError(f"found {cpu_count_physical} physical cores < 1")
except Exception as e:
exception = e
cpu_count_physical = "not found"
# Put the result in cache
physical_cores_cache[cache_key] = cpu_count_physical
return cpu_count_physical, exception
def _count_physical_cores_linux(cpu_set=None):
"""Return the number of distinct physical cores on Linux, by parsing
/proc/cpuinfo.
A physical core is identified by its (physical id, core id) pair: core
id alone is only guaranteed unique within a physical package, so a
multi-socket machine can otherwise under-count cores that share the
same core id across sockets.
If `cpu_set` is not None, only the logical CPUs it contains are
considered, which also collapses SMT/hyper-threading sibling logical
CPUs that share the same physical core.
"""
with open("/proc/cpuinfo") as f:
cpuinfo = f.read()
cores = set()
processor = core_id = physical_id = None
for line in cpuinfo.splitlines() + [""]:
if not line.strip():
# blank line: end of the current logical CPU block
if core_id is not None and (
cpu_set is None or processor in cpu_set
):
cores.add((physical_id, core_id))
processor = core_id = physical_id = None
continue
key, sep, value = line.partition(":")
if not sep:
continue
key = key.strip()
value = value.strip()
if key == "processor":
processor = int(value)
elif key == "core id":
core_id = value
elif key == "physical id":
physical_id = value
if not cores:
raise ValueError("could not find any physical core in /proc/cpuinfo")
return len(cores)
def _count_physical_cores_win32():
try:
return _count_physical_cores_win32_ctypes()
except Exception:
pass # fallback to powershell
try:
return _count_physical_cores_win32_powershell()
except Exception:
pass # fallback to wmic (older Windows versions; deprecated now)
cpu_info = subprocess.run(
"wmic CPU Get NumberOfCores /Format:csv".split(),
capture_output=True,
text=True,
creationflags=subprocess.CREATE_NO_WINDOW,
)
cpu_info = cpu_info.stdout.splitlines()
cpu_info = [
l.split(",")[1] for l in cpu_info if (l and l != "Node,NumberOfCores")
]
return sum(map(int, cpu_info))
def _count_physical_cores_win32_powershell():
cmd = "-NoProfile -Command (Get-CimInstance -ClassName Win32_Processor).NumberOfCores"
cpu_info = subprocess.run(
f"powershell.exe {cmd}".split(),
capture_output=True,
text=True,
creationflags=subprocess.CREATE_NO_WINDOW,
)
cpu_info = cpu_info.stdout.splitlines()
return sum(map(int, cpu_info))
def _count_physical_cores_win32_ctypes():
ERROR_INSUFFICIENT_BUFFER = 122
RelationProcessorCore = 0
kernel32 = ctypes.WinDLL("kernel32", use_last_error=True)
get_logical_processor_information = (
kernel32.GetLogicalProcessorInformationEx
)
get_logical_processor_information.argtypes = [
wintypes.DWORD,
ctypes.c_void_p,
ctypes.POINTER(wintypes.DWORD),
]
get_logical_processor_information.restype = wintypes.BOOL
# Mirror the header of SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX. The full
# structure is variable-sized, and only Relationship and Size are needed.
# https://learn.microsoft.com/en-us/windows/win32/api/winnt/ns-winnt-system_logical_processor_information_ex
class SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX(ctypes.Structure):
_fields_ = [
("Relationship", wintypes.DWORD),
("Size", wintypes.DWORD),
]
# First obtain the required buffer size. This call is expected to fail with
# ERROR_INSUFFICIENT_BUFFER and set returned_length.
# https://learn.microsoft.com/en-us/windows/win32/api/sysinfoapi/nf-sysinfoapi-getlogicalprocessorinformationex
returned_length = wintypes.DWORD()
returned_length_ref = ctypes.byref(returned_length)
if get_logical_processor_information(
RelationProcessorCore, None, returned_length_ref
):
raise RuntimeError("unexpected successful buffer sizing call")
error = ctypes.get_last_error()
if error != ERROR_INSUFFICIENT_BUFFER:
raise ctypes.WinError(error)
buf = ctypes.create_string_buffer(returned_length.value)
if not get_logical_processor_information(
RelationProcessorCore, buf, returned_length_ref
):
raise ctypes.WinError(ctypes.get_last_error())
offset = 0
physical_core_count = 0
header_size = ctypes.sizeof(SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX)
while offset < returned_length.value:
remaining = returned_length.value - offset
if remaining < header_size:
raise RuntimeError("truncated processor information record")
processor_core_info = (
SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX.from_buffer(buf, offset)
)
record_size = processor_core_info.Size
if record_size < header_size or record_size > remaining:
raise RuntimeError("invalid processor information record size")
if processor_core_info.Relationship != RelationProcessorCore:
raise RuntimeError("unexpected logical processor relationship")
physical_core_count += 1
offset += record_size
if physical_core_count == 0:
raise RuntimeError("Windows reported no active physical cores")
if physical_core_count < 1:
raise RuntimeError(
"GetLogicalProcessorInformationEx returned no physical cores"
)
return physical_core_count
def _count_physical_cores_darwin():
cpu_info = subprocess.run(
"sysctl -n hw.physicalcpu".split(),
capture_output=True,
text=True,
)
cpu_info = cpu_info.stdout
return int(cpu_info)
def _count_physical_cores_freebsd():
cpu_info = subprocess.run(
"sysctl -n kern.smp.cores".split(),
capture_output=True,
text=True,
)
cpu_info = cpu_info.stdout
return int(cpu_info)
class LokyContext(BaseContext):
"""Context relying on the LokyProcess."""
_name = "loky"
Process = LokyProcess
cpu_count = staticmethod(cpu_count)
def Queue(self, maxsize=0, reducers=None):
"""Returns a queue object"""
from .queues import Queue
return Queue(maxsize, reducers=reducers, ctx=self.get_context())
def SimpleQueue(self, reducers=None):
"""Returns a queue object"""
from .queues import SimpleQueue
return SimpleQueue(reducers=reducers, ctx=self.get_context())
if sys.platform != "win32":
"""For Unix platform, use our custom implementation of synchronize
ensuring that we use the loky.backend.resource_tracker to clean-up
the semaphores in case of a worker crash.
"""
def Semaphore(self, value=1):
"""Returns a semaphore object"""
from .synchronize import Semaphore
return Semaphore(value=value)
def BoundedSemaphore(self, value):
"""Returns a bounded semaphore object"""
from .synchronize import BoundedSemaphore
return BoundedSemaphore(value)
def Lock(self):
"""Returns a lock object"""
from .synchronize import Lock
return Lock()
def RLock(self):
"""Returns a recurrent lock object"""
from .synchronize import RLock
return RLock()
def Condition(self, lock=None):
"""Returns a condition object"""
from .synchronize import Condition
return Condition(lock)
def Event(self):
"""Returns an event object"""
from .synchronize import Event
return Event()
class LokyInitMainContext(LokyContext):
"""Extra context with LokyProcess, which does load the main module
This context is used for compatibility in the case ``cloudpickle`` is not
present on the running system. This permits to load functions defined in
the ``main`` module, using proper safeguards. The declaration of the
``executor`` should be protected by ``if __name__ == "__main__":`` and the
functions and variable used from main should be out of this block.
This mimics the default behavior of multiprocessing under Windows and the
behavior of the ``spawn`` start method on a posix system.
For more details, see the end of the following section of python doc
https://docs.python.org/3/library/multiprocessing.html#multiprocessing-programming
"""
_name = "loky_init_main"
Process = LokyInitMainProcess
# Register loky context so it works with multiprocessing.get_context
ctx_loky = LokyContext()
mp.context._concrete_contexts["loky"] = ctx_loky
mp.context._concrete_contexts["loky_init_main"] = LokyInitMainContext()
joblib-loky-51247f4/loky/backend/fork_exec.py 0000664 0000000 0000000 00000004417 15256713413 0021106 0 ustar 00root root 0000000 0000000 ###############################################################################
# Launch a subprocess using forkexec and make sure only the needed fd are
# shared in the two process.
#
# author: Thomas Moreau and Olivier Grisel
#
import sys
import os
import subprocess
def fork_exec(cmd, keep_fds, env=None):
import _posixsubprocess
# Encoded command args as bytes:
cmd = [os.fsencode(arg) for arg in cmd]
# Copy the environment variables to set in the child process (also encoded
# as bytes).
env = env or {}
env = {**os.environ, **env}
encoded_env = []
for key, value in env.items():
encoded_env.append(os.fsencode(f"{key}={value}"))
# Fds with fileno larger than 3 (stdin=0, stdout=1, stderr=2) are be closed
# in the child process, except for those passed in keep_fds.
keep_fds = tuple(sorted(map(int, keep_fds)))
errpipe_read, errpipe_write = os.pipe()
if sys.version_info >= (3, 14):
# Python >= 3.14 removed allow_vfork from _posixsubprocess.fork_exec,
# see https://github.com/python/cpython/pull/121383
pgid_to_set = [-1]
allow_vfork = []
elif sys.version_info >= (3, 11):
# Python 3.11 - 3.13 has allow_vfork in _posixsubprocess.fork_exec
pgid_to_set = [-1]
allow_vfork = [subprocess._USE_VFORK]
else:
# Python < 3.11
pgid_to_set = []
allow_vfork = []
try:
return _posixsubprocess.fork_exec(
cmd, # args
cmd[0:1], # executable_list
True, # close_fds
keep_fds, # pass_fds
None, # cwd
encoded_env, # env
-1, # p2cread
-1, # p2cwrite
-1, # c2pread
-1, # c2pwrite
-1, # errread
-1, # errwrite
errpipe_read, # errpipe_read
errpipe_write, # errpipe_write
False, # restore_signal
False, # call_setsid
*pgid_to_set, # pgid_to_set
None, # gid
None, # extra_groups
None, # uid
-1, # child_umask
None, # preexec_fn
*allow_vfork, # extra flag if vfork is available
)
finally:
os.close(errpipe_read)
os.close(errpipe_write)
joblib-loky-51247f4/loky/backend/popen_loky_posix.py 0000664 0000000 0000000 00000012645 15256713413 0022544 0 ustar 00root root 0000000 0000000 ###############################################################################
# Popen for LokyProcess.
#
# author: Thomas Moreau and Olivier Grisel
#
import os
import sys
import signal
import pickle
from io import BytesIO
from multiprocessing import util, process
from multiprocessing.connection import wait
from multiprocessing.context import set_spawning_popen
from . import reduction, resource_tracker, spawn
__all__ = ["Popen"]
#
# Wrapper for an fd used while launching a process
#
class _DupFd:
def __init__(self, fd):
self.fd = reduction._mk_inheritable(fd)
def detach(self):
return self.fd
#
# Start child process using subprocess.Popen
#
class Popen:
method = "loky"
DupFd = _DupFd
def __init__(self, process_obj):
sys.stdout.flush()
sys.stderr.flush()
self.returncode = None
self._fds = []
self._launch(process_obj)
def duplicate_for_child(self, fd):
self._fds.append(fd)
return reduction._mk_inheritable(fd)
def poll(self, flag=os.WNOHANG):
if self.returncode is None:
while True:
try:
pid, sts = os.waitpid(self.pid, flag)
except OSError:
# Child process not yet created. See #1731717
# e.errno == errno.ECHILD == 10
return None
else:
break
if pid == self.pid:
if os.WIFSIGNALED(sts):
self.returncode = -os.WTERMSIG(sts)
else:
assert os.WIFEXITED(sts)
self.returncode = os.WEXITSTATUS(sts)
return self.returncode
def wait(self, timeout=None):
if self.returncode is None:
if timeout is not None:
if not wait([self.sentinel], timeout):
return None
# This shouldn't block if wait() returned successfully.
return self.poll(os.WNOHANG if timeout == 0.0 else 0)
return self.returncode
def terminate(self):
if self.returncode is None:
try:
os.kill(self.pid, signal.SIGTERM)
except ProcessLookupError:
pass
except OSError:
if self.wait(timeout=0.1) is None:
raise
def _launch(self, process_obj):
tracker_fd = resource_tracker._resource_tracker.getfd()
fp = BytesIO()
set_spawning_popen(self)
try:
prep_data = spawn.get_preparation_data(
process_obj._name,
getattr(process_obj, "init_main_module", True),
)
reduction.dump(prep_data, fp)
reduction.dump(process_obj, fp)
finally:
set_spawning_popen(None)
try:
parent_r, child_w = os.pipe()
child_r, parent_w = os.pipe()
# for fd in self._fds:
# _mk_inheritable(fd)
cmd_python = [sys.executable]
cmd_python += ["-m", self.__module__]
cmd_python += ["--process-name", str(process_obj.name)]
cmd_python += ["--pipe", str(reduction._mk_inheritable(child_r))]
reduction._mk_inheritable(child_w)
reduction._mk_inheritable(tracker_fd)
self._fds += [child_r, child_w, tracker_fd]
if os.name == "posix":
mp_tracker_fd = prep_data["mp_tracker_fd"]
self.duplicate_for_child(mp_tracker_fd)
from .fork_exec import fork_exec
pid = fork_exec(cmd_python, self._fds, env=process_obj.env)
util.debug(
f"launched python with pid {pid} and cmd:\n{cmd_python}"
)
self.sentinel = parent_r
method = "getbuffer"
if not hasattr(fp, method):
method = "getvalue"
with os.fdopen(parent_w, "wb") as f:
f.write(getattr(fp, method)())
self.pid = pid
finally:
if parent_r is not None:
util.Finalize(self, os.close, (parent_r,))
for fd in (child_r, child_w):
if fd is not None:
os.close(fd)
@staticmethod
def thread_is_spawning():
return True
if __name__ == "__main__":
import argparse
parser = argparse.ArgumentParser("Command line parser")
parser.add_argument(
"--pipe", type=int, required=True, help="File handle for the pipe"
)
parser.add_argument(
"--process-name",
type=str,
default=None,
help="Identifier for debugging purpose",
)
args = parser.parse_args()
info = {}
exitcode = 1
try:
with os.fdopen(args.pipe, "rb") as from_parent:
process.current_process()._inheriting = True
try:
prep_data = pickle.load(from_parent)
spawn.prepare(prep_data)
process_obj = pickle.load(from_parent)
finally:
del process.current_process()._inheriting
exitcode = process_obj._bootstrap()
except Exception:
print("\n\n" + "-" * 80)
print(f"{args.process_name} failed with traceback: ")
print("-" * 80)
import traceback
print(traceback.format_exc())
print("\n" + "-" * 80)
finally:
if from_parent is not None:
from_parent.close()
sys.exit(exitcode)
joblib-loky-51247f4/loky/backend/popen_loky_win32.py 0000664 0000000 0000000 00000012274 15256713413 0022342 0 ustar 00root root 0000000 0000000 import os
import sys
import msvcrt
import _winapi
from pickle import load
from multiprocessing import process, util
from multiprocessing.context import set_spawning_popen
from multiprocessing.popen_spawn_win32 import Popen as _Popen
from . import reduction, spawn
__all__ = ["Popen"]
#
#
#
def _path_eq(p1, p2):
return p1 == p2 or os.path.normcase(p1) == os.path.normcase(p2)
WINENV = hasattr(sys, "_base_executable") and not _path_eq(
sys.executable, sys._base_executable
)
def _close_handles(*handles):
for handle in handles:
_winapi.CloseHandle(handle)
#
# We define a Popen class similar to the one from subprocess, but
# whose constructor takes a process object as its argument.
#
class Popen(_Popen):
"""
Start a subprocess to run the code of a process object.
We differ from cpython implementation with the way we handle environment
variables, in order to be able to modify then in the child processes before
importing any library, in order to control the number of threads in C-level
threadpools.
We also use the loky preparation data, in particular to handle main_module
inits and the loky resource tracker.
"""
method = "loky"
def __init__(self, process_obj):
prep_data = spawn.get_preparation_data(
process_obj._name, getattr(process_obj, "init_main_module", True)
)
# read end of pipe will be duplicated by the child process
# -- see spawn_main() in spawn.py.
#
# bpo-33929: Previously, the read end of pipe was "stolen" by the child
# process, but it leaked a handle if the child process had been
# terminated before it could steal the handle from the parent process.
rhandle, whandle = _winapi.CreatePipe(None, 0)
wfd = msvcrt.open_osfhandle(whandle, 0)
cmd = get_command_line(parent_pid=os.getpid(), pipe_handle=rhandle)
python_exe = spawn.get_executable()
# copy the environment variables to set in the child process
child_env = {**os.environ, **process_obj.env}
# bpo-35797: When running in a venv, we bypass the redirect
# executor and launch our base Python.
if WINENV and _path_eq(python_exe, sys.executable):
cmd[0] = python_exe = sys._base_executable
child_env["__PYVENV_LAUNCHER__"] = sys.executable
cmd = " ".join(f'"{x}"' for x in cmd)
with open(wfd, "wb") as to_child:
# start process
try:
hp, ht, pid, _ = _winapi.CreateProcess(
python_exe,
cmd,
None,
None,
False,
0,
child_env,
None,
None,
)
_winapi.CloseHandle(ht)
except BaseException:
_winapi.CloseHandle(rhandle)
raise
# set attributes of self
self.pid = pid
self.returncode = None
self._handle = hp
self.sentinel = int(hp)
self.finalizer = util.Finalize(
self, _close_handles, (self.sentinel, int(rhandle))
)
# send information to child
set_spawning_popen(self)
try:
reduction.dump(prep_data, to_child)
reduction.dump(process_obj, to_child)
finally:
set_spawning_popen(None)
def get_command_line(pipe_handle, parent_pid, **kwds):
"""Returns prefix of command line used for spawning a child process."""
if getattr(sys, "frozen", False):
return [sys.executable, "--multiprocessing-fork", pipe_handle]
else:
prog = (
"from loky.backend.popen_loky_win32 import main; "
f"main(pipe_handle={pipe_handle}, parent_pid={parent_pid})"
)
opts = util._args_from_interpreter_flags()
return [
spawn.get_executable(),
*opts,
"-c",
prog,
"--multiprocessing-fork",
]
def is_forking(argv):
"""Return whether commandline indicates we are forking."""
if len(argv) >= 2 and argv[1] == "--multiprocessing-fork":
return True
else:
return False
def main(pipe_handle, parent_pid=None):
"""Run code specified by data received over pipe."""
assert is_forking(sys.argv), "Not forking"
if parent_pid is not None:
source_process = _winapi.OpenProcess(
_winapi.SYNCHRONIZE | _winapi.PROCESS_DUP_HANDLE, False, parent_pid
)
else:
source_process = None
new_handle = reduction.duplicate(
pipe_handle, source_process=source_process
)
fd = msvcrt.open_osfhandle(new_handle, os.O_RDONLY)
parent_sentinel = source_process
with os.fdopen(fd, "rb", closefd=True) as from_parent:
process.current_process()._inheriting = True
try:
preparation_data = load(from_parent)
spawn.prepare(preparation_data, parent_sentinel)
self = load(from_parent)
finally:
del process.current_process()._inheriting
exitcode = self._bootstrap(parent_sentinel)
sys.exit(exitcode)
joblib-loky-51247f4/loky/backend/process.py 0000664 0000000 0000000 00000003742 15256713413 0020617 0 ustar 00root root 0000000 0000000 ###############################################################################
# LokyProcess implementation
#
# authors: Thomas Moreau and Olivier Grisel
#
# based on multiprocessing/process.py (17/02/2017)
#
import sys
from multiprocessing.context import assert_spawning
from multiprocessing.process import BaseProcess
class LokyProcess(BaseProcess):
_start_method = "loky"
def __init__(
self,
group=None,
target=None,
name=None,
args=(),
kwargs={},
daemon=None,
init_main_module=False,
env=None,
):
super().__init__(
group=group,
target=target,
name=name,
args=args,
kwargs=kwargs,
daemon=daemon,
)
self.env = {} if env is None else env
self.authkey = self.authkey
self.init_main_module = init_main_module
@staticmethod
def _Popen(process_obj):
if sys.platform == "win32":
from .popen_loky_win32 import Popen
else:
from .popen_loky_posix import Popen
return Popen(process_obj)
class LokyInitMainProcess(LokyProcess):
_start_method = "loky_init_main"
def __init__(
self,
group=None,
target=None,
name=None,
args=(),
kwargs={},
daemon=None,
):
super().__init__(
group=group,
target=target,
name=name,
args=args,
kwargs=kwargs,
daemon=daemon,
init_main_module=True,
)
#
# We subclass bytes to avoid accidental transmission of auth keys over network
#
class AuthenticationKey(bytes):
def __reduce__(self):
try:
assert_spawning(self)
except RuntimeError:
raise TypeError(
"Pickling an AuthenticationKey object is "
"disallowed for security reasons"
)
return AuthenticationKey, (bytes(self),)
joblib-loky-51247f4/loky/backend/queues.py 0000664 0000000 0000000 00000016232 15256713413 0020446 0 ustar 00root root 0000000 0000000 ###############################################################################
# Queue and SimpleQueue implementation for loky
#
# authors: Thomas Moreau, Olivier Grisel
#
# based on multiprocessing/queues.py (16/02/2017)
# * Add some custom reducers for the Queues/SimpleQueue to tweak the
# pickling process. (overload Queue._feed/SimpleQueue.put)
#
import os
import sys
import errno
import weakref
import threading
from multiprocessing import util
from multiprocessing.queues import (
Full,
Queue as mp_Queue,
SimpleQueue as mp_SimpleQueue,
_sentinel,
)
from multiprocessing.context import assert_spawning
from .reduction import dumps
__all__ = ["Queue", "SimpleQueue", "Full"]
class Queue(mp_Queue):
def __init__(self, maxsize=0, reducers=None, ctx=None):
super().__init__(maxsize=maxsize, ctx=ctx)
self._reducers = reducers
# Use custom queue set/get state to be able to reduce the custom reducers
def __getstate__(self):
assert_spawning(self)
return (
self._ignore_epipe,
self._maxsize,
self._reader,
self._writer,
self._reducers,
self._rlock,
self._wlock,
self._sem,
self._opid,
)
def __setstate__(self, state):
(
self._ignore_epipe,
self._maxsize,
self._reader,
self._writer,
self._reducers,
self._rlock,
self._wlock,
self._sem,
self._opid,
) = state
if sys.version_info >= (3, 9):
self._reset()
else:
self._after_fork()
# Overload _start_thread to correctly call our custom _feed
def _start_thread(self):
util.debug("Queue._start_thread()")
# Start thread which transfers data from buffer to pipe
self._buffer.clear()
self._thread = threading.Thread(
target=Queue._feed,
args=(
self._buffer,
self._notempty,
self._send_bytes,
self._wlock,
self._writer.close,
self._reducers,
self._ignore_epipe,
self._on_queue_feeder_error,
self._sem,
),
name="QueueFeederThread",
)
self._thread.daemon = True
util.debug("doing self._thread.start()")
self._thread.start()
util.debug("... done self._thread.start()")
# On process exit we will wait for data to be flushed to pipe.
#
# However, if this process created the queue then all
# processes which use the queue will be descendants of this
# process. Therefore waiting for the queue to be flushed
# is pointless once all the child processes have been joined.
created_by_this_process = self._opid == os.getpid()
if not self._joincancelled and not created_by_this_process:
self._jointhread = util.Finalize(
self._thread,
Queue._finalize_join,
[weakref.ref(self._thread)],
exitpriority=-5,
)
# Send sentinel to the thread queue object when garbage collected
self._close = util.Finalize(
self,
Queue._finalize_close,
[self._buffer, self._notempty],
exitpriority=10,
)
# Overload the _feed methods to use our custom pickling strategy.
@staticmethod
def _feed(
buffer,
notempty,
send_bytes,
writelock,
close,
reducers,
ignore_epipe,
onerror,
queue_sem,
):
util.debug("starting thread to feed data to pipe")
nacquire = notempty.acquire
nrelease = notempty.release
nwait = notempty.wait
bpopleft = buffer.popleft
sentinel = _sentinel
if sys.platform != "win32":
wacquire = writelock.acquire
wrelease = writelock.release
else:
wacquire = None
while True:
try:
nacquire()
try:
if not buffer:
nwait()
finally:
nrelease()
try:
while True:
obj = bpopleft()
if obj is sentinel:
util.debug("feeder thread got sentinel -- exiting")
close()
return
# serialize the data before acquiring the lock
obj_ = dumps(obj, reducers=reducers)
if wacquire is None:
send_bytes(obj_)
else:
wacquire()
try:
send_bytes(obj_)
finally:
wrelease()
# Remove references early to avoid leaking memory
del obj, obj_
except IndexError:
pass
except BaseException as e:
if ignore_epipe and getattr(e, "errno", 0) == errno.EPIPE:
return
# Since this runs in a daemon thread the resources it uses
# may be become unusable while the process is cleaning up.
# We ignore errors which happen after the process has
# started to cleanup.
if util.is_exiting():
util.info(f"error in queue thread: {e}")
return
else:
queue_sem.release()
onerror(e, obj)
def _on_queue_feeder_error(self, e, obj):
"""
Private API hook called when feeding data in the background thread
raises an exception. For overriding by concurrent.futures.
"""
import traceback
traceback.print_exc()
class SimpleQueue(mp_SimpleQueue):
def __init__(self, reducers=None, ctx=None):
super().__init__(ctx=ctx)
# Add possiblity to use custom reducers
self._reducers = reducers
def close(self):
self._reader.close()
self._writer.close()
# Use custom queue set/get state to be able to reduce the custom reducers
def __getstate__(self):
assert_spawning(self)
return (
self._reader,
self._writer,
self._reducers,
self._rlock,
self._wlock,
)
def __setstate__(self, state):
(
self._reader,
self._writer,
self._reducers,
self._rlock,
self._wlock,
) = state
# Overload put to use our customizable reducer
def put(self, obj):
# serialize the data before acquiring the lock
obj = dumps(obj, reducers=self._reducers)
if self._wlock is None:
# writes to a message oriented win32 pipe are atomic
self._writer.send_bytes(obj)
else:
with self._wlock:
self._writer.send_bytes(obj)
joblib-loky-51247f4/loky/backend/reduction.py 0000664 0000000 0000000 00000015347 15256713413 0021141 0 ustar 00root root 0000000 0000000 ###############################################################################
# Customizable Pickler with some basic reducers
#
# author: Thomas Moreau
#
# adapted from multiprocessing/reduction.py (17/02/2017)
# * Replace the ForkingPickler with a similar _LokyPickler,
# * Add CustomizableLokyPickler to allow customizing pickling process
# on the fly.
#
import copyreg
import io
import functools
import types
import sys
import os
from multiprocessing import util
from pickle import loads, HIGHEST_PROTOCOL
###############################################################################
# Enable custom pickling in Loky.
_dispatch_table = {}
def register(type_, reduce_function):
_dispatch_table[type_] = reduce_function
###############################################################################
# Registers extra pickling routines to improve picklization for loky
# make methods picklable
def _reduce_method(m):
if m.__self__ is None:
return getattr, (m.__class__, m.__func__.__name__)
else:
return getattr, (m.__self__, m.__func__.__name__)
class _C:
def f(self):
pass
@classmethod
def h(cls):
pass
register(type(_C().f), _reduce_method)
register(type(_C.h), _reduce_method)
def _reduce_method_descriptor(m):
return getattr, (m.__objclass__, m.__name__)
register(type(list.append), _reduce_method_descriptor)
register(type(int.__add__), _reduce_method_descriptor)
# Make partial func pickable
def _reduce_partial(p):
return _rebuild_partial, (p.func, p.args, p.keywords or {})
def _rebuild_partial(func, args, keywords):
return functools.partial(func, *args, **keywords)
register(functools.partial, _reduce_partial)
if sys.platform != "win32":
from ._posix_reduction import _mk_inheritable # noqa: F401
else:
from . import _win_reduction # noqa: F401
# global variable to change the pickler behavior
try:
import cloudpickle # noqa: F401
DEFAULT_ENV = "cloudpickle"
except ImportError:
# If cloudpickle is not present, fallback to pickle
DEFAULT_ENV = "pickle"
ENV_LOKY_PICKLER = os.environ.get("LOKY_PICKLER", DEFAULT_ENV)
_LokyPickler = None
_loky_pickler_name = None
def set_loky_pickler(loky_pickler=None):
global _LokyPickler, _loky_pickler_name
if loky_pickler is None:
loky_pickler = ENV_LOKY_PICKLER
loky_pickler_cls = None
# The default loky_pickler is cloudpickle
if loky_pickler in ["", None]:
loky_pickler = "cloudpickle"
if loky_pickler == _loky_pickler_name:
return
if loky_pickler == "cloudpickle":
from cloudpickle import CloudPickler as loky_pickler_cls
else:
try:
from importlib import import_module
module_pickle = import_module(loky_pickler)
loky_pickler_cls = module_pickle.Pickler
except (ImportError, AttributeError) as e:
extra_info = (
"\nThis error occurred while setting loky_pickler to"
f" '{loky_pickler}', as required by the env variable "
"LOKY_PICKLER or the function set_loky_pickler."
)
e.args = (e.args[0] + extra_info,) + e.args[1:]
e.msg = e.args[0]
raise e
util.debug(
f"Using '{loky_pickler if loky_pickler else 'cloudpickle'}' for "
"serialization."
)
class CustomizablePickler(loky_pickler_cls):
_loky_pickler_cls = loky_pickler_cls
def _set_dispatch_table(self, dispatch_table):
for ancestor_class in self._loky_pickler_cls.mro():
dt_attribute = getattr(ancestor_class, "dispatch_table", None)
if isinstance(dt_attribute, types.MemberDescriptorType):
# Ancestor class (typically _pickle.Pickler) has a
# member_descriptor for its "dispatch_table" attribute. Use
# it to set the dispatch_table as a member instead of a
# dynamic attribute in the __dict__ of the instance,
# otherwise it will not be taken into account by the C
# implementation of the dump method if a subclass defines a
# class-level dispatch_table attribute as was done in
# cloudpickle 1.6.0:
# https://github.com/joblib/loky/pull/260
dt_attribute.__set__(self, dispatch_table)
break
# On top of member descriptor set, also use setattr such that code
# that directly access self.dispatch_table gets a consistent view
# of the same table.
self.dispatch_table = dispatch_table
def __init__(self, writer, reducers=None, protocol=HIGHEST_PROTOCOL):
loky_pickler_cls.__init__(self, writer, protocol=protocol)
if reducers is None:
reducers = {}
if hasattr(self, "dispatch_table"):
# Force a copy that we will update without mutating the
# any class level defined dispatch_table.
loky_dt = dict(self.dispatch_table)
else:
# Use standard reducers as bases
loky_dt = copyreg.dispatch_table.copy()
# Register loky specific reducers
loky_dt.update(_dispatch_table)
# Set the new dispatch table, taking care of the fact that we
# need to use the member_descriptor when we inherit from a
# subclass of the C implementation of the Pickler base class
# with an class level dispatch_table attribute.
self._set_dispatch_table(loky_dt)
# Register the reducers
for type, reduce_func in reducers.items():
self.register(type, reduce_func)
def register(self, type, reduce_func):
"""Attach a reducer function to a given type in the dispatch table."""
self.dispatch_table[type] = reduce_func
_LokyPickler = CustomizablePickler
_loky_pickler_name = loky_pickler
def get_loky_pickler_name():
global _loky_pickler_name
return _loky_pickler_name
def get_loky_pickler():
global _LokyPickler
return _LokyPickler
# Set it to its default value
set_loky_pickler()
def dump(obj, file, reducers=None, protocol=None):
"""Replacement for pickle.dump() using _LokyPickler."""
global _LokyPickler
_LokyPickler(file, reducers=reducers, protocol=protocol).dump(obj)
def dumps(obj, reducers=None, protocol=None):
global _LokyPickler
buf = io.BytesIO()
dump(obj, buf, reducers=reducers, protocol=protocol)
return buf.getbuffer()
__all__ = ["dump", "dumps", "loads", "register", "set_loky_pickler"]
if sys.platform == "win32":
from multiprocessing.reduction import duplicate
__all__ += ["duplicate"]
joblib-loky-51247f4/loky/backend/resource_tracker.py 0000664 0000000 0000000 00000050510 15256713413 0022476 0 ustar 00root root 0000000 0000000 ###############################################################################
# Server process to keep track of unlinked resources, like folders and
# semaphores and clean them.
#
# author: Thomas Moreau
#
# Adapted from multiprocessing/resource_tracker.py
# * add some VERBOSE logging,
# * add support to track folders,
# * add Windows support,
# * refcounting scheme to avoid unlinking resources still in use.
#
# On Unix we run a server process which keeps track of unlinked
# resources. The server ignores SIGINT and SIGTERM and reads from a
# pipe. The resource_tracker implements a reference counting scheme: each time
# a Python process anticipates the shared usage of a resource by another
# process, it signals the resource_tracker of this shared usage, and in return,
# the resource_tracker increments the resource's reference count by 1.
# Similarly, when access to a resource is closed by a Python process, the
# process notifies the resource_tracker by asking it to decrement the
# resource's reference count by 1. When the reference count drops to 0, the
# resource_tracker attempts to clean up the underlying resource.
# Finally, every other process connected to the resource tracker has a copy of
# the writable end of the pipe used to communicate with it, so the resource
# tracker gets EOF when all other processes have exited. Then the
# resource_tracker process unlinks any remaining leaked resources (with
# reference count above 0)
# For semaphores, this is important because the system only supports a limited
# number of named semaphores, and they will not be automatically removed till
# the next reboot. Without this resource tracker process, "killall python"
# would probably leave unlinked semaphores.
# Note that this behavior differs from CPython's resource_tracker, which only
# implements list of shared resources, and not a proper refcounting scheme.
# Also, CPython's resource tracker will only attempt to cleanup those shared
# resources once all processes connected to the resource tracker have exited.
#
# Source code organization and maintenance:
#
# This file is derived from the matching file in the standard library with
# modifications to add the new features described above.
#
# It defines a subclass of a vendored copy of the ResourceTracker from the
# standard library to make loky less likely to break when internals of
# change in a new version of CPython.
#
# The new features are implemented in the overriden method of the ResourceTracker
# class as well as in the custom `main` function.
#
# The vendored copy is minimally patched and should not be edited by hand.
# Instead, it should be revendored from time to time using the script
# in the `tools/` folder at the root of this repo.
#
# When vendoring a new copy of the resource_tracker module of the standard
# library, it is important to check of the `main` function has evolved
# upstream. If so, doing a diff of that particular function in the upstream
# file and in the current file might be helpful.
#
# Make sure to update the inline comments to help future maintainers understand
# what loky-specific changes were made.
import os
import shutil
import sys
import signal
import warnings
from multiprocessing import util
import base64
import json
import threading
from .stdlib_py314_resource_tracker import (
ResourceTracker as StdLibResourceTracker,
_decode_message,
)
from . import spawn
if sys.platform == "win32":
import _winapi
import msvcrt
from multiprocessing.reduction import duplicate
# To minimize diff vs stdlib
# fmt:off
__all__ = ['ensure_running', 'register', 'unregister']
_HAVE_SIGMASK = hasattr(signal, 'pthread_sigmask')
_IGNORED_SIGNALS = (signal.SIGINT, signal.SIGTERM)
def cleanup_noop(name):
raise RuntimeError('noop should never be registered or cleaned up')
_CLEANUP_FUNCS = {
'noop': cleanup_noop,
'dummy': lambda name: None, # Dummy resource used in tests
# loky: add 'folder' and 'file' resources
'folder': shutil.rmtree,
'file': os.unlink,
}
if os.name == 'posix':
import _multiprocessing
# Use sem_unlink() to clean up named semaphores.
#
# sem_unlink() may be missing if the Python build process detected the
# absence of POSIX named semaphores. In that case, no named semaphores were
# ever opened, so no cleanup would be necessary.
if hasattr(_multiprocessing, 'sem_unlink'):
_CLEANUP_FUNCS.update(
{
'semlock': _multiprocessing.sem_unlink,
}
)
# fmt: on
# loky: logging
VERBOSE = False
# loky: compatibility for CPython versions that don't have _RLock._recursion_count
# This was done in CPython 3.13 in
# 'Fix reentrancy issue in multiprocessing resource_tracker'
# https://github.com/python/cpython/pull/109629
# This was back-ported in 3.11.6 and 3.12.1
# TODO Remove work-around when Python 3.13 is our minimum supported version
class LokyRLock(type(threading.RLock())):
def _recursion_count(self):
return 1
class ResourceTracker(StdLibResourceTracker):
"""Resource tracker with refcounting scheme.
This class is an extension of the multiprocessing ResourceTracker class
which implements a reference counting scheme to avoid unlinking shared
resources still in use in other processes.
This feature is notably used by `joblib.Parallel` to share temporary
folders and memory mapped files between the main process and the worker
processes.
The actual implementation of the refcounting scheme is in the main
function, which is run in a dedicated process.
"""
def __init__(self):
super().__init__()
# Windows process handle returned by CreateProcess. A pid cannot be
# passed to os.waitpid on Windows because the underlying _cwait expects
# a process handle.
self._proc_handle = None
# TODO Remove block when Python 3.13 is our minimum supported version
# see above comment about _recursion_count
if not hasattr(self._lock, "_recursion_count"):
self._lock = LokyRLock()
def maybe_unlink(self, name, rtype):
"""Decrement the refcount of a resource, and delete it if it hits 0"""
self._send("MAYBE_UNLINK", name, rtype)
def _teardown_dead_process(self):
if os.name == "posix":
super()._teardown_dead_process()
elif sys.platform == "win32":
os.close(self._fd)
if (proc_handle := self._proc_handle) is not None:
_winapi.CloseHandle(proc_handle)
# All 3 lines copied from stdlib _teardown_dead_processes
self._fd = None
self._pid = None
self._exitcode = None
self._proc_handle = None
warnings.warn(
"resource_tracker: process died unexpectedly, "
"relaunching. Some resources might leak."
)
# To minimize the diff with stdlib ResourceTracker._launch
# fmt: off
def _launch(self):
# This is copied from Python 3.14.7 with loky additions/modifications
# mostly for Windows support and logging.
# Added or changed lines have a comment that starts with "# loky:"
fds_to_pass = []
try:
fds_to_pass.append(sys.stderr.fileno())
except Exception:
pass
r, w = os.pipe()
# loky: Windows support
if sys.platform == "win32":
_r = duplicate(msvcrt.get_osfhandle(r), inheritable=True)
os.close(r)
r = _r
try:
fds_to_pass.append(r)
# process will out live us, so no need to wait on pid
exe = spawn.get_executable()
args = [
exe,
*util._args_from_interpreter_flags(),
'-c',
# loky: use loky main function rather than stdlib one
f'from {main.__module__} import main; main({r}, {VERBOSE})'
]
# loky: logging
util.debug(f"launching resource tracker: {args}")
# bpo-33613: Register a signal mask that will block the signals.
# This signal mask will be inherited by the child that is going
# to be spawned and will protect the child from a race condition
# that can make the child die before it registers signal handlers
# for SIGINT and SIGTERM. The mask is unregistered after spawning
# the child.
prev_sigmask = None
try:
if _HAVE_SIGMASK:
prev_sigmask = signal.pthread_sigmask(signal.SIG_BLOCK, _IGNORED_SIGNALS)
# loky: call loky spawnv_passfds which supports Windows
pid, proc_handle = spawnv_passfds(exe, args, fds_to_pass)
finally:
if prev_sigmask is not None:
signal.pthread_sigmask(signal.SIG_SETMASK, prev_sigmask)
except:
os.close(w)
raise
else:
self._fd = w
self._pid = pid
self._proc_handle = proc_handle
finally:
# loky: Windows support
if sys.platform == "win32":
_winapi.CloseHandle(r)
else:
os.close(r)
# fmt: on
if sys.platform == "win32":
# stdlib ResourceTracker._stop_locked is POSIX-specific, so we override
# to be able to use Windows-specific primitives.
# This is loosely inspired from stdlib ResourceTracker._stop_locked but
# there are a number of changes. TODO: could the structure be closer?
def _stop_locked(
self,
close=os.close,
wait_timeout=None,
wait_for_single_object=_winapi.WaitForSingleObject,
get_exit_code_process=_winapi.GetExitCodeProcess,
close_handle=_winapi.CloseHandle,
winapi_infinite=_winapi.INFINITE,
wait_timeout_code=_winapi.WAIT_TIMEOUT,
):
# This shouldn't happen (it might when called by a finalizer)
# so we check for it anyway.
if self._lock._recursion_count() > 1:
raise self._reentrant_call_error()
if self._fd is None:
# not running
return
if self._pid is None:
return
# Closing the "alive" file descriptor asks the tracker to stop.
close(self._fd)
self._fd = None
proc_handle = self._proc_handle
try:
if proc_handle is not None:
timeout_ms = (
winapi_infinite
if wait_timeout is None
else round(wait_timeout * 1000)
)
wait_result = wait_for_single_object(
proc_handle, timeout_ms
)
if wait_result == wait_timeout_code:
self._pid = None
self._exitcode = None
self._waitpid_timed_out = True
return
self._pid = None
self._exitcode = get_exit_code_process(proc_handle)
else:
self._pid = None
self._exitcode = None
finally:
if proc_handle is not None:
close_handle(proc_handle)
self._proc_handle = None
# Copied from Python 3.14.7
# fmt: off
_resource_tracker = ResourceTracker()
ensure_running = _resource_tracker.ensure_running
register = _resource_tracker.register
maybe_unlink = _resource_tracker.maybe_unlink
unregister = _resource_tracker.unregister
getfd = _resource_tracker.getfd
# gh-146313: See _after_fork_in_child docstring.
if hasattr(os, 'register_at_fork'):
os.register_at_fork(after_in_child=_resource_tracker._after_fork_in_child)
# fmt: on
# fmt: off
# The main function has been copied from Python 3.14.7 and modified, mostly for
# Windows support, logging and refcount functionality.
# Added or changed lines have a comment that starts with "# loky:"
# loky: add verbose argument for logging
def main(fd, verbose=0):
'''Run resource tracker.'''
# loky: logging
if verbose:
util.log_to_stderr(level=util.DEBUG)
# protect the process from ^C and "killall python" etc
signal.signal(signal.SIGINT, signal.SIG_IGN)
signal.signal(signal.SIGTERM, signal.SIG_IGN)
if _HAVE_SIGMASK:
signal.pthread_sigmask(signal.SIG_UNBLOCK, _IGNORED_SIGNALS)
for f in (sys.stdin, sys.stdout):
try:
f.close()
except Exception:
pass
# loky: logging
if verbose:
util.debug("Main resource tracker is running")
# loky: change for refcount functionality we want a dict[str, dict] rather
# than a dict[str, set] so that cache[folder]['resource'] is the refcount
# associated with it
cache = {rtype: dict() for rtype in _CLEANUP_FUNCS.keys()}
exit_code = 0
try:
# loky: Windows support
if sys.platform == "win32":
fd = msvcrt.open_osfhandle(fd, os.O_RDONLY)
# keep track of registered/unregistered resources
with open(fd, 'rb') as f:
for line in f:
try:
cmd, rtype, name = _decode_message(line)
cleanup_func = _CLEANUP_FUNCS.get(rtype, None)
if cleanup_func is None:
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type ({rtype})'
# loky: additional info for possible keys
'. Resource type should be one of the following: '
f'{list(_CLEANUP_FUNCS.keys())}'
)
if cmd == 'REGISTER':
# loky: refcount functionality
if name not in cache[rtype]:
cache[rtype][name] = 1
else:
cache[rtype][name] += 1
# loky: logging
if verbose:
util.debug(
'[ResourceTracker] incremented refcount of '
f'{rtype} {name} '
f'(current {cache[rtype][name]})'
)
elif cmd == 'UNREGISTER':
# loky: refcount functionality
del cache[rtype][name]
# loky: logging
if verbose:
util.debug(
f'[ResourceTracker] unregister {name} {rtype}: '
f'cache({len(cache)})'
)
elif cmd == 'PROBE':
pass
# loky: refcount functionality with logging
elif cmd == 'MAYBE_UNLINK':
cache[rtype][name] -= 1
if verbose:
util.debug(
'[ResourceTracker] decremented refcount of '
f'{rtype} {name} '
f'(current {cache[rtype][name]})'
)
if cache[rtype][name] == 0:
del cache[rtype][name]
try:
if verbose:
util.debug(
f'[ResourceTracker] unlink {name}'
)
_CLEANUP_FUNCS[rtype](name)
except Exception as e:
warnings.warn(
f"resource_tracker: {name}: {e!r}"
)
else:
raise RuntimeError('unrecognized command %r' % cmd)
except Exception:
exit_code = 3
try:
sys.excepthook(*sys.exc_info())
except:
pass
finally:
# all processes have terminated; cleanup any remaining resources
# loky: loky wants to clean ressources first and folder last because
# there can be tracked resources inside tracked folders.
# _unlink_resources is the stdlib code with some additional logging, it
# is called for all resources except folders and then at the end for
# all folders
def _unlink_resources(rtype_cache, rtype):
nonlocal exit_code
if rtype_cache:
try:
exit_code = 1
if rtype == 'dummy':
# The test 'dummy' resource is expected to leak.
# We skip the warning (and *only* the warning) for it.
pass
else:
warnings.warn(
f'resource_tracker: There appear to be '
f'{len(rtype_cache)} leaked {rtype} objects to '
f'clean up at shutdown: {rtype_cache}'
)
except Exception:
pass
for name in rtype_cache:
# For some reason the process which created and registered this
# resource has failed to unregister it. Presumably it has
# died. We therefore unlink it.
try:
try:
_CLEANUP_FUNCS[rtype](name)
# loky: logging
if verbose:
util.debug(f'[ResourceTracker] unlink {name}')
except Exception as e:
exit_code = 2
# loky: tweaked formatting (%r instead of %s for
# exception and %s instead of %s for name) I guess you
# get the exact exception type with %r. %s instead of
# %r for name, may be because on Windows %r doubles the
# backslashes for path-like ressources
warnings.warn('resource_tracker: %s: %r' % (name, e))
finally:
pass
# loky: 2 stage cleaning process
# The default cleanup routine for folders deletes everything inside
# those folders recursively, which can include other resources tracked
# by the resource tracker). To limit the risk of the resource tracker
# attempting to delete twice a resource (once as part of a tracked
# folder, and once as a resource), we delete the folders after all
# other resource types.
for rtype, rtype_cache in cache.items():
if rtype == 'folder':
continue
_unlink_resources(rtype_cache, rtype)
if 'folder' in cache:
_unlink_resources(cache['folder'], 'folder')
# loky: logging
if verbose:
util.debug("resource tracker shut down")
# TODO not sure about exit_code management with 2-stage clean-up
# (first non-folder resources then folder resources) but seems good enough
# for now. We did not have any exit code management until
# https://github.com/joblib/loky/pull/472
sys.exit(exit_code)
# fmt: on
def spawnv_passfds(path, args, passfds):
"""Loky version multiprocessing.util.spawnv_passfds with added Windows support.
Returns (pid, process_handle) because os.waitpid needs handle on Windows.
On Linux process_handle is None.
"""
if sys.platform != "win32":
# loky: additional encoding needed here
# TODO We should fix loky.backend.spawn.get_executable to return bytes
# on POSIX so that this can be removed
path = path.encode("utf-8")
return util.spawnv_passfds(path, args, passfds), None
else:
# loky: Windows support
passfds = sorted(passfds)
cmd = " ".join(f'"{x}"' for x in args)
hp, ht, pid, _ = _winapi.CreateProcess(
path, cmd, None, None, True, 0, None, None, None
)
_winapi.CloseHandle(ht)
# Keep the process handle for a safe wait during teardown. Pids and
# handles are different namespaces on Windows.
return pid, hp
joblib-loky-51247f4/loky/backend/spawn.py 0000664 0000000 0000000 00000020662 15256713413 0020271 0 ustar 00root root 0000000 0000000 ###############################################################################
# Prepares and processes the data to setup the new process environment
#
# author: Thomas Moreau and Olivier Grisel
#
# adapted from multiprocessing/spawn.py (17/02/2017)
# * Improve logging data
#
import os
import sys
import runpy
import textwrap
import types
from multiprocessing import process, util
if sys.platform != "win32":
WINEXE = False
WINSERVICE = False
else:
import msvcrt
from multiprocessing.reduction import duplicate
WINEXE = sys.platform == "win32" and getattr(sys, "frozen", False)
WINSERVICE = sys.executable.lower().endswith("pythonservice.exe")
if WINSERVICE:
_python_exe = os.path.join(sys.exec_prefix, "python.exe")
else:
_python_exe = sys.executable
def get_executable():
return _python_exe
def _check_not_importing_main():
if getattr(process.current_process(), "_inheriting", False):
raise RuntimeError(
textwrap.dedent(
"""\
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable."""
)
)
def get_preparation_data(name, init_main_module=True):
"""Return info about parent needed by child to unpickle process object."""
_check_not_importing_main()
d = dict(
log_to_stderr=util._log_to_stderr,
authkey=bytes(process.current_process().authkey),
name=name,
sys_argv=sys.argv,
orig_dir=process.ORIGINAL_DIR,
dir=os.getcwd(),
)
# Send sys_path and make sure the current directory will not be changed
d["sys_path"] = [p if p != "" else process.ORIGINAL_DIR for p in sys.path]
# Make sure to pass the information if the multiprocessing logger is active
if util._logger is not None:
d["log_level"] = util._logger.getEffectiveLevel()
if util._logger.handlers:
h = util._logger.handlers[0]
d["log_fmt"] = h.formatter._fmt
# Tell the child how to communicate with the resource_tracker
from .resource_tracker import _resource_tracker
_resource_tracker.ensure_running()
if sys.platform == "win32":
d["tracker_fd"] = msvcrt.get_osfhandle(_resource_tracker._fd)
else:
d["tracker_fd"] = _resource_tracker._fd
if os.name == "posix":
# joblib/loky#242: allow loky processes to retrieve the resource
# tracker of their parent in case the child processes depickles
# shared_memory objects, that are still tracked by multiprocessing's
# resource_tracker by default.
# XXX: this is a workaround that may be error prone: in the future, it
# would be better to have loky subclass multiprocessing's shared_memory
# to force registration of shared_memory segments via loky's
# resource_tracker.
from multiprocessing.resource_tracker import (
_resource_tracker as mp_resource_tracker,
)
# multiprocessing's resource_tracker must be running before loky
# process is created (othewise the child won't be able to use it if it
# is created later on)
mp_resource_tracker.ensure_running()
d["mp_tracker_fd"] = mp_resource_tracker._fd
# Figure out whether to initialise main in the subprocess as a module
# or through direct execution (or to leave it alone entirely)
if init_main_module:
main_module = sys.modules["__main__"]
try:
main_mod_name = getattr(main_module.__spec__, "name", None)
except BaseException:
main_mod_name = None
if main_mod_name is not None:
d["init_main_from_name"] = main_mod_name
elif sys.platform != "win32" or (not WINEXE and not WINSERVICE):
main_path = getattr(main_module, "__file__", None)
if main_path is not None:
if (
not os.path.isabs(main_path)
and process.ORIGINAL_DIR is not None
):
main_path = os.path.join(process.ORIGINAL_DIR, main_path)
d["init_main_from_path"] = os.path.normpath(main_path)
return d
#
# Prepare current process
#
old_main_modules = []
def prepare(data, parent_sentinel=None):
"""Try to get current process ready to unpickle process object."""
if "name" in data:
process.current_process().name = data["name"]
if "authkey" in data:
process.current_process().authkey = data["authkey"]
if "log_to_stderr" in data and data["log_to_stderr"]:
util.log_to_stderr()
if "log_level" in data:
util.get_logger().setLevel(data["log_level"])
if "log_fmt" in data:
import logging
util.get_logger().handlers[0].setFormatter(
logging.Formatter(data["log_fmt"])
)
if "sys_path" in data:
sys.path = data["sys_path"]
if "sys_argv" in data:
sys.argv = data["sys_argv"]
if "dir" in data:
os.chdir(data["dir"])
if "orig_dir" in data:
process.ORIGINAL_DIR = data["orig_dir"]
if "mp_tracker_fd" in data:
from multiprocessing.resource_tracker import (
_resource_tracker as mp_resource_tracker,
)
mp_resource_tracker._fd = data["mp_tracker_fd"]
if "tracker_fd" in data:
from .resource_tracker import _resource_tracker
if sys.platform == "win32":
handle = data["tracker_fd"]
handle = duplicate(handle, source_process=parent_sentinel)
_resource_tracker._fd = msvcrt.open_osfhandle(handle, os.O_RDONLY)
else:
_resource_tracker._fd = data["tracker_fd"]
if "init_main_from_name" in data:
_fixup_main_from_name(data["init_main_from_name"])
elif "init_main_from_path" in data:
_fixup_main_from_path(data["init_main_from_path"])
# Multiprocessing module helpers to fix up the main module in
# spawned subprocesses
def _fixup_main_from_name(mod_name):
# __main__.py files for packages, directories, zip archives, etc, run
# their "main only" code unconditionally, so we don't even try to
# populate anything in __main__, nor do we make any changes to
# __main__ attributes
current_main = sys.modules["__main__"]
if mod_name == "__main__" or mod_name.endswith(".__main__"):
return
# If this process was forked, __main__ may already be populated
if getattr(current_main.__spec__, "name", None) == mod_name:
return
# Otherwise, __main__ may contain some non-main code where we need to
# support unpickling it properly. We rerun it as __mp_main__ and make
# the normal __main__ an alias to that
old_main_modules.append(current_main)
main_module = types.ModuleType("__mp_main__")
main_content = runpy.run_module(
mod_name, run_name="__mp_main__", alter_sys=True
)
main_module.__dict__.update(main_content)
sys.modules["__main__"] = sys.modules["__mp_main__"] = main_module
def _fixup_main_from_path(main_path):
# If this process was forked, __main__ may already be populated
current_main = sys.modules["__main__"]
# Unfortunately, the main ipython launch script historically had no
# "if __name__ == '__main__'" guard, so we work around that
# by treating it like a __main__.py file
# See https://github.com/ipython/ipython/issues/4698
main_name = os.path.splitext(os.path.basename(main_path))[0]
if main_name == "ipython":
return
# Otherwise, if __file__ already has the setting we expect,
# there's nothing more to do
if getattr(current_main, "__file__", None) == main_path:
return
# If the parent process has sent a path through rather than a module
# name we assume it is an executable script that may contain
# non-main code that needs to be executed
old_main_modules.append(current_main)
main_module = types.ModuleType("__mp_main__")
main_content = runpy.run_path(main_path, run_name="__mp_main__")
main_module.__dict__.update(main_content)
sys.modules["__main__"] = sys.modules["__mp_main__"] = main_module
joblib-loky-51247f4/loky/backend/stdlib_py314_resource_tracker.py 0000664 0000000 0000000 00000046445 15256713413 0025013 0 ustar 00root root 0000000 0000000 ###############################################################################
# Server process to keep track of unlinked resources (like shared memory
# segments, semaphores etc.) and clean them.
#
# On Unix we run a server process which keeps track of unlinked
# resources. The server ignores SIGINT and SIGTERM and reads from a
# pipe. Every other process of the program has a copy of the writable
# end of the pipe, so we get EOF when all other processes have exited.
# Then the server process unlinks any remaining resource names.
#
# This is important because there may be system limits for such resources: for
# instance, the system only supports a limited number of named semaphores, and
# shared-memory segments live in the RAM. If a python process leaks such a
# resource, this resource will not be removed till the next reboot. Without
# this resource tracker process, "killall python" would probably leave unlinked
# resources.
import base64
import os
import signal
import sys
import threading
import time
import warnings
from collections import deque
from types import SimpleNamespace
import json
# loky: fix imports
from multiprocessing import spawn
from multiprocessing import util
__all__ = ['ensure_running', 'register', 'unregister']
_HAVE_SIGMASK = hasattr(signal, 'pthread_sigmask')
_IGNORED_SIGNALS = (signal.SIGINT, signal.SIGTERM)
def cleanup_noop(name):
raise RuntimeError('noop should never be registered or cleaned up')
_CLEANUP_FUNCS = {
'noop': cleanup_noop,
'dummy': lambda name: None, # Dummy resource used in tests
}
if os.name == 'posix':
import _multiprocessing
import _posixshmem
# Use sem_unlink() to clean up named semaphores.
#
# sem_unlink() may be missing if the Python build process detected the
# absence of POSIX named semaphores. In that case, no named semaphores were
# ever opened, so no cleanup would be necessary.
if hasattr(_multiprocessing, 'sem_unlink'):
_CLEANUP_FUNCS['semaphore'] = _multiprocessing.sem_unlink
_CLEANUP_FUNCS['shared_memory'] = _posixshmem.shm_unlink
class ReentrantCallError(RuntimeError):
pass
class ResourceTracker(object):
def __init__(self):
self._lock = threading.RLock()
self._fd = None
self._pid = None
self._exitcode = None
self._reentrant_messages = deque()
# True to use colon-separated lines, rather than JSON lines,
# for internal communication. (Mainly for testing).
# Filenames not supported by the simple format will always be sent
# using JSON.
# The reader should understand all formats.
self._use_simple_format = True
# Set to True by _stop_locked() if the waitpid polling loop ran to
# its timeout without reaping the tracker. Exposed for tests.
self._waitpid_timed_out = False
def _reentrant_call_error(self):
# gh-109629: this happens if an explicit call to the ResourceTracker
# gets interrupted by a garbage collection, invoking a finalizer (*)
# that itself calls back into ResourceTracker.
# (*) for example the SemLock finalizer
raise ReentrantCallError(
"Reentrant call into the multiprocessing resource tracker")
def __del__(self):
# making sure child processess are cleaned before ResourceTracker
# gets destructed.
# see https://github.com/python/cpython/issues/88887
# gh-146313: use a timeout to avoid deadlocking if a forked child
# still holds the pipe's write end open.
self._stop(use_blocking_lock=False, wait_timeout=1.0)
def _after_fork_in_child(self):
# gh-146313: Called in the child right after os.fork().
#
# The tracker process is a child of the *parent*, not of us, so we
# could never waitpid() it anyway. Clearing _pid means our __del__
# becomes a no-op (the early return for _pid is None).
#
# Whether we keep the inherited _fd depends on who forked us:
#
# - multiprocessing.Process with the 'fork' start method sets
# _fork_intent.preserve_fd before forking. The child keeps the
# fd and reuses the parent's tracker (gh-80849). This is safe
# because multiprocessing's atexit handler joins all children
# before the parent's __del__ runs, so by then the fd copies
# are gone and the parent can reap the tracker promptly.
#
# - A raw os.fork() leaves the flag unset. We close the fd in the child after forking so
# the parent's __del__ can reap the tracker without waiting
# for the child to exit. If we later need a tracker, ensure_running()
# will launch a fresh one.
self._lock._at_fork_reinit()
self._reentrant_messages.clear()
self._pid = None
self._exitcode = None
if (self._fd is not None and
not getattr(_fork_intent, 'preserve_fd', False)):
fd = self._fd
self._fd = None
try:
os.close(fd)
except OSError:
pass
def _stop(self, use_blocking_lock=True, wait_timeout=None):
if use_blocking_lock:
with self._lock:
self._stop_locked(wait_timeout=wait_timeout)
else:
acquired = self._lock.acquire(blocking=False)
try:
self._stop_locked(wait_timeout=wait_timeout)
finally:
if acquired:
self._lock.release()
def _stop_locked(
self,
close=os.close,
waitpid=os.waitpid,
waitstatus_to_exitcode=os.waitstatus_to_exitcode,
monotonic=time.monotonic,
sleep=time.sleep,
WNOHANG=getattr(os, 'WNOHANG', None),
wait_timeout=None,
):
# This shouldn't happen (it might when called by a finalizer)
# so we check for it anyway.
if self._lock._recursion_count() > 1:
raise self._reentrant_call_error()
if self._fd is None:
# not running
return
if self._pid is None:
return
# closing the "alive" file descriptor stops main()
close(self._fd)
self._fd = None
try:
if wait_timeout is None:
_, status = waitpid(self._pid, 0)
else:
# gh-146313: A forked child may still hold the pipe's write
# end open, preventing the tracker from seeing EOF and
# exiting. Poll with WNOHANG to avoid blocking forever.
deadline = monotonic() + wait_timeout
delay = 0.001
while True:
result_pid, status = waitpid(self._pid, WNOHANG)
if result_pid != 0:
break
remaining = deadline - monotonic()
if remaining <= 0:
# The tracker is still running; it will be
# reparented to PID 1 (or the nearest subreaper)
# when we exit, and reaped there once all pipe
# holders release their fd.
self._pid = None
self._exitcode = None
self._waitpid_timed_out = True
return
delay = min(delay * 2, remaining, 0.1)
sleep(delay)
except ChildProcessError:
self._pid = None
self._exitcode = None
return
self._pid = None
try:
self._exitcode = waitstatus_to_exitcode(status)
except ValueError:
# os.waitstatus_to_exitcode may raise an exception for invalid values
self._exitcode = None
def getfd(self):
self.ensure_running()
return self._fd
def ensure_running(self):
'''Make sure that resource tracker process is running.
This can be run from any process. Usually a child process will use
the resource created by its parent.'''
return self._ensure_running_and_write()
def _teardown_dead_process(self):
os.close(self._fd)
# Clean-up to avoid dangling processes.
try:
# _pid can be None if this process is a child from another
# python process, which has started the resource_tracker.
if self._pid is not None:
os.waitpid(self._pid, 0)
except ChildProcessError:
# The resource_tracker has already been terminated.
pass
self._fd = None
self._pid = None
self._exitcode = None
warnings.warn('resource_tracker: process died unexpectedly, '
'relaunching. Some resources might leak.')
def _launch(self):
fds_to_pass = []
try:
fds_to_pass.append(sys.stderr.fileno())
except Exception:
pass
r, w = os.pipe()
try:
fds_to_pass.append(r)
# process will out live us, so no need to wait on pid
exe = spawn.get_executable()
args = [
exe,
*util._args_from_interpreter_flags(),
'-c',
f'from multiprocessing.resource_tracker import main;main({r})',
]
# bpo-33613: Register a signal mask that will block the signals.
# This signal mask will be inherited by the child that is going
# to be spawned and will protect the child from a race condition
# that can make the child die before it registers signal handlers
# for SIGINT and SIGTERM. The mask is unregistered after spawning
# the child.
prev_sigmask = None
try:
if _HAVE_SIGMASK:
prev_sigmask = signal.pthread_sigmask(signal.SIG_BLOCK, _IGNORED_SIGNALS)
pid = util.spawnv_passfds(exe, args, fds_to_pass)
finally:
if prev_sigmask is not None:
signal.pthread_sigmask(signal.SIG_SETMASK, prev_sigmask)
except:
os.close(w)
raise
else:
self._fd = w
self._pid = pid
finally:
os.close(r)
def _make_probe_message(self):
"""Return a probe message."""
if self._use_simple_format:
return b'PROBE:0:noop\n'
return (
json.dumps(
{"cmd": "PROBE", "rtype": "noop"},
ensure_ascii=True,
separators=(",", ":"),
)
+ "\n"
).encode("ascii")
def _ensure_running_and_write(self, msg=None):
with self._lock:
if self._lock._recursion_count() > 1:
# The code below is certainly not reentrant-safe, so bail out
if msg is None:
raise self._reentrant_call_error()
return self._reentrant_messages.append(msg)
if self._fd is not None:
# resource tracker was launched before, is it still running?
if msg is None:
to_send = self._make_probe_message()
else:
to_send = msg
try:
self._write(to_send)
except OSError:
self._teardown_dead_process()
self._launch()
msg = None # message was sent in probe
else:
self._launch()
while True:
try:
reentrant_msg = self._reentrant_messages.popleft()
except IndexError:
break
self._write(reentrant_msg)
if msg is not None:
self._write(msg)
def _check_alive(self):
'''Check that the pipe has not been closed by sending a probe.'''
try:
# We cannot use send here as it calls ensure_running, creating
# a cycle.
os.write(self._fd, self._make_probe_message())
except OSError:
return False
else:
return True
def register(self, name, rtype):
'''Register name of resource with resource tracker.'''
self._send('REGISTER', name, rtype)
def unregister(self, name, rtype):
'''Unregister name of resource with resource tracker.'''
self._send('UNREGISTER', name, rtype)
def _write(self, msg):
nbytes = os.write(self._fd, msg)
assert nbytes == len(msg), f"{nbytes=} != {len(msg)=}"
def _send(self, cmd, name, rtype):
if self._use_simple_format and '\n' not in name:
msg = f"{cmd}:{name}:{rtype}\n".encode("ascii")
if len(msg) > 512:
# posix guarantees that writes to a pipe of less than PIPE_BUF
# bytes are atomic, and that PIPE_BUF >= 512
raise ValueError('msg too long')
self._ensure_running_and_write(msg)
return
# POSIX guarantees that writes to a pipe of less than PIPE_BUF (512 on Linux)
# bytes are atomic. Therefore, we want the message to be shorter than 512 bytes.
# POSIX shm_open() and sem_open() require the name, including its leading slash,
# to be at most NAME_MAX bytes (255 on Linux)
# With json.dump(..., ensure_ascii=True) every non-ASCII byte becomes a 6-char
# escape like \uDC80.
# As we want the overall message to be kept atomic and therefore smaller than 512,
# we encode encode the raw name bytes with URL-safe Base64 - so a 255 long name
# will not exceed 340 bytes.
b = name.encode('utf-8', 'surrogateescape')
if len(b) > 255:
raise ValueError('shared memory name too long (max 255 bytes)')
b64 = base64.urlsafe_b64encode(b).decode('ascii')
payload = {"cmd": cmd, "rtype": rtype, "base64_name": b64}
msg = (json.dumps(payload, ensure_ascii=True, separators=(",", ":")) + "\n").encode("ascii")
# The entire JSON message is guaranteed < PIPE_BUF (512 bytes) by construction.
assert len(msg) <= 512, f"internal error: message too long ({len(msg)} bytes)"
assert msg.startswith(b'{')
self._ensure_running_and_write(msg)
# loky: We want to use multiprocessing.resource_tracker._fork_intent rather
# than create a separate variable.
# multiprocessing.resource_tracker._fork_intent is used by
# multiprocessing.popen_fork to distinguish raw forks from multiprocessing
# forks.
try:
from multiprocessing.resource_tracker import _fork_intent
except ImportError:
# Older Python versions cannot distinguish multiprocessing forks from raw
# os.fork() calls. Preserve the inherited tracker fd in both cases.
# TODO remove when Python 3.15 is the minimum supported version. For more details about preserve_fd see
# https://github.com/python/cpython/pull/146316
# which has been merged in Python 3.15 and backported in 3.14.5 and 3.13.14
_fork_intent = SimpleNamespace(preserve_fd=True)
# loky: begin commented out section
# Remove module-level globals and logic which
# are unused and can cause confusing errors at interpreter exit because of
# discrepancies in the code from vendored Python version and used Python
# version
# _resource_tracker = ResourceTracker()
# ensure_running = _resource_tracker.ensure_running
# register = _resource_tracker.register
# unregister = _resource_tracker.unregister
# getfd = _resource_tracker.getfd
# # gh-146313: See _after_fork_in_child docstring.
# if hasattr(os, 'register_at_fork'):
# os.register_at_fork(after_in_child=_resource_tracker._after_fork_in_child)
# loky: end commented out section
def _decode_message(line):
if line.startswith(b'{'):
try:
obj = json.loads(line.decode('ascii'))
except Exception as e:
raise ValueError("malformed resource_tracker message: %r" % (line,)) from e
cmd = obj["cmd"]
rtype = obj["rtype"]
b64 = obj.get("base64_name", "")
if not isinstance(cmd, str) or not isinstance(rtype, str) or not isinstance(b64, str):
raise ValueError("malformed resource_tracker fields: %r" % (obj,))
try:
name = base64.urlsafe_b64decode(b64).decode('utf-8', 'surrogateescape')
except ValueError as e:
raise ValueError("malformed resource_tracker base64_name: %r" % (b64,)) from e
else:
cmd, rest = line.strip().decode('ascii').split(':', maxsplit=1)
name, rtype = rest.rsplit(':', maxsplit=1)
return cmd, rtype, name
def main(fd):
'''Run resource tracker.'''
# protect the process from ^C and "killall python" etc
signal.signal(signal.SIGINT, signal.SIG_IGN)
signal.signal(signal.SIGTERM, signal.SIG_IGN)
if _HAVE_SIGMASK:
signal.pthread_sigmask(signal.SIG_UNBLOCK, _IGNORED_SIGNALS)
for f in (sys.stdin, sys.stdout):
try:
f.close()
except Exception:
pass
cache = {rtype: set() for rtype in _CLEANUP_FUNCS.keys()}
exit_code = 0
try:
# keep track of registered/unregistered resources
with open(fd, 'rb') as f:
for line in f:
try:
cmd, rtype, name = _decode_message(line)
cleanup_func = _CLEANUP_FUNCS.get(rtype, None)
if cleanup_func is None:
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
if cmd == 'REGISTER':
cache[rtype].add(name)
elif cmd == 'UNREGISTER':
cache[rtype].remove(name)
elif cmd == 'PROBE':
pass
else:
raise RuntimeError('unrecognized command %r' % cmd)
except Exception:
exit_code = 3
try:
sys.excepthook(*sys.exc_info())
except:
pass
finally:
# all processes have terminated; cleanup any remaining resources
for rtype, rtype_cache in cache.items():
if rtype_cache:
try:
exit_code = 1
if rtype == 'dummy':
# The test 'dummy' resource is expected to leak.
# We skip the warning (and *only* the warning) for it.
pass
else:
warnings.warn(
f'resource_tracker: There appear to be '
f'{len(rtype_cache)} leaked {rtype} objects to '
f'clean up at shutdown: {rtype_cache}'
)
except Exception:
pass
for name in rtype_cache:
# For some reason the process which created and registered this
# resource has failed to unregister it. Presumably it has
# died. We therefore unlink it.
try:
try:
_CLEANUP_FUNCS[rtype](name)
except Exception as e:
exit_code = 2
warnings.warn('resource_tracker: %r: %s' % (name, e))
finally:
pass
sys.exit(exit_code)
joblib-loky-51247f4/loky/backend/synchronize.py 0000664 0000000 0000000 00000026770 15256713413 0021522 0 ustar 00root root 0000000 0000000 ###############################################################################
# Synchronization primitives based on our SemLock implementation
#
# author: Thomas Moreau and Olivier Grisel
#
# adapted from multiprocessing/synchronize.py (17/02/2017)
# * Remove ctx argument for compatibility reason
# * Registers a cleanup function with the loky resource_tracker to remove the
# semaphore when the process dies instead.
#
# TODO: investigate which Python version is required to be able to use
# multiprocessing.resource_tracker and therefore multiprocessing.synchronize
# instead of a loky-specific fork.
import os
import sys
import tempfile
import threading
import _multiprocessing
from time import time as _time
from multiprocessing import process, util
from multiprocessing.context import assert_spawning
from . import resource_tracker
__all__ = [
"Lock",
"RLock",
"Semaphore",
"BoundedSemaphore",
"Condition",
"Event",
]
# Try to import the mp.synchronize module cleanly, if it fails
# raise ImportError for platforms lacking a working sem_open implementation.
# See issue 3770
try:
from _multiprocessing import SemLock as _SemLock
from _multiprocessing import sem_unlink
except ImportError:
raise ImportError(
"This platform lacks a functioning sem_open"
" implementation, therefore, the required"
" synchronization primitives needed will not"
" function, see issue 3770."
)
#
# Constants
#
RECURSIVE_MUTEX, SEMAPHORE = range(2)
SEM_VALUE_MAX = _multiprocessing.SemLock.SEM_VALUE_MAX
#
# Base class for semaphores and mutexes; wraps `_multiprocessing.SemLock`
#
class SemLock:
_rand = tempfile._RandomNameSequence()
def __init__(self, kind, value, maxvalue, name=None):
# unlink_now is only used on win32 or when we are using fork.
unlink_now = False
if name is None:
# Try to find an unused name for the SemLock instance.
for _ in range(100):
try:
self._semlock = _SemLock(
kind, value, maxvalue, SemLock._make_name(), unlink_now
)
except FileExistsError: # pragma: no cover
pass
else:
break
else: # pragma: no cover
raise FileExistsError("cannot find name for semaphore")
else:
self._semlock = _SemLock(kind, value, maxvalue, name, unlink_now)
self.name = name
util.debug(
f"created semlock with handle {self._semlock.handle} and name "
f'"{self.name}"'
)
self._make_methods()
def _after_fork(obj):
obj._semlock._after_fork()
util.register_after_fork(self, _after_fork)
# When the object is garbage collected or the
# process shuts down we unlink the semaphore name
resource_tracker.register(self._semlock.name, "semlock")
util.Finalize(
self, SemLock._cleanup, (self._semlock.name,), exitpriority=0
)
@staticmethod
def _cleanup(name):
try:
sem_unlink(name)
except FileNotFoundError:
# Already unlinked, possibly by user code: ignore and make sure to
# unregister the semaphore from the resource tracker.
pass
finally:
resource_tracker.unregister(name, "semlock")
def _make_methods(self):
self.acquire = self._semlock.acquire
self.release = self._semlock.release
def __enter__(self):
return self._semlock.acquire()
def __exit__(self, *args):
return self._semlock.release()
def __getstate__(self):
assert_spawning(self)
sl = self._semlock
h = sl.handle
return (h, sl.kind, sl.maxvalue, sl.name)
def __setstate__(self, state):
self._semlock = _SemLock._rebuild(*state)
util.debug(
f'recreated blocker with handle {state[0]!r} and name "{state[3]}"'
)
self._make_methods()
@staticmethod
def _make_name():
# OSX does not support long names for semaphores
return f"/loky-{os.getpid()}-{next(SemLock._rand)}"
#
# Semaphore
#
class Semaphore(SemLock):
def __init__(self, value=1):
SemLock.__init__(self, SEMAPHORE, value, SEM_VALUE_MAX)
def get_value(self):
if sys.platform == "darwin":
raise NotImplementedError("OSX does not implement sem_getvalue")
return self._semlock._get_value()
def __repr__(self):
try:
value = self._semlock._get_value()
except Exception:
value = "unknown"
return f"<{self.__class__.__name__}(value={value})>"
#
# Bounded semaphore
#
class BoundedSemaphore(Semaphore):
def __init__(self, value=1):
SemLock.__init__(self, SEMAPHORE, value, value)
def __repr__(self):
try:
value = self._semlock._get_value()
except Exception:
value = "unknown"
return (
f"<{self.__class__.__name__}(value={value}, "
f"maxvalue={self._semlock.maxvalue})>"
)
#
# Non-recursive lock
#
class Lock(SemLock):
def __init__(self):
super().__init__(SEMAPHORE, 1, 1)
def __repr__(self):
try:
if self._semlock._is_mine():
name = process.current_process().name
if threading.current_thread().name != "MainThread":
name = f"{name}|{threading.current_thread().name}"
elif self._semlock._get_value() == 1:
name = "None"
elif self._semlock._count() > 0:
name = "SomeOtherThread"
else:
name = "SomeOtherProcess"
except Exception:
name = "unknown"
return f"<{self.__class__.__name__}(owner={name})>"
#
# Recursive lock
#
class RLock(SemLock):
def __init__(self):
super().__init__(RECURSIVE_MUTEX, 1, 1)
def __repr__(self):
try:
if self._semlock._is_mine():
name = process.current_process().name
if threading.current_thread().name != "MainThread":
name = f"{name}|{threading.current_thread().name}"
count = self._semlock._count()
elif self._semlock._get_value() == 1:
name, count = "None", 0
elif self._semlock._count() > 0:
name, count = "SomeOtherThread", "nonzero"
else:
name, count = "SomeOtherProcess", "nonzero"
except Exception:
name, count = "unknown", "unknown"
return f"<{self.__class__.__name__}({name}, {count})>"
#
# Condition variable
#
class Condition:
def __init__(self, lock=None):
self._lock = lock or RLock()
self._sleeping_count = Semaphore(0)
self._woken_count = Semaphore(0)
self._wait_semaphore = Semaphore(0)
self._make_methods()
def __getstate__(self):
assert_spawning(self)
return (
self._lock,
self._sleeping_count,
self._woken_count,
self._wait_semaphore,
)
def __setstate__(self, state):
(
self._lock,
self._sleeping_count,
self._woken_count,
self._wait_semaphore,
) = state
self._make_methods()
def __enter__(self):
return self._lock.__enter__()
def __exit__(self, *args):
return self._lock.__exit__(*args)
def _make_methods(self):
self.acquire = self._lock.acquire
self.release = self._lock.release
def __repr__(self):
try:
num_waiters = (
self._sleeping_count._semlock._get_value()
- self._woken_count._semlock._get_value()
)
except Exception:
num_waiters = "unknown"
return f"<{self.__class__.__name__}({self._lock}, {num_waiters})>"
def wait(self, timeout=None):
assert (
self._lock._semlock._is_mine()
), "must acquire() condition before using wait()"
# indicate that this thread is going to sleep
self._sleeping_count.release()
# release lock
count = self._lock._semlock._count()
for _ in range(count):
self._lock.release()
try:
# wait for notification or timeout
return self._wait_semaphore.acquire(True, timeout)
finally:
# indicate that this thread has woken
self._woken_count.release()
# reacquire lock
for _ in range(count):
self._lock.acquire()
def notify(self):
assert self._lock._semlock._is_mine(), "lock is not owned"
assert not self._wait_semaphore.acquire(False)
# to take account of timeouts since last notify() we subtract
# woken_count from sleeping_count and rezero woken_count
while self._woken_count.acquire(False):
res = self._sleeping_count.acquire(False)
assert res
if self._sleeping_count.acquire(False): # try grabbing a sleeper
self._wait_semaphore.release() # wake up one sleeper
self._woken_count.acquire() # wait for the sleeper to wake
# rezero _wait_semaphore in case a timeout just happened
self._wait_semaphore.acquire(False)
def notify_all(self):
assert self._lock._semlock._is_mine(), "lock is not owned"
assert not self._wait_semaphore.acquire(False)
# to take account of timeouts since last notify*() we subtract
# woken_count from sleeping_count and rezero woken_count
while self._woken_count.acquire(False):
res = self._sleeping_count.acquire(False)
assert res
sleepers = 0
while self._sleeping_count.acquire(False):
self._wait_semaphore.release() # wake up one sleeper
sleepers += 1
if sleepers:
for _ in range(sleepers):
self._woken_count.acquire() # wait for a sleeper to wake
# rezero wait_semaphore in case some timeouts just happened
while self._wait_semaphore.acquire(False):
pass
def wait_for(self, predicate, timeout=None):
result = predicate()
if result:
return result
if timeout is not None:
endtime = _time() + timeout
else:
endtime = None
waittime = None
while not result:
if endtime is not None:
waittime = endtime - _time()
if waittime <= 0:
break
self.wait(waittime)
result = predicate()
return result
#
# Event
#
class Event:
def __init__(self):
self._cond = Condition(Lock())
self._flag = Semaphore(0)
def is_set(self):
with self._cond:
if self._flag.acquire(False):
self._flag.release()
return True
return False
def set(self):
with self._cond:
self._flag.acquire(False)
self._flag.release()
self._cond.notify_all()
def clear(self):
with self._cond:
self._flag.acquire(False)
def wait(self, timeout=None):
with self._cond:
if self._flag.acquire(False):
self._flag.release()
else:
self._cond.wait(timeout)
if self._flag.acquire(False):
self._flag.release()
return True
return False
joblib-loky-51247f4/loky/backend/utils.py 0000664 0000000 0000000 00000013301 15256713413 0020271 0 ustar 00root root 0000000 0000000 import os
import sys
import time
import errno
import signal
import warnings
import subprocess
import traceback
try:
import psutil
except ImportError:
psutil = None
def kill_process_tree(process, use_psutil=True):
"""Terminate process and its descendants with SIGKILL"""
if use_psutil and psutil is not None:
_kill_process_tree_with_psutil(process)
else:
_kill_process_tree_without_psutil(process)
def recursive_terminate(process, use_psutil=True):
warnings.warn(
"recursive_terminate is deprecated in loky 3.2, use kill_process_tree"
"instead",
DeprecationWarning,
)
kill_process_tree(process, use_psutil=use_psutil)
def _kill_process_tree_with_psutil(process):
try:
descendants = psutil.Process(process.pid).children(recursive=True)
except psutil.NoSuchProcess:
return
# Kill the descendants in reverse order to avoid killing the parents before
# the descendant in cases where there are more processes nested.
for descendant in descendants[::-1]:
try:
descendant.kill()
except psutil.NoSuchProcess:
pass
try:
psutil.Process(process.pid).kill()
except psutil.NoSuchProcess:
pass
process.join()
def _kill_process_tree_without_psutil(process):
"""Terminate a process and its descendants."""
try:
if sys.platform == "win32":
_windows_taskkill_process_tree(process.pid)
else:
_posix_recursive_kill(process.pid)
except Exception: # pragma: no cover
details = traceback.format_exc()
warnings.warn(
"Failed to kill subprocesses on this platform. Please install"
"psutil: https://github.com/giampaolo/psutil\n"
f"Details:\n{details}"
)
# In case we cannot introspect or kill the descendants, we fall back to
# only killing the main process.
#
# Note: on Windows, process.kill() is an alias for process.terminate()
# which in turns calls the Win32 API function TerminateProcess().
process.kill()
process.join()
def _windows_taskkill_process_tree(pid):
# On windows, the taskkill function with option `/T` terminate a given
# process pid and its children.
try:
subprocess.check_output(
["taskkill", "/F", "/T", "/PID", str(pid)],
stderr=None,
creationflags=subprocess.CREATE_NO_WINDOW,
)
except subprocess.CalledProcessError as e:
# In Windows, taskkill returns 128, 255 for no process found.
if e.returncode not in [128, 255]:
# Let's raise to let the caller log the error details in a
# warning and only kill the root process.
raise # pragma: no cover
def _kill(pid):
# Not all systems (e.g. Windows) have a SIGKILL, but the C specification
# mandates a SIGTERM signal. While Windows is handled specifically above,
# let's try to be safe for other hypothetic platforms that only have
# SIGTERM without SIGKILL.
kill_signal = getattr(signal, "SIGKILL", signal.SIGTERM)
try:
os.kill(pid, kill_signal)
except OSError as e:
# if OSError is raised with [Errno 3] no such process, the process
# is already terminated, else, raise the error and let the top
# level function raise a warning and retry to kill the process.
if e.errno != errno.ESRCH:
raise # pragma: no cover
def _posix_recursive_kill(pid):
"""Recursively kill the descendants of a process before killing it."""
try:
children_pids = subprocess.check_output(
["pgrep", "-P", str(pid)], stderr=None, text=True
)
except subprocess.CalledProcessError as e:
# `ps` returns 1 when no child process has been found
if e.returncode == 1:
children_pids = ""
else:
raise # pragma: no cover
# Decode the result, split the cpid and remove the trailing line
for cpid in children_pids.splitlines():
cpid = int(cpid)
_posix_recursive_kill(cpid)
_kill(pid)
def get_exitcodes_terminated_worker(processes):
"""Return a formatted string with the exitcodes of terminated workers.
If necessary, wait (up to .25s) for the system to correctly set the
exitcode of one terminated worker.
"""
patience = 5
# Catch the exitcode of the terminated workers. There should at least be
# one. If not, wait a bit for the system to correctly set the exitcode of
# the terminated worker.
exitcodes = [
p.exitcode for p in list(processes.values()) if p.exitcode is not None
]
while not exitcodes and patience > 0:
patience -= 1
exitcodes = [
p.exitcode
for p in list(processes.values())
if p.exitcode is not None
]
time.sleep(0.05)
return _format_exitcodes(exitcodes)
def _format_exitcodes(exitcodes):
"""Format a list of exit code with names of the signals if possible"""
str_exitcodes = [
f"{_get_exitcode_name(e)}({e})" for e in exitcodes if e is not None
]
return "{" + ", ".join(str_exitcodes) + "}"
def _get_exitcode_name(exitcode):
if sys.platform == "win32":
# The exitcode are unreliable on windows (see bpo-31863).
# For this case, return UNKNOWN
return "UNKNOWN"
if exitcode < 0:
try:
import signal
return signal.Signals(-exitcode).name
except ValueError:
return "UNKNOWN"
elif exitcode != 255:
# The exitcode are unreliable on forkserver were 255 is always returned
# (see bpo-30589). For this case, return UNKNOWN
return "EXIT"
return "UNKNOWN"
joblib-loky-51247f4/loky/cloudpickle_wrapper.py 0000664 0000000 0000000 00000007010 15256713413 0021600 0 ustar 00root root 0000000 0000000 import inspect
from functools import partial
from cloudpickle import dumps, loads
WRAP_CACHE = {}
class CloudpickledObjectWrapper:
def __init__(self, obj, keep_wrapper=False):
self._obj = obj
self._keep_wrapper = keep_wrapper
def __reduce__(self):
_pickled_object = dumps(self._obj)
if not self._keep_wrapper:
return loads, (_pickled_object,)
return _reconstruct_wrapper, (_pickled_object, self._keep_wrapper)
def __getattr__(self, attr):
# Ensure that the wrapped object can be used seemlessly as the
# previous object.
if attr not in ["_obj", "_keep_wrapper"]:
return getattr(self._obj, attr)
return getattr(self, attr)
# Make sure the wrapped object conserves the callable property
class CallableObjectWrapper(CloudpickledObjectWrapper):
def __call__(self, *args, **kwargs):
return self._obj(*args, **kwargs)
def _wrap_non_picklable_objects(obj, keep_wrapper):
if callable(obj):
return CallableObjectWrapper(obj, keep_wrapper=keep_wrapper)
return CloudpickledObjectWrapper(obj, keep_wrapper=keep_wrapper)
def _reconstruct_wrapper(_pickled_object, keep_wrapper):
obj = loads(_pickled_object)
return _wrap_non_picklable_objects(obj, keep_wrapper)
def _wrap_objects_when_needed(obj):
# Function to introspect an object and decide if it should be wrapped or
# not.
need_wrap = "__main__" in getattr(obj, "__module__", "")
if isinstance(obj, partial):
return partial(
_wrap_objects_when_needed(obj.func),
*[_wrap_objects_when_needed(a) for a in obj.args],
**{
k: _wrap_objects_when_needed(v)
for k, v in obj.keywords.items()
},
)
if callable(obj):
# Need wrap if the object is a function defined in a local scope of
# another function.
func_code = getattr(obj, "__code__", "")
need_wrap |= getattr(func_code, "co_flags", 0) & inspect.CO_NESTED
# Need wrap if the obj is a lambda expression
func_name = getattr(obj, "__name__", "")
need_wrap |= "" in func_name
if not need_wrap:
return obj
wrapped_obj = WRAP_CACHE.get(obj)
if wrapped_obj is None:
wrapped_obj = _wrap_non_picklable_objects(obj, keep_wrapper=False)
WRAP_CACHE[obj] = wrapped_obj
return wrapped_obj
def wrap_non_picklable_objects(obj, keep_wrapper=True):
"""Wrapper for non-picklable object to use cloudpickle to serialize them.
Note that this wrapper tends to slow down the serialization process as it
is done with cloudpickle which is typically slower compared to pickle. The
proper way to solve serialization issues is to avoid defining functions and
objects in the main scripts and to implement __reduce__ functions for
complex classes.
"""
# If obj is a class, create a CloudpickledClassWrapper which instantiates
# the object internally and wrap it directly in a CloudpickledObjectWrapper
if inspect.isclass(obj):
class CloudpickledClassWrapper(CloudpickledObjectWrapper):
def __init__(self, *args, **kwargs):
self._obj = obj(*args, **kwargs)
self._keep_wrapper = keep_wrapper
CloudpickledClassWrapper.__name__ = obj.__name__
return CloudpickledClassWrapper
# If obj is an instance of a class, just wrap it in a regular
# CloudpickledObjectWrapper
return _wrap_non_picklable_objects(obj, keep_wrapper=keep_wrapper)
joblib-loky-51247f4/loky/initializers.py 0000664 0000000 0000000 00000005007 15256713413 0020254 0 ustar 00root root 0000000 0000000 import warnings
def _viztracer_init(init_kwargs):
"""Initialize viztracer's profiler in worker processes"""
from viztracer import VizTracer
tracer = VizTracer(**init_kwargs)
tracer.register_exit()
tracer.start()
def _make_viztracer_initializer_and_initargs():
try:
import viztracer
tracer = viztracer.get_tracer()
if tracer is not None and getattr(tracer, "enable", False):
# Profiler is active: introspect its configuration to
# initialize the workers with the same configuration.
return _viztracer_init, (tracer.init_kwargs,)
except ImportError:
# viztracer is not installed: nothing to do
pass
except Exception as e:
# In case viztracer's API evolve, we do not want to crash loky but
# we want to know about it to be able to update loky.
warnings.warn(f"Unable to introspect viztracer state: {e}")
return None, ()
class _ChainedInitializer:
"""Compound worker initializer
This is meant to be used in conjunction with _chain_initializers to
produce the necessary chained_args list to be passed to __call__.
"""
def __init__(self, initializers):
self._initializers = initializers
def __call__(self, *chained_args):
for initializer, args in zip(self._initializers, chained_args):
initializer(*args)
def _chain_initializers(initializer_and_args):
"""Convenience helper to combine a sequence of initializers.
If some initializers are None, they are filtered out.
"""
filtered_initializers = []
filtered_initargs = []
for initializer, initargs in initializer_and_args:
if initializer is not None:
filtered_initializers.append(initializer)
filtered_initargs.append(initargs)
if not filtered_initializers:
return None, ()
elif len(filtered_initializers) == 1:
return filtered_initializers[0], filtered_initargs[0]
else:
return _ChainedInitializer(filtered_initializers), filtered_initargs
def _prepare_initializer(initializer, initargs):
if initializer is not None and not callable(initializer):
raise TypeError(
f"initializer must be a callable, got: {initializer!r}"
)
# Introspect runtime to determine if we need to propagate the viztracer
# profiler information to the workers:
return _chain_initializers(
[
(initializer, initargs),
_make_viztracer_initializer_and_initargs(),
]
)
joblib-loky-51247f4/loky/process_executor.py 0000664 0000000 0000000 00000160745 15256713413 0021155 0 ustar 00root root 0000000 0000000 ###############################################################################
# Re-implementation of the ProcessPoolExecutor more robust to faults
#
# author: Thomas Moreau and Olivier Grisel
#
# adapted from concurrent/futures/process_pool_executor.py (17/02/2017)
# * Add an extra management thread to detect executor_manager_thread failures,
# * Improve the shutdown process to avoid deadlocks,
# * Add timeout for workers,
# * More robust pickling process.
#
# Copyright 2009 Brian Quinlan. All Rights Reserved.
# Licensed to PSF under a Contributor Agreement.
"""Implements ProcessPoolExecutor.
The follow diagram and text describe the data-flow through the system:
|======================= In-process =====================|== Out-of-process ==|
+----------+ +----------+ +--------+ +-----------+ +---------+
| | => | Work Ids | | | | Call Q | | Process |
| | +----------+ | | +-----------+ | Pool |
| | | ... | | | | ... | +---------+
| | | 6 | => | | => | 5, call() | => | |
| | | 7 | | | | ... | | |
| Process | | ... | | Local | +-----------+ | Process |
| Pool | +----------+ | Worker | | #1..n |
| Executor | | Thread | | |
| | +----------- + | | +-----------+ | |
| | <=> | Work Items | <=> | | <= | Result Q | <= | |
| | +------------+ | | +-----------+ | |
| | | 6: call() | | | | ... | | |
| | | future | +--------+ | 4, result | | |
| | | ... | | 3, except | | |
+----------+ +------------+ +-----------+ +---------+
Executor.submit() called:
- creates a uniquely numbered _WorkItem and adds it to the "Work Items" dict
- adds the id of the _WorkItem to the "Work Ids" queue
Local worker thread:
- reads work ids from the "Work Ids" queue and looks up the corresponding
WorkItem from the "Work Items" dict: if the work item has been cancelled then
it is simply removed from the dict, otherwise it is repackaged as a
_CallItem and put in the "Call Q". New _CallItems are put in the "Call Q"
until "Call Q" is full. NOTE: the size of the "Call Q" is kept small because
calls placed in the "Call Q" can no longer be cancelled with Future.cancel().
- reads _ResultItems from "Result Q", updates the future stored in the
"Work Items" dict and deletes the dict entry
Process #1..n:
- reads _CallItems from "Call Q", executes the calls, and puts the resulting
_ResultItems in "Result Q"
"""
__author__ = "Thomas Moreau (thomas.moreau.2010@gmail.com)"
import faulthandler
import os
import gc
import sys
import queue
import struct
import weakref
import warnings
import itertools
import traceback
import threading
from time import time, sleep
import multiprocessing as mp
from functools import partial
from pickle import PicklingError
from concurrent.futures import Executor, InvalidStateError
from concurrent.futures._base import LOGGER
from concurrent.futures.process import BrokenProcessPool as _BPPException
from multiprocessing.connection import wait
from ._base import Future
from .backend import get_context
from .backend.context import cpu_count, _MAX_WINDOWS_WORKERS
from .backend.queues import Queue, SimpleQueue
from .backend.reduction import set_loky_pickler, get_loky_pickler_name
from .backend.utils import kill_process_tree, get_exitcodes_terminated_worker
from .initializers import _prepare_initializer
# Mechanism to prevent infinite process spawning. When a worker of a
# ProcessPoolExecutor nested in MAX_DEPTH Executor tries to create a new
# Executor, a LokyRecursionError is raised
MAX_DEPTH = int(os.environ.get("LOKY_MAX_DEPTH", 10))
_CURRENT_DEPTH = 0
# Minimum time interval between two consecutive memory leak protection checks.
_MEMORY_LEAK_CHECK_DELAY = 1.0
# Number of bytes of memory usage allowed over the reference process size.
# Workloads whose tasks differ a lot in size can legitimately exceed the default
# without leaking, so it is configurable; it is read in the worker processes,
# which inherit the environment of the process that created the executor.
_MAX_MEMORY_LEAK_SIZE = int(
float(os.environ.get("LOKY_MAX_MEMORY_LEAK_SIZE", 3e8))
)
try:
from psutil import Process
_USE_PSUTIL = True
def _get_memory_usage(pid, force_gc=False):
if force_gc:
gc.collect()
mem_size = Process(pid).memory_info().rss
mp.util.debug(f"psutil return memory size: {mem_size}")
return mem_size
except ImportError:
_USE_PSUTIL = False
class _RecycledWorkerPid(int):
"""PID of a worker that the executor itself decided to shut down.
Behaves like the plain PID the workers usually send back, but lets the
manager thread tell a recycling it triggered on purpose from a worker that
stopped on its own.
"""
class _ThreadWakeup:
def __init__(self):
self._closed = False
self._reader, self._writer = mp.Pipe(duplex=False)
def close(self):
if not self._closed:
self._closed = True
self._writer.close()
self._reader.close()
def wakeup(self):
if not self._closed:
self._writer.send_bytes(b"")
def clear(self):
if not self._closed:
while self._reader.poll():
self._reader.recv_bytes()
class _ExecutorFlags:
"""necessary references to maintain executor states without preventing gc
It permits to keep the information needed by executor_manager_thread
and crash_detection_thread to maintain the pool without preventing the
garbage collection of unreferenced executors.
"""
def __init__(self, shutdown_lock):
self.shutdown = False
self.broken = None
self.kill_workers = False
self.shutdown_lock = shutdown_lock
def flag_as_shutting_down(self, kill_workers=None):
with self.shutdown_lock:
self.shutdown = True
if kill_workers is not None:
self.kill_workers = kill_workers
def flag_as_broken(self, broken):
with self.shutdown_lock:
self.shutdown = True
self.broken = broken
# Prior to 3.9, executor_manager_thread is created as daemon thread. This means
# that it is not joined automatically when the interpreter is shutting down.
# To work around this problem, an exit handler is installed to tell the
# thread to exit when the interpreter is shutting down and then waits until
# it finishes. The thread needs to be daemonized because the atexit hooks are
# called after all non daemonized threads are joined.
#
# Starting 3.9, there exists a specific atexit hook to be called before joining
# the threads so the executor_manager_thread does not need to be daemonized
# anymore.
#
# The atexit hooks are registered when starting the first ProcessPoolExecutor
# to avoid import having an effect on the interpreter.
_global_shutdown = False
_global_shutdown_lock = threading.Lock()
_threads_wakeups = weakref.WeakKeyDictionary()
def _python_exit():
global _global_shutdown
_global_shutdown = True
# Materialize the list of items to avoid error due to iterating over
# changing size dictionary.
items = list(_threads_wakeups.items())
if len(items) > 0:
mp.util.debug(
f"Interpreter shutting down. Waking up {len(items)}"
f"executor_manager_thread:\n{items}"
)
# Wake up the executor_manager_thread's so they can detect the interpreter
# is shutting down and exit.
for _, (shutdown_lock, thread_wakeup) in items:
with shutdown_lock:
thread_wakeup.wakeup()
# Collect the executor_manager_thread's to make sure we exit cleanly.
for thread, _ in items:
# This locks is to prevent situations where an executor is gc'ed in one
# thread while the atexit finalizer is running in another thread.
with _global_shutdown_lock:
thread.join()
# With the fork context, _thread_wakeups is propagated to children.
# Clear it after fork to avoid some situation that can cause some
# freeze when joining the workers.
mp.util.register_after_fork(_threads_wakeups, lambda obj: obj.clear())
# Module variable to register the at_exit call
process_pool_executor_at_exit = None
# Controls how many more calls than processes will be queued in the call queue.
# A smaller number will mean that processes spend more time idle waiting for
# work while a larger number will make Future.cancel() succeed less frequently
# (Futures in the call queue cannot be cancelled).
EXTRA_QUEUED_CALLS = 1
class _RemoteTraceback(Exception):
"""Embed stringification of remote traceback in local traceback"""
def __init__(self, tb=None):
self.tb = f'\n"""\n{tb}"""'
def __str__(self):
return self.tb
# Do not inherit from BaseException to mirror
# concurrent.futures.process._ExceptionWithTraceback
class _ExceptionWithTraceback:
def __init__(self, exc):
tb = getattr(exc, "__traceback__", None)
if tb is None:
_, _, tb = sys.exc_info()
tb = traceback.format_exception(type(exc), exc, tb)
tb = "".join(tb)
self.exc = exc
self.tb = tb
def __reduce__(self):
return _rebuild_exc, (self.exc, self.tb)
def _rebuild_exc(exc, tb):
exc.__cause__ = _RemoteTraceback(tb)
return exc
class _WorkItem:
__slots__ = ["future", "fn", "args", "kwargs"]
def __init__(self, future, fn, args, kwargs):
self.future = future
self.fn = fn
self.args = args
self.kwargs = kwargs
class _ResultItem:
def __init__(self, work_id, exception=None, result=None):
self.work_id = work_id
self.exception = exception
self.result = result
class _CallItem:
def __init__(self, work_id, fn, args, kwargs):
self.work_id = work_id
self.fn = fn
self.args = args
self.kwargs = kwargs
# Store the current loky_pickler so it is correctly set in the worker
self.loky_pickler = get_loky_pickler_name()
def __call__(self):
set_loky_pickler(self.loky_pickler)
return self.fn(*self.args, **self.kwargs)
def __repr__(self):
return (
f"CallItem({self.work_id}, {self.fn}, {self.args}, {self.kwargs})"
)
class _SafeQueue(Queue):
"""Safe Queue set exception to the future object linked to a job"""
def __init__(
self,
max_size=0,
ctx=None,
pending_work_items=None,
running_work_items=None,
thread_wakeup=None,
shutdown_lock=None,
executor_flags=None,
reducers=None,
):
self.thread_wakeup = thread_wakeup
self.shutdown_lock = shutdown_lock
self.executor_flags = executor_flags
self.pending_work_items = pending_work_items
self.running_work_items = running_work_items
super().__init__(max_size, reducers=reducers, ctx=ctx)
def _on_queue_feeder_error(self, e, obj):
if not isinstance(obj, _CallItem):
super()._on_queue_feeder_error(e, obj)
return
# work_item can be None if another process terminated. In this
# case, the executor_manager_thread fails all work_items with
# BrokenProcessPool
work_item = self.pending_work_items.pop(obj.work_id, None)
try:
# format traceback only works on python3
if isinstance(e, struct.error):
raised_error = RuntimeError(
"The task could not be sent to the workers as it is too "
"large for `send_bytes`."
)
else:
raised_error = PicklingError(
"Could not pickle the task to send it to the workers."
)
tb = traceback.format_exception(
type(e), e, getattr(e, "__traceback__", None)
)
raised_error.__cause__ = _RemoteTraceback("".join(tb))
self.running_work_items.remove(obj.work_id)
except BaseException as hook_exc:
# An error here would kill the feeder thread silently and leave
# every later task unsent: flag the executor as broken instead
raised_error = BrokenProcessPool(
"The call queue feeder thread crashed: the executor is "
"broken and the pending tasks have been cancelled."
)
tb = traceback.format_exception(
type(hook_exc), hook_exc, hook_exc.__traceback__
)
raised_error.__cause__ = _RemoteTraceback("".join(tb))
LOGGER.critical(
"Exception in call queue feeder error hook:", exc_info=True
)
self.executor_flags.flag_as_broken(raised_error)
if work_item is not None:
work_item.future.set_exception(raised_error)
del work_item
with self.shutdown_lock:
self.thread_wakeup.wakeup()
def _get_chunks(chunksize, *iterables):
"""Iterates over zip()ed iterables in chunks."""
it = zip(*iterables)
while True:
chunk = tuple(itertools.islice(it, chunksize))
if not chunk:
return
yield chunk
def _process_chunk(fn, chunk):
"""Processes a chunk of an iterable passed to map.
Runs the function passed to map() on a chunk of the
iterable passed to map.
This function is run in a separate process.
"""
return [fn(*args) for args in chunk]
def _sendback_result(result_queue, work_id, result=None, exception=None):
"""Safely send back the given result or exception"""
try:
result_queue.put(
_ResultItem(work_id, result=result, exception=exception)
)
except BaseException as e:
exc = _ExceptionWithTraceback(e)
result_queue.put(_ResultItem(work_id, exception=exc))
def _enable_faulthandler_if_needed():
if "PYTHONFAULTHANDLER" in os.environ:
# Respect the environment variable to configure faulthandler. This
# makes it possible to never enable faulthandler in the loky workers by
# setting PYTHONFAULTHANDLER=0 explicitly in the environment.
mp.util.debug(
f"faulthandler explicitly configured by environment variable: "
f"PYTHONFAULTHANDLER={os.environ['PYTHONFAULTHANDLER']}."
)
else:
if faulthandler.is_enabled():
# Fault handler is already enabled, possibly via a custom
# initializer to customize the behavior.
mp.util.debug("faulthandler already enabled.")
else:
# Enable faulthandler by default with default paramaters otherwise.
mp.util.debug(
"Enabling faulthandler to report tracebacks on worker crashes."
)
faulthandler.enable()
def _process_worker(
call_queue,
result_queue,
initializer,
initargs,
processes_management_lock,
timeout,
worker_exit_lock,
current_depth,
):
"""Evaluates calls from call_queue and places the results in result_queue.
This worker is run in a separate process.
Args:
call_queue: A ctx.Queue of _CallItems that will be read and
evaluated by the worker.
result_queue: A ctx.Queue of _ResultItems that will written
to by the worker.
initializer: A callable initializer, or None
initargs: A tuple of args for the initializer
processes_management_lock: A ctx.Lock avoiding worker timeout while
some workers are being spawned.
timeout: maximum time to wait for a new item in the call_queue. If that
time is expired, the worker will shutdown.
worker_exit_lock: Lock to avoid flagging the executor as broken on
workers timeout.
current_depth: Nested parallelism level, to avoid infinite spawning.
"""
if initializer is not None:
try:
initializer(*initargs)
except BaseException:
LOGGER.critical("Exception in initializer:", exc_info=True)
# The parent will notice that the process stopped and
# mark the pool broken
return
# set the global _CURRENT_DEPTH mechanism to limit recursive call
global _CURRENT_DEPTH
_CURRENT_DEPTH = current_depth
_process_reference_size = None
_last_memory_leak_check = None
pid = os.getpid()
mp.util.debug(f"Worker started with timeout={timeout}")
_enable_faulthandler_if_needed()
while True:
try:
call_item = call_queue.get(block=True, timeout=timeout)
if call_item is None:
mp.util.info("Shutting down worker on sentinel")
except queue.Empty:
mp.util.info(f"Shutting down worker after timeout {timeout:0.3f}s")
if processes_management_lock.acquire(block=False):
processes_management_lock.release()
call_item = None
else:
mp.util.info("Could not acquire processes_management_lock")
continue
except BaseException:
previous_tb = traceback.format_exc()
try:
result_queue.put(_RemoteTraceback(previous_tb))
except BaseException:
# If we cannot format correctly the exception, at least print
# the traceback.
print(previous_tb)
mp.util.debug("Exiting with code 1")
sys.exit(1)
if call_item is None:
# Notify queue management thread about worker shutdown
result_queue.put(pid)
is_clean = worker_exit_lock.acquire(True, timeout=30)
# Early notify any loky executor running in this worker process
# (nested parallelism) that this process is about to shutdown to
# avoid a deadlock waiting undifinitely for the worker to finish.
_python_exit()
if is_clean:
mp.util.debug("Exited cleanly")
else:
mp.util.info("Main process did not release worker_exit")
return
try:
r = call_item()
except BaseException as e:
exc = _ExceptionWithTraceback(e)
result_queue.put(_ResultItem(call_item.work_id, exception=exc))
else:
_sendback_result(result_queue, call_item.work_id, result=r)
del r
# Free the resource as soon as possible, to avoid holding onto
# open files or shared memory that is not needed anymore
del call_item
if _USE_PSUTIL:
if _process_reference_size is None:
# Make reference measurement after the first call
_process_reference_size = _get_memory_usage(pid, force_gc=True)
_last_memory_leak_check = time()
continue
if time() - _last_memory_leak_check > _MEMORY_LEAK_CHECK_DELAY:
mem_usage = _get_memory_usage(pid)
_last_memory_leak_check = time()
if mem_usage - _process_reference_size < _MAX_MEMORY_LEAK_SIZE:
# Memory usage stays within bounds: everything is fine.
continue
# Check again memory usage; this time take the measurement
# after a forced garbage collection to break any reference
# cycles.
mem_usage = _get_memory_usage(pid, force_gc=True)
_last_memory_leak_check = time()
if mem_usage - _process_reference_size < _MAX_MEMORY_LEAK_SIZE:
# The GC managed to free the memory: everything is fine.
continue
# The process is leaking memory: let the master process
# know that we need to start a new worker.
mp.util.info("Memory leak detected: shutting down worker")
result_queue.put(_RecycledWorkerPid(pid))
with worker_exit_lock:
mp.util.debug("Exit due to memory leak")
return
else:
# if psutil is not installed, trigger gc.collect events
# regularly to limit potential memory leaks due to reference cycles
if _last_memory_leak_check is None or (
time() - _last_memory_leak_check > _MEMORY_LEAK_CHECK_DELAY
):
gc.collect()
_last_memory_leak_check = time()
class _ExecutorManagerThread(threading.Thread):
"""Manages the communication between this process and the worker processes.
The manager is run in a local thread.
Args:
executor: A reference to the ProcessPoolExecutor that owns
this thread. A weakref will be own by the manager as well as
references to internal objects used to introspect the state of
the executor.
"""
def __init__(self, executor):
# Store references to necessary internals of the executor.
# A _ThreadWakeup to allow waking up the executor_manager_thread from
# the main Thread and avoid deadlocks caused by permanently
# locked queues.
self.thread_wakeup = executor._executor_manager_thread_wakeup
self.shutdown_lock = executor._shutdown_lock
# A weakref.ref to the ProcessPoolExecutor that owns this thread. Used
# to determine if the ProcessPoolExecutor has been garbage collected
# and that the manager can exit.
# When the executor gets garbage collected, the weakref callback
# will wake up the queue management thread so that it can terminate
# if there is no pending work item.
def weakref_cb(
_,
thread_wakeup=self.thread_wakeup,
shutdown_lock=self.shutdown_lock,
):
if mp is not None:
# At this point, the multiprocessing module can already be
# garbage collected. We only log debug info when still
# possible.
mp.util.debug(
"Executor collected: triggering callback for"
" QueueManager wakeup"
)
with shutdown_lock:
thread_wakeup.wakeup()
self.executor_reference = weakref.ref(executor, weakref_cb)
# The flags of the executor
self.executor_flags = executor._flags
# A list of the ctx.Process instances used as workers.
self.processes = executor._processes
# A ctx.Queue that will be filled with _CallItems derived from
# _WorkItems for processing by the process workers.
self.call_queue = executor._call_queue
# A ctx.SimpleQueue of _ResultItems generated by the process workers.
self.result_queue = executor._result_queue
# A queue.Queue of work ids e.g. Queue([5, 6, ...]).
self.work_ids_queue = executor._work_ids
# A dict mapping work ids to _WorkItems e.g.
# {5: <_WorkItem...>, 6: <_WorkItem...>, ...}
self.pending_work_items = executor._pending_work_items
# A list of the work_ids that are currently running
self.running_work_items = executor._running_work_items
# A lock to avoid concurrent shutdown of workers on timeout and spawn
# of new processes or shut down
self.processes_management_lock = executor._processes_management_lock
# Long-running executors can recycle workers many times over their
# lifetime; the user only needs to hear about it once.
self.recycling_warned = False
super().__init__(name="ExecutorManagerThread")
if sys.version_info < (3, 9):
self.daemon = True
def run(self):
# Main loop for the executor manager thread.
try:
self._run_loop()
except BaseException as exc:
# Nothing in the loop is expected to raise, so this is a bug, a
# warning filter turning one of the warnings below into an error,
# or a failure of the runtime itself, such as a queue that cannot
# start its feeder thread while the interpreter is shutting down
# (python/cpython#109047). Letting the exception kill this thread
# would leave every pending future unresolved and the executor
# looking healthy, so a caller waiting on a result would hang
# forever.
# Flag the executor as broken instead, with the error as cause.
bpe = BrokenProcessPool(
"The executor manager thread crashed: the executor is broken "
"and the pending tasks have been cancelled."
)
tb = traceback.format_exception(type(exc), exc, exc.__traceback__)
bpe.__cause__ = _RemoteTraceback("".join(tb))
# Also log it: with no pending future, nothing else reports it
LOGGER.critical(
"Exception in executor manager thread:", exc_info=True
)
self.terminate_broken(bpe)
def _run_loop(self):
while True:
self.add_call_item_to_queue()
result_item, is_broken, bpe = self.wait_result_broken_or_wakeup()
# The call queue feeder thread flags the executor broken and
# wakes us up when it cannot report a task failure itself
if not is_broken and self.executor_flags.broken is not None:
is_broken, bpe = True, self.executor_flags.broken
if is_broken:
self.terminate_broken(bpe)
return
if result_item is not None:
self.process_result_item(result_item)
# Delete reference to result_item to avoid keeping references
# while waiting on new results.
del result_item
if self.is_shutting_down():
self.flag_executor_shutting_down()
# When only canceled futures remain in pending_work_items, our
# next call to wait_result_broken_or_wakeup would hang forever.
# This makes sure we have some running futures or none at all.
self.add_call_item_to_queue()
# Since no new work items can be added, it is safe to shutdown
# this thread if there are no pending work items.
if not self.pending_work_items:
self.join_executor_internals()
return
def add_call_item_to_queue(self):
# Fills call_queue with _WorkItems from pending_work_items.
# This function never blocks.
while True:
if self.call_queue.full():
return
try:
work_id = self.work_ids_queue.get(block=False)
except queue.Empty:
return
else:
work_item = self.pending_work_items[work_id]
if work_item.future.set_running_or_notify_cancel():
self.running_work_items += [work_id]
self.call_queue.put(
_CallItem(
work_id,
work_item.fn,
work_item.args,
work_item.kwargs,
),
block=True,
)
else:
del self.pending_work_items[work_id]
continue
def wait_result_broken_or_wakeup(self):
# Wait for a result to be ready in the result_queue while checking
# that all worker processes are still running, or for a wake up
# signal send. The wake up signals come either from new tasks being
# submitted, from the executor being shutdown/gc-ed, or from the
# shutdown of the python interpreter.
result_reader = self.result_queue._reader
wakeup_reader = self.thread_wakeup._reader
readers = [result_reader, wakeup_reader]
worker_sentinels = [p.sentinel for p in list(self.processes.values())]
ready = wait(readers + worker_sentinels)
bpe = None
is_broken = True
result_item = None
if result_reader in ready:
try:
result_item = result_reader.recv()
if isinstance(result_item, _RemoteTraceback):
bpe = BrokenProcessPool(
"A task has failed to un-serialize. Please ensure that"
" the arguments of the function are all picklable."
)
bpe.__cause__ = result_item
else:
is_broken = False
except BaseException as e:
bpe = BrokenProcessPool(
"A result has failed to un-serialize. Please ensure that "
"the objects returned by the function are always "
"picklable."
)
tb = traceback.format_exception(
type(e), e, getattr(e, "__traceback__", None)
)
bpe.__cause__ = _RemoteTraceback("".join(tb))
elif wakeup_reader in ready:
# This is simply a wake-up event that might either trigger putting
# more tasks in the queue or trigger the clean up of resources.
is_broken = False
else:
# A worker has terminated and we don't know why, set the state of
# the executor as broken
exit_codes = ""
if sys.platform != "win32":
# In Windows, introspecting terminated workers exitcodes seems
# unstable, therefore they are not appended in the exception
# message.
exit_codes = (
"\nThe exit codes of the workers are "
f"{get_exitcodes_terminated_worker(self.processes)}"
)
mp.util.debug(
"A worker unexpectedly terminated. Workers that "
"might have caused the breakage: "
+ str(
{
p.name: p.exitcode
for p in list(self.processes.values())
if p is not None and p.sentinel in ready
}
)
)
bpe = TerminatedWorkerError(
"A worker process managed by the executor was unexpectedly "
"terminated. This could be caused by a segmentation fault "
"while calling the function or by an excessive memory usage "
"causing the Operating System to kill the worker.\n"
f"{exit_codes}\n"
"Detailed tracebacks of the workers should have been printed "
"to stderr in the executor process if faulthandler was not "
"disabled."
)
self.thread_wakeup.clear()
return result_item, is_broken, bpe
def process_result_item(self, result_item):
# Process the received a result_item. This can be either the PID of a
# worker that exited gracefully or a _ResultItem
if isinstance(result_item, int):
# Clean shutdown of a worker using its PID, either on request
# by the executor.shutdown method or by the timeout of the worker
# itself: we should not mark the executor as broken.
with self.processes_management_lock:
p = self.processes.pop(result_item, None)
# p can be None if the executor is concurrently shutting down.
if p is not None:
p._worker_exit_lock.release()
mp.util.debug(
f"joining {p.name} when processing {p.pid} as result_item"
)
p.join()
del p
# Make sure the executor have the right number of worker, even if a
# worker timeout while some jobs were submitted. If some work is
# pending or there is less processes than running items, we need to
# start a new Process and raise a warning.
n_pending = len(self.pending_work_items)
n_running = len(self.running_work_items)
if n_pending - n_running > 0 or n_running > len(self.processes):
executor = self.executor_reference()
if (
executor is not None
and len(self.processes) < executor._max_workers
):
with executor._processes_management_lock:
executor._adjust_process_count()
executor = None
# Warn only once the pool is back to full strength, so
# that a filter turning this into an error costs no worker
if not isinstance(result_item, _RecycledWorkerPid):
warnings.warn(
"A worker stopped while some jobs were given to "
"the executor. This can be caused by a too short "
"worker timeout or by a memory leak triggered by the "
"OS-level out-of-memory (OOM) process killer.",
UserWarning,
)
elif not self.recycling_warned:
# Recycling is a deliberate decision of the executor and
# it is transparent to the caller, so repeating it for
# every worker of a long-running executor is just noise.
self.recycling_warned = True
warnings.warn(
"A worker was restarted while some jobs were given "
"to the executor because it was using more than "
f"{_MAX_MEMORY_LEAK_SIZE / 1e6:.0f} MB more memory "
"than when it started, which usually indicates a "
"memory leak. Further restarts of this executor "
"will not be reported.",
UserWarning,
)
else:
# Received a _ResultItem so mark the future as completed.
work_item = self.pending_work_items.pop(result_item.work_id, None)
# work_item can be None if another process terminated (see above)
if work_item is not None:
if result_item.exception is not None:
work_item.future.set_exception(result_item.exception)
else:
work_item.future.set_result(result_item.result)
self.running_work_items.remove(result_item.work_id)
def is_shutting_down(self):
# Check whether we should start shutting down the executor.
executor = self.executor_reference()
# No more work items can be added if:
# - The interpreter is shutting down OR
# - The executor that owns this thread is not broken AND
# * The executor that owns this worker has been collected OR
# * The executor that owns this worker has been shutdown.
# If the executor is broken, it should be detected in the next loop.
return _global_shutdown or (
(executor is None or self.executor_flags.shutdown)
and not self.executor_flags.broken
)
def terminate_broken(self, bpe):
# Terminate the executor because it is in a broken state. The bpe
# argument can be used to display more information on the error that
# lead the executor into becoming broken.
# Mark the process pool broken so that submits fail right now.
self.executor_flags.flag_as_broken(bpe)
# Mark pending tasks as failed.
for work_item in self.pending_work_items.values():
try:
work_item.future.set_exception(bpe)
except InvalidStateError:
# set_exception() fails if the future is cancelled: ignore it.
# Trying to check if the future is cancelled before calling
# set_exception() would leave a race condition if the future is
# cancelled between the check and set_exception().
pass
# Delete references to object. See issue16284
del work_item
self.pending_work_items.clear()
# Terminate remaining workers forcibly: the queues or their
# locks may be in a dirty state and block forever.
self.kill_workers(reason="broken executor")
# clean up resources
self.join_executor_internals()
def flag_executor_shutting_down(self):
# Flag the executor as shutting down and cancel remaining tasks if
# requested as early as possible if it is not gc-ed yet.
self.executor_flags.flag_as_shutting_down()
# Cancel pending work items if requested.
if self.executor_flags.kill_workers:
# Drain the queued ids first: add_call_item_to_queue runs again
# right after this and looks each of them up in pending_work_items
while True:
try:
self.work_ids_queue.get(block=False)
except queue.Empty:
break
while self.pending_work_items:
_, work_item = self.pending_work_items.popitem()
work_item.future.set_exception(
ShutdownExecutorError(
"The Executor was shutdown with `kill_workers=True` "
"before this job could complete."
)
)
del work_item
# Kill the remaining worker forcibly to no waste time joining them
self.kill_workers(reason="executor shutting down")
def kill_workers(self, reason=""):
# Terminate the remaining workers using SIGKILL. This function also
# terminates descendant workers of the children in case there is some
# nested parallelism.
while self.processes:
_, p = self.processes.popitem()
mp.util.debug(f"terminate process {p.name}, reason: {reason}")
try:
kill_process_tree(p)
except ProcessLookupError: # pragma: no cover
pass
def shutdown_workers(self):
# shutdown all workers in self.processes
# Create a list to avoid RuntimeError due to concurrent modification of
# processes. nb_children_alive is thus an upper bound. Also release the
# processes' _worker_exit_lock to accelerate the shutdown procedure, as
# there is no need for hand-shake here.
with self.processes_management_lock:
n_children_to_stop = 0
for p in list(self.processes.values()):
mp.util.debug(f"releasing worker exit lock on {p.name}")
p._worker_exit_lock.release()
n_children_to_stop += 1
mp.util.debug(f"found {n_children_to_stop} processes to stop")
# Send the right number of sentinels, to make sure all children are
# properly terminated. Do it with a mechanism that avoid hanging on
# Full queue when all workers have already been shutdown.
n_sentinels_sent = 0
cooldown_time = 0.001
while (
n_sentinels_sent < n_children_to_stop
and self.get_n_children_alive() > 0
):
for _ in range(n_children_to_stop - n_sentinels_sent):
try:
self.call_queue.put_nowait(None)
n_sentinels_sent += 1
except queue.Full as e:
if cooldown_time > 5.0:
mp.util.info(
"failed to send all sentinels and exit with error."
f"\ncall_queue size={self.call_queue._maxsize}; "
f" full is {self.call_queue.full()}; "
)
raise e
mp.util.info(
"full call_queue prevented to send all sentinels at "
"once, waiting..."
)
sleep(cooldown_time)
cooldown_time *= 1.2
break
mp.util.debug(f"sent {n_sentinels_sent} sentinels to the call queue")
def join_executor_internals(self):
self.shutdown_workers()
# Release the queue's resources as soon as possible. Flag the feeder
# thread for clean exit to avoid having the crash detection thread flag
# the Executor as broken during the shutdown. This is safe as either:
# * We don't need to communicate with the workers anymore
# * There is nothing left in the Queue buffer except None sentinels
mp.util.debug("closing call_queue")
self.call_queue.close()
self.call_queue.join_thread()
# Closing result_queue
mp.util.debug("closing result_queue")
self.result_queue.close()
mp.util.debug("closing thread_wakeup")
with self.shutdown_lock:
self.thread_wakeup.close()
# If .join() is not called on the created processes then
# some ctx.Queue methods may deadlock on macOS.
with self.processes_management_lock:
mp.util.debug(f"joining {len(self.processes)} processes")
n_joined_processes = 0
while True:
try:
pid, p = self.processes.popitem()
mp.util.debug(f"joining process {p.name} with pid {pid}")
p.join()
n_joined_processes += 1
except KeyError:
break
mp.util.debug(
"executor management thread clean shutdown of "
f"{n_joined_processes} workers"
)
def get_n_children_alive(self):
# This is an upper bound on the number of children alive.
with self.processes_management_lock:
return sum(p.is_alive() for p in list(self.processes.values()))
_system_limits_checked = False
_system_limited = None
def _check_system_limits():
global _system_limits_checked, _system_limited
if _system_limits_checked and _system_limited:
raise NotImplementedError(_system_limited)
_system_limits_checked = True
try:
nsems_max = os.sysconf("SC_SEM_NSEMS_MAX")
except (AttributeError, ValueError):
# sysconf not available or setting not available
return
if nsems_max == -1:
# undetermined limit, assume that limit is determined
# by available memory only
return
if nsems_max >= 256:
# minimum number of semaphores available
# according to POSIX
return
_system_limited = (
f"system provides too few semaphores ({nsems_max} available, "
"256 necessary)"
)
raise NotImplementedError(_system_limited)
def _chain_from_iterable_of_lists(iterable):
"""
Specialized implementation of itertools.chain.from_iterable.
Each item in *iterable* should be a list. This function is
careful not to keep references to yielded objects.
"""
for element in iterable:
element.reverse()
while element:
yield element.pop()
def _check_max_depth(context):
# Limit the maxmal recursion level
global _CURRENT_DEPTH
if context.get_start_method() == "fork" and _CURRENT_DEPTH > 0:
raise LokyRecursionError(
"Could not spawn extra nested processes at depth superior to "
"MAX_DEPTH=1. It is not possible to increase this limit when "
"using the 'fork' start method."
)
if 0 < MAX_DEPTH and _CURRENT_DEPTH + 1 > MAX_DEPTH:
raise LokyRecursionError(
"Could not spawn extra nested processes at depth superior to "
f"MAX_DEPTH={MAX_DEPTH}. If this is intendend, you can change "
"this limit with the LOKY_MAX_DEPTH environment variable."
)
class LokyRecursionError(RuntimeError):
"""A process tries to spawn too many levels of nested processes."""
class BrokenProcessPool(_BPPException):
"""
Raised when the executor is broken while a future was in the running state.
The cause can an error raised when unpickling the task in the worker
process or when unpickling the result value in the parent process. It can
also be caused by a worker process being terminated unexpectedly.
"""
class TerminatedWorkerError(BrokenProcessPool):
"""
Raised when a process in a ProcessPoolExecutor terminated abruptly
while a future was in the running state.
"""
# Alias for backward compat (for code written for loky 1.1.4 and earlier). Do
# not use in new code.
BrokenExecutor = BrokenProcessPool
class ShutdownExecutorError(RuntimeError):
"""
Raised when a ProcessPoolExecutor is shutdown while a future was in the
running or pending state.
"""
class ProcessPoolExecutor(Executor):
_at_exit = None
def __init__(
self,
max_workers=None,
job_reducers=None,
result_reducers=None,
timeout=None,
context=None,
initializer=None,
initargs=(),
env=None,
):
"""Initializes a new ProcessPoolExecutor instance.
Args:
max_workers: int, optional (default: cpu_count())
The maximum number of processes that can be used to execute the
given calls. If None or not given then as many worker processes
will be created as the number of CPUs the current process
can use.
job_reducers, result_reducers: dict(type: reducer_func)
Custom reducer for pickling the jobs and the results from the
Executor. If only `job_reducers` is provided, `result_reducer`
will use the same reducers
timeout: int, optional (default: None)
Idle workers exit after timeout seconds. If a new job is
submitted after the timeout, the executor will start enough
new Python processes to make sure the pool of workers is full.
context: A multiprocessing context to launch the workers. This
object should provide SimpleQueue, Queue and Process.
initializer: An callable used to initialize worker processes.
initargs: A tuple of arguments to pass to the initializer.
env: A dict of environment variable to overwrite in the child
process. The environment variables are set before any module is
loaded. Note that this only works with the loky context.
"""
_check_system_limits()
if max_workers is None:
self._max_workers = cpu_count()
else:
if max_workers <= 0:
raise ValueError("max_workers must be greater than 0")
self._max_workers = max_workers
if (
sys.platform == "win32"
and self._max_workers > _MAX_WINDOWS_WORKERS
):
warnings.warn(
f"On Windows, max_workers cannot exceed {_MAX_WINDOWS_WORKERS} "
"due to limitations of the operating system."
)
self._max_workers = _MAX_WINDOWS_WORKERS
if context is None:
context = get_context()
self._context = context
self._env = env
self._initializer, self._initargs = _prepare_initializer(
initializer, initargs
)
_check_max_depth(self._context)
if result_reducers is None:
result_reducers = job_reducers
# Timeout
self._timeout = timeout
# Management thread
self._executor_manager_thread = None
# Map of pids to processes
self._processes = {}
# Internal variables of the ProcessPoolExecutor
self._processes = {}
self._queue_count = 0
self._pending_work_items = {}
self._running_work_items = []
self._work_ids = queue.Queue()
self._processes_management_lock = self._context.Lock()
self._executor_manager_thread = None
self._shutdown_lock = threading.Lock()
# _ThreadWakeup is a communication channel used to interrupt the wait
# of the main loop of executor_manager_thread from another thread (e.g.
# when calling executor.submit or executor.shutdown). We do not use the
# _result_queue to send wakeup signals to the executor_manager_thread
# as it could result in a deadlock if a worker process dies with the
# _result_queue write lock still acquired.
#
# _shutdown_lock must be locked to access _ThreadWakeup.wakeup.
self._executor_manager_thread_wakeup = _ThreadWakeup()
# Flag to hold the state of the Executor. This permits to introspect
# the Executor state even once it has been garbage collected.
self._flags = _ExecutorFlags(self._shutdown_lock)
# Finally setup the queues for interprocess communication
self._setup_queues(job_reducers, result_reducers)
mp.util.debug("ProcessPoolExecutor is setup")
def _setup_queues(self, job_reducers, result_reducers, queue_size=None):
# Make the call queue slightly larger than the number of processes to
# prevent the worker processes from idling. But don't make it too big
# because futures in the call queue cannot be cancelled.
if queue_size is None:
queue_size = 2 * self._max_workers + EXTRA_QUEUED_CALLS
self._call_queue = _SafeQueue(
max_size=queue_size,
pending_work_items=self._pending_work_items,
running_work_items=self._running_work_items,
thread_wakeup=self._executor_manager_thread_wakeup,
shutdown_lock=self._shutdown_lock,
executor_flags=self._flags,
reducers=job_reducers,
ctx=self._context,
)
# Killed worker processes can produce spurious "broken pipe"
# tracebacks in the queue's own worker thread. But we detect killed
# processes anyway, so silence the tracebacks.
self._call_queue._ignore_epipe = True
self._result_queue = SimpleQueue(
reducers=result_reducers, ctx=self._context
)
def _start_executor_manager_thread(self):
if self._executor_manager_thread is None:
mp.util.debug("_start_executor_manager_thread called")
# Start the processes so that their sentinels are known.
self._executor_manager_thread = _ExecutorManagerThread(self)
self._executor_manager_thread.start()
# register this executor in a mechanism that ensures it will wakeup
# when the interpreter is exiting.
_threads_wakeups[self._executor_manager_thread] = (
self._shutdown_lock,
self._executor_manager_thread_wakeup,
)
global process_pool_executor_at_exit
if process_pool_executor_at_exit is None:
# Ensure that the _python_exit function will be called before
# the multiprocessing.Queue._close finalizers which have an
# exitpriority of 10.
if sys.version_info < (3, 9):
process_pool_executor_at_exit = mp.util.Finalize(
None, _python_exit, exitpriority=20
)
else:
process_pool_executor_at_exit = threading._register_atexit(
_python_exit
)
def _adjust_process_count(self):
while len(self._processes) < self._max_workers:
worker_exit_lock = self._context.BoundedSemaphore(1)
args = (
self._call_queue,
self._result_queue,
self._initializer,
self._initargs,
self._processes_management_lock,
self._timeout,
worker_exit_lock,
_CURRENT_DEPTH + 1,
)
worker_exit_lock.acquire()
try:
# Try to spawn the process with some environment variable to
# overwrite but it only works with the loky context for now.
p = self._context.Process(
target=_process_worker, args=args, env=self._env
)
except TypeError:
p = self._context.Process(target=_process_worker, args=args)
p._worker_exit_lock = worker_exit_lock
p.start()
self._processes[p.pid] = p
mp.util.debug(
f"Adjusted process count to {self._max_workers}: "
f"{[(p.name, pid) for pid, p in self._processes.items()]}"
)
def _ensure_executor_running(self):
"""ensures all workers and management thread are running"""
with self._processes_management_lock:
if len(self._processes) != self._max_workers:
self._adjust_process_count()
self._start_executor_manager_thread()
def submit(self, fn, *args, **kwargs):
with self._flags.shutdown_lock:
if self._flags.broken is not None:
raise self._flags.broken
if self._flags.shutdown:
raise ShutdownExecutorError(
"cannot schedule new futures after shutdown"
)
# Cannot submit a new calls once the interpreter is shutting down.
# This check avoids spawning new processes at exit.
if _global_shutdown:
raise RuntimeError(
"cannot schedule new futures after interpreter shutdown"
)
f = Future()
w = _WorkItem(f, fn, args, kwargs)
self._pending_work_items[self._queue_count] = w
self._work_ids.put(self._queue_count)
self._queue_count += 1
# Wake up queue management thread
self._executor_manager_thread_wakeup.wakeup()
self._ensure_executor_running()
return f
submit.__doc__ = Executor.submit.__doc__
def map(self, fn, *iterables, **kwargs):
"""Returns an iterator equivalent to map(fn, iter).
Args:
fn: A callable that will take as many arguments as there are
passed iterables.
timeout: The maximum number of seconds to wait. If None, then there
is no limit on the wait time.
chunksize: If greater than one, the iterables will be chopped into
chunks of size chunksize and submitted to the process pool.
If set to one, the items in the list will be sent one at a
time.
Returns:
An iterator equivalent to: map(func, *iterables) but the calls may
be evaluated out-of-order.
Raises:
TimeoutError: If the entire result iterator could not be generated
before the given timeout.
Exception: If fn(*args) raises for any values.
"""
timeout = kwargs.get("timeout", None)
chunksize = kwargs.get("chunksize", 1)
if chunksize < 1:
raise ValueError("chunksize must be >= 1.")
results = super().map(
partial(_process_chunk, fn),
_get_chunks(chunksize, *iterables),
timeout=timeout,
)
return _chain_from_iterable_of_lists(results)
def shutdown(self, wait=True, kill_workers=False):
mp.util.debug(f"shutting down executor {self}")
self._flags.flag_as_shutting_down(kill_workers)
executor_manager_thread = self._executor_manager_thread
executor_manager_thread_wakeup = self._executor_manager_thread_wakeup
if executor_manager_thread_wakeup is not None:
# Wake up queue management thread
with self._shutdown_lock:
self._executor_manager_thread_wakeup.wakeup()
if executor_manager_thread is not None and wait:
# This locks avoids concurrent join if the interpreter
# is shutting down.
with _global_shutdown_lock:
executor_manager_thread.join()
_threads_wakeups.pop(executor_manager_thread, None)
# To reduce the risk of opening too many files, remove references to
# objects that use file descriptors.
self._executor_manager_thread = None
self._executor_manager_thread_wakeup = None
self._call_queue = None
self._result_queue = None
self._processes_management_lock = None
shutdown.__doc__ = Executor.shutdown.__doc__
joblib-loky-51247f4/loky/reusable_executor.py 0000664 0000000 0000000 00000024245 15256713413 0021273 0 ustar 00root root 0000000 0000000 ###############################################################################
# Reusable ProcessPoolExecutor
#
# author: Thomas Moreau and Olivier Grisel
#
from __future__ import annotations
import time
import warnings
import threading
import multiprocessing as mp
from typing import Any
from .process_executor import ProcessPoolExecutor, EXTRA_QUEUED_CALLS
from .backend.context import cpu_count
from .backend import get_context
__all__ = ["get_reusable_executor"]
class _ExecutorStorage(threading.local):
"""Cache a thread-local executor"""
executor: _ReusablePoolExecutor | None = None
executor_kwargs: dict[str, Any] | None = None
# Lock for id management:
_executor_lock = threading.RLock()
_next_executor_id = 0
# Thread-local executor storage:
_executor_storage = _ExecutorStorage()
def _get_next_executor_id():
"""Ensure that each successive executor instance has a unique, monotonic id.
The purpose of this monotonic id is to help debug and test automated
instance creation.
"""
global _next_executor_id
with _executor_lock:
executor_id = _next_executor_id
_next_executor_id += 1
return executor_id
def get_reusable_executor(
max_workers=None,
context=None,
timeout=10,
kill_workers=False,
reuse="auto",
job_reducers=None,
result_reducers=None,
initializer=None,
initargs=(),
env=None,
):
"""Return the current ReusableExecutor instance for the current thread.
Start a new instance if it has not been started already or if the previous
instance was left in a broken state.
If the previous instance does not have the requested number of workers, the
executor is dynamically resized to adjust the number of workers prior to
returning.
Reusing an instance (per thread) spares the overhead of starting new
worker processes and importing common python packages each time.
``max_workers`` controls the maximum number of tasks that can be running in
parallel in worker processes. By default this is set to the number of
CPUs on the host.
Setting ``timeout`` (in seconds) makes idle workers automatically shutdown
so as to release system resources. New workers are respawn upon submission
of new tasks so that ``max_workers`` are available to accept the newly
submitted tasks. Setting ``timeout`` to around 100 times the time required
to spawn new processes and import packages in them (on the order of 100ms)
ensures that the overhead of spawning workers is negligible.
Setting ``kill_workers=True`` makes it possible to forcibly interrupt
previously spawned jobs to get a new instance of the reusable executor
with new constructor argument values.
The ``job_reducers`` and ``result_reducers`` are used to customize the
pickling of tasks and results send to the executor.
When provided, the ``initializer`` is run first in newly spawned
processes with argument ``initargs``.
The environment variable in the child process are a copy of the values in
the main process. One can provide a dict ``{ENV: VAL}`` where ``ENV`` and
``VAL`` are string literals to overwrite the environment variable ``ENV``
in the child processes to value ``VAL``. The environment variables are set
in the children before any module is loaded. This only works with the
``loky`` context.
"""
_executor, _ = _ReusablePoolExecutor.get_reusable_executor(
max_workers=max_workers,
context=context,
timeout=timeout,
kill_workers=kill_workers,
reuse=reuse,
job_reducers=job_reducers,
result_reducers=result_reducers,
initializer=initializer,
initargs=initargs,
env=env,
)
return _executor
class _ReusablePoolExecutor(ProcessPoolExecutor):
def __init__(
self,
max_workers=None,
context=None,
timeout=None,
executor_id=0,
job_reducers=None,
result_reducers=None,
initializer=None,
initargs=(),
env=None,
):
super().__init__(
max_workers=max_workers,
context=context,
timeout=timeout,
job_reducers=job_reducers,
result_reducers=result_reducers,
initializer=initializer,
initargs=initargs,
env=env,
)
self.executor_id = executor_id
@classmethod
def get_reusable_executor(
cls,
max_workers=None,
context=None,
timeout=10,
kill_workers=False,
reuse="auto",
job_reducers=None,
result_reducers=None,
initializer=None,
initargs=(),
env=None,
):
executor = _executor_storage.executor
if max_workers is None:
if reuse is True and executor is not None:
max_workers = executor._max_workers
else:
max_workers = cpu_count()
elif max_workers <= 0:
raise ValueError(
f"max_workers must be greater than 0, got {max_workers}."
)
if isinstance(context, str):
context = get_context(context)
if context is not None and context.get_start_method() == "fork":
raise ValueError(
"Cannot use reusable executor with the 'fork' context"
)
kwargs = dict(
context=context,
timeout=timeout,
job_reducers=job_reducers,
result_reducers=result_reducers,
initializer=initializer,
initargs=initargs,
env=env,
)
if executor is None:
is_reused = False
mp.util.debug(f"Create a executor with max_workers={max_workers}.")
executor_id = _get_next_executor_id()
_executor_storage.executor_kwargs = kwargs
_executor_storage.executor = executor = cls(
max_workers=max_workers,
executor_id=executor_id,
**kwargs,
)
else:
if reuse == "auto":
reuse = kwargs == _executor_storage.executor_kwargs
if (
executor._flags.broken
or executor._flags.shutdown
or not reuse
or executor.queue_size < max_workers
):
if executor._flags.broken:
reason = "broken"
elif executor._flags.shutdown:
reason = "shutdown"
elif executor.queue_size < max_workers:
# Do not reuse the executor if the queue size is too
# small as this would lead to limited parallelism.
reason = "queue size is too small"
else:
reason = "arguments have changed"
mp.util.debug(
"Creating a new executor with max_workers="
f"{max_workers} as the previous instance cannot be "
f"reused ({reason})."
)
executor.shutdown(wait=True, kill_workers=kill_workers)
_executor_storage.executor = executor = (
_executor_storage.executor_kwargs
) = None
return cls.get_reusable_executor(
max_workers=max_workers, **kwargs
)
else:
mp.util.debug(
"Reusing existing executor with "
f"max_workers={executor._max_workers}."
)
is_reused = True
executor._resize(max_workers)
return executor, is_reused
def _resize(self, max_workers):
if max_workers is None:
raise ValueError("Trying to resize with max_workers=None")
elif max_workers == self._max_workers:
return
if self._executor_manager_thread is None:
# If the executor_manager_thread has not been started
# then no processes have been spawned and we can just
# update _max_workers and return
self._max_workers = max_workers
return
self._wait_job_completion()
# Some process might have returned due to timeout so check how many
# children are still alive. Use the _process_management_lock to
# ensure that no process are spawned or timeout during the resize.
with self._processes_management_lock:
processes = list(self._processes.values())
nb_children_alive = sum(p.is_alive() for p in processes)
self._max_workers = max_workers
for _ in range(max_workers, nb_children_alive):
self._call_queue.put(None)
while len(self._processes) > max_workers and not self._flags.broken:
time.sleep(1e-3)
self._adjust_process_count()
processes = list(self._processes.values())
while not all(p.is_alive() for p in processes):
time.sleep(1e-3)
def _wait_job_completion(self):
"""Wait for the cache to be empty before resizing the pool."""
# Issue a warning to the user about the bad effect of this usage.
if self._pending_work_items:
warnings.warn(
"Trying to resize an executor with running jobs: "
"waiting for jobs completion before resizing.",
UserWarning,
)
mp.util.debug(
f"Executor {self.executor_id} waiting for jobs completion "
"before resizing"
)
# Wait for the completion of the jobs
while self._pending_work_items:
time.sleep(1e-3)
def _setup_queues(self, job_reducers, result_reducers):
# As this executor can be resized, use a large queue size to avoid
# underestimating capacity and introducing overhead
# Also handle the case where the user set max_workers to a value larger
# than cpu_count(), to avoid limiting the number of parallel jobs.
min_queue_size = max(cpu_count(), self._max_workers)
self.queue_size = 2 * min_queue_size + EXTRA_QUEUED_CALLS
super()._setup_queues(
job_reducers, result_reducers, queue_size=self.queue_size
)
joblib-loky-51247f4/pyproject.toml 0000664 0000000 0000000 00000004415 15256713413 0017134 0 ustar 00root root 0000000 0000000 [build-system]
requires = ["setuptools>=61.2"]
build-backend = "setuptools.build_meta"
[project]
name = "loky"
description= "A robust implementation of concurrent.futures.ProcessPoolExecutor"
authors = [{name = "Thomas Moreau", email = "thomas.moreau.2010@gmail.com"}]
license= {file = "LICENSE.txt"}
dependencies = ["cloudpickle"]
requires-python = ">=3.10"
classifiers=[
"Development Status :: 5 - Production/Stable",
"Environment :: Console",
"Intended Audience :: Developers",
"Intended Audience :: Science/Research",
"License :: OSI Approved :: BSD License",
"Operating System :: OS Independent",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
"Programming Language :: Python :: 3.14",
"Topic :: Scientific/Engineering",
"Topic :: Utilities",
"Topic :: Software Development :: Libraries",
]
dynamic = ["version"]
[project.optional-dependencies]
tests = ["packaging", "pytest", "psutil"]
[project.readme]
file = "README.md"
content-type = "text/markdown"
[project.urls]
Homepage = "http://loky.readthedocs.io"
Source = "https://github.com/joblib/loky"
[tool.setuptools]
packages = [
"loky",
"loky.backend",
]
platforms = ["any"]
include-package-data = false
[tool.setuptools.dynamic]
version = {attr = "loky.__version__"}
[tool.black]
line-length = 79
include = '\.pyi?$'
exclude = '''
/(
\.git
| \.hg
| \.mypy_cache
| \.tox
| \.venv
| _build
| buck-out
| build
| dist
)/
'''
# Because pre-commit uses explicit filenames, you need to use force-exclude in
# order for pre-commit black to leave the stdlib files untouched
force-exclude = '''
loky/backend/stdlib_py.*resource_tracker\.py
'''
[tool.pytest.ini_options]
# No test should last longer than 1 minute, even on the CI. Otherwise, dump
# per-thread tracebacks. Note that this value can be overriden from the
# commandline using: pytest -o faulthandler_timeout= ...
faulthandler_timeout=60
# Interrupt in pytest in case of deadlock. This will only work once the
# following PR has been merged and released:
# https://github.com/pytest-dev/pytest/pull/13679
faulthandler_exit_on_timeout=true
joblib-loky-51247f4/tests/ 0000775 0000000 0000000 00000000000 15256713413 0015356 5 ustar 00root root 0000000 0000000 joblib-loky-51247f4/tests/__init__.py 0000664 0000000 0000000 00000000000 15256713413 0017455 0 ustar 00root root 0000000 0000000 joblib-loky-51247f4/tests/_executor_mixin.py 0000664 0000000 0000000 00000025031 15256713413 0021132 0 ustar 00root root 0000000 0000000 import sys
import time
import math
import pytest
import threading
from time import sleep
import random
from loky import TimeoutError, get_reusable_executor
from loky.backend import get_context
try:
import psutil
psutil_exceptions = (psutil.NoSuchProcess, psutil.AccessDenied)
except ImportError:
psutil = None
psutil_exceptions = ()
# Set a large timeout as it should only be reached in case of deadlocks
TIMEOUT = 40
_test_event = None
def initializer_event(event):
"""Initializer that set a global test event for test synchronization"""
global _test_event
_test_event = event
# Inject some randomness in the initialization to reveal race conditions.
if random.random() < 0.2:
sleep(random.random() * 0.1) # 0-100ms
def _direct_children_with_cmdline(p):
"""Helper to fetch cmdline from children process list"""
children_with_cmdline = []
for c in p.children():
try:
cmdline = " ".join(c.cmdline())
if not c.is_running() or not cmdline:
# Under linux is_running() can return True even though
# the command line data can no longer be read from
# /proc//cmdline. This looks like a race condition
# between /proc//stat and /proc//cmdline
# when the process is being terminated by the OS.
continue
children_with_cmdline.append((c, cmdline))
# SystemError is needed for psutil <= 7.2.2, which on macOS returns a
# result with a pending exception when the process exits mid-call, see
# https://github.com/giampaolo/psutil/pull/2854.
except (OSError, SystemError) + psutil_exceptions:
# These errors indicate that the process has terminated while
# we were processing the info. Just discard it.
pass
return children_with_cmdline
def _running_children_pids_with_cmdline(p):
all_children = _direct_children_with_cmdline(p)
workers = [
(c, cmdline)
for c, cmdline in all_children
if (
"semaphore_tracker" not in cmdline
and "resource_tracker" not in cmdline
and "multiprocessing.forkserver" not in cmdline
)
]
forkservers = [
c
for c, cmdline in all_children
if "multiprocessing.forkserver" in cmdline
]
for fs in forkservers:
workers += _direct_children_with_cmdline(fs)
return [(w.pid, cmdline) for w, cmdline in workers]
def _check_subprocesses_number(
executor=None,
expected_process_number=None,
expected_max_process_number=None,
patience=10,
):
if not psutil:
# psutil is not installed, we cannot check the number of subprocesses
return
for trial_idx in range(patience):
try:
# Wait for terminating processes to disappear
pids_cmdlines = _running_children_pids_with_cmdline(
psutil.Process()
)
children_pids = {pid for pid, _ in pids_cmdlines}
if executor is not None:
worker_pids = set(executor._processes.keys())
# Consistency check: all workers should be in the children list
assert worker_pids.issubset(children_pids)
else:
# Bypass pids checks when executor has been garbage collected
worker_pids = children_pids
if expected_process_number is not None:
assert (
len(children_pids) == expected_process_number
), pids_cmdlines
assert (
len(worker_pids) == expected_process_number
), pids_cmdlines
if expected_max_process_number is not None:
assert (
len(children_pids) <= expected_max_process_number
), pids_cmdlines
assert (
len(worker_pids) <= expected_max_process_number
), pids_cmdlines
return
except AssertionError as e:
# Sometimes executor._processes or psutil seems to report
# out-of-sync information. Let's wait a bit an try again later to see if:
# - worker_pids is consistent with the psutil children list
# - the number of children is consistent with the expected (max) number
if trial_idx == patience - 1:
raise e
sleep(0.1)
def _check_executor_started(executor):
# Submit a small job to make sure that the pool is an working state
res = executor.submit(id, None)
try:
res.result(timeout=TIMEOUT)
except TimeoutError:
print(
"\n" * 3,
res.done(),
executor._call_queue.empty(),
executor._result_queue.empty(),
)
print(executor._processes)
print(threading.enumerate())
from faulthandler import dump_traceback
dump_traceback()
executor.submit(dump_traceback).result(TIMEOUT)
raise RuntimeError("Executor took too long to run basic task.")
class ExecutorMixin:
worker_count = 5
@classmethod
def setup_class(cls):
print(f"setup class with {cls.context}")
global _test_event
if _test_event is None:
_test_event = cls.context.Event()
@classmethod
def teardown_class(cls):
print(f"teardown class with {cls.context}")
global _test_event
if _test_event is not None:
_test_event = None
@pytest.fixture(autouse=True)
def setup_method(self):
global _test_event
assert _test_event is not None
try:
self.executor = self.executor_type(
max_workers=self.worker_count,
context=self.context,
initializer=initializer_event,
initargs=(_test_event,),
)
except NotImplementedError as e:
self.skipTest(str(e))
_check_executor_started(self.executor)
_check_subprocesses_number(self.executor, self.worker_count)
def teardown_method(self, method):
# Make sure executor is not broken if it should not be
executor = getattr(self, "executor", None)
if executor is not None:
try:
# old pytest markers:
expect_broken_pool = hasattr(method, "broken_pool")
# new pytest markers:
for mark in getattr(method, "pytestmark", []):
if mark.name == "broken_pool":
expect_broken_pool = True
if expect_broken_pool:
for _ in range(10):
# The executor manager thread can take some time to
# mark the executor broken.
is_actually_broken = executor._flags.broken is not None
if is_actually_broken:
break
sleep(0.1)
else:
raise AssertionError(
"The executor was not flagged broken at the end of "
f" {method.__qualname__} as expected."
)
else:
# Check that the executor is not broken right away to avoid
# wasting CI time. False negative should be very rare.
is_actually_broken = executor._flags.broken is not None
assert not is_actually_broken
finally:
# Always shutdown the executor, with SIGKILL if the executor
# is actually broken.
kill_workers = executor._flags.broken is not None
t_start = time.time()
executor.shutdown(wait=True, kill_workers=kill_workers)
dt = time.time() - t_start
assert dt < 10, "Executor took too long to shutdown"
_check_subprocesses_number(executor, 0)
def _prime_executor(self):
# Make sure that the executor is ready to do work before running the
# tests. This should reduce the probability of timeouts in the tests.
futures = [
self.executor.submit(time.sleep, 0.1)
for _ in range(self.worker_count)
]
for f in futures:
f.result()
@classmethod
def check_no_running_workers(cls, patience=5, sleep_duration=0.01):
if psutil is None:
return
deadline = time.time() + patience
while time.time() <= deadline:
time.sleep(sleep_duration)
p = psutil.Process()
workers = _running_children_pids_with_cmdline(p)
if not workers:
return
# Patience exhausted: log the remaining workers command line and
# raise error.
print("Remaining worker processes command lines:", file=sys.stderr)
for w, cmdline in workers:
print(w.pid, w.status(), end="\n", file=sys.stderr)
print(cmdline, end="\n\n", file=sys.stderr)
raise AssertionError(
f"Expected no more running worker processes but got {len(workers)}"
f" after waiting {patience:0.3f}s."
)
class ReusableExecutorMixin:
def setup_method(self, method):
default_start_method = get_context().get_start_method()
assert default_start_method == "loky", default_start_method
executor = get_reusable_executor(max_workers=2)
_check_executor_started(executor)
# There can be less than 2 workers because of the worker timeout
_check_subprocesses_number(executor, expected_max_process_number=2)
# Check that there no other running subprocesses beyond the workers
# of the reusable executor.
_check_subprocesses_number(
executor=None, expected_max_process_number=2
)
def teardown_method(self, method):
"""Make sure the executor can be recovered after the tests"""
executor = get_reusable_executor(max_workers=2)
assert executor.submit(math.sqrt, 1).result() == 1
# There can be less than 2 workers because of the worker timeout
_check_subprocesses_number(executor, expected_max_process_number=2)
# Check that there no other running subprocesses beyond the workers
# of the reusable executor.
_check_subprocesses_number(
executor=None, expected_max_process_number=2
)
@classmethod
def teardown_class(cls):
executor = get_reusable_executor(max_workers=2)
executor.shutdown(wait=True)
joblib-loky-51247f4/tests/_openmp_test_helper/ 0000775 0000000 0000000 00000000000 15256713413 0021411 5 ustar 00root root 0000000 0000000 joblib-loky-51247f4/tests/_openmp_test_helper/.gitignore 0000664 0000000 0000000 00000000020 15256713413 0023371 0 ustar 00root root 0000000 0000000 *.c
*.so
/build
joblib-loky-51247f4/tests/_openmp_test_helper/__init__.py 0000664 0000000 0000000 00000000000 15256713413 0023510 0 ustar 00root root 0000000 0000000 joblib-loky-51247f4/tests/_openmp_test_helper/parallel_sum.pyx 0000664 0000000 0000000 00000000661 15256713413 0024636 0 ustar 00root root 0000000 0000000 # cython: freethreading_compatible = True
import cython
cimport openmp
from cython.parallel import prange
from libc.stdlib cimport malloc, free
def parallel_sum(int N):
cdef double Ysum = 0
cdef int i, num_threads
cdef double* X = malloc(N*cython.sizeof(double))
for i in prange(N, nogil=True):
num_threads = openmp.omp_get_num_threads()
Ysum += X[i]
free(X)
return num_threads
joblib-loky-51247f4/tests/_openmp_test_helper/setup.py 0000664 0000000 0000000 00000001362 15256713413 0023125 0 ustar 00root root 0000000 0000000 """Helper module to test OpenMP support on Continuous Integration"""
import os
import sys
from Cython.Build import cythonize
from setuptools import setup
from setuptools.extension import Extension
if sys.platform == "darwin":
extra_compile_args = ["-Xpreprocessor", "-fopenmp"]
extra_link_args = ["-lomp"]
elif sys.platform == "win32":
extra_compile_args = ["/openmp"]
extra_link_args = None
else:
extra_compile_args = ["-fopenmp"]
extra_link_args = ["-fopenmp"]
ext_modules = [
Extension(
"parallel_sum",
["parallel_sum.pyx"],
extra_compile_args=extra_compile_args,
extra_link_args=extra_link_args,
)
]
setup(
name="_openmp_test_helper",
ext_modules=cythonize(ext_modules),
)
joblib-loky-51247f4/tests/_test_process_executor.py 0000664 0000000 0000000 00000137007 15256713413 0022532 0 ustar 00root root 0000000 0000000 import sysconfig
from loky import process_executor
import os
import gc
import sys
import time
import shutil
import platform
import pytest
import weakref
import tempfile
import threading
import faulthandler
import warnings
from math import sqrt
from pickle import PicklingError
from threading import Thread
from collections import defaultdict
from concurrent import futures
from concurrent.futures._base import (
PENDING,
RUNNING,
CANCELLED,
CANCELLED_AND_NOTIFIED,
FINISHED,
)
import loky
from loky.process_executor import (
LokyRecursionError,
BrokenProcessPool,
ShutdownExecutorError,
TerminatedWorkerError,
)
from loky._base import Future
from . import _executor_mixin
from .utils import id_sleep, check_subprocess_call, filter_match
from .test_reusable_executor import ErrorAtPickle, ExitAtPickle, c_exit
def create_future(state=PENDING, exception=None, result=None):
f = Future()
f._state = state
f._exception = exception
f._result = result
return f
PENDING_FUTURE = create_future(state=PENDING)
RUNNING_FUTURE = create_future(state=RUNNING)
CANCELLED_FUTURE = create_future(state=CANCELLED)
CANCELLED_AND_NOTIFIED_FUTURE = create_future(state=CANCELLED_AND_NOTIFIED)
EXCEPTION_FUTURE = create_future(state=FINISHED, exception=OSError())
SUCCESSFUL_FUTURE = create_future(state=FINISHED, result=42)
def mul(x, y):
return x * y
def sleep_and_print(t, msg):
time.sleep(t)
print(msg)
sys.stdout.flush()
def sleep_and_return(delay, x):
time.sleep(delay)
return x
def sleep_and_write(t, filename, msg):
time.sleep(t)
with open(filename, "w") as f:
f.write(str(msg))
class FalseyError(Exception):
"""An exception that is falsey, to check it is not mistaken for a result."""
def __bool__(self):
return False
def raise_falsey_error():
raise FalseyError("falsey errors must still propagate")
class MyObject:
def __init__(self, value=0):
self.value = value
def __repr__(self):
return f"MyObject({self.value})"
def my_method(self):
pass
def _assert_no_error(stderr):
if sys.platform == "darwin":
# On macOS, ignore UserWarning related to their broken semaphore
# implementation.
stderr = "\n".join(
line
for line in stderr.splitlines()
if "increase its maximal value" not in line
)
assert len(stderr) == 0, stderr
class ExecutorShutdownTest:
def test_run_after_shutdown(self):
self.executor.shutdown()
with pytest.raises(RuntimeError):
self.executor.submit(pow, 2, 5)
def test_shutdown_with_pickle_error(self):
self.executor.shutdown()
# Iterate a few times to catch deadlocks/race conditions in the
# executor shutdown.
for _ in range(5):
with self.executor_type(max_workers=4) as e:
with pytest.raises(PicklingError):
e.submit(id, ErrorAtPickle()).result()
def test_shutdown_with_sys_exit_at_pickle(self):
self.executor.shutdown()
# Iterate a few times to catch deadlocks/race conditions in the
# executor shutdown.
for _ in range(5):
with self.executor_type(max_workers=4) as e:
with pytest.raises(PicklingError):
e.submit(id, ExitAtPickle()).result()
def test_interpreter_shutdown(self):
# Free resources to avoid random timeout in CI
self.executor.shutdown(wait=True, kill_workers=True)
executor_type = self.executor_type.__name__
start_method = self.context.get_start_method()
tempdir = tempfile.mkdtemp(prefix="loky_").replace("\\", "/")
try:
n_jobs = 4
code = f"""if True:
from loky.process_executor import {executor_type}
from loky.backend import get_context
from tests._test_process_executor import sleep_and_write
context = get_context("{start_method}")
e = {executor_type}({n_jobs}, context=context)
e.submit(id, 42).result()
task_ids = list(range(2 * {n_jobs}))
filenames = [f'{tempdir}/task_{{i:02}}.log'
for i in task_ids]
e.map(sleep_and_write, [0.1] * 2 * {n_jobs},
filenames, task_ids)
# Do not wait for the results: garbage collect executor and
# shutdown main Python interpreter while letting the worker
# processes finish in the background.
"""
_, stderr = check_subprocess_call(
[sys.executable, "-c", code], timeout=55
)
_assert_no_error(stderr)
# The workers should have completed their work before the main
# process exits:
expected_filenames = [
f"task_{i:02d}.log" for i in range(2 * n_jobs)
]
# Apparently files can take some time to appear under windows
# on AppVeyor
for _ in range(20):
filenames = sorted(os.listdir(tempdir))
if len(filenames) != len(expected_filenames):
time.sleep(1)
else:
break
assert filenames == expected_filenames
for i, filename in enumerate(filenames):
with open(os.path.join(tempdir, filename), "rb") as f:
assert int(f.read().strip()) == i
finally:
shutil.rmtree(tempdir)
def test_hang_issue12364(self):
fs = [self.executor.submit(time.sleep, 0.01) for _ in range(50)]
self.executor.shutdown()
for f in fs:
f.result()
def test_processes_terminate(self):
self.executor.submit(mul, 21, 2)
self.executor.submit(mul, 6, 7)
self.executor.submit(mul, 3, 14)
assert len(self.executor._processes) == self.worker_count
processes = self.executor._processes
self.executor.shutdown()
for p in processes.values():
p.join()
def test_processes_terminate_on_executor_gc(self):
results = self.executor.map(sleep_and_return, [0.1] * 10, range(10))
assert len(self.executor._processes) == self.worker_count
processes = self.executor._processes
executor_flags = self.executor._flags
# The following should trigger GC and therefore shutdown of workers.
# However the shutdown wait for all the pending jobs to complete
# first.
executor_reference = weakref.ref(self.executor)
self.executor = None
# Make sure that there is not other reference to the executor object.
# We have to be patient as _thread_management_worker might have a
# reference when we deleted self.executor.
t_deadline = time.time() + 1
while executor_reference() is not None and time.time() < t_deadline:
time.sleep(0.001)
assert executor_reference() is None
# The remaining jobs should still be processed in the background
for result, expected in zip(results, range(10)):
assert result == expected
# Once all pending jobs have completed the executor and threads should
# terminate automatically. Note that the effective time for a Python
# process to completely shutdown can vary a lot especially on loaded CI
# machines with and the atexit callbacks that writes test coverage data
# to disk. Let's be patient.
self.check_no_running_workers(patience=5)
assert executor_flags.shutdown, processes
assert not executor_flags.broken, processes
@classmethod
def _wait_and_crash(cls):
_executor_mixin._test_event.wait()
faulthandler._sigsegv()
def test_processes_crash_handling_after_executor_gc(self):
# Start 5 easy jobs on 5 workers
results = self.executor.map(sleep_and_return, [0.01] * 5, range(5))
# Enqueue a job that will trigger a crash of one of the workers.
# Make sure this crash does not happen before the non-failing jobs
# have returned their results by using and multiprocessing Event
# instance
crash_result = self.executor.submit(self._wait_and_crash)
assert len(self.executor._processes) == self.worker_count
processes = self.executor._processes
executor_flags = self.executor._flags
# The following should trigger the GC and therefore shutdown of
# workers. However the shutdown wait for all the pending jobs to
# complete first.
executor_reference = weakref.ref(self.executor)
self.executor = None
# Make sure that there is not other reference to the executor object.
assert executor_reference() is None
# The remaining jobs should still be processed in the background
for result, expected in zip(results, range(5)):
assert result == expected
# Let the crash job know that it can crash now
_executor_mixin._test_event.set()
# The crashing job should be executed after the non-failing jobs
# have completed. The crash should be detected.
match = filter_match("SIGSEGV")
with pytest.raises(TerminatedWorkerError, match=match):
crash_result.result()
_executor_mixin._test_event.clear()
# The executor flag should have been set at this point.
assert executor_flags.broken, processes
# Since the executor is broken, all workers should be SIGKILLed on
# POSIX or terminated one Windows. Usually this should be fast but
# let's be patient just in case the CI is overloaded.
self.check_no_running_workers(patience=5)
def test_context_manager_shutdown(self):
with self.executor_type(max_workers=5, context=self.context) as e:
processes = e._processes
assert list(e.map(abs, range(-5, 5))) == [
5,
4,
3,
2,
1,
0,
1,
2,
3,
4,
]
for p in processes.values():
p.join()
def test_del_shutdown(self):
executor = self.executor_type(max_workers=5, context=self.context)
list(executor.map(abs, range(-5, 5)))
executor_manager_thread = executor._executor_manager_thread
processes = executor._processes
del executor
executor_manager_thread.join()
for p in processes.values():
p.join()
@classmethod
def _wait_and_return(cls, x):
# This _test_event is passed globally through an initializer to
# the executor.
_executor_mixin._test_event.wait()
return x
def test_shutdown_no_wait(self):
# Ensure that the executor cleans up the processes when calling
# shutdown with wait=False
# Stores executor internals to be able to check that the executor
# shutdown correctly
processes = self.executor._processes
call_queue = self.executor._call_queue
executor_manager_thread = self.executor._executor_manager_thread
# submit tasks that will finish after the shutdown and make sure they
# were started
res = [
self.executor.submit(self._wait_and_return, x)
for x in range(-5, 5)
]
self.executor.shutdown(wait=False)
with pytest.raises(ShutdownExecutorError):
# It's no longer possible to submit any new tasks to this
# executor after shutdown.
self.executor.submit(lambda x: x, 42)
# Check that even after shutdown, all futures are still running
assert all(f._state in (PENDING, RUNNING) for f in res)
# Let the futures finish and make sure that all the executor resources
# were properly cleaned by the shutdown process
_executor_mixin._test_event.set()
executor_manager_thread.join()
for p in processes.values():
p.join()
call_queue.join_thread()
# Make sure the results were all computed before the executor
# resources were freed.
assert [f.result() for f in res] == list(range(-5, 5))
def test_shutdown_deadlock_pickle(self):
# Test that the pool calling shutdown with wait=False does not cause
# a deadlock if a task fails at pickle after the shutdown call.
# Reported in bpo-39104.
self.executor.shutdown(wait=True)
with self.executor_type(
max_workers=2, context=self.context
) as executor:
self.executor = executor # Allow clean up in fail_on_deadlock
# Start the executor and get the executor_manager_thread to collect
# the threads and avoid dangling thread that should be cleaned up
# asynchronously.
executor.submit(id, 42).result()
executor_manager = executor._executor_manager_thread
# Submit a task that fails at pickle and shutdown the executor
# without waiting
f = executor.submit(id, ErrorAtPickle())
executor.shutdown(wait=False)
with pytest.raises(PicklingError):
f.result()
# Make sure the executor is eventually shutdown and do not leave
# dangling threads
executor_manager.join()
def test_hang_gh94440(self):
"""shutdown(wait=True) doesn't hang when a submitted future is
cancelled right before shutdown.
See https://github.com/python/cpython/issues/94440.
"""
executor_type = self.executor_type.__name__
start_method = self.context.get_start_method()
code = f"""if True:
from loky.process_executor import {executor_type}
from loky.backend import get_context
context = get_context("{start_method}")
e = {executor_type}(1, context=context)
e.submit(int).result()
e.submit(int).cancel()
e.shutdown(wait=True)
"""
# stderr is not checked here: only the hang matters (see #642 noise)
check_subprocess_call([sys.executable, "-c", code], timeout=30)
def test_hang_issue39205(self):
"""shutdown(wait=False) doesn't hang at exit with running futures.
See https://bugs.python.org/issue39205.
"""
executor_type = self.executor_type.__name__
start_method = self.context.get_start_method()
code = f"""if True:
from loky.process_executor import {executor_type}
from loky.backend import get_context
from tests._test_process_executor import sleep_and_print
context = get_context("{start_method}")
e = {executor_type}(3, context=context)
e.submit(sleep_and_print, 1.0, "apple")
e.shutdown(wait=False)
"""
stdout, stderr = check_subprocess_call(
[sys.executable, "-c", code], timeout=55
)
_assert_no_error(stderr)
assert stdout.strip() == "apple"
@classmethod
def _test_shutdown_and_kill_workers(cls, depth):
executor = cls.executor_type(
max_workers=2,
context=cls.context,
initializer=_executor_mixin.initializer_event,
initargs=(_executor_mixin._test_event,),
)
assert executor.submit(sleep_and_return, 0, 42).result() == 42
if depth >= 2:
_executor_mixin._test_event.set()
executor.submit(sleep_and_return, 30, 42).result()
executor.shutdown()
else:
f = executor.submit(cls._test_shutdown_and_kill_workers, depth + 1)
f.result()
def test_shutdown_and_kill_workers(self):
f = self.executor.submit(self._test_shutdown_and_kill_workers, 1)
# Wait for the nested executors to be started
_executor_mixin._test_event.wait()
# Forcefully shutdown the executor and kill the workers
t_start = time.time()
self.executor.shutdown(wait=True, kill_workers=True)
msg = "Failed to quickly kill nested executor"
t_shutdown = time.time() - t_start
assert t_shutdown < 5, msg
with pytest.raises(ShutdownExecutorError):
f.result()
_executor_mixin._check_subprocesses_number(self.executor, 0)
class WaitTests:
def test_first_completed(self):
future1 = self.executor.submit(mul, 21, 2)
future2 = self.executor.submit(time.sleep, 1.5)
done, not_done = futures.wait(
[CANCELLED_FUTURE, future1, future2],
return_when=futures.FIRST_COMPLETED,
)
assert {future1} == done
assert {CANCELLED_FUTURE, future2} == not_done
def test_first_completed_some_already_completed(self):
future1 = self.executor.submit(time.sleep, 1.5)
finished, pending = futures.wait(
[CANCELLED_AND_NOTIFIED_FUTURE, SUCCESSFUL_FUTURE, future1],
return_when=futures.FIRST_COMPLETED,
)
assert {CANCELLED_AND_NOTIFIED_FUTURE, SUCCESSFUL_FUTURE} == finished
assert {future1} == pending
@classmethod
def wait_and_raise(cls, t):
_executor_mixin._test_event.wait(t)
raise Exception("this is an exception")
@classmethod
def wait_and_return(cls, t):
_executor_mixin._test_event.wait()
return True
def test_first_exception(self):
future1 = self.executor.submit(mul, 2, 21)
future2 = self.executor.submit(self.wait_and_raise, 1.5)
future3 = self.executor.submit(time.sleep, 3)
def cb_done(f):
_executor_mixin._test_event.set()
future1.add_done_callback(cb_done)
finished, pending = futures.wait(
[future1, future2, future3], return_when=futures.FIRST_EXCEPTION
)
assert _executor_mixin._test_event.is_set()
assert {future1, future2} == finished
assert {future3} == pending
_executor_mixin._test_event.clear()
def test_first_exception_some_already_complete(self):
future1 = self.executor.submit(divmod, 21, 0)
future2 = self.executor.submit(time.sleep, 1.5)
finished, pending = futures.wait(
[
SUCCESSFUL_FUTURE,
CANCELLED_FUTURE,
CANCELLED_AND_NOTIFIED_FUTURE,
future1,
future2,
],
return_when=futures.FIRST_EXCEPTION,
)
assert {
SUCCESSFUL_FUTURE,
CANCELLED_AND_NOTIFIED_FUTURE,
future1,
} == finished
assert {CANCELLED_FUTURE, future2} == pending
def test_first_exception_one_already_failed(self):
future1 = self.executor.submit(time.sleep, 2)
finished, pending = futures.wait(
[EXCEPTION_FUTURE, future1], return_when=futures.FIRST_EXCEPTION
)
assert {EXCEPTION_FUTURE} == finished
assert {future1} == pending
def test_all_completed(self):
future1 = self.executor.submit(divmod, 2, 0)
future2 = self.executor.submit(mul, 2, 21)
finished, pending = futures.wait(
[
SUCCESSFUL_FUTURE,
EXCEPTION_FUTURE,
CANCELLED_AND_NOTIFIED_FUTURE,
future1,
future2,
],
return_when=futures.ALL_COMPLETED,
)
assert {
SUCCESSFUL_FUTURE,
CANCELLED_AND_NOTIFIED_FUTURE,
EXCEPTION_FUTURE,
future1,
future2,
} == finished
assert not pending
def test_timeout(self):
# Make sure the executor has already started to avoid timeout happening
# before future1 returns
assert self.executor.submit(id_sleep, 42).result() == 42
future1 = self.executor.submit(mul, 6, 7)
future2 = self.executor.submit(self.wait_and_return, 5)
assert future1.result() == 42
finished, pending = futures.wait(
[
CANCELLED_AND_NOTIFIED_FUTURE,
EXCEPTION_FUTURE,
SUCCESSFUL_FUTURE,
future1,
future2,
],
timeout=0.1,
return_when=futures.ALL_COMPLETED,
)
assert {
CANCELLED_AND_NOTIFIED_FUTURE,
EXCEPTION_FUTURE,
SUCCESSFUL_FUTURE,
future1,
} == finished
assert {future2} == pending
_executor_mixin._test_event.set()
assert future2.result(timeout=10)
_executor_mixin._test_event.clear()
class AsCompletedTests:
# TODO(brian@sweetapp.com): Should have a test with a non-zero timeout.
def test_no_timeout(self):
future1 = self.executor.submit(mul, 2, 21)
future2 = self.executor.submit(mul, 7, 6)
completed = set(
futures.as_completed(
[
CANCELLED_AND_NOTIFIED_FUTURE,
EXCEPTION_FUTURE,
SUCCESSFUL_FUTURE,
future1,
future2,
]
)
)
assert {
CANCELLED_AND_NOTIFIED_FUTURE,
EXCEPTION_FUTURE,
SUCCESSFUL_FUTURE,
future1,
future2,
} == completed
def test_zero_timeout(self):
future1 = self.executor.submit(time.sleep, 2)
completed_futures = set()
with pytest.raises(futures.TimeoutError):
for future in futures.as_completed(
[
CANCELLED_AND_NOTIFIED_FUTURE,
EXCEPTION_FUTURE,
SUCCESSFUL_FUTURE,
future1,
],
timeout=0,
):
completed_futures.add(future)
assert {
CANCELLED_AND_NOTIFIED_FUTURE,
EXCEPTION_FUTURE,
SUCCESSFUL_FUTURE,
} == completed_futures
def test_duplicate_futures(self):
# Issue 20367. Duplicate futures should not raise exceptions or give
# duplicate responses.
future1 = self.executor.submit(time.sleep, 0.1)
completed = list(futures.as_completed([future1, future1]))
assert len(completed) == 1
class ExecutorTest:
# Executor.shutdown() and context manager usage is tested by
# ExecutorShutdownTest.
def test_submit(self):
future = self.executor.submit(pow, 2, 8)
assert 256 == future.result()
def test_submit_keyword(self):
future = self.executor.submit(mul, 2, y=8)
assert 16 == future.result()
def test_map(self):
assert list(self.executor.map(pow, range(10), range(10))) == list(
map(pow, range(10), range(10))
)
def test_map_exception(self):
i = self.executor.map(divmod, [1, 1, 1, 1], [2, 3, 0, 5])
assert next(i), 0 == 1
assert next(i), 0 == 1
with pytest.raises(ZeroDivisionError):
next(i)
def test_map_timeout(self):
results = []
with pytest.raises(futures.TimeoutError):
for i in self.executor.map(time.sleep, [0, 0, 5], timeout=1):
results.append(i)
assert [None, None] == results
def test_shutdown_race_issue12456(self):
# Issue #12456: race condition at shutdown where trying to post a
# sentinel in the call queue blocks (the queue is full while processes
# have exited).
self.executor.map(str, [2] * (self.worker_count + 1))
self.executor.shutdown()
@pytest.mark.skipif(
platform.python_implementation() != "CPython",
reason="cpython specific non-regression test (fixed starting 3.8.2)",
)
def test_no_stale_references(self):
# Issue #16284: check that the executors don't unnecessarily hang onto
# references.
# This test has to be skipped on early Python 3.8 versions because of a
# low-level reference cycle inside the pickle module for early versions
# of Python 3.8 preventing stale references from being collected. See
# cloudpipe/cloudpickle#327 as well as
# https://bugs.python.org/issue39492
my_object = MyObject()
collect = threading.Event()
_ref = weakref.ref(my_object, lambda obj: collect.set()) # noqa
# Deliberately discarding the future.
self.executor.submit(my_object.my_method)
del my_object
collected = False
for _ in range(5):
if hasattr(sys, "_is_gil_enabled") and not sys._is_gil_enabled():
# XXX: full GC collect is needed on free-threading CPython to
# actually trigger the garbage collection of my_object once the
# executor is done with it. Not sure if this can be considered
# a bug or not.
gc.collect()
collected = collect.wait(timeout=1.0)
if collected:
return
assert collected, "Stale reference not collected within timeout."
def test_max_workers_negative(self):
for number in (0, -1):
with pytest.raises(ValueError) as infos:
self.executor_type(max_workers=number)
assert infos.value.args[0] == "max_workers must be greater than 0"
@pytest.mark.broken_pool
def test_killed_child(self):
# When a child process is abruptly terminated, the whole pool gets
# "broken".
future = self.executor.submit(time.sleep, 30)
# Get one of the processes, and terminate (kill) it
p = next(iter(self.executor._processes.values()))
p.terminate()
match = filter_match("SIGTERM")
with pytest.raises(TerminatedWorkerError, match=match):
future.result()
# Submitting other jobs fails as well.
with pytest.raises(TerminatedWorkerError, match=match):
self.executor.submit(pow, 2, 8)
@pytest.mark.broken_pool
def test_killed_child_with_cancelled_work_items(self):
# A cancelled work item must not stop the executor manager thread from
# failing the remaining ones, see
# https://github.com/python/cpython/issues/107219.
futures = [
self.executor.submit(time.sleep, 30)
for _ in range(2 * self.worker_count + 50)
]
# Wait for the manager thread to fill the bounded call queue and settle
# back into wait(): only then do the leftovers stay in
# pending_work_items long enough to be cancelled behind its back.
deadline = time.time() + 30
while (
len(self.executor._running_work_items) < self.worker_count
and time.time() < deadline
):
time.sleep(0.01)
assert sum(f.cancel() for f in futures), "expected pending work items"
p = next(iter(self.executor._processes.values()))
p.terminate()
match = filter_match("SIGTERM")
with pytest.raises(TerminatedWorkerError, match=match):
futures[0].result(timeout=60)
# Without the fix the manager thread dies on the first cancelled work
# item, so the work items behind it are never failed and callers
# waiting on them block forever.
self.executor._executor_manager_thread.join(60)
assert not self.executor._pending_work_items
def test_falsey_exception(self):
# An exception is an exception even if it is falsey, see
# https://github.com/python/cpython/issues/132063.
future = self.executor.submit(raise_falsey_error)
# Checked with exception() rather than result(): Future.__get_result()
# has the same truthiness bug and was only fixed in CPython 3.13, so on
# older versions result() returns None whatever loky does.
assert isinstance(future.exception(), FalseyError)
def test_map_chunksize(self):
def bad_map():
list(self.executor.map(pow, range(40), range(40), chunksize=-1))
ref = list(map(pow, range(40), range(40)))
assert (
list(self.executor.map(pow, range(40), range(40), chunksize=6))
== ref
)
assert (
list(self.executor.map(pow, range(40), range(40), chunksize=50))
== ref
)
assert (
list(self.executor.map(pow, range(40), range(40), chunksize=40))
== ref
)
with pytest.raises(ValueError):
bad_map()
@classmethod
def _test_traceback(cls):
raise RuntimeError(123) # some comment
def test_traceback(self):
# We want ensure that the traceback from the child process is
# contained in the traceback raised in the main process.
future = self.executor.submit(self._test_traceback)
with pytest.raises(Exception) as cm:
future.result()
exc = cm.value
assert type(exc) is RuntimeError
assert exc.args == (123,)
cause = exc.__cause__
assert type(cause) is process_executor._RemoteTraceback
assert "raise RuntimeError(123) # some comment" in cause.tb
#
# The following tests are new additions to the test suite originally
# backported from the Python 3 concurrent.futures package.
#
def _test_thread_safety(self, thread_idx, results, timeout=30.0):
try:
# submit a mix of very simple tasks with map and submit,
# cancel some of them and check the results
map_future_1 = self.executor.map(sqrt, range(40), timeout=timeout)
if thread_idx % 2 == 0:
# Make it more likely for scheduling threads to overtake one
# another
time.sleep(0.001)
submit_futures = [
self.executor.submit(time.sleep, 0.0001) for _ in range(20)
]
for i, f in enumerate(submit_futures):
if i % 2 == 0:
f.cancel()
map_future_2 = self.executor.map(sqrt, range(40), timeout=timeout)
assert list(map_future_1) == [sqrt(x) for x in range(40)]
assert list(map_future_2) == [sqrt(i) for i in range(40)]
for i, f in enumerate(submit_futures):
if i % 2 == 1 or not f.cancelled():
assert f.result(timeout=timeout) is None
results[thread_idx] = "ok"
except Exception as e:
# Ensure that py.test can report the content of the exception
# by raising it in the main test thread
results[thread_idx] = e
def test_thread_safety(self):
# Check that our process-pool executor can be shared to schedule work
# by concurrent threads
results = [None] * 10
threads = [
Thread(target=self._test_thread_safety, args=(i, results))
for i in range(len(results))
]
for t in threads:
t.start()
for t in threads:
t.join()
for result in results:
if isinstance(result, Exception):
raise result
assert result == "ok"
@classmethod
def return_inputs(cls, *args):
return args
def test_submit_from_callback(self):
collected = defaultdict(list)
executor = self.executor
def _collect_and_submit_next(future):
name, count = future.result()
collected[name].append(count)
if count > 0:
future = executor.submit(self.return_inputs, name, count - 1)
future.add_done_callback(_collect_and_submit_next)
# Start 3 concurrent callbacks chains
fa = executor.submit(self.return_inputs, "chain a", 100)
fa.add_done_callback(_collect_and_submit_next)
fb = executor.submit(self.return_inputs, "chain b", 50)
fb.add_done_callback(_collect_and_submit_next)
fc = executor.submit(self.return_inputs, "chain c", 60)
fc.add_done_callback(_collect_and_submit_next)
assert fa.result() == ("chain a", 100)
assert fb.result() == ("chain b", 50)
assert fc.result() == ("chain c", 60)
# Wait a maximum of 5s for the asynchronous callback chains to complete
patience = 500
while True:
if (
collected["chain a"] == list(range(100, -1, -1))
and collected["chain b"] == list(range(50, -1, -1))
and collected["chain c"] == list(range(60, -1, -1))
):
# the recursive callback chains have completed successfully
break
elif patience < 0:
raise AssertionError(
f"callback submit chains stalled at: {collected!r}"
)
else:
patience -= 1
time.sleep(0.01)
@pytest.mark.timeout(60)
def test_worker_timeout(self):
self.executor.shutdown(wait=True)
self.check_no_running_workers(patience=5)
timeout = getattr(self, "min_worker_timeout", 0.01)
try:
self.executor = self.executor_type(
max_workers=4, context=self.context, timeout=timeout
)
except NotImplementedError as e:
self.skipTest(str(e))
for _ in range(5):
with warnings.catch_warnings():
# It's ok to get a warning about the worker interrupted by the
# short timeout while tasks are pending in the queue on
# overloaded CI hosts.
warnings.simplefilter("ignore", category=UserWarning)
# Trigger worker spawn for lazy executor implementations
for _ in self.executor.map(id, range(8)):
pass
# Check that all workers shutdown (via timeout) when waiting a bit:
# note that the effective time for a Python process to completely
# shutdown can vary a lot especially on loaded CI machines with and
# the atexit callbacks that writes test coverage data to disk.
# Let's be patient.
self.check_no_running_workers(patience=5)
@classmethod
def reducer_in(cls, obj):
return MyObject, (obj.value + 5,)
@classmethod
def reducer_out(cls, obj):
return MyObject, (7 * obj.value,)
def test_serialization(self):
"""Test custom serialization for process_executor"""
self.executor.shutdown(wait=True)
# Use non commutative operation to check correct order
job_reducers = {}
job_reducers[MyObject] = self.reducer_in
result_reducers = {}
result_reducers[MyObject] = self.reducer_out
# Create a new executor to ensure that we did not mess with the
# existing module level serialization
executor = self.executor_type(
max_workers=2,
context=self.context,
job_reducers=job_reducers,
result_reducers=result_reducers,
)
self.executor = self.executor_type(max_workers=2, context=self.context)
obj = MyObject(1)
try:
ret_obj_custom = executor.submit(self.return_inputs, obj).result()[
0
]
ret_obj = self.executor.submit(self.return_inputs, obj).result()[0]
assert ret_obj.value == 1
assert ret_obj_custom.value == 42
finally:
executor.shutdown(wait=True)
@classmethod
def _test_max_depth(cls, max_depth=10, kill_workers=False, ctx=None):
if max_depth == 0:
return 42
executor = cls.executor_type(1, context=ctx)
f = executor.submit(cls._test_max_depth, max_depth - 1, ctx)
try:
return f.result()
finally:
executor.shutdown(wait=True, kill_workers=kill_workers)
@pytest.mark.parametrize("kill_workers", [True, False])
def test_max_depth(self, kill_workers):
from loky.process_executor import MAX_DEPTH
if self.context.get_start_method() == "fork":
# For 'fork', we do not allow nested process as the threads ends
# up in messy states
with pytest.raises(LokyRecursionError):
self._test_max_depth(max_depth=2, ctx=self.context)
return
assert (
self._test_max_depth(
max_depth=MAX_DEPTH,
kill_workers=kill_workers,
ctx=self.context,
)
== 42
)
with pytest.raises(LokyRecursionError):
self._test_max_depth(
max_depth=MAX_DEPTH + 1,
kill_workers=kill_workers,
ctx=self.context,
)
@pytest.mark.high_memory
@pytest.mark.skipif(
sys.maxsize < 2**32,
reason="Test requires a 64 bit version of Python",
)
def test_no_failure_on_large_data_send(self):
data = b"\x00" * int(2.2e9)
self.executor.submit(id, data).result()
def test_memory_leak_protection(self):
pytest.importorskip("psutil") # cannot work without psutil
self.executor.shutdown(wait=True)
executor = self.executor_type(1, context=self.context)
def _leak_some_memory(size=int(3e6), delay=0.001):
"""function that leaks some memory"""
from loky import process_executor
process_executor._MEMORY_LEAK_CHECK_DELAY = 0.1
if getattr(os, "_loky_leak", None) is None:
os._loky_leak = []
os._loky_leak.append(b"\x00" * size)
# Leave enough time for the memory leak detector to kick-in:
# by default the process does not check its memory usage
# more than once per second.
time.sleep(delay)
leaked_size = sum(len(buffer) for buffer in os._loky_leak)
return os.getpid(), leaked_size
with pytest.warns(UserWarning, match="memory leak") as record:
# Total run time should be 3s which is way over the 1s cooldown
# period between two consecutive memory checks in the worker.
futures = [executor.submit(_leak_some_memory) for _ in range(300)]
executor.shutdown(wait=True)
results = [f.result() for f in futures]
# The pid of the worker has changed when restarting the worker
first_pid, last_pid = results[0][0], results[-1][0]
assert first_pid != last_pid
# The restart happened after 100 MB of leak over the
# default process size + what has leaked since the last
# memory check.
for _, leak_size in results:
assert leak_size / 1e6 < 650
# The worker gets recycled repeatedly but that is only reported once
assert sum("memory leak" in str(w.message) for w in record) == 1
def test_reference_cycle_collection(self):
# make the parallel call create a reference cycle and make
# a weak reference to be able to track the garbage collected objects
self.executor.shutdown(wait=True)
executor = self.executor_type(1, context=self.context)
def _create_cyclic_reference(delay=0.001):
"""function that creates a cyclic reference"""
from loky import process_executor
process_executor._USE_PSUTIL = False
process_executor._MEMORY_LEAK_CHECK_DELAY = 0.1
class A:
def __init__(self, size=int(1e6)):
self.data = b"\x00" * size
self.a = self
if getattr(os, "_loky_cyclic_weakrefs", None) is None:
os._loky_cyclic_weakrefs = []
a = A()
time.sleep(delay)
os._loky_cyclic_weakrefs.append(weakref.ref(a))
return sum(1 for r in os._loky_cyclic_weakrefs if r() is not None)
# Total run time should be 3s which is way over the 1s cooldown
# period between two consecutive memory checks in the worker.
futures = [
executor.submit(_create_cyclic_reference) for _ in range(300)
]
executor.shutdown(wait=True)
max_active_refs_count = max(f.result() for f in futures)
assert max_active_refs_count < 150
assert max_active_refs_count != 1
@pytest.mark.broken_pool
def test_exited_child(self):
# When a child process is abruptly terminated, the whole pool gets
# "broken".
print(self.context.get_start_method())
match = filter_match(r"EXIT\(42\)")
future = self.executor.submit(c_exit, 42)
with pytest.raises(TerminatedWorkerError, match=match):
future.result()
# Submitting other jobs fails as well.
with pytest.raises(TerminatedWorkerError, match=match):
self.executor.submit(pow, 2, 8)
@staticmethod
def _test_child_env(var_name):
return os.environ.get(var_name, "unset")
def test_child_env_executor(self):
# Test that for loky context, setting argument env correctly overwrite
# the environment of the child process.
if self.context.get_start_method() != "loky":
pytest.skip("Only work with loky context")
var_name = "loky_child_env_executor"
var_value = "variable set"
executor = self.executor_type(1, env={var_name: var_value})
var_child = executor.submit(self._test_child_env, var_name).result()
assert var_child == var_value
executor.shutdown(wait=True)
def test_viztracer_profiler(self):
# Check that viztracer profiler is initialzed in workers when
# installed.
viztracer = pytest.importorskip("viztracer")
def check_viztracer_active():
tracer = viztracer.get_tracer()
if tracer is None:
return False
return tracer.enable
active_in_main_process = check_viztracer_active()
with self.executor_type(1, context=self.context) as e:
active_in_child_process = e.submit(check_viztracer_active).result()
assert active_in_main_process == active_in_child_process
if not active_in_main_process:
tracer = viztracer.VizTracer()
try:
tracer.start()
with self.executor_type(1, context=self.context) as e:
assert e.submit(check_viztracer_active).result()
finally:
tracer.stop()
# Once the tracer has been stopped, should be no side effect on
# workers started in new executors.
with self.executor_type(1, context=self.context) as e:
assert not e.submit(check_viztracer_active).result()
def test_viztracer_profiler_with_custom_init(self):
# Check that viztracer profiler is initialzed in workers when
# installed.
viztracer = pytest.importorskip("viztracer")
# Make sure the auto-viztracer initialization works even when
# the call pass their own init.
def check_viztracer_active_and_custom_init():
assert loky._custom_global_var == 42
tracer = viztracer.get_tracer()
if tracer is None:
return False
return tracer.enable
existing_tracer = viztracer.get_tracer()
if existing_tracer is not None and existing_tracer.enable:
pytest.skip("Cannot run this test if viztracer is active")
tracer = viztracer.VizTracer()
try:
tracer.start()
with self.executor_type(
1, context=self.context, initializer=_custom_initializer
) as e:
assert e.submit(
check_viztracer_active_and_custom_init
).result()
finally:
tracer.stop()
def test_exception_cause_with_tblib(self):
"Ensure tampering with exception pickling do not break __cause__ propagation"
tblib_pickling_support = pytest.importorskip("tblib.pickling_support")
error_message = "This is the error message"
def raise_value_error():
tblib_pickling_support.install()
raise ValueError(error_message)
executor = self.executor_type(max_workers=2)
f = executor.submit(raise_value_error)
try:
f.result()
except ValueError as e:
assert e.__cause__ is not None
assert error_message in str(e.__cause__)
executor.shutdown(wait=True)
def test_no_deprecation_warning_is_raised_on_fork(self):
# On POSIX, recent versions of Python detect if the process has native
# threads running when calling `os.fork` and similar. All supported
# process executors should not trigger this warning by not calling
# `os.fork` (and `os.execve`) manually but instead using the compound
# _posixsubprocess.fork_exec.
try:
from _testcapi import _spawn_pthread_waiter, _end_spawned_pthread
except ImportError:
pytest.skip("Cannot test without _testcapi module")
_spawn_pthread_waiter()
try:
with warnings.catch_warnings(
category=DeprecationWarning, record=True
) as w:
warnings.simplefilter("always", category=DeprecationWarning)
executor = self.executor_type(max_workers=2)
try:
list(executor.map(sqrt, range(10)))
finally:
executor.shutdown(wait=True)
assert len(w) == 0, [w.message for w in w]
finally:
_end_spawned_pthread()
@pytest.mark.broken_pool
def test_manager_thread_crash_breaks_executor(self):
"""An unexpected error in the manager thread breaks the executor
rather than leaving the submitted futures pending forever."""
self.executor.submit(int).result() # starts the manager thread
manager = self.executor._executor_manager_thread
def crash():
raise RuntimeError("manager thread bug")
# The manager thread may crash before or after the submit below
# depending on scheduling: both paths raise the same exception.
manager.wait_result_broken_or_wakeup = crash
with pytest.raises(
BrokenProcessPool, match="manager thread crashed"
) as exc_info:
self.executor.submit(int).result(timeout=_executor_mixin.TIMEOUT)
assert "manager thread bug" in str(exc_info.value.__cause__)
manager.join(_executor_mixin.TIMEOUT)
assert not manager.is_alive()
with pytest.raises(BrokenProcessPool):
self.executor.submit(int)
@pytest.mark.broken_pool
def test_feeder_error_hook_crash_breaks_executor(self):
"""A failure in the call queue's feeder error hook breaks the
executor rather than leaving the submitted futures pending forever."""
self.executor.submit(int).result() # starts the feeder thread
manager = self.executor._executor_manager_thread
# Make the hook itself fail when it handles the pickling error, as
# when the manager thread already dropped the work id: the ``remove``
# of the id from this list then raises ValueError
self.executor._call_queue.running_work_items = []
with pytest.raises(
BrokenProcessPool, match="feeder thread crashed"
) as exc_info:
self.executor.submit(id, ErrorAtPickle()).result(
timeout=_executor_mixin.TIMEOUT
)
assert "ValueError" in str(exc_info.value.__cause__)
manager.join(_executor_mixin.TIMEOUT)
assert not manager.is_alive()
with pytest.raises(BrokenProcessPool):
self.executor.submit(int)
def _custom_initializer():
"""_custom_initializer is module function to be picklable
This is necessary for executor implementations that do not
use cloudpickle to pickle the initializer.
"""
loky._custom_global_var = 42
joblib-loky-51247f4/tests/conftest.py 0000664 0000000 0000000 00000002742 15256713413 0017562 0 ustar 00root root 0000000 0000000 import pytest
import logging
import warnings
from multiprocessing.util import log_to_stderr
def pytest_addoption(parser):
parser.addoption(
"--loky-verbosity",
type=int,
default=logging.DEBUG,
help="log-level: integer, SUBDEBUG(5) - INFO(20)",
)
parser.addoption(
"--skip-high-memory",
action="store_true",
help="skip high-memory test to avoid conflict on CI.",
)
def log_lvl(request):
"""Choose logging level for multiprocessing"""
return request.config.getoption("--loky-verbosity")
def pytest_configure(config):
"""Setup multiprocessing logging for loky testing"""
logging._levelToName[5] = "SUBDEBUG"
log = log_to_stderr(config.getoption("--loky-verbosity"))
log.handlers[0].setFormatter(
logging.Formatter(
"[%(levelname)s:%(processName)s:%(threadName)s] %(message)s"
)
)
warnings.simplefilter("always")
config.addinivalue_line("markers", "timeout")
config.addinivalue_line("markers", "broken_pool")
config.addinivalue_line("markers", "high_memory")
def pytest_collection_modifyitems(config, items):
if not config.getoption("--skip-high-memory"):
# --skip-high-memory given in cli: skip high-memory tests
return
skip_high_memory = pytest.mark.skip(
reason="--skip-high-memory option was provided"
)
for item in items:
if "high_memory" in item.keywords:
item.add_marker(skip_high_memory)
joblib-loky-51247f4/tests/test_cloudpickle_wrapper.py 0000664 0000000 0000000 00000023157 15256713413 0023035 0 ustar 00root root 0000000 0000000 import os
import sys
import pytest
import io
from pickle import loads
from tempfile import mkstemp
from loky import set_loky_pickler
from loky.backend.reduction import get_loky_pickler
from .utils import check_subprocess_call
class TestCloudpickleWrapper:
def test_isolated_pickler_dispatch_tables(self):
class A:
pass
class B:
pass
iob1 = io.BytesIO()
p1 = get_loky_pickler()(iob1, reducers={A: lambda obj: (int, (42,))})
iob2 = io.BytesIO()
p2 = get_loky_pickler()(iob2, reducers={B: lambda obj: (int, (42,))})
assert p1.dispatch_table is not p2.dispatch_table
assert A in p1.dispatch_table
assert A not in p2.dispatch_table
assert B in p2.dispatch_table
assert B not in p1.dispatch_table
p1.dump((A(), B()))
a, b = loads(iob1.getvalue())
assert a == 42
assert isinstance(b, B)
p2.dump((A(), B()))
a, b = loads(iob2.getvalue())
assert isinstance(a, A)
assert b == 42
def test_serialization_function_from_main(self):
# check that the init_main_module parameter works properly
# when using -c option, we don't need the safeguard if __name__ ..
# and thus test LokyProcess without the extra argument. For running
# a script, it is necessary to use init_main_module=False.
code = """if True:
from loky import get_reusable_executor
def test_func(x):
pass
e = get_reusable_executor()
e.submit(test_func, 42).result()
print("ok")
"""
cmd = [sys.executable]
try:
fid, filename = mkstemp(suffix="_joblib.py")
os.close(fid)
with open(filename, mode="w") as f:
f.write(code)
cmd += [filename]
check_subprocess_call(cmd, stdout_regex="ok", timeout=10)
# Makes sure that if LOKY_PICKLER is set to default pickle, the
# tasks are not wrapped with cloudpickle and it is not possible
# using functions from the main module.
env = {"LOKY_PICKLER": "pickle"}
with pytest.raises(ValueError, match="A task has failed to un-s"):
check_subprocess_call(cmd, timeout=10, env=env)
finally:
os.unlink(filename)
def test_serialization_class_from_main(self):
# check that the init_main_module parameter works properly
# when using -c option, we don't need the safeguard if __name__ ..
# and thus test LokyProcess without the extra argument. For running
# a script, it is necessary to use init_main_module=False.
code = """if True:
from loky import get_reusable_executor
class Test:
def __init__(self, x=42):
self.x = x
def test_func(self, x):
return 42
def pass_all(*args, **kwargs):
pass
e = get_reusable_executor()
e.submit(pass_all, Test()).result()
e.submit(pass_all, x=Test()).result()
e.submit(pass_all, 1, 2, a=0, x=Test()).result()
assert e.submit(Test().test_func, 0).result() == 42
print("ok")
"""
cmd = [sys.executable]
try:
fid, filename = mkstemp(suffix="_joblib.py")
os.close(fid)
with open(filename, mode="w") as f:
f.write(code)
cmd += [filename]
check_subprocess_call(cmd, stdout_regex="ok", timeout=10)
finally:
os.unlink(filename)
def test_cloudpickle_flag_wrapper(self):
# check that the wrap_non_picklable_objects works properly on functions
# and classes.
code = """if True:
import pytest
from loky import get_reusable_executor
from loky.cloudpickle_wrapper import wrap_non_picklable_objects
@wrap_non_picklable_objects
def test_func(x):
return x
@wrap_non_picklable_objects
class Test:
def __init__(self):
self.x = 42
def return_func(self):
return self.x
test_obj = Test()
# Make sure the function and object behave correctly
assert test_obj.x == 42
assert test_func(42) == 42
assert test_obj.return_func() == 42
assert test_func(test_obj.return_func)() == 42
# Make sure the wrapper do not make the object callable
with pytest.raises(TypeError,
match="'Test' object is not callable"):
test_obj()
assert not callable(test_obj)
# Make sure it is picklable even when the executor does not rely on
# cloudpickle.
e = get_reusable_executor()
result_obj = e.submit(test_func, 42).result()
result_obj = e.submit(id, test_obj).result()
result_obj = e.submit(test_func, test_obj).result()
assert result_obj.return_func() == 42
print("ok")
"""
cmd = [sys.executable]
try:
fid, filename = mkstemp(suffix="_joblib.py")
os.close(fid)
with open(filename, mode="w") as f:
f.write(code)
cmd += [filename]
env = {"LOKY_PICKLER": "pickle"}
check_subprocess_call(cmd, stdout_regex="ok", timeout=10, env=env)
finally:
os.unlink(filename)
@pytest.mark.parametrize(
"loky_pickler", [None, "''", "'cloudpickle'", "'pickle'"]
)
def test_set_loky_pickler(self, loky_pickler):
# Test that the function set_loky_pickler correctly changes the pickler
# used in loky.
code = f"""if True:
from loky import set_loky_pickler
from loky import get_reusable_executor
from loky import wrap_non_picklable_objects
from loky.backend.reduction import get_loky_pickler
from loky.backend.reduction import get_loky_pickler_name
from pickle import Pickler
from cloudpickle import CloudPickler
# Check the default loky_pickler is cloudpickle
current_loky_pickler_name = get_loky_pickler_name()
assert current_loky_pickler_name == 'cloudpickle', (
f"default got loky_pickler={{current_loky_pickler_name}}"
)
assert issubclass(get_loky_pickler(), CloudPickler)
# Check that setting loky pickler to a value is working
loky_pickler = {loky_pickler}
set_loky_pickler(loky_pickler)
if loky_pickler in [None, '', 'cloudpickle']:
expected_loky_pickler = CloudPickler
expected_loky_pickler_name = 'cloudpickle'
elif loky_pickler == 'pickle':
expected_loky_pickler = Pickler
expected_loky_pickler_name = 'pickle'
else:
raise RuntimeError(
f"unexpected value {{loky_pickler}} for loky_pickler")
current_loky_pickler_name = get_loky_pickler_name()
assert current_loky_pickler_name == expected_loky_pickler_name, (
f"Expected 'pickle' and got {{current_loky_pickler_name}}")
assert issubclass(get_loky_pickler(), expected_loky_pickler)
# Make sure that the default behavior is restored when
# set_loky_pickler is used without arguments
set_loky_pickler()
current_loky_pickler_name = get_loky_pickler_name()
assert current_loky_pickler_name == 'cloudpickle', (
f"default got loky_pickler={{current_loky_pickler_name}}")
assert issubclass(get_loky_pickler(), CloudPickler)
# Check that the loky pickler in the workers is the correct one.
set_loky_pickler(loky_pickler)
e = get_reusable_executor()
worker_loky_pickler = e.submit(get_loky_pickler_name).result()
assert worker_loky_pickler == expected_loky_pickler_name, (
f"expected {{loky_pickler}} "
f"but got {{worker_loky_pickler}} for the worker loky pickler"
)
# Check that for cloudpickle, this does not fail and for pickle, it
# fails with the correct Error.
def test_func(x):
return x
e = get_reusable_executor()
assert e.submit(test_func, 42).result() == 42
print("ok")
"""
cmd = [sys.executable]
try:
fid, filename = mkstemp(suffix="_joblib.py")
os.close(fid)
with open(filename, mode="w") as f:
f.write(code)
cmd += [filename]
if loky_pickler == "'pickle'":
match = "(Can't get|has no) attribute 'test_func'"
with pytest.raises(ValueError, match=match):
check_subprocess_call(cmd, timeout=10)
else:
check_subprocess_call(cmd, stdout_regex="ok", timeout=10)
finally:
os.unlink(filename)
def test_set_loky_pickler_failures(self):
# Check that `set_loky_pickler` fails when a non-existing module is
# required.
with pytest.raises(ImportError, match="loky_pickler to 'no_module'"):
set_loky_pickler("no_module")
# The module passed to `set_loky_pickler` should have a Pickler
# attribute.
with pytest.raises(AttributeError, match="loky_pickler to 'os'"):
set_loky_pickler("os")
joblib-loky-51247f4/tests/test_futures.py 0000664 0000000 0000000 00000021373 15256713413 0020472 0 ustar 00root root 0000000 0000000 import time
import pytest
import threading
from concurrent import futures
from concurrent.futures._base import (
PENDING,
RUNNING,
CANCELLED,
CANCELLED_AND_NOTIFIED,
FINISHED,
)
from loky._base import Future
from .utils import captured_stderr
def create_future(state=PENDING, exception=None, result=None):
f = Future()
f._state = state
f._exception = exception
f._result = result
return f
PENDING_FUTURE = create_future(state=PENDING)
RUNNING_FUTURE = create_future(state=RUNNING)
CANCELLED_FUTURE = create_future(state=CANCELLED)
CANCELLED_AND_NOTIFIED_FUTURE = create_future(state=CANCELLED_AND_NOTIFIED)
EXCEPTION_FUTURE = create_future(state=FINISHED, exception=OSError())
SUCCESSFUL_FUTURE = create_future(state=FINISHED, result=42)
class TestsFuture:
def test_done_callback_with_result(self):
callback_result = [None]
def fn(callback_future):
callback_result[0] = callback_future.result()
f = Future()
f.add_done_callback(fn)
f.set_result(5)
assert 5 == callback_result[0]
def test_done_callback_with_exception(self):
callback_exception = [None]
def fn(callback_future):
callback_exception[0] = callback_future.exception()
f = Future()
f.add_done_callback(fn)
f.set_exception(Exception("test"))
assert ("test",) == callback_exception[0].args
def test_done_callback_with_cancel(self):
was_cancelled = [None]
def fn(callback_future):
was_cancelled[0] = callback_future.cancelled()
f = Future()
f.add_done_callback(fn)
assert f.cancel()
assert was_cancelled[0]
def test_done_callback_raises(self):
with captured_stderr() as stderr:
import logging
log = logging.getLogger("concurrent.futures")
log.addHandler(logging.StreamHandler())
raising_was_called = [False]
fn_was_called = [False]
def raising_fn(callback_future):
raising_was_called[0] = True
raise Exception("foobar")
def fn(callback_future):
fn_was_called[0] = True
f = Future()
f.add_done_callback(raising_fn)
f.add_done_callback(fn)
f.set_result(5)
assert raising_was_called[0]
assert fn_was_called[0]
assert "Exception: foobar" in stderr.getvalue()
del log.handlers[:]
def test_done_callback_already_successful(self):
callback_result = [None]
def fn(callback_future):
callback_result[0] = callback_future.result()
f = Future()
f.set_result(5)
f.add_done_callback(fn)
assert 5 == callback_result[0]
def test_done_callback_already_failed(self):
callback_exception = [None]
def fn(callback_future):
callback_exception[0] = callback_future.exception()
f = Future()
f.set_exception(Exception("test"))
f.add_done_callback(fn)
assert ("test",) == callback_exception[0].args
def test_done_callback_already_cancelled(self):
was_cancelled = [None]
def fn(callback_future):
was_cancelled[0] = callback_future.cancelled()
f = Future()
assert f.cancel()
f.add_done_callback(fn)
assert was_cancelled[0]
def test_repr(self):
import re
assert (
re.match(
"", repr(PENDING_FUTURE)
).pos
> -1
)
assert (
re.match(
"", repr(RUNNING_FUTURE)
).pos
> -1
)
assert (
re.match(
"",
repr(CANCELLED_FUTURE),
).pos
> -1
)
assert (
re.match(
"",
repr(CANCELLED_AND_NOTIFIED_FUTURE),
).pos
> -1
)
assert (
re.match(
"",
repr(EXCEPTION_FUTURE),
).pos
> -1
)
assert (
re.match(
"",
repr(SUCCESSFUL_FUTURE),
).pos
> -1
)
def test_cancel(self):
f1 = create_future(state=PENDING)
f2 = create_future(state=RUNNING)
f3 = create_future(state=CANCELLED)
f4 = create_future(state=CANCELLED_AND_NOTIFIED)
f5 = create_future(state=FINISHED, exception=OSError())
f6 = create_future(state=FINISHED, result=5)
assert f1.cancel()
assert f1._state == CANCELLED
assert not f2.cancel()
assert f2._state == RUNNING
assert f3.cancel()
assert f3._state == CANCELLED
assert f4.cancel()
assert f4._state == CANCELLED_AND_NOTIFIED
assert not f5.cancel()
assert f5._state == FINISHED
assert not f6.cancel()
assert f6._state == FINISHED
def test_cancelled(self):
assert not PENDING_FUTURE.cancelled()
assert not RUNNING_FUTURE.cancelled()
assert CANCELLED_FUTURE.cancelled()
assert CANCELLED_AND_NOTIFIED_FUTURE.cancelled()
assert not EXCEPTION_FUTURE.cancelled()
assert not SUCCESSFUL_FUTURE.cancelled()
def test_done(self):
assert not PENDING_FUTURE.done()
assert not RUNNING_FUTURE.done()
assert CANCELLED_FUTURE.done()
assert CANCELLED_AND_NOTIFIED_FUTURE.done()
assert EXCEPTION_FUTURE.done()
assert SUCCESSFUL_FUTURE.done()
def test_running(self):
assert not PENDING_FUTURE.running()
assert RUNNING_FUTURE.running()
assert not CANCELLED_FUTURE.running()
assert not CANCELLED_AND_NOTIFIED_FUTURE.running()
assert not EXCEPTION_FUTURE.running()
assert not SUCCESSFUL_FUTURE.running()
def test_result_with_timeout(self):
with pytest.raises(futures.TimeoutError):
PENDING_FUTURE.result(timeout=0)
with pytest.raises(futures.TimeoutError):
RUNNING_FUTURE.result(timeout=0)
with pytest.raises(futures.CancelledError):
CANCELLED_FUTURE.result(timeout=0)
with pytest.raises(futures.CancelledError):
CANCELLED_AND_NOTIFIED_FUTURE.result(timeout=0)
with pytest.raises(OSError):
EXCEPTION_FUTURE.result(timeout=0)
assert SUCCESSFUL_FUTURE.result(timeout=0) == 42
def test_result_with_success(self):
def notification(ready):
# Wait until the main thread is waiting for the result.
ready.wait(1)
time.sleep(0.1)
f1.set_result(42)
ready = threading.Event()
f1 = create_future(state=PENDING)
t = threading.Thread(target=notification, args=(ready,))
t.start()
ready.set()
assert f1.result(timeout=5) == 42
def test_result_with_cancel(self):
def notification(ready):
# Wait until the main thread is waiting for the result.
ready.wait(1)
time.sleep(0.1)
f1.cancel()
ready = threading.Event()
f1 = create_future(state=PENDING)
t = threading.Thread(target=notification, args=(ready,))
t.start()
ready.set()
with pytest.raises(futures.CancelledError):
f1.result(timeout=5)
def test_exception_with_timeout(self):
with pytest.raises(futures.TimeoutError):
PENDING_FUTURE.exception(timeout=0)
with pytest.raises(futures.TimeoutError):
RUNNING_FUTURE.exception(timeout=0)
with pytest.raises(futures.CancelledError):
CANCELLED_FUTURE.exception(timeout=0)
with pytest.raises(futures.CancelledError):
CANCELLED_AND_NOTIFIED_FUTURE.exception(timeout=0)
assert isinstance(EXCEPTION_FUTURE.exception(timeout=0), OSError)
assert SUCCESSFUL_FUTURE.exception(timeout=0) is None
def test_exception_with_success(self):
def notification(ready):
# Wait until the main thread is waiting for the exception.
ready.wait(1)
time.sleep(0.1)
with f1._condition:
f1._state = FINISHED
f1._exception = OSError()
f1._condition.notify_all()
ready = threading.Event()
f1 = create_future(state=PENDING)
t = threading.Thread(target=notification, args=(ready,))
t.start()
ready.set()
assert isinstance(f1.exception(timeout=5), OSError)
joblib-loky-51247f4/tests/test_loky_backend.py 0000664 0000000 0000000 00000060631 15256713413 0021422 0 ustar 00root root 0000000 0000000 import os
import sys
import time
import pytest
import signal
import pickle
import socket
import multiprocessing as mp
from multiprocessing.connection import wait
from tempfile import mkstemp
from loky.backend import get_context
from loky.backend.context import START_METHODS
from loky.backend.utils import kill_process_tree
from .utils import (
TimingWrapper,
check_subprocess_call,
with_parallel_sum,
_run_openmp_parallel_sum,
)
if not hasattr(socket, "socketpair"):
def socketpair():
s1 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s2 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s1.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s1.bind((socket.gethostname(), 8080))
s1.listen(1)
s2.connect((socket.gethostname(), 8080))
conn, _ = s1.accept()
return conn, s2
socket.socketpair = socketpair
DELTA = 0.1
ctx_loky = get_context("loky")
HAVE_SEND_HANDLE = sys.platform == "win32" or (
hasattr(socket, "CMSG_LEN")
and hasattr(socket, "SCM_RIGHTS")
and hasattr(socket.socket, "sendmsg")
)
HAVE_FROM_FD = hasattr(socket, "fromfd")
class TestLokyBackend:
# loky processes
Process = staticmethod(ctx_loky.Process)
current_process = staticmethod(mp.current_process)
active_children = staticmethod(mp.active_children)
# interprocess communication objects
Pipe = staticmethod(ctx_loky.Pipe)
Manager = staticmethod(ctx_loky.Manager)
# synchronization primitives
Lock = staticmethod(ctx_loky.Lock)
RLock = staticmethod(ctx_loky.RLock)
Semaphore = staticmethod(ctx_loky.Semaphore)
BoundedSemaphore = staticmethod(ctx_loky.BoundedSemaphore)
Condition = staticmethod(ctx_loky.Condition)
Event = staticmethod(ctx_loky.Event)
Queue = staticmethod(ctx_loky.Queue)
SimpleQueue = staticmethod(ctx_loky.SimpleQueue)
@classmethod
def teardown_class(cls):
"""Clean up the test environment from any remaining subprocesses."""
for child_process in cls.active_children():
kill_process_tree(child_process)
def test_current(self):
current = self.current_process()
authkey = current.authkey
assert current.is_alive()
assert not current.daemon
assert isinstance(authkey, bytes)
assert authkey
assert current.ident == os.getpid()
assert current.exitcode is None
def test_daemon_argument(self):
# By default uses the current process's daemon flag.
proc0 = self.Process(target=self._test_process)
assert proc0.daemon == self.current_process().daemon
proc1 = self.Process(target=self._test_process, daemon=True)
assert proc1.daemon
proc2 = self.Process(target=self._test_process, daemon=False)
assert not proc2.daemon
@classmethod
def _test_process(cls, q, sq, *args, **kwds):
current = cls.current_process()
q.put(args, timeout=1)
sq.put(args)
q.put(kwds, timeout=1)
q.put(current.name, timeout=1)
q.put(bytes(current.authkey))
q.put(current.pid)
@pytest.mark.parametrize("context_name", ["loky", "loky_init_main"])
def test_process(self, context_name):
"""behavior of Process variables and functional connection objects"""
q = self.Queue()
sq = self.SimpleQueue()
args = (q, sq, 1, 2)
kwargs = {"hello": 23, "bye": 2.54}
name = "TestLokyProcess"
ctx = get_context(context_name)
p = ctx.Process(
target=self._test_process, args=args, kwargs=kwargs, name=name
)
p.daemon = True
current = self.current_process()
assert p.authkey == current.authkey
assert not p.is_alive()
assert p.daemon
assert p not in self.active_children()
assert type(self.active_children()) is list
assert p.exitcode is None
# Make sure we do not break security
with pytest.raises(TypeError):
pickle.dumps(p.authkey)
# Make sure we detect bad pickling
with pytest.raises(RuntimeError):
pickle.dumps(q)
p.start()
assert p.exitcode is None
assert p.is_alive()
assert p in self.active_children()
assert q.get() == args[2:]
assert sq.get() == args[2:]
assert q.get() == kwargs
assert q.get() == p.name
assert q.get() == current.authkey
assert q.get() == p.pid
p.join()
assert p.exitcode == 0
assert not p.is_alive()
assert p not in self.active_children()
@classmethod
def _test_connection(cls, conn):
"""Make sure a connection object is functional"""
if hasattr(conn, "get"):
conn = conn.get()
if hasattr(conn, "accept"):
msg = conn.recv(2)
conn.send(msg)
else:
msg = conn.recv_bytes()
conn.send_bytes(msg)
conn.close()
def test_socket(self):
"""sockets can be pickled at spawn and are able to send/recv"""
server, client = socket.socketpair()
p = self.Process(target=self._test_connection, args=(server,))
p.start()
client.settimeout(5)
msg = b"42"
client.send(msg)
assert client.recv(2) == msg
p.join()
assert p.exitcode == 0
client.close()
server.close()
@pytest.mark.skipif(
not HAVE_SEND_HANDLE or not HAVE_FROM_FD,
reason="This system cannot send handle between process"
". Connections object should be shared at spawning.",
)
def test_socket_queue(self):
"""sockets can be pickled in a queue and are able to send/recv"""
q = self.SimpleQueue()
p = self.Process(target=self._test_connection, args=(q,))
p.start()
server, client = socket.socketpair()
q.put(server)
msg = b"42"
client.settimeout(5)
client.send(msg)
assert client.recv(2) == msg
p.join()
assert p.exitcode == 0
client.close()
server.close()
def test_connection(self):
"""connections can be pickled at spawn and are able to send/recv"""
parent_connection, child_connection = self.Pipe(duplex=True)
p = self.Process(
target=self._test_connection, args=(child_connection,)
)
p.start()
msg = b"42"
parent_connection.send(msg)
assert parent_connection.recv() == msg
p.join()
assert p.exitcode == 0
parent_connection.close()
child_connection.close()
@pytest.mark.skipif(
not HAVE_SEND_HANDLE,
reason="This system cannot send handle between. "
"Connections object should be shared at spawning.",
)
def test_connection_queue(self):
"""connections can be pickled in a queue and are able to send/recv"""
q = self.SimpleQueue()
p = self.Process(target=self._test_connection, args=(q,))
p.start()
parent_connection, child_connection = self.Pipe(duplex=True)
q.put(child_connection)
msg = b"42"
parent_connection.send(msg)
assert parent_connection.recv() == msg
p.join()
assert p.exitcode == 0
parent_connection.close()
child_connection.close()
@staticmethod
def _test_child_env(key, queue):
queue.put(os.environ.get(key, "not set"))
def test_child_env_process(self):
key = "loky_child_env_process"
value = "loky works"
out_queue = self.SimpleQueue()
try:
# Test that the environment variable is correctly copied in the
# child process.
os.environ[key] = value
p = self.Process(
target=self._test_child_env, args=(key, out_queue)
)
p.start()
child_var = out_queue.get()
p.join()
assert child_var == value
# Test that the environment variable is correctly overwritted by
# using the `env` argument in Process.
new_value = "loky rocks"
p = self.Process(
target=self._test_child_env,
args=(key, out_queue),
env={key: new_value},
)
p.start()
child_var = out_queue.get()
p.join()
assert child_var == new_value, p.env
finally:
del os.environ[key]
@classmethod
def _test_terminate(cls, event):
# Notify the main process that child process started
event.set()
time.sleep(100)
def test_terminate(self):
manager = self.Manager()
event = manager.Event()
p = self.Process(target=self._test_terminate, args=(event,))
p.daemon = True
p.start()
assert p.is_alive()
assert p in self.active_children()
assert p.exitcode is None
join = TimingWrapper(p.join)
assert join(0) is None
join.assert_timing_almost_zero()
assert p.is_alive()
assert join(-1) is None
join.assert_timing_almost_zero()
assert p.is_alive()
# wait for child process to be fully setup
event.wait(5)
p.terminate()
MAX_JOIN_TIME = 10
if hasattr(signal, "alarm"):
# On the Gentoo buildbot waitpid() often seems to block forever.
# We use alarm() to interrupt it if it blocks for too long.
def handler(*args):
raise RuntimeError(f"join took too long: {p}")
old_handler = signal.signal(signal.SIGALRM, handler)
try:
signal.alarm(MAX_JOIN_TIME)
assert join() is None
finally:
signal.alarm(0)
signal.signal(signal.SIGALRM, old_handler)
else:
assert join() is None
join.assert_timing_lower_than(MAX_JOIN_TIME)
assert not p.is_alive()
assert p not in self.active_children()
p.join()
manager.shutdown()
# XXX sometimes get p.exitcode == 0 on Windows ...
# assert p.exitcode == -signal.SIGTERM
def test_active_children(self):
assert type(self.active_children()) == list
p = self.Process(target=time.sleep, args=(DELTA,))
assert p not in self.active_children()
p.daemon = True
p.start()
assert p in self.active_children()
p.join()
assert p not in self.active_children()
assert p.exitcode == 0
@classmethod
def _test_recursion(cls, wconn, l):
wconn.send(l)
if len(l) < 2:
for i in range(2):
p = cls.Process(
target=cls._test_recursion, args=(wconn, l + [i])
)
p.start()
p.join()
assert p.exitcode == 0
def test_recursion(self):
rconn, wconn = self.Pipe(duplex=False)
self._test_recursion(wconn, [])
time.sleep(DELTA)
result = []
while rconn.poll():
a = rconn.recv()
result.append(a)
expected = [[], [0], [0, 0], [0, 1], [1], [1, 0], [1, 1]]
assert result == expected
@classmethod
def _test_sentinel(cls, event):
event.wait(30.0)
def test_sentinel(self):
event = self.Event()
p = self.Process(target=self._test_sentinel, args=(event,))
with pytest.raises(ValueError):
p.sentinel
p.start()
assert not wait_for_handle(p.sentinel, timeout=0.0)
event.set()
p.join()
assert p.exitcode == 0
assert wait_for_handle(p.sentinel, timeout=1)
@classmethod
def _test_wait_sentinel(cls):
from signal import SIGTERM
time.sleep(0.1)
os.kill(os.getpid(), SIGTERM)
def test_wait_sentinel(self):
p = self.Process(target=self._test_wait_sentinel)
with pytest.raises(ValueError):
p.sentinel
p.start()
assert isinstance(p.sentinel, int)
assert not wait([p.sentinel], timeout=0.0)
assert wait([p.sentinel], timeout=5), p.exitcode
expected_code = 15 if sys.platform == "win32" else -15
p.join() # force refresh of p.exitcode
assert p.exitcode == expected_code
@classmethod
def _high_number_pipe(cls):
"""Create a Pipe with 2 high numbered file descriptors"""
fds = [fd for _ in range(50) for fd in os.pipe()]
r, w = cls.Pipe(duplex=False)
for fd in fds:
os.close(fd)
return r, w
@classmethod
def _test_sync_object_handling(cls, started, stop, conn, w):
"""Check validity of parents args and Create semaphores to clean up
started, stop: Event
make sure the main Process use lsof when this Process is setup
conn: Connection
an open pipe that should be closed at exit
w: int
fileno of the writable end of the Pipe, it should be closed
"""
to_clean_up = [
cls.Semaphore(0),
cls.BoundedSemaphore(1), # noqa: F841
cls.Lock(),
cls.RLock(),
cls.Condition(),
cls.Event(),
]
started.set()
assert conn.recv_bytes() == b"foo"
with pytest.raises(OSError):
os.fstat(w)
stop.wait(5)
def _check_fds(self, pid, w):
"""List all the open files and check no extra files are presents.
Return a list of open named semaphores
"""
import subprocess
try:
out = subprocess.check_output(
f"lsof -a -Fftn -p {pid} -d ^txt,^cwd,^rtd".split(), text=True
)
lines = out.splitlines()[1:]
except (FileNotFoundError, OSError):
print(
"lsof does not exist on this platform. Skip open files"
"check."
)
return []
n_pipe = 0
named_sem = []
unexpected = []
for fd, t, name in zip(lines[::3], lines[1::3], lines[2::3]):
# Check if fd is a standard IO file.
is_std = fd in ["f0", "f1", "f2"]
# Check if fd is a pipe.
is_pipe = t in ["tPIPE", "tFIFO"]
n_pipe += is_pipe
# Check if fd is open for the rng. This can happen on different
# platform and depending of the python version.
is_rng = name == "n/dev/urandom"
# Not sure if this really expected to be open or not...
is_dev_null = name == "n/dev/null"
# Check if fd is a semaphore or an open library. Store all the
# named semaphore.
is_mem = fd in ["fmem", "fDEL"]
if sys.platform == "darwin":
is_mem |= "n/loky-" in name
if is_mem and "n/dev/shm/sem." in name:
named_sem += [name[1:]]
# No other files should be opened at this stage in the process.
if not (is_pipe or is_std or is_rng or is_mem or is_dev_null):
unexpected.append((fd, t, name))
assert len(unexpected) == 0
# there should be:
# - one pipe for communication with main process
# - loky's resource_tracker pipe
# - the Connection pipe
# - multiprocessing's resource_tracker pipe (POSIX only)
if os.name == "posix":
n_expected_pipes = 4
else:
n_expected_pipes = 3
msg = (
"Some pipes were not properly closed during the child process "
"setup."
)
assert n_pipe == n_expected_pipes, msg
# assert that the writable part of the Pipe (not passed to child),
# have been properly closed.
assert len(set(f"f{w}").intersection(lines)) == 0
return named_sem
def test_sync_object_handling(self):
"""Check the correct handling of semaphores and pipes with loky
We use a Pipe object to check the stated of file descriptors in parent
and child. To make sure there is no interference in the fd numbers, we
use high number fd, so newly created fd should be inferior.
To ensure we have the right number of fd in the child Process, we used
`lsof` as it is compatible with Unix systems.
"""
# TODO generate high numbered mp.Pipe directly
# -> can be used on windows
r, w = self._high_number_pipe()
tmp_fname = "/tmp/foobar" if sys.platform != "win32" else ".foobar"
with open(tmp_fname, "w"):
# Process creating semaphore and pipes before stopping
started, stop = self.Event(), self.Event()
p = self.Process(
target=self._test_sync_object_handling,
args=(started, stop, r, w.fileno()),
)
named_sem = []
try:
p.start()
assert started.wait(5), "The process took too long to start"
r.close()
w.send_bytes(b"foo")
if sys.platform != "win32":
named_sem = self._check_fds(p.pid, w.fileno())
finally:
stop.set()
p.join()
# ensure that Pipe->r was closed when the child process exited
with pytest.raises(IOError):
w.send_bytes(b"foo")
w.close()
if sys.platform == "linux":
# On linux, check that the named semaphores created in the
# child process have been unlinked when it terminated.
pid = str(os.getpid())
for sem in named_sem:
if pid not in sem:
assert not os.path.exists(sem), (
"Some named semaphore are not properly cleaned"
" up"
)
assert p.exitcode == 0
@with_parallel_sum
def test_compatibility_openmp(self):
# Use openMP before launching subprocesses. With fork backend, some fds
# are nto correctly clean up, causing a freeze. No freeze should be
# detected with loky.
_run_openmp_parallel_sum(10)
p = self.Process(target=_run_openmp_parallel_sum, args=(100,))
p.start()
p.join()
assert p.exitcode == 0
@pytest.mark.parametrize("run_file", [True, False])
def test_interactively_define_process_no_main(self, run_file):
# check that the init_main_module parameter works properly
# when using -c option, we don't need the safeguard if __name__ ..
# and thus test LokyProcess without the extra argument. For running
# a script, it is necessary to use init_main_module=False.
code = "\n".join(
[
"from loky.backend.process import LokyProcess",
"p = LokyProcess(target=id, args=(1,), ",
f" init_main_module={not run_file})",
"p.start()",
"p.join()",
'msg = "LokyProcess failed to load without safeguard"',
"assert p.exitcode == 0, msg",
'print("ok")',
]
)
cmd = [sys.executable]
try:
if run_file:
fid, filename = mkstemp(suffix="_joblib.py")
os.close(fid)
with open(filename, mode="w") as f:
f.write(code)
cmd += [filename]
else:
cmd += ["-c", code]
check_subprocess_call(cmd, stdout_regex="ok", timeout=10)
finally:
if run_file:
os.unlink(filename)
def test_interactively_define_process_fail_main(self):
# check that the default behavior of the LokyProcess is correct
code = "\n".join(
[
"from loky.backend.process import LokyProcess",
"p = LokyProcess(target=id, args=(1,),",
" init_main_module=True)",
"p.start()",
"p.join()",
'msg = "LokyProcess succeed without safeguards"',
"assert p.exitcode != 0, msg",
]
)
fid, filename = mkstemp(suffix="_joblib.py")
os.close(fid)
try:
with open(filename, mode="w") as f:
f.write(code)
stdout, stderr = check_subprocess_call(
[sys.executable, filename], timeout=10
)
if sys.platform == "win32":
assert "RuntimeError:" in stderr
else:
assert "RuntimeError:" in stdout
finally:
os.unlink(filename)
def test_loky_get_context(self):
# check the behavior of get_context
ctx_default = get_context()
assert ctx_default.get_start_method() == "loky"
ctx_loky = get_context("loky")
assert ctx_loky.get_start_method() == "loky"
ctx_loky_init_main = get_context("loky_init_main")
assert ctx_loky_init_main.get_start_method() == "loky_init_main"
with pytest.raises(ValueError):
get_context("not_available")
def test_interactive_contex_no_main(self):
# Ensure that loky context is working properly
code = "\n".join(
[
"from loky.backend import get_context",
"ctx = get_context()",
'assert ctx.get_start_method() == "loky"',
"p = ctx.Process(target=id, args=(1,))",
"p.start()",
"p.join()",
'msg = "loky context failed to load without safeguard"',
"assert p.exitcode == 0, msg",
'print("ok")',
]
)
try:
fid, filename = mkstemp(suffix="_joblib.py")
os.close(fid)
with open(filename, mode="w") as f:
f.write(code)
check_subprocess_call(
[sys.executable, filename], stdout_regex="ok", timeout=10
)
finally:
os.unlink(filename)
def wait_for_handle(handle, timeout):
if timeout is not None and timeout < 0.0:
timeout = None
return wait([handle], timeout)
def _run_nested_delayed(depth, delay, event):
if depth > 0:
p = ctx_loky.Process(
target=_run_nested_delayed, args=(depth - 1, delay, event)
)
p.start()
p.join()
else:
event.set()
time.sleep(delay)
@pytest.mark.parametrize("use_psutil", [True, False])
def test_kill_process_tree(use_psutil):
psutil = pytest.importorskip("psutil")
event = ctx_loky.Event()
p = ctx_loky.Process(target=_run_nested_delayed, args=(4, 1000, event))
p.start()
# Wait for all the processes to be launched
if not event.wait(30):
kill_process_tree(p, use_psutil=use_psutil)
raise RuntimeError(
"test_kill_process_tree was not able to launch "
"all nested processes."
)
children = psutil.Process(pid=p.pid).children(recursive=True)
kill_process_tree(p, use_psutil=use_psutil)
# The process can take some time finishing so we should wait up to 5s
gone, alive = psutil.wait_procs(children, timeout=5)
msg = f"Should be no descendant left but found:\n{alive}"
assert len(alive) == 0, msg
def _test_default_subcontext(queue):
queue.put(mp.get_start_method())
@pytest.mark.parametrize("method", START_METHODS)
def test_default_subcontext(method):
code = f"""if True:
import sys
from loky.backend.context import get_context, set_start_method
from tests.test_loky_backend import _test_default_subcontext
set_start_method('{method}')
ctx = get_context()
assert ctx.get_start_method() == '{method}'
queue = ctx.SimpleQueue()
p = ctx.Process(target=_test_default_subcontext, args=(queue,))
p.start()
p.join()
start_method = queue.get()
assert start_method == '{method}', start_method
try:
set_start_method('loky')
except RuntimeError:
pass
else:
raise AssertionError("Did not raise RuntimeError when resetting"
"start_method without force")
set_start_method(None, force=True)
ctx = get_context()
assert ctx.get_start_method() == 'loky'
"""
cmd = [sys.executable, "-c", code]
check_subprocess_call(cmd, timeout=10)
ctx_default = get_context()
assert ctx_default.get_start_method() == "loky"
joblib-loky-51247f4/tests/test_loky_module.py 0000664 0000000 0000000 00000041704 15256713413 0021320 0 ustar 00root root 0000000 0000000 import multiprocessing as mp
import os
import sys
import shutil
import subprocess
import warnings
from subprocess import check_output
from unittest.mock import patch, mock_open
import pytest
import loky
from loky import cpu_count
from loky.backend.context import _MAX_WINDOWS_WORKERS
def test_version():
assert hasattr(
loky, "__version__"
), "There are no __version__ argument on the loky module"
def test_cpu_count(monkeypatch):
# Monkeypatch subprocess.run to simulate the absence of CIM on windows to
# test the different code paths in _cpu_count_physical.
old_run = subprocess.run
def mock_run(*args, **kwargs):
if (
"powershell.exe" in args[0]
and os.environ.get("LOKY_TEST_NO_CIM") == "true"
):
raise RuntimeError("Cim not available")
return old_run(*args, **kwargs)
monkeypatch.setattr(subprocess, "run", mock_run)
cpus = cpu_count()
assert type(cpus) is int
assert cpus >= 1
cpus_physical = cpu_count(only_physical_cores=True)
assert type(cpus_physical) is int
assert 1 <= cpus_physical <= cpus
# again to check that it's correctly cached
cpus_physical = cpu_count(only_physical_cores=True)
assert type(cpus_physical) is int
assert 1 <= cpus_physical <= cpus
@pytest.mark.skipif(sys.platform != "win32", reason="Windows specific test")
def test_windows_max_cpu_count():
assert cpu_count() <= _MAX_WINDOWS_WORKERS
@pytest.mark.skipif(sys.platform != "win32", reason="Windows specific test")
@pytest.mark.parametrize(
"implementation_name",
[
"_count_physical_cores_win32_ctypes",
"_count_physical_cores_win32_powershell",
],
)
def test_windows_physical_cores(implementation_name):
psutil = pytest.importorskip("psutil")
implementation = getattr(loky.backend.context, implementation_name)
expected = psutil.cpu_count(logical=False)
assert expected is not None
assert implementation() == expected
@pytest.mark.skipif(sys.platform != "win32", reason="Windows specific test")
def test_windows_physical_cores_falls_back_to_powershell():
from loky.backend.context import _count_physical_cores_win32
with patch(
"loky.backend.context._count_physical_cores_win32_ctypes",
side_effect=RuntimeError("ctypes failed"),
):
with patch(
"loky.backend.context._count_physical_cores_win32_powershell",
return_value=8,
) as mock_powershell:
count = _count_physical_cores_win32()
assert count == 8
mock_powershell.assert_called_once()
def test_windows_physical_cores_powershell_sums_sockets_mine(monkeypatch):
from loky.backend.context import _count_physical_cores_win32_powershell
completed_process = subprocess.CompletedProcess([], 0, stdout="4\n4\n")
monkeypatch.setattr(subprocess, "CREATE_NO_WINDOW", 0, raising=False)
monkeypatch.setattr(
subprocess, "run", lambda *args, **kwargs: completed_process
)
assert _count_physical_cores_win32_powershell() == 8
cpu_count_cmd = (
"from loky.backend.context import cpu_count;" "print(cpu_count({args}))"
)
def _patch_proc_cpuinfo(monkeypatch, content=None, error=None):
# Monkeypatch open() so that reading /proc/cpuinfo returns `content` (or
# raises `error`), without disturbing other files opened while computing
# cpu_count() (e.g. Cgroup files).
real_open = open
def fake_open(path, *args, **kwargs):
if path == "/proc/cpuinfo":
if error is not None:
raise error
return mock_open(read_data=content)()
return real_open(path, *args, **kwargs)
monkeypatch.setattr("builtins.open", fake_open)
def test_cpu_count_os_sched_getaffinity():
if not hasattr(os, "sched_getaffinity") or not hasattr(shutil, "which"):
pytest.skip()
taskset_bin = shutil.which("taskset")
python_bin = shutil.which("python")
if taskset_bin is None or python_bin is None:
raise pytest.skip()
try:
os.sched_getaffinity(0)
except NotImplementedError:
pytest.skip()
res = check_output(
[
taskset_bin,
"-c",
"0",
python_bin,
"-c",
cpu_count_cmd.format(args=""),
],
text=True,
)
res_physical = check_output(
[
taskset_bin,
"-c",
"0",
python_bin,
"-c",
cpu_count_cmd.format(args="only_physical_cores=True"),
],
text=True,
)
assert res.strip() == "1"
assert res_physical.strip() == "1"
def test_cpu_count_psutil_affinity():
psutil = pytest.importorskip("psutil")
p = psutil.Process()
if not hasattr(p, "cpu_affinity"):
pytest.skip("psutil does not provide cpu_affinity on this platform")
original_affinity = p.cpu_affinity()
assert cpu_count() <= len(original_affinity)
try:
new_affinity = original_affinity[:1]
p.cpu_affinity(new_affinity)
assert cpu_count() == 1
finally:
p.cpu_affinity(original_affinity)
def test_cpu_count_cgroup_limit():
if sys.platform == "win32":
pytest.skip()
if not hasattr(shutil, "which"):
pytest.skip()
docker_bin = shutil.which("docker")
if docker_bin is None:
raise pytest.skip("docker is required to run this test")
loky_module_path = os.path.abspath(os.path.dirname(loky.__file__))
loky_project_path = os.path.abspath(
os.path.join(loky_module_path, os.pardir)
)
# Check if Docker can actually set cgroup CPU limits in this environment
# by verifying that --cpus flag writes to cgroup files
cgroup_check = check_output(
f'{docker_bin} run --rm --cpus 0.5 python:3.10 python3 -c "'
"import os; "
"v2 = '/sys/fs/cgroup/cpu.max'; "
"v1_quota = '/sys/fs/cgroup/cpu/cpu.cfs_quota_us'; "
"v2_content = open(v2).read().strip() if os.path.exists(v2) else ''; "
"v1_content = open(v1_quota).read().strip() if os.path.exists(v1_quota) else ''; "
"print('ok' if (v2_content and v2_content != 'max') or (v1_content and v1_content != '-1') else 'skip')"
'"',
shell=True,
text=True,
).strip()
if cgroup_check != "ok":
pytest.skip(
"Docker doesn't properly set cgroup CPU limits in this environment"
)
# The following will always run using the Python 3.10 docker image.
# We mount the loky source as /loky inside the container,
# so it can be imported when running commands under /
# Tell docker to configure the Cgroup quota to use 0.5 CPU, loky will
# always detect 1 CPU because it rounds up to the next integer.
res_500_mCPU = int(
check_output(
f"{docker_bin} run --rm --cpus 0.5 -v {loky_project_path}:/loky python:3.10 "
f"/bin/bash -c 'pip install --quiet -e /loky ; "
f"python -c \"{cpu_count_cmd.format(args='')}\"'",
shell=True,
text=True,
).strip()
)
assert res_500_mCPU == 1
# Limiting to 1.5 CPUs can lead to 1 if there is only 1 CPU on the machine or
# 2 if there are 2 CPUs or more.
res_1500_mCPU = int(
check_output(
f"{docker_bin} run --rm --cpus 1.5 -v {loky_project_path}:/loky python:3.10 "
f"/bin/bash -c 'pip install --quiet -e /loky ; "
f"python -c \"{cpu_count_cmd.format(args='')}\"'",
shell=True,
text=True,
).strip()
)
assert res_1500_mCPU in (1, 2)
# By default there is no limit: use all available CPUs.
res_default = int(
check_output(
f"{docker_bin} run --rm -v {loky_project_path}:/loky python:3.10 "
f"/bin/bash -c 'pip install --quiet -e /loky ; "
f"python -c \"{cpu_count_cmd.format(args='')}\"'",
shell=True,
text=True,
).strip()
)
assert res_default >= res_1500_mCPU
def test_only_physical_cores_error(monkeypatch):
# Check the warning issued by cpu_count(only_physical_cores=True) when
# unable to retrieve the number of physical cores.
if sys.platform != "linux":
pytest.skip()
# if number of available cpus is already restricted, cpu_count will return
# that value and no warning is issued even if only_physical_cores == True.
# (tested in another test: test_only_physical_cores_with_user_limitation
cpu_count_mp = mp.cpu_count()
if cpu_count() < cpu_count_mp:
pytest.skip()
# Simulate /proc/cpuinfo being unreadable so that the physical core
# count cannot be found.
_patch_proc_cpuinfo(
monkeypatch, error=OSError("simulated /proc/cpuinfo read failure")
)
# clear the cache otherwise the warning is not triggered
import loky.backend.context
monkeypatch.setattr(loky.backend.context, "physical_cores_cache", {})
with pytest.warns(
UserWarning,
match="Could not find the number of physical cores",
):
# Falls back to the logical CPU count when the physical core count
# cannot be found.
assert cpu_count(only_physical_cores=True) == cpu_count_mp
# Should not warn the second time
with warnings.catch_warnings():
warnings.simplefilter("error")
assert cpu_count(only_physical_cores=True) == cpu_count_mp
def test_only_physical_cores_enforced_non_linux(monkeypatch):
# On non-Linux platforms, only_physical_cores=True is enforced by taking
# the minimum with the whole-machine physical core count, even when
# another constraint (here, a Cgroup CPU bandwidth limit) already
# restricts the usable CPU count below the machine's total logical CPU
# count. Unlike on Linux, CPU affinity is not taken into account there:
# _count_physical_cores is called with cpu_set=None.
import loky.backend.context as context
monkeypatch.setattr(sys, "platform", "darwin")
monkeypatch.setattr(os, "cpu_count", lambda: 8)
monkeypatch.setattr(context, "_cpu_count_affinity_set", lambda: None)
monkeypatch.setattr(context, "_cpu_count_cgroup", lambda os_cpu_count: 6)
def _fake_count_physical_cores(cpu_set=None):
assert cpu_set is None
return 4, None
monkeypatch.setattr(
context, "_count_physical_cores", _fake_count_physical_cores
)
assert context.cpu_count() == 6
assert context.cpu_count(only_physical_cores=True) == 4
def test_cpu_count_only_physical_cores_smt_siblings_affinity(monkeypatch):
# Regression test for https://github.com/joblib/loky/issues/639:
# only_physical_cores=True should collapse SMT/hyper-threading sibling
# logical CPUs sharing the same physical core when the usable CPUs are
# restricted through CPU affinity (e.g. `taskset`), instead of just
# returning the (affinity-restricted) logical CPU count unchanged.
if sys.platform != "linux":
pytest.skip("Linux specific test")
import loky.backend.context as context
cpu_count = context.cpu_count
# Simulate a 4 logical CPU machine: CPU 0 and 1 are SMT siblings of
# physical core 0; CPU 2 and 3 are SMT siblings of physical core 1.
fake_cpuinfo = (
"processor\t: 0\nphysical id\t: 0\ncore id\t: 0\n\n"
"processor\t: 1\nphysical id\t: 0\ncore id\t: 0\n\n"
"processor\t: 2\nphysical id\t: 0\ncore id\t: 1\n\n"
"processor\t: 3\nphysical id\t: 0\ncore id\t: 1\n\n"
)
_patch_proc_cpuinfo(monkeypatch, content=fake_cpuinfo)
monkeypatch.setattr(os, "cpu_count", lambda: 4)
monkeypatch.setattr(context, "physical_cores_cache", {})
# With unconstrained affinity, cpu_count should report
# 2 physical cores and 4 logical cpus:
monkeypatch.setattr(
os, "sched_getaffinity", lambda pid: {0, 1, 2, 3}, raising=False
)
assert cpu_count() == 4
assert cpu_count(only_physical_cores=True) == 2
monkeypatch.setattr(
os, "sched_getaffinity", lambda pid: {0, 1}, raising=False
)
# taskset -c 0,1 pins the process to 2 logical CPUs that are SMT
# siblings of a single physical core.
assert cpu_count() == 2
assert cpu_count(only_physical_cores=True) == 1
# Changing the affinity to 2 logical CPUs that belong to different
# physical cores (0 and 2) must not collapse them, and must use a
# separate cache entry than the previous affinity set.
monkeypatch.setattr(
os, "sched_getaffinity", lambda pid: {0, 2}, raising=False
)
assert cpu_count() == cpu_count(only_physical_cores=True) == 2
# Back to unconstrained affinity:
monkeypatch.setattr(
os, "sched_getaffinity", lambda pid: {0, 1, 2, 3}, raising=False
)
# Restrict solely through LOKY_MAX_CPU_COUNT
monkeypatch.setenv("LOKY_MAX_CPU_COUNT", "3")
assert cpu_count() == 3
assert cpu_count(only_physical_cores=True) == 2
monkeypatch.setenv("LOKY_MAX_CPU_COUNT", "1")
assert cpu_count() == cpu_count(only_physical_cores=True) == 1
monkeypatch.delenv("LOKY_MAX_CPU_COUNT")
# Same as above, but the restriction now comes solely from a Cgroup CPU
# bandwidth limit instead of LOKY_MAX_CPU_COUNT.
monkeypatch.setattr(context, "_cpu_count_cgroup", lambda os_cpu_count: 3)
assert cpu_count() == 3
assert cpu_count(only_physical_cores=True) == 2
monkeypatch.setattr(context, "_cpu_count_cgroup", lambda os_cpu_count: 1)
assert cpu_count() == cpu_count(only_physical_cores=True) == 1
def test_count_physical_cores_linux_multi_socket(monkeypatch):
# Regression test: a physical core must be identified by its
# (physical id, core id) pair, not by core id alone, otherwise cores
# sharing the same core id across sockets on a multi-socket machine
# get under-counted.
if sys.platform != "linux":
pytest.skip("Linux specific test")
from loky.backend.context import _count_physical_cores_linux
# Simulate a 2-socket machine with 2 physical cores per socket, where
# each socket reuses core id 0 and 1.
fake_cpuinfo = "".join(
f"processor\t: {cpu}\nphysical id\t: {cpu // 2}\ncore id\t: {cpu % 2}\n\n"
for cpu in range(4)
)
_patch_proc_cpuinfo(monkeypatch, content=fake_cpuinfo)
assert _count_physical_cores_linux() == 4
def test_cpu_count_os_sched_getaffinity_smt_siblings():
# End-to-end version of the test above: actually pin the current
# process to 2 SMT sibling logical CPUs of the same physical core (if
# such a pair can be found on the machine running the test) and check
# that only_physical_cores=True correctly reports 1 physical core while
# the plain logical CPU count reports 2.
if sys.platform != "linux":
pytest.skip("Linux specific test")
if not hasattr(os, "sched_getaffinity"):
pytest.skip()
psutil = pytest.importorskip("psutil")
p = psutil.Process()
if not hasattr(p, "cpu_affinity"):
pytest.skip("psutil does not provide cpu_affinity on this platform")
def _expand(cpu_range):
if "-" in cpu_range:
start, end = cpu_range.split("-")
return list(range(int(start), int(end) + 1))
return [int(cpu_range)]
smt_pair = None
for cpu in range(os.cpu_count() or 0):
siblings_path = (
f"/sys/devices/system/cpu/cpu{cpu}/topology/core_cpus_list"
)
try:
with open(siblings_path) as f:
content = f.read().strip()
except OSError:
pytest.skip("CPU topology information not available")
siblings = sorted(
{c for part in content.split(",") for c in _expand(part)}
)
if len(siblings) >= 2:
smt_pair = siblings[:2]
break
if smt_pair is None:
pytest.skip(
"could not find 2 SMT sibling logical CPUs on this machine"
)
original_affinity = p.cpu_affinity()
try:
p.cpu_affinity(smt_pair)
assert cpu_count() == 2
assert cpu_count(only_physical_cores=True) == 1
finally:
p.cpu_affinity(original_affinity)
@pytest.mark.parametrize(
"read_data,description",
[
("", "empty file"),
("max\n", "max value"),
],
)
def test_cpu_count_cgroup_invalid_content(read_data, description):
# Test that invalid cgroup cpu.max file content is handled gracefully
# and doesn't cause a ValueError when trying to unpack values
if sys.platform != "linux":
pytest.skip()
from loky.backend.context import _cpu_count_cgroup
os_cpu_count = mp.cpu_count()
# Mock the file with the provided read_data
with patch("builtins.open", mock_open(read_data=read_data)):
with patch("os.path.exists") as mock_exists:
# cpu.max exists, but other files don't
mock_exists.side_effect = (
lambda path: path == "/sys/fs/cgroup/cpu.max"
)
# This should not raise ValueError and return os_cpu_count
result = _cpu_count_cgroup(os_cpu_count)
assert (
result == os_cpu_count
), f"cpu.max with {description} should return os_cpu_count"
joblib-loky-51247f4/tests/test_process_executor_forkserver.py 0000664 0000000 0000000 00000001714 15256713413 0024636 0 ustar 00root root 0000000 0000000 import sys
from loky import process_executor
from loky.backend import get_context
from ._executor_mixin import ExecutorMixin
if sys.platform != "win32":
from ._test_process_executor import (
AsCompletedTests,
ExecutorShutdownTest,
ExecutorTest,
WaitTests,
)
class ProcessPoolForkserverMixin(ExecutorMixin):
executor_type = process_executor.ProcessPoolExecutor
context = get_context("forkserver")
class TestsProcessPoolForkserverShutdown(
ProcessPoolForkserverMixin, ExecutorShutdownTest
):
def _prime_executor(self):
pass
class TestsProcessPoolForkserverWait(
ProcessPoolForkserverMixin, WaitTests
):
pass
class TestsProcessPoolForkserverAsCompleted(
ProcessPoolForkserverMixin, AsCompletedTests
):
pass
class TestsProcessPoolForkserverExecutor(
ProcessPoolForkserverMixin, ExecutorTest
):
pass
joblib-loky-51247f4/tests/test_process_executor_loky.py 0000664 0000000 0000000 00000001430 15256713413 0023417 0 ustar 00root root 0000000 0000000 from loky import process_executor
from loky.backend import get_context
from ._executor_mixin import ExecutorMixin
from ._test_process_executor import (
AsCompletedTests,
ExecutorShutdownTest,
ExecutorTest,
WaitTests,
)
class ProcessPoolLokyMixin(ExecutorMixin):
# Makes sure that the context is defined
executor_type = process_executor.ProcessPoolExecutor
context = get_context("loky")
class TestsProcessPoolLokyShutdown(ProcessPoolLokyMixin, ExecutorShutdownTest):
def _prime_executor(self):
pass
class TestsProcessPoolLokyWait(ProcessPoolLokyMixin, WaitTests):
pass
class TestsProcessPoolLokyAsCompleted(ProcessPoolLokyMixin, AsCompletedTests):
pass
class TestsProcessPoolLokyExecutor(ProcessPoolLokyMixin, ExecutorTest):
pass
joblib-loky-51247f4/tests/test_process_executor_spawn.py 0000664 0000000 0000000 00000001402 15256713413 0023570 0 ustar 00root root 0000000 0000000 from loky import process_executor
from loky.backend import get_context
from ._executor_mixin import ExecutorMixin
from ._test_process_executor import (
AsCompletedTests,
ExecutorShutdownTest,
ExecutorTest,
WaitTests,
)
class ProcessPoolSpawnMixin(ExecutorMixin):
executor_type = process_executor.ProcessPoolExecutor
context = get_context("spawn")
class TestsProcessPoolSpawnShutdown(
ProcessPoolSpawnMixin, ExecutorShutdownTest
):
def _prime_executor(self):
pass
class TestsProcessPoolSpawnWait(ProcessPoolSpawnMixin, WaitTests):
pass
class TestsProcessPoolSpawnAsCompleted(
ProcessPoolSpawnMixin, AsCompletedTests
):
pass
class TestsProcessPoolSpawnExecutor(ProcessPoolSpawnMixin, ExecutorTest):
pass
joblib-loky-51247f4/tests/test_resource_tracker.py 0000664 0000000 0000000 00000046056 15256713413 0022344 0 ustar 00root root 0000000 0000000 """Tests for the ResourceTracker class"""
import base64
import errno
import gc
import json
import multiprocessing
import os
import pytest
import re
import signal
import subprocess
import sys
import time
import warnings
import weakref
from loky import ProcessPoolExecutor
import loky.backend.resource_tracker as resource_tracker
from loky.backend.context import get_context
from .utils import resource_unlink, create_resource, resource_exists
def _resource_unlink(name, rtype):
resource_tracker._CLEANUP_FUNCS[rtype](name)
def get_rtracker_fd():
resource_tracker.ensure_running()
return resource_tracker._resource_tracker._fd
def get_rtracker_fd_identity():
stat = os.fstat(get_rtracker_fd())
return stat.st_dev, stat.st_ino
class _RecordingWinapi:
"""Stand-in for _winapi so Windows paths can be tested everywhere."""
INFINITE = 0xFFFFFFFF
WAIT_OBJECT_0 = 0
WAIT_TIMEOUT = 258
def __init__(
self,
create_process=None,
wait_result=WAIT_OBJECT_0,
exit_code=0,
):
self.waited = []
self.closed = []
self._create_process = create_process
self._wait_result = wait_result
self._exit_code = exit_code
def WaitForSingleObject(self, handle, timeout):
self.waited.append((handle, timeout))
return self._wait_result
def GetExitCodeProcess(self, handle):
return self._exit_code
def CloseHandle(self, handle):
self.closed.append(handle)
def CreateProcess(self, *args):
return self._create_process
def _make_stopped_tracker(handle=42):
tracker = resource_tracker.ResourceTracker()
r, w = os.pipe()
os.close(r)
tracker._fd, tracker._pid, tracker._proc_handle = w, 123456, handle
return tracker, w
class TestResourceTracker:
@pytest.mark.parametrize("rtype", ["file", "folder", "semlock"])
def test_resource_utils(self, rtype):
# Check that the resouce utils work as expected in the main process
if sys.platform == "win32" and rtype == "semlock":
pytest.skip("no semlock on windows")
name = create_resource(rtype)
assert resource_exists(name, rtype)
resource_unlink(name, rtype)
assert not resource_exists(name, rtype)
def test_child_retrieves_resource_tracker(self):
# First simple fd retrieval check (see #200)
# checking fd only work on posix for now
if sys.platform != "win32":
try:
parent_rtracker_fd = get_rtracker_fd()
executor = ProcessPoolExecutor(max_workers=2)
child_rtracker_fd = executor.submit(get_rtracker_fd).result()
assert child_rtracker_fd == parent_rtracker_fd
finally:
executor.shutdown()
# Register a resource in the parent process, and un-register it in the
# child process. If the two processes do not share the same
# resource_tracker, a cache KeyError should be printed in stderr.
cmd = """if 1:
import os, sys
from loky import ProcessPoolExecutor
from loky.backend import resource_tracker
from tempfile import NamedTemporaryFile
tmpfile = NamedTemporaryFile(delete=False)
tmpfile.close()
filename = tmpfile.name
resource_tracker.VERBOSE = True
resource_tracker.register(filename, "file")
def maybe_unlink(name, rtype):
# resource_tracker.maybe_unlink is actually a bound method of the
# ResourceTracker. We need a custom wrapper to avoid object
# serialization.
from loky.backend import resource_tracker
resource_tracker.maybe_unlink(name, rtype)
print(filename)
e = ProcessPoolExecutor(1)
e.submit(maybe_unlink, filename, "file").result()
e.shutdown()
"""
p = subprocess.run(
[sys.executable, "-E", "-c", cmd],
capture_output=True,
text=True,
)
filename = p.stdout.strip()
pattern = f"decremented refcount of file {filename}"
assert pattern in p.stderr
assert "leaked" not in p.stderr
pattern = f"KeyError: '{filename}'"
assert pattern not in p.stderr
# The following four tests are inspired from cpython _test_multiprocessing
@pytest.mark.parametrize("rtype", ["file", "folder", "semlock"])
def test_resource_tracker(self, rtype):
#
# Check that killing process does not leak named resources
#
if (sys.platform == "win32") and rtype == "semlock":
pytest.skip("no semlock on windows")
cmd = f"""if 1:
import time, os, tempfile, sys
from loky.backend import resource_tracker
from utils import create_resource
for _ in range(2):
rname = create_resource("{rtype}")
resource_tracker.register(rname, "{rtype}")
# give the resource_tracker time to register the new resource
time.sleep(0.5)
sys.stdout.write(f"{{rname}}\\n")
sys.stdout.flush()
time.sleep(10)
"""
env = {**os.environ, "PYTHONPATH": os.path.dirname(__file__)}
p = subprocess.Popen(
[sys.executable, "-c", cmd],
stderr=subprocess.PIPE,
stdout=subprocess.PIPE,
env=env,
text=True,
)
name1 = p.stdout.readline().rstrip()
name2 = p.stdout.readline().rstrip()
# subprocess holding a reference to lock1 is still alive, so this call
# should succeed
_resource_unlink(name1, rtype)
p.terminate()
p.wait()
# wait for the resource_tracker to cleanup the leaked resources
time.sleep(2.0)
with pytest.raises(OSError) as ctx:
_resource_unlink(name2, rtype)
# docs say it should be ENOENT, but OSX seems to give EINVAL
assert ctx.value.errno in (errno.ENOENT, errno.EINVAL)
err = p.stderr.read()
p.stderr.close()
p.stdout.close()
expected = f"resource_tracker: There appear to be 2 leaked {rtype}"
assert re.search(expected, err) is not None
# resource 1 is still registered, but was destroyed externally: the
# tracker is expected to complain.
if sys.platform == "win32":
errno_map = {"file": 2, "folder": 3}
expected = (
f"resource_tracker: {re.escape(name1)}: "
f"(WindowsError\\(({errno_map[rtype]})|FileNotFoundError)"
)
else:
expected = (
f"resource_tracker: {re.escape(name1)}: "
f"(OSError\\({errno.ENOENT}|FileNotFoundError)"
)
assert re.search(expected, err) is not None
@pytest.mark.parametrize("rtype", ["file", "folder", "semlock"])
def test_resource_tracker_refcounting(self, rtype):
if sys.platform == "win32" and rtype == "semlock":
pytest.skip("no semlock on windows")
cmd = f"""if 1:
import os
import tempfile
import time
from loky.backend import resource_tracker
from utils import resource_unlink, create_resource, resource_exists
resource_tracker.VERBOSE = True
try:
name = create_resource("{rtype}")
assert resource_exists(name, "{rtype}")
from loky.backend.resource_tracker import _resource_tracker
_resource_tracker.register(name, "{rtype}")
_resource_tracker.register(name, "{rtype}")
# Forget all information about the resource, but do not try to
# remove it
_resource_tracker.unregister(name, "{rtype}")
time.sleep(1)
assert resource_exists(name, "{rtype}")
_resource_tracker.register(name, "{rtype}")
_resource_tracker.register(name, "{rtype}")
_resource_tracker.maybe_unlink(name, "{rtype}")
time.sleep(1)
assert resource_exists(name, "{rtype}")
_resource_tracker.maybe_unlink(name, "{rtype}")
for _ in range(100):
if not resource_exists(name, "{rtype}"):
break
time.sleep(.1)
else:
raise AssertionError(f"{{name}} was not unlinked in time")
finally:
try:
if resource_exists(name, "{rtype}"):
resource_unlink(name, "{rtype}")
except NameError:
# "name" is not defined because create_resource has failed
pass
"""
env = {**os.environ, "PYTHONPATH": os.path.dirname(__file__)}
p = subprocess.run(
[sys.executable, "-c", cmd], capture_output=True, env=env
)
assert p.returncode == 0, p.stderr
def check_resource_tracker_death(self, signum, should_die):
# bpo-31310: if the semaphore tracker process has died, it should
# be restarted implicitly.
from loky.backend.resource_tracker import _resource_tracker
pid = _resource_tracker._pid
if pid is not None:
os.kill(pid, signal.SIGKILL)
os.waitpid(pid, 0)
with warnings.catch_warnings():
warnings.simplefilter("ignore")
_resource_tracker.ensure_running()
pid = _resource_tracker._pid
os.kill(pid, signum)
time.sleep(1.0) # give it time to die
ctx = get_context("loky")
with warnings.catch_warnings(record=True) as all_warn:
warnings.simplefilter("always")
# remove unrelated MacOS warning messages first
warnings.filterwarnings(
"ignore", message="semaphore are broken on OSX"
)
sem = ctx.Semaphore()
sem.acquire()
sem.release()
wr = weakref.ref(sem)
# ensure `sem` gets collected, which triggers communication with
# the resource_tracker
del sem
gc.collect()
assert wr() is None
if should_die:
assert len(all_warn) == 1
the_warn = all_warn[0]
assert issubclass(the_warn.category, UserWarning)
assert "resource_tracker: process died" in str(
the_warn.message
)
else:
assert len(all_warn) == 0, [w.message for w in all_warn]
@pytest.mark.skipif(
sys.platform == "win32", reason="Limited signal support on Windows"
)
def test_resource_tracker_sigint(self):
# Catchable signal (ignored by resource tracker)
self.check_resource_tracker_death(signal.SIGINT, False)
@pytest.mark.skipif(
sys.platform == "win32", reason="Limited signal support on Windows"
)
def test_resource_tracker_sigterm(self):
# Catchable signal (ignored by resource tracker)
self.check_resource_tracker_death(signal.SIGTERM, False)
@pytest.mark.skipif(
sys.platform == "win32", reason="Limited signal support on Windows"
)
def test_resource_tracker_sigkill(self):
# Uncatchable signal.
self.check_resource_tracker_death(signal.SIGKILL, True)
def test_resource_tracker_keeps_process_handle(self):
resource_tracker.ensure_running()
handle = resource_tracker._resource_tracker._proc_handle
assert (handle is not None) == (sys.platform == "win32")
@pytest.mark.skipif(
not hasattr(os, "register_at_fork"),
reason="os.register_at_fork unavailable",
)
def test_resource_tracker_at_fork_callback_is_registered(self):
# Ensure that `_resource_tracker._after_fork_in_child` is called to
# perform the fd handling specified when the `_fork_intent` module-level
# variable from the multiprocessing module of the stdlib (rather than
# the loky vendored copy) is set to False.
#
# After a raw os.fork(), the child's ResourceTracker._pid must be None.
# That only happens if os.register_at_fork registered
# _resource_tracker._after_fork_in_child
resource_tracker.ensure_running()
assert resource_tracker._resource_tracker._pid is not None
read_fd, write_fd = os.pipe()
pid = os.fork()
if pid == 0:
os.close(read_fd)
child_pid = resource_tracker._resource_tracker._pid
os.write(write_fd, str(child_pid).encode("ascii"))
os.close(write_fd)
os._exit(0)
os.close(write_fd)
try:
child_pid = os.read(read_fd, 32)
finally:
os.close(read_fd)
os.waitpid(pid, 0)
assert child_pid == b"None"
@pytest.mark.skipif(
"fork" not in multiprocessing.get_all_start_methods(),
reason="fork start method unavailable",
)
def test_multiprocessing_fork_preserves_resource_tracker_fd(self):
# Check that multiprocessing's `preserve_fd=True` is respected.
parent_fd_identity = get_rtracker_fd_identity()
ctx = multiprocessing.get_context("fork")
with ctx.Pool(1) as pool:
child_fd_identity = pool.apply(get_rtracker_fd_identity)
assert child_fd_identity == parent_fd_identity
@pytest.mark.skipif(
sys.platform != "win32", reason="Windows-specific test"
)
def test_resource_tracker_stop_win32_waits_on_handle(self):
winapi = _RecordingWinapi(exit_code=7)
tracker, w = _make_stopped_tracker()
timeout_in_seconds = 0.5
timeout_in_milliseconds = 1000 * timeout_in_seconds
tracker._stop_locked(
wait_for_single_object=winapi.WaitForSingleObject,
get_exit_code_process=winapi.GetExitCodeProcess,
close_handle=winapi.CloseHandle,
wait_timeout_code=winapi.WAIT_TIMEOUT,
wait_timeout=timeout_in_seconds,
)
assert winapi.waited == [(42, timeout_in_milliseconds)]
assert winapi.closed == [42]
assert tracker._fd is None
assert tracker._pid is None
assert tracker._exitcode == 7
assert tracker._proc_handle is None
with pytest.raises(OSError):
os.close(w)
@pytest.mark.skipif(
sys.platform != "win32", reason="Windows-specific test"
)
def test_resource_tracker_stop_win32_timeout(self):
winapi = _RecordingWinapi(wait_result=_RecordingWinapi.WAIT_TIMEOUT)
tracker, _ = _make_stopped_tracker()
tracker._stop_locked(
wait_for_single_object=winapi.WaitForSingleObject,
get_exit_code_process=winapi.GetExitCodeProcess,
close_handle=winapi.CloseHandle,
wait_timeout_code=winapi.WAIT_TIMEOUT,
wait_timeout=0.5,
)
assert tracker._pid is None
assert tracker._exitcode is None
assert tracker._waitpid_timed_out
assert tracker._proc_handle is None
assert winapi.closed == [42]
@pytest.mark.parametrize("handle", [42, None])
def test_resource_tracker_relaunch_closes_handle(
self, monkeypatch, handle
):
winapi = _RecordingWinapi()
monkeypatch.setattr(resource_tracker, "_winapi", winapi, raising=False)
monkeypatch.setattr(os, "name", "nt")
monkeypatch.setattr(sys, "platform", "win32")
tracker, _ = _make_stopped_tracker(handle=handle)
with pytest.warns(UserWarning, match="died unexpectedly"):
tracker._teardown_dead_process()
assert winapi.closed == ([] if handle is None else [handle])
assert tracker._proc_handle is None
def test_spawnv_passfds_keeps_process_handle(self, monkeypatch):
winapi = _RecordingWinapi(create_process=(42, 43, 123456, 0))
monkeypatch.setattr(resource_tracker, "_winapi", winapi, raising=False)
monkeypatch.setattr(sys, "platform", "win32")
assert resource_tracker.spawnv_passfds("exe", ["exe"], []) == (
123456,
42,
)
assert winapi.closed == [43]
def test_loky_process_inherit_multiprocessing_resource_tracker(self):
cmd = """if 1:
from loky import get_reusable_executor
from multiprocessing.shared_memory import SharedMemory
def mp_rtracker_getfd():
from multiprocessing.resource_tracker import (
_resource_tracker as mp_resource_tracker
)
return mp_resource_tracker._fd
if __name__ == '__main__':
executor = get_reusable_executor(max_workers=1)
# warm up
f = executor.submit(id, 1).result()
# loky forces the creation of the resource tracker at process
# creation so that loky processes can inherit its file descriptor.
parent_fd = mp_rtracker_getfd()
child_fd = executor.submit(mp_rtracker_getfd).result()
assert child_fd == parent_fd
# non-regression test for #242: unlinking in a loky process a
# shared_memory segment tracked by multiprocessing and created its
# parent should not generate warnings.
shm = SharedMemory(create=True, size=10)
f = executor.submit(shm.unlink).result()
"""
p = subprocess.run(
[sys.executable, "-c", cmd], capture_output=True, text=True
)
assert not p.stdout, p.stdout
assert not p.stderr, p.stderr
def test_shutdown_cleans_resources_once_and_folders_last(tmp_path):
folder = tmp_path / "tracked-folder"
folder.mkdir()
filename = folder / "tracked-file"
filename.touch()
cmd = "\n".join(
[
"from loky.backend import resource_tracker",
"",
f"resource_tracker.register({str(folder)!r}, 'folder')",
f"resource_tracker.register({str(filename)!r}, 'file')",
]
)
result = subprocess.run(
[sys.executable, "-c", cmd],
capture_output=True,
text=True,
)
assert result.returncode == 0
assert not folder.exists()
assert not filename.exists()
assert "FileNotFoundError" not in result.stderr
@pytest.mark.skipif(sys.platform == "win32", reason="pass_fds is POSIX-only")
def test_shutdown_cleanup_failure_sets_exit_code():
read_fd, write_fd = os.pipe()
cmd = "from loky.backend.resource_tracker import main; " f"main({read_fd})"
process = subprocess.Popen(
[sys.executable, "-c", cmd],
pass_fds=(read_fd,),
stderr=subprocess.PIPE,
text=True,
)
os.close(read_fd)
try:
os.write(write_fd, b"REGISTER:test-resource:noop\n")
finally:
os.close(write_fd)
_, stderr = process.communicate()
assert process.returncode == 2, stderr
def test_decode_json_message_with_newline_in_name():
name = "folder/name\nwith-newline"
encoded_name = base64.urlsafe_b64encode(name.encode("utf-8")).decode(
"ascii"
)
message = json.dumps(
{
"cmd": "REGISTER",
"rtype": "file",
"base64_name": encoded_name,
}
).encode("ascii")
assert resource_tracker._decode_message(message) == (
"REGISTER",
"file",
name,
)
joblib-loky-51247f4/tests/test_reusable_executor.py 0000664 0000000 0000000 00000134212 15256713413 0022512 0 ustar 00root root 0000000 0000000 import os
import subprocess
import sys
import gc
import ctypes
from concurrent.futures import ThreadPoolExecutor
from tempfile import NamedTemporaryFile
import pytest
import warnings
import threading
from time import sleep
from multiprocessing import util, current_process, Manager
from pickle import PicklingError, UnpicklingError
import cloudpickle
from packaging.version import Version
import loky
from loky import cpu_count
from loky import get_reusable_executor
from loky.process_executor import _RemoteTraceback, TerminatedWorkerError
from loky.process_executor import BrokenProcessPool, ShutdownExecutorError
from loky.reusable_executor import _ReusablePoolExecutor
from loky.backend.context import _MAX_WINDOWS_WORKERS
try:
import psutil
except ImportError:
psutil = None
from ._executor_mixin import ReusableExecutorMixin
from .utils import TimingWrapper, id_sleep, check_python_subprocess_call
from .utils import filter_match
cloudpickle_version = Version(cloudpickle.__version__)
# Compat windows
if sys.platform == "win32":
from signal import SIGTERM as SIGKILL
libc = ctypes.cdll.msvcrt
else:
from signal import SIGKILL
from ctypes.util import find_library
libc = ctypes.CDLL(find_library("c"))
try:
import numpy as np
except ImportError:
np = None
def clean_warning_registry():
"""Safe way to reset warnings."""
warnings.resetwarnings()
registry_name = "__warningregistry__"
for mod in list(sys.modules.values()):
warning_registry = getattr(mod, registry_name, None)
if warning_registry is not None:
warning_registry.clear()
def wait_dead(worker, n_tries=1000, delay=0.001):
"""Wait for process pid to die"""
for _ in range(n_tries):
if worker.exitcode is not None:
return
sleep(delay)
raise RuntimeError(
f"Process {worker.pid} failed to die for at least "
f"{delay * n_tries:0.3f}s"
)
def crash():
"""Induces a segfault"""
import faulthandler
faulthandler._sigsegv()
def exit():
"""Induces a sys exit with exitcode 0"""
sys.exit(0)
def c_exit(exitcode=0):
"""Induces a libc exit with exitcode 0"""
libc.exit(exitcode)
def sleep_then_check_pids_exist(arg):
"""Sleep for some time and the check if all the passed pids exist"""
time, pids = arg
sleep(time)
res = True
for p in pids:
res &= psutil.pid_exists(p)
return res
def kill_friend(pid, delay=0):
"""Function that send SIGKILL at process pid"""
sleep(delay)
try:
os.kill(pid, SIGKILL)
except (PermissionError, ProcessLookupError) as e:
if psutil.pid_exists(pid):
util.debug("Fail to kill an alive process?!?")
raise e
util.debug(f"process {pid} was already dead")
def raise_error(etype=UnpicklingError, message=None):
"""Function that raises an Exception in process"""
raise etype(message)
def return_instance(cls):
"""Function that returns a instance of cls"""
return cls()
class SayWhenError(ValueError):
pass
def exception_throwing_generator(total, when):
for i in range(total):
if i == when:
raise SayWhenError("Somebody said when")
yield i
def do_nothing(arg):
"""Function that return True, test passing argument"""
return True
class CrashAtPickle:
"""Bad object that triggers a segfault at pickling time."""
def __reduce__(self):
crash()
class CrashAtUnpickle:
"""Bad object that triggers a segfault at unpickling time."""
def __reduce__(self):
return crash, ()
class ExitAtPickle:
"""Bad object that triggers a segfault at pickling time."""
def __reduce__(self):
exit()
class ExitAtUnpickle:
"""Bad object that triggers a process exit at unpickling time."""
def __reduce__(self):
return exit, ()
class CExitAtPickle:
"""Bad object that triggers a segfault at pickling time."""
def __reduce__(self):
c_exit()
class CExitAtUnpickle:
"""Bad object that triggers a process exit at unpickling time."""
def __reduce__(self):
return c_exit, ()
class ErrorAtPickle:
"""Bad object that raises an error at pickling time."""
def __init__(self, fail=True):
self.fail = fail
def __reduce__(self):
if self.fail:
raise PicklingError("Error in pickle")
else:
return id, (42,)
class ErrorAtUnpickle:
"""Bad object that triggers a process exit at unpickling time."""
def __init__(self, etype=UnpicklingError, message="the error message"):
self.etype = etype
self.message = message
def __reduce__(self):
return raise_error, (self.etype, self.message)
class CrashAtGCInWorker:
"""Bad object that triggers a segfault at call item GC time"""
def __del__(self):
if current_process().name != "MainProcess":
crash()
class CExitAtGCInWorker:
"""Exit worker at call item GC time"""
def __del__(self):
if current_process().name != "MainProcess":
c_exit()
class TestExecutorDeadLock(ReusableExecutorMixin):
crash_cases = [
# Check problem occuring while pickling a task in
(id, (ExitAtPickle(),), PicklingError, None),
(id, (ErrorAtPickle(),), PicklingError, None),
# Check problem occuring while unpickling a task on workers
(id, (ExitAtUnpickle(),), BrokenProcessPool, "SystemExit"),
(id, (CExitAtUnpickle(),), TerminatedWorkerError, r"EXIT\(0\)"),
(id, (ErrorAtUnpickle(),), BrokenProcessPool, "UnpicklingError"),
(id, (CrashAtUnpickle(),), TerminatedWorkerError, "SIGSEGV"),
# Check problem occuring during function execution on workers
(crash, (), TerminatedWorkerError, "SIGSEGV"),
(exit, (), SystemExit, None),
(c_exit, (), TerminatedWorkerError, r"EXIT\(0\)"),
(raise_error, (RuntimeError,), RuntimeError, None),
# Check problem occuring while pickling a task result
# on workers
(return_instance, (CrashAtPickle,), TerminatedWorkerError, "SIGSEGV"),
(return_instance, (ExitAtPickle,), SystemExit, None),
(
return_instance,
(CExitAtPickle,),
TerminatedWorkerError,
r"EXIT\(0\)",
),
(return_instance, (ErrorAtPickle,), PicklingError, None),
# Check problem occuring while unpickling a task in
# the result_handler thread
(return_instance, (ExitAtUnpickle,), BrokenProcessPool, "SystemExit"),
(
return_instance,
(ErrorAtUnpickle,),
BrokenProcessPool,
"UnpicklingError",
),
]
@pytest.mark.parametrize("func, args, expected_err, match", crash_cases)
def test_crashes(self, func, args, expected_err, match):
"""Test various reusable_executor crash handling"""
executor = get_reusable_executor(max_workers=2)
res = executor.submit(func, *args)
match_err = None
if expected_err is TerminatedWorkerError:
match_err = filter_match(match)
match = None
with pytest.raises(expected_err, match=match_err) as exc_info:
res.result()
# For remote traceback, ensure that the cause contains the original
# error
if match is not None:
with pytest.raises(_RemoteTraceback, match=match):
raise exc_info.value.__cause__
@pytest.mark.parametrize("func, args, expected_err, match", crash_cases)
def test_in_callback_submit_with_crash(
self, func, args, expected_err, match
):
"""Test the recovery from callback crash"""
executor = get_reusable_executor(max_workers=2, timeout=12)
def in_callback_submit(future):
future2 = get_reusable_executor(max_workers=2, timeout=12).submit(
func, *args
)
# Store the future of the job submitted in the callback to make it
# easy to introspect.
future.callback_future = future2
future.callback_done.set()
# Make sure the first submitted job last a bit to make sure that
# the callback will be called in the queue manager thread and not
# immediately in the main thread.
delay = 0.1
f = executor.submit(id_sleep, 42, delay)
f.callback_done = threading.Event()
f.add_done_callback(in_callback_submit)
assert f.result() == 42
if not f.callback_done.wait(timeout=3):
raise AssertionError("callback not done before timeout")
match_err = None
if expected_err is TerminatedWorkerError:
match_err = filter_match(match)
match = None
with pytest.raises(expected_err, match=match_err) as exc_info:
f.callback_future.result()
# For remote traceback, ensure that the cause contains the original
# error
if match is not None:
with pytest.raises(_RemoteTraceback, match=match):
raise exc_info.value.__cause__
def test_callback_crash_on_submit(self):
"""Errors in the callback execution directly in queue manager thread.
This case can break the process executor and we want to make sure
that we can detect the issue and recover by calling
get_reusable_executor.
"""
executor = get_reusable_executor(max_workers=2)
# Make sure the first submitted job last a bit to make sure that
# the callback will be called in the queue manager thread and not
# immediately in the main thread.
delay = 0.1
f = executor.submit(id_sleep, 42, delay)
f.add_done_callback(lambda _: exit())
assert f.result() == 42
assert executor.submit(id_sleep, 42, 0.1).result() == 42
executor = get_reusable_executor(max_workers=2)
f = executor.submit(id_sleep, 42, delay)
f.add_done_callback(lambda _: raise_error())
assert f.result() == 42
assert executor.submit(id_sleep, 42, 0.0).result() == 42
def test_deadlock_kill(self):
"""Test deadlock recovery for reusable_executor"""
executor = get_reusable_executor(max_workers=1, timeout=None)
# trigger the spawning of the worker process
executor.submit(sleep, 0.1)
worker = next(iter(executor._processes.values()))
with pytest.warns(UserWarning) as recorded_warnings:
executor = get_reusable_executor(max_workers=2, timeout=None)
assert len(recorded_warnings) == 1
expected_msg = (
"Trying to resize an executor with running jobs:"
" waiting for jobs completion before resizing."
)
assert recorded_warnings[0].message.args[0] == expected_msg
os.kill(worker.pid, SIGKILL)
wait_dead(worker)
# wait for the executor to be able to detect the issue and set itself
# in broken state:
sleep(0.5)
with pytest.raises(
TerminatedWorkerError, match=filter_match("SIGKILL")
):
executor.submit(id_sleep, 42, 0.1).result()
# the get_reusable_executor factory should be able to create a new
# working instance
executor = get_reusable_executor(max_workers=2, timeout=None)
assert executor.submit(id_sleep, 42, 0.0).result() == 42
@pytest.mark.parametrize("n_proc", [1, 2, 5, 13])
def test_crash_races(self, n_proc):
"""Test the race conditions in reusable_executor crash handling"""
pytest.importorskip("psutil") # required for kill_friend & co
if sys.platform == "win32" and n_proc > 5:
pytest.skip(
"On win32, the paging size can be too small to import numpy "
"multiple times in the sub-processes (imported when loading "
"this file). Skipping while no better solution is found. See "
"https://github.com/joblib/loky/issues/279 for more details."
)
# Test for external crash signal comming from neighbor
# with various race setup
executor = get_reusable_executor(max_workers=n_proc, timeout=None)
executor.map(id, range(n_proc)) # trigger the creation of the workers
pids = list(executor._processes.keys())
assert len(pids) == n_proc
assert None not in pids
res = executor.map(
sleep_then_check_pids_exist,
[(0.0001 * (j // 2), pids) for j in range(2 * n_proc)],
)
assert all(res)
with pytest.raises(
TerminatedWorkerError, match=filter_match("SIGKILL")
):
res = executor.map(kill_friend, pids[::-1])
list(res)
def test_imap_handle_iterable_exception(self):
# The catch of the errors in imap generation depend on the
# builded version of python
executor = get_reusable_executor(max_workers=2)
with pytest.raises(SayWhenError):
executor.map(
id_sleep, exception_throwing_generator(10, 3), chunksize=1
)
# SayWhenError seen at start of problematic chunk's results
executor = get_reusable_executor(max_workers=2)
with pytest.raises(SayWhenError):
executor.map(
id_sleep, exception_throwing_generator(20, 7), chunksize=2
)
executor = get_reusable_executor(max_workers=2)
with pytest.raises(SayWhenError):
executor.map(
id_sleep, exception_throwing_generator(20, 7), chunksize=4
)
def test_queue_full_deadlock(self):
with warnings.catch_warnings():
warnings.simplefilter("ignore", UserWarning)
# Ignore any UserWarning about the executor being resized
executor = get_reusable_executor(max_workers=1)
executor.submit(id, 42)
fs_fail = [
executor.submit(do_nothing, ErrorAtPickle(True))
for _ in range(100)
]
fs = [
executor.submit(do_nothing, ErrorAtPickle(False))
for _ in range(100)
]
with pytest.raises(PicklingError):
fs_fail[99].result()
assert fs[99].result()
def test_informative_error_when_fail_at_unpickle(self):
executor = get_reusable_executor(max_workers=2)
obj = ErrorAtUnpickle(RuntimeError, "message raised in child")
f = executor.submit(id, obj)
with pytest.raises(BrokenProcessPool) as exc_info:
f.result()
assert "RuntimeError" in str(exc_info.value.__cause__)
assert "message raised in child" in str(exc_info.value.__cause__)
@pytest.mark.skipif(np is None, reason="requires numpy")
def test_numpy_dot_parent_and_child_no_freeze(self):
"""Test that no freeze happens in child process when numpy's thread
pool is started in the parent.
"""
a = np.random.randn(1000, 1000)
np.dot(a, a) # trigger the thread pool init in the parent process
executor = get_reusable_executor(max_workers=2)
executor.submit(np.dot, a, a).result()
executor.shutdown(wait=True)
class TestTerminateExecutor(ReusableExecutorMixin):
def test_shutdown_kill(self):
"""Test reusable_executor termination handling"""
from itertools import repeat
executor = get_reusable_executor(max_workers=5)
res1 = executor.map(id_sleep, range(100), repeat(0.001))
res2 = executor.map(id_sleep, range(100), repeat(1))
assert list(res1) == list(range(100))
shutdown = TimingWrapper(executor.shutdown)
shutdown(wait=True, kill_workers=True)
assert shutdown.elapsed < 5
# We should get an error as the executor shutdowned before we fetched
# all the results from the long running operation.
with pytest.raises(ShutdownExecutorError):
list(res2)
def test_shutdown_deadlock(self):
"""Test recovery if killed after resize call"""
pytest.importorskip("psutil") # required for kill_friend & co
# Test the executor.shutdown call do not cause deadlock
executor = get_reusable_executor(max_workers=2, timeout=None)
executor.map(id, range(2)) # start the worker processes
executor.submit(
kill_friend, (next(iter(executor._processes.keys())), 0.0)
)
sleep(0.01)
executor.shutdown(wait=True)
def test_kill_workers_on_new_options(self):
# submit a long running job with no timeout
executor = get_reusable_executor(max_workers=2, timeout=None)
f = executor.submit(sleep, 10000)
# change the constructor parameter while requesting not to wait
# for the long running task to complete (the workers will get
# shutdown forcibly)
executor = get_reusable_executor(
max_workers=2, timeout=5, kill_workers=True
)
with pytest.raises(ShutdownExecutorError):
f.result()
f2 = executor.submit(id_sleep, 42, 0)
assert f2.result() == 42
@pytest.mark.parametrize(
"bad_object, match",
[(CrashAtGCInWorker, "SIGSEGV"), (CExitAtGCInWorker, r"EXIT\(0\)")],
)
def test_call_item_gc_crash_or_exit(self, bad_object, match):
executor = get_reusable_executor(max_workers=1)
bad_object = bad_object()
f = executor.submit(id, bad_object)
# The worker will successfully send back its result to the master
# process before crashing so this future can always be collected:
assert f.result() is not None
# The executor should automatically detect that the worker has crashed
# when processing subsequently dispatched tasks:
with pytest.raises(TerminatedWorkerError, match=filter_match(match)):
executor.submit(gc.collect).result()
for _ in executor.map(sleep, [0.1] * 100):
pass
def test_sigkill_shutdown_leaks_workers(self):
# Create a parent process which will report its workers pids
# and sigkill itself.
code = """if True:
import os
import loky
import psutil
parent_pid = os.getpid()
with loky.get_reusable_executor(timeout=1, kill_workers=True) as p:
list(p.map(lambda x: x, range(100)))
for pid in p._processes.keys():
print(f'worker_pid:{pid}')
print(f'parent_pid:{parent_pid}')
psutil.Process(os.getpid()).kill()
"""
with NamedTemporaryFile(
mode="w", suffix="_joblib.py", delete=True
) as f:
f.write(code)
f.flush()
cmd = [sys.executable, f.name]
out = subprocess.run(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
text=True,
timeout=60,
)
running_workers = []
for line in out.stdout.split("\n"):
if line.startswith("worker_pid"):
worker_pid = int(line.split(":")[1])
try:
psutil.Process(worker_pid)
running_workers.append(worker_pid)
except psutil.NoSuchProcess:
pass
assert not len(
running_workers
), f"There are running workers left: {running_workers}"
class TestResizeExecutor(ReusableExecutorMixin):
def test_reusable_executor_resize(self):
"""Test reusable_executor resizing"""
# required by sleep_then_check_pids_exist
pytest.importorskip("psutil")
executor = get_reusable_executor(max_workers=2, timeout=None)
executor.map(id, range(2))
# Decreasing the executor should drop a single process and keep one of
# the old one as it is still in a good shape. The resize should not
# occur while there are on going works.
pids = list(executor._processes.keys())
res1 = executor.submit(sleep_then_check_pids_exist, (0.3, pids))
clean_warning_registry()
with warnings.catch_warnings(record=True) as w:
# Cause all warnings to always be triggered.
warnings.simplefilter("always")
executor = get_reusable_executor(max_workers=1, timeout=None)
assert len(w) == 1
expected_msg = "Trying to resize an executor with running jobs"
assert expected_msg in str(w[0].message)
assert res1.result(), (
"Resize should wait for current processes " " to finish"
)
assert len(executor._processes) == 1
assert next(iter(executor._processes.keys())) in pids
# Requesting the same number of process should not impact the executor
# nor kill the processed
old_pid = next(iter(executor._processes.keys()))
unchanged_executor = get_reusable_executor(max_workers=1, timeout=None)
assert len(unchanged_executor._processes) == 1
assert unchanged_executor is executor
assert next(iter(unchanged_executor._processes.keys())) == old_pid
# Growing the executor again should add a single process and keep the
# old one as it is still in a good shape
executor = get_reusable_executor(max_workers=2, timeout=None)
assert len(executor._processes) == 2
assert old_pid in list(executor._processes.keys())
@pytest.mark.parametrize("reuse", [True, False])
@pytest.mark.parametrize("kill_workers", [True, False])
def test_reusable_executor_resize_many_times(self, kill_workers, reuse):
# Tentative non-regression test for a deadlock when shutting down
# the workers of an executor prior to resizing it.
kwargs = {
"timeout": None,
"kill_workers": kill_workers,
"reuse": reuse,
}
with warnings.catch_warnings(record=True):
# Cause all warnings to always be triggered.
warnings.simplefilter("always")
for size in [12, 2, 1, 12, 6, 1, 8, 5]:
executor = get_reusable_executor(max_workers=size, **kwargs)
executor.map(sleep, [0.01] * 6)
# Do not wait for the tasks to complete.
executor.shutdown()
def test_kill_after_resize_call(self):
"""Test recovery if killed after resize call"""
pytest.importorskip("psutil") # required for kill_friend & co
# Test the executor resizing called before a kill arrive
executor = get_reusable_executor(max_workers=2, timeout=None)
executor.map(id, range(2)) # trigger the creation of worker processes
pid = next(iter(executor._processes.keys()))
executor.submit(kill_friend, (pid, 0.1))
with pytest.warns(UserWarning) as recorded_warnings:
warnings.simplefilter("always")
executor = get_reusable_executor(max_workers=1, timeout=None)
assert len(recorded_warnings) == 1
expected_msg = (
"Trying to resize an executor with running jobs:"
" waiting for jobs completion before resizing."
)
assert recorded_warnings[0].message.args[0] == expected_msg
assert executor.submit(id_sleep, 42, 0.0).result() == 42
executor.shutdown()
def test_resize_after_timeout(self):
with warnings.catch_warnings(record=True) as recorded_warnings:
warnings.simplefilter("always")
executor = get_reusable_executor(max_workers=2, timeout=0.001)
assert executor.submit(id_sleep, 42, 0.0).result() == 42
sleep(0.1)
executor = get_reusable_executor(max_workers=8, timeout=0.001)
assert executor.submit(id_sleep, 42, 0.0).result() == 42
sleep(0.1)
executor = get_reusable_executor(max_workers=2, timeout=0.001)
assert executor.submit(id_sleep, 42, 0.0).result() == 42
if len(recorded_warnings) > 1:
expected_msg = "A worker stopped"
assert expected_msg in recorded_warnings[0].message.args[0]
class TestGetReusableExecutor(ReusableExecutorMixin):
def test_invalid_process_number(self):
"""Raise error on invalid process number"""
with pytest.raises(ValueError):
get_reusable_executor(max_workers=0)
with pytest.raises(ValueError):
get_reusable_executor(max_workers=-1)
executor = get_reusable_executor()
with pytest.raises(ValueError):
executor._resize(max_workers=None)
@pytest.mark.skipif(sys.platform == "win32", reason="No fork on windows")
def test_invalid_context(self):
"""Raise error on invalid context"""
with pytest.warns(UserWarning):
with pytest.raises(ValueError):
get_reusable_executor(max_workers=2, context="fork")
def test_pass_start_method_name_as_context(self):
executor = get_reusable_executor(max_workers=2, context="loky")
assert executor.submit(id, 42).result() >= 0
with pytest.raises(ValueError):
get_reusable_executor(max_workers=2, context="bad_start_method")
def test_interactively_defined_executor_no_main(self):
# check that the init_main_module parameter works properly
# when using -c option, we don't need the safeguard if __name__ ..
# and thus test LokyProcess without the extra argument. For running
# a script, it is necessary to use init_main_module=False.
code = """if True:
from loky import get_reusable_executor
e = get_reusable_executor()
e.submit(id, 42).result()
print("ok")
"""
check_python_subprocess_call(code, stdout_regex="ok")
def test_reused_flag(self):
executor, _ = _ReusablePoolExecutor.get_reusable_executor(
max_workers=2
)
executor, reused = _ReusablePoolExecutor.get_reusable_executor(
max_workers=2
)
assert reused
executor.shutdown(kill_workers=True)
executor, reused = _ReusablePoolExecutor.get_reusable_executor(
max_workers=2
)
assert not reused
@pytest.mark.xfail(
cloudpickle_version >= Version("0.5.4")
and cloudpickle_version <= Version("0.7.0"),
reason="Known issue in cloudpickle",
)
# https://github.com/cloudpipe/cloudpickle/pull/240
def test_interactively_defined_nested_functions(self):
# Check that it's possible to call nested interactively defined
# functions and furthermore that changing the code interactively
# is taken into account by the single worker process.
code = """if True:
from loky import get_reusable_executor
e = get_reusable_executor(max_workers=1)
# Force a start of the children process:
e.submit(id, 42).result()
# Test that it's possible to call interactively defined, nested
# functions:
def inner_func(x):
return -x
def outer_func(x):
return inner_func(x)
assert e.submit(outer_func, 1).result() == outer_func(1) == -1
# Test that changes to the definition of the inner function are
# taken into account in subsequent calls to the outer function.
def inner_func(x):
return x
assert e.submit(outer_func, 1).result() == outer_func(1) == 1
print("ok")
"""
check_python_subprocess_call(code, stdout_regex="ok")
def test_interactively_defined_recursive_functions(self):
# Check that it's possible to call a recursive function defined
# in a closure.
# Also check that calling several function that stems from the same
# factory with different closure states results in the expected result:
# the function definitions should not collapse in the single worker
# process.
code = """if True:
from loky import get_reusable_executor
e = get_reusable_executor(max_workers=1)
# Force a start of the children process:
e.submit(id, 42).result()
def make_func(seed):
def func(x):
if x <= 0:
return seed
return func(x - 1) + 1
return func
func = make_func(0)
assert e.submit(func, 5).result() == func(5) == 5
func = make_func(1)
assert e.submit(func, 5).result() == func(5) == 6
print("ok")
"""
check_python_subprocess_call(code, stdout_regex="ok")
def test_no_deadlock_on_nested_reusable_exector(self):
# Non-regression test for a deadlock that occurred when the main
# process was waiting for level 1 children processes to terminate while
# they had reusable executors still managing level 2 worker processes
# waiting for more work to arrive (untill their long worker timeout).
# https://github.com/joblib/loky/issues/363
code = """if True:
from loky import get_reusable_executor
import logging
import multiprocessing as mp
from time import sleep
import faulthandler
def enable_mp_logging():
# Enable verbose logging for nested loky executor and workers in
# order to make this test easier to debug in case of failure.
log = mp.util.log_to_stderr(logging.DEBUG)
log.handlers[0].setFormatter(logging.Formatter(
'[%(levelname)s:%(processName)s:%(threadName)s] %(message)s'))
return log
def inner_parallel_func(j):
sleep(0.1) # to avoid termination of parent before all workers
mp.util.debug(f"inner_parallel_func({j}) done")
# If this test fails, uncommenting the following line will
# help to debug the issue:
# faulthandler.dump_traceback_later(10, exit=True)
return j ** 2
def outer_parallel_func(i):
executor = get_reusable_executor(max_workers=2)
list(executor.map(inner_parallel_func, range(2)))
enable_mp_logging()
mp.util.debug(f"outer_parallel_func({i}) done")
enable_mp_logging()
executor = get_reusable_executor(max_workers=2)
list(executor.map(outer_parallel_func, range(2)))
print("ok")
"""
check_python_subprocess_call(code, stdout_regex="ok", timeout=30)
def test_compat_with_concurrent_futures_exception(self):
# It should be possible to use a loky process pool executor as a dropin
# replacement for a ProcessPoolExecutor, including when catching
# exceptions:
concurrent = pytest.importorskip("concurrent")
from concurrent.futures.process import BrokenProcessPool as BPPExc
with pytest.raises(BPPExc):
get_reusable_executor(max_workers=2).submit(crash).result()
e = get_reusable_executor(max_workers=2)
f = e.submit(id, 42)
# Ensure that loky.Future are compatible with concurrent.futures
# (see #155)
assert isinstance(f, concurrent.futures.Future)
_, running = concurrent.futures.wait([f], timeout=15)
assert len(running) == 0
thread_configurations = [
("constant", "clean_start"),
("constant", "broken_start"),
("varying", "clean_start"),
("varying", "broken_start"),
]
@pytest.mark.parametrize("workers, executor_state", thread_configurations)
def test_reusable_executor_thread_safety(self, workers, executor_state):
if executor_state == "clean_start":
# Create a new shared executor and ensures that it's workers are
# ready:
get_reusable_executor(reuse=False).submit(id, 42).result()
else:
# Break the shared executor before launching the threads:
with pytest.raises(
TerminatedWorkerError, match=filter_match("SIGSEGV")
):
executor = get_reusable_executor(reuse=False)
executor.submit(return_instance, CrashAtPickle).result()
def helper_func(
output_collector, max_workers=2, n_outer_steps=5, n_inner_steps=10
):
with warnings.catch_warnings(): # ignore resize warnings
warnings.simplefilter("always")
executor = get_reusable_executor(max_workers=max_workers)
for _ in range(n_outer_steps):
results = executor.map(
lambda x: x**2, range(n_inner_steps)
)
expected_result = [x**2 for x in range(n_inner_steps)]
assert list(results) == expected_result
output_collector.append("ok")
if workers == "constant":
max_workers = [2] * 10
else:
max_workers = [(i % 4) + 1 for i in range(10)]
# Use the same executor with the same number of workers concurrently
# in different threads:
output_collector = []
threads = [
threading.Thread(
target=helper_func,
args=(output_collector, w),
name=f"test_thread_{i:02d}_max_workers_{w:d}",
)
for i, w in enumerate(max_workers)
]
with warnings.catch_warnings(record=True):
for t in threads:
t.start()
for t in threads:
t.join()
assert output_collector == ["ok"] * len(threads)
def test_reusable_executor_submit_during_shutdown_race(self):
resolved = threading.Event()
barrier = threading.Barrier(2)
results = []
errors = []
def submit_on_resolved_executor():
try:
executor = get_reusable_executor(max_workers=1, env={"a": "1"})
resolved.set()
barrier.wait(timeout=10)
results.append(
executor.submit(id_sleep, 1, 0).result(timeout=10)
)
except Exception as e:
errors.append(e)
thread = threading.Thread(target=submit_on_resolved_executor)
thread.start()
assert resolved.wait(timeout=10)
get_reusable_executor(max_workers=1, env={"a": "2"})
barrier.wait(timeout=10)
thread.join(timeout=10)
assert not thread.is_alive()
assert not any(isinstance(e, ShutdownExecutorError) for e in errors)
if errors:
raise errors[0]
assert results == [1]
@pytest.mark.parametrize("iteration", range(2))
def test_reusable_executor_submit_with_concurrent_env_changes(
self, iteration
):
def submit_with_env(i):
executor = get_reusable_executor(
max_workers=1, env={"a": f"{iteration}-{i}"}
)
return executor.submit(id_sleep, i, 0).result(timeout=10)
n_submissions = 20
with ThreadPoolExecutor(max_workers=10) as executor:
results = list(executor.map(submit_with_env, range(n_submissions)))
assert results == list(range(n_submissions))
def test_max_memory_leak_size_env_var(self):
pytest.importorskip("psutil") # cannot work without psutil
def _leak_some_memory(size=int(3e6), delay=0.001):
"""function that leaks some memory"""
from loky import process_executor
process_executor._MEMORY_LEAK_CHECK_DELAY = 0.1
if getattr(os, "_loky_leak", None) is None:
os._loky_leak = []
os._loky_leak.append(b"\x00" * size)
sleep(delay)
return os.getpid()
# These submissions leak far more than the 300 MB default, so the
# worker only survives them if the raised threshold reaches the worker.
executor = get_reusable_executor(
max_workers=1, env={"LOKY_MAX_MEMORY_LEAK_SIZE": str(int(2e9))}
)
with warnings.catch_warnings(record=True) as record:
warnings.simplefilter("always")
futures = [executor.submit(_leak_some_memory) for _ in range(300)]
pids = {f.result() for f in futures}
executor.shutdown(wait=True)
assert len(pids) == 1, "the worker was recycled despite the threshold"
assert not [w for w in record if "memory leak" in str(w.message)]
def test_reusable_executor_reuse_true(self):
executor = get_reusable_executor(max_workers=3, timeout=42)
executor.submit(id, 42).result()
assert len(executor._processes) == 3
assert executor._timeout == 42
executor2 = get_reusable_executor(reuse=True)
executor2.submit(id, 42).result()
assert len(executor2._processes) == 3
assert executor2._timeout == 42
assert executor2 is executor
executor3 = get_reusable_executor()
executor3.submit(id, 42).result()
assert len(executor3._processes) == cpu_count()
assert executor3._timeout == 10
assert executor3 is not executor
executor4 = get_reusable_executor()
assert executor4 is executor3
def test_no_crash_on_stdin_access_in_child_subprocess_run(self):
# non-regression test for #420.
def call_subprocess():
sleep(0.01)
subprocess.run(
[sys.executable, "-c", "import sys; sys.stdin.read(0)"]
).check_returncode()
executor = get_reusable_executor(max_workers=4, timeout=2)
executor.submit(call_subprocess).result()
def test_faulthandler_enabled(self):
cmd = """if 1:
from loky import get_reusable_executor
import faulthandler
def f(i):
if {expect_enabled}:
assert faulthandler.is_enabled()
else:
assert not faulthandler.is_enabled()
if i == 5:
faulthandler._sigsegv()
if {enable_faulthandler_via_initializer}:
executor = get_reusable_executor(max_workers=2, initializer=faulthandler.enable)
else:
executor = get_reusable_executor(max_workers=2)
list(executor.map(f, range(10)))
# This should always trigger a crash.
"""
def check_faulthandler_output(
expect_enabled=True, enable_faulthandler_via_initializer=False
):
p = subprocess.Popen(
[
sys.executable,
"-c",
cmd.format(
expect_enabled=expect_enabled,
enable_faulthandler_via_initializer=enable_faulthandler_via_initializer,
),
],
stderr=subprocess.PIPE,
stdout=subprocess.PIPE,
)
p.wait()
out, err = p.communicate()
# The worker is always expected to crash, irrespective of whether
# faulthandler is enabled or not.
assert p.returncode == 1, out.decode()
if expect_enabled:
assert b"Current thread" in err, err.decode()
original_pythonfaulthandler_env = os.environ.pop(
"PYTHONFAULTHANDLER", None
)
try:
# Case 1: faulthandler should be automatically enabled by default.
check_faulthandler_output(expect_enabled=True)
# Case 2: faulthandler should also be enabled when
# PYTHONFAULTHANDLER=1 is set.
os.environ["PYTHONFAULTHANDLER"] = "1"
check_faulthandler_output(expect_enabled=True)
# Case 3: faulthandler should not be enabled when
# PYTHONFAULTHANDLER=0 is set explicitly.
os.environ["PYTHONFAULTHANDLER"] = "0"
check_faulthandler_output(expect_enabled=False)
# Case 4: faulthandler can also be enabled manually via the initializer.
del os.environ["PYTHONFAULTHANDLER"]
check_faulthandler_output(
expect_enabled=True, enable_faulthandler_via_initializer=True
)
finally:
if original_pythonfaulthandler_env is None:
os.environ.pop("PYTHONFAULTHANDLER", None)
else:
os.environ["PYTHONFAULTHANDLER"] = (
original_pythonfaulthandler_env
)
@pytest.mark.parametrize("n_cpus", [1, 2, 4])
def test_reusable_executor_queue_size(self, n_cpus):
# Check that the reusable executor can be lauched with more workers
# than the number of CPUs with no limitation on the queue_size.
#
# Non-regression test for https://github.com/joblib/loky/issues/396.
get_reusable_executor().shutdown()
import os
original_n_cpus = os.environ.get("LOKY_MAX_CPU_COUNT", None)
try:
os.environ["LOKY_MAX_CPU_COUNT"] = f"{n_cpus}"
n_workers = 2 * n_cpus + 2
manager = Manager()
barrier = manager.Barrier(n_workers, timeout=1)
def check_running(i, barrier):
print(f"TASK {i} running")
barrier.wait()
return True
executor = get_reusable_executor(max_workers=n_workers)
futures = [
executor.submit(check_running, i, barrier)
for i in range(n_workers)
]
assert all(r.result() for r in futures)
finally:
manager.shutdown()
executor.shutdown()
if original_n_cpus is None:
os.environ.pop("LOKY_MAX_CPU_COUNT", None)
else:
os.environ["LOKY_MAX_CPU_COUNT"] = original_n_cpus
class TestExecutorInitializer(ReusableExecutorMixin):
def _initializer(self, x):
loky._initialized_state = x
def _test_initializer(self, delay=0):
sleep(delay)
return getattr(loky, "_initialized_state", "uninitialized")
def test_reusable_initializer(self):
executor = get_reusable_executor(
max_workers=2, initializer=self._initializer, initargs=("done",)
)
assert executor.submit(self._test_initializer).result() == "done"
# when the initializer change, the executor is re-spawned
executor = get_reusable_executor(
max_workers=2, initializer=self._initializer, initargs=(42,)
)
assert executor.submit(self._test_initializer).result() == 42
# With reuse=True, the executor use the same initializer
executor = get_reusable_executor(max_workers=4, reuse=True)
for x in executor.map(self._test_initializer, delay=0.1):
assert x == 42
# With reuse='auto', the initializer is not used anymore
executor = get_reusable_executor(max_workers=4)
for x in executor.map(self._test_initializer, delay=0.1):
assert x == "uninitialized"
# XXX: not sure if #369 has been fixed or not. Let's not make it XFAIL
# for now to see if it is still flaky on CI.
# @pytest.mark.xfail(reason="https://github.com/joblib/loky/issues/369")
def test_error_in_nested_call_keeps_resource_tracker_silent(self):
# Safety smoke test: test that nested parallel calls don't yield noisy
# resource_tracker outputs when the grandchild errors out.
cmd = """if 1:
from loky import get_reusable_executor
def raise_error(i):
raise ValueError
def nested_loop(f):
executor = get_reusable_executor(max_workers=2)
list(executor.map(f, range(10))
executor = get_reusable_executor(max_workers=2)
list(executor.map(nested_loop, [raise_error])
"""
p = subprocess.Popen(
[sys.executable, "-c", cmd],
stderr=subprocess.PIPE,
stdout=subprocess.PIPE,
)
p.wait()
out, err = p.communicate()
assert p.returncode == 1, out.decode()
assert b"resource_tracker" not in err, err.decode()
def test_no_crash_max_workers_on_windows():
# Check that loky's reusable process pool executor does not crash when the
# user asks for more workers than the maximum number of workers supported
# by the platform.
# To make sure we have the proper size for the call_queue, we
# shutdown the existing executor.
# See https://github.com/joblib/loky#396
get_reusable_executor().shutdown()
# Note: on overloaded CI hosts, spawning many processes can take a long
# time. We need to increase the timeout to avoid spurious failures when
# making assertions on `len(executor._processes)`.
idle_worker_timeout = 5 * 60
with warnings.catch_warnings(record=True) as record:
executor = get_reusable_executor(
max_workers=_MAX_WINDOWS_WORKERS + 1,
timeout=idle_worker_timeout,
)
assert executor.submit(lambda: None).result() is None
if sys.platform == "win32":
assert len(record) == 1
assert "max_workers" in str(record[0].message)
assert len(executor._processes) == _MAX_WINDOWS_WORKERS
else:
assert len(record) == 0
assert len(executor._processes) == _MAX_WINDOWS_WORKERS + 1
# Downsizing should never raise a warning.
before_downsizing_executor = executor
with warnings.catch_warnings(record=True) as record:
executor = get_reusable_executor(
max_workers=_MAX_WINDOWS_WORKERS, timeout=idle_worker_timeout
)
assert executor.submit(lambda: None).result() is None
# No warning on any OS when max_workers does not exceed the limit.
assert len(record) == 0, [w.message for w in record]
assert before_downsizing_executor is executor
assert len(executor._processes) == _MAX_WINDOWS_WORKERS
executor.shutdown()
def test_thread_local() -> None:
"""``get_reusable_executor()`` returns a thread-local executor."""
def in_thread():
executors = set()
for _ in range(100):
executors.add(id(get_reusable_executor()))
assert len(executors) == 1
executor_ids.add(executors.pop())
executor_ids = set()
threads = []
for _ in range(10):
t = threading.Thread(target=in_thread)
threads.append(t)
t.start()
for t in threads:
t.join()
# Should have 10 different IDs:
assert len(executor_ids) == 10
joblib-loky-51247f4/tests/test_synchronize.py 0000664 0000000 0000000 00000024227 15256713413 0021351 0 ustar 00root root 0000000 0000000 import os
import sys
import time
import pytest
import signal
import threading
from loky.backend import get_context
from .utils import TimingWrapper
loky_context = get_context("loky")
DELTA = 0.1
TIMEOUT1 = 0.1
TIMEOUT2 = 0.3
@pytest.mark.skipif(sys.platform == "win32", reason="UNIX test")
def test_semlock_failure():
from loky.backend.synchronize import SemLock, sem_unlink
name = "loky-test-semlock"
try:
sem_unlink(name)
except FileNotFoundError:
pass
try:
sl = SemLock(0, 1, 1, name=name)
assert sl.name == name
with pytest.raises(FileExistsError):
SemLock(0, 1, 1, name=name)
finally:
# Always clean-up the test semaphore to make this test independent of
# previous runs (successful or not).
sem_unlink(name)
with pytest.raises(FileNotFoundError):
sl._semlock._rebuild(0, 0, 0, name)
def assert_sem_value_equal(sem, value):
try:
assert sem.get_value() == value
except NotImplementedError:
pass
def assert_timing_almost_equal(t1, t2=0):
assert abs(t1 - t2) < 2e-1
class TestLock:
def test_lock(self):
lock = loky_context.Lock()
assert lock.acquire()
assert not lock.acquire(False)
assert lock.release() is None
with pytest.raises(ValueError) as excinfo:
lock.release()
assert "released too many times" in str(excinfo.value)
def test_rlock(self):
lock = loky_context.RLock()
assert lock.acquire()
assert lock.acquire()
assert lock.acquire()
assert lock.release() is None
assert lock.release() is None
assert lock.release() is None
with pytest.raises(AssertionError) as excinfo:
lock.release()
assert "not owned by thread" in str(excinfo.value)
def test_lock_context(self):
with loky_context.Lock():
pass
class TestSemaphore:
def _test_semaphore(self, sem):
assert_sem_value_equal(sem, 2)
assert sem.acquire()
assert_sem_value_equal(sem, 1)
assert sem.acquire()
assert_sem_value_equal(sem, 0)
assert not sem.acquire(False)
assert_sem_value_equal(sem, 0)
assert sem.release() is None
assert_sem_value_equal(sem, 1)
assert sem.release() is None
assert_sem_value_equal(sem, 2)
def test_semaphore(self):
sem = loky_context.Semaphore(2)
self._test_semaphore(sem)
assert sem.release() is None
assert_sem_value_equal(sem, 3)
assert sem.release() is None
assert_sem_value_equal(sem, 4)
@pytest.mark.skipif(
sys.platform == "darwin", reason="OSX have borken `get_value`"
)
def test_bounded_semaphore(self):
sem = loky_context.BoundedSemaphore(2)
self._test_semaphore(sem)
with pytest.raises(ValueError):
sem.release()
assert_sem_value_equal(sem, 2)
def test_timeout(self):
sem = loky_context.Semaphore(0)
acquire = TimingWrapper(sem.acquire)
assert not acquire(False)
assert_timing_almost_equal(acquire.elapsed)
assert not acquire(False, None)
assert_timing_almost_equal(acquire.elapsed)
assert not acquire(False, TIMEOUT1)
assert_timing_almost_equal(acquire.elapsed)
assert not acquire(True, TIMEOUT1)
assert_timing_almost_equal(acquire.elapsed, TIMEOUT1)
assert not acquire(True, TIMEOUT2)
assert_timing_almost_equal(acquire.elapsed, TIMEOUT2)
class TestCondition:
@classmethod
def _test_notify(cls, cond, sleeping, woken, timeout=None):
cond.acquire()
sleeping.release()
cond.wait(timeout)
woken.release()
cond.release()
def check_invariant(self, cond):
# this is only supposed to succeed when there are no sleepers
try:
sleepers = (
cond._sleeping_count.get_value()
- cond._woken_count.get_value()
)
sleepers == 0
cond._wait_semaphore.get_value() == 0
except NotImplementedError:
pass
def test_notify(self):
cond = loky_context.Condition()
sleeping = loky_context.Semaphore(0)
woken = loky_context.Semaphore(0)
p = loky_context.Process(
target=self._test_notify, args=(cond, sleeping, woken)
)
p.daemon = True
p.start()
p = threading.Thread(
target=self._test_notify, args=(cond, sleeping, woken)
)
p.daemon = True
p.start()
# wait for both children to start sleeping
sleeping.acquire()
sleeping.acquire()
# check no process/thread has woken up
time.sleep(DELTA)
assert_sem_value_equal(woken, 0)
# wake up one process/thread
cond.acquire()
cond.notify()
cond.release()
# check one process/thread has woken up
time.sleep(DELTA)
assert_sem_value_equal(woken, 1)
# wake up another
cond.acquire()
cond.notify()
cond.release()
# check other has woken up
time.sleep(DELTA)
assert_sem_value_equal(woken, 2)
# check state is not mucked up
self.check_invariant(cond)
p.join()
def test_notify_all(self):
cond = loky_context.Condition()
sleeping = loky_context.Semaphore(0)
woken = loky_context.Semaphore(0)
# start some threads/processes which will timeout
for _ in range(3):
p = loky_context.Process(
target=self._test_notify,
args=(cond, sleeping, woken, TIMEOUT1),
)
p.daemon = True
p.start()
t = threading.Thread(
target=self._test_notify,
args=(cond, sleeping, woken, TIMEOUT1),
)
t.daemon = True
t.start()
# wait for them all to sleep
for _ in range(6):
sleeping.acquire()
# check they have all timed out
for _ in range(6):
woken.acquire()
assert_sem_value_equal(woken, 0)
# check state is not mucked up
self.check_invariant(cond)
# start some more threads/processes
for _ in range(3):
p = loky_context.Process(
target=self._test_notify, args=(cond, sleeping, woken)
)
p.daemon = True
p.start()
t = threading.Thread(
target=self._test_notify, args=(cond, sleeping, woken)
)
t.daemon = True
t.start()
# wait for them to all sleep
for _ in range(6):
sleeping.acquire()
# check no process/thread has woken up
time.sleep(DELTA)
assert_sem_value_equal(woken, 0)
# wake them all up
cond.acquire()
cond.notify_all()
cond.release()
# check they have all woken
for _ in range(50):
try:
if woken.get_value() == 6:
break
except NotImplementedError:
break
time.sleep(DELTA)
assert_sem_value_equal(woken, 6)
# check state is not mucked up
self.check_invariant(cond)
def test_timeout(self):
cond = loky_context.Condition()
wait = TimingWrapper(cond.wait)
cond.acquire()
res = wait(TIMEOUT1)
cond.release()
assert not res
assert abs(wait.elapsed - TIMEOUT1) < 1e-1
@classmethod
def _test_waitfor_f(cls, cond, state):
with cond:
state.release()
cond.notify()
result = cond.wait_for(lambda: state.get_value() == 5)
if not result or state.get_value() != 5:
sys.exit(1)
def test_waitfor(self):
# based on test in test/lock_tests.py
cond = loky_context.Condition()
state = loky_context.Semaphore(0)
try:
state.get_value()
except NotImplementedError:
pytest.skip("`sem_get_value not implemented")
p = loky_context.Process(
target=self._test_waitfor_f, args=(cond, state)
)
p.daemon = True
p.start()
with cond:
result = cond.wait_for(lambda: state.get_value() == 1)
assert result
assert state.get_value() == 1
for _ in range(4):
time.sleep(0.01)
with cond:
state.release()
cond.notify()
p.join(5)
assert not p.is_alive()
assert p.exitcode == 0
@classmethod
def _test_wait_result(cls, c, pid):
with c:
c.notify()
time.sleep(1)
if pid is not None:
os.kill(pid, signal.SIGINT)
def test_wait_result(self):
if sys.platform != "win32":
pid = os.getpid()
else:
pid = None
c = loky_context.Condition()
with c:
assert not c.wait(0)
assert not c.wait(0.1)
p = loky_context.Process(
target=self._test_wait_result, args=(c, pid)
)
p.start()
assert c.wait(10)
if pid is not None:
with pytest.raises(KeyboardInterrupt):
c.wait(10)
p.join()
class TestEvent:
@classmethod
def _test_event(cls, event):
time.sleep(TIMEOUT1)
event.set()
def test_event(self):
event = loky_context.Event()
wait = TimingWrapper(event.wait)
assert not event.is_set()
assert not wait(0.0)
assert_timing_almost_equal(wait.elapsed)
assert not wait(TIMEOUT1)
assert_timing_almost_equal(wait.elapsed, TIMEOUT1)
event.set()
assert event.is_set()
assert wait()
assert_timing_almost_equal(wait.elapsed)
assert wait(TIMEOUT1)
assert_timing_almost_equal(wait.elapsed)
event.clear()
assert not event.is_set()
p = loky_context.Process(target=self._test_event, args=(event,))
p.daemon = True
p.start()
assert wait()
p.join()
joblib-loky-51247f4/tests/test_worker_timeout.py 0000664 0000000 0000000 00000010533 15256713413 0022050 0 ustar 00root root 0000000 0000000 import sys
import time
import pytest
import threading
import warnings
import multiprocessing as mp
from loky.backend import get_context
from loky import ProcessPoolExecutor
from loky import get_reusable_executor
from loky.backend.queues import SimpleQueue
from loky.backend.reduction import dumps
class SlowlyPickling:
"""Simulate slowly pickling object, e.g. large numpy array or dict"""
def __init__(self, delay=1):
self.delay = delay
def __reduce__(self):
time.sleep(self.delay)
return SlowlyPickling, (self.delay,)
class DelayedSimpleQueue(SimpleQueue):
def __init__(self, reducers=None, ctx=None, delay=0.1):
super().__init__(reducers=reducers, ctx=ctx)
self.out_queue = SimpleQueue(ctx=ctx)
self._inner_reader = self._reader
self._reader = self.out_queue._reader
self._start_thread(delay)
# Overload _start_thread to correctly call our custom _feed
def _start_thread(self, delay):
self._thread = threading.Thread(
target=DelayedSimpleQueue._feed,
args=(
self._rlock,
self._inner_reader,
self.out_queue._writer,
delay,
),
name="QueueDeFeederThread",
)
self._thread.daemon = True
self._thread.start()
def get(self):
return self.out_queue.get()
def close(self):
self.put(None)
self._thread.join()
@staticmethod
def _feed(readlock, reader, writer, delay):
PICKLE_NONE = dumps(None)
while True:
with readlock:
res = reader.recv_bytes()
if res == PICKLE_NONE:
break
time.sleep(delay)
writer.send_bytes(res)
mp.util.debug("Defeeder clean exit")
class TestTimeoutExecutor:
def test_worker_timeout_mock(self):
timeout = 0.001
context = get_context()
executor = ProcessPoolExecutor(
max_workers=4, context=context, timeout=timeout
)
result_queue = DelayedSimpleQueue(ctx=context, delay=0.001)
executor._result_queue = result_queue
with pytest.warns(
UserWarning, match="^A worker stopped while some jobs"
):
for _ in range(5):
# Trigger worker spawn for lazy executor implementations
for _ in executor.map(id, range(8)):
pass
executor.shutdown()
result_queue.close()
def test_worker_timeout_with_slowly_pickling_objects(self, n_tasks=5):
"""Check that the worker timeout can be low without deadlocking
In particular if dispatching call items to the queue is slow because of
pickling large arguments, the executor should ensure that there is an
appropriate amount of workers to move one and not get stalled.
"""
with pytest.warns(
UserWarning, match="^A worker stopped while some jobs"
):
for timeout, delay in [
(0.01, 0.02),
(0.01, 0.1),
(0.1, 0.1),
(0.001, 0.1),
]:
executor = get_reusable_executor(
max_workers=2, timeout=timeout
)
# First make sure the executor is started
executor.submit(id, 42).result()
results = list(
executor.map(id, [SlowlyPickling(delay)] * n_tasks)
)
assert len(results) == n_tasks
def test_worker_timeout_shutdown_no_deadlock(self):
"""Check that worker timeout don't cause deadlock when shutting down."""
with warnings.catch_warnings(record=True) as record:
with get_reusable_executor(max_workers=2, timeout=0.001) as e:
# First make sure the executor is started
e.submit(id, 42).result()
# Then give a task that will take more time to be pickled so
# we are sure that the worker timeout.
f = e.submit(id, SlowlyPickling(1))
f.result()
# We can expect to observe warnings related to worker timeout while
# there are still pending tasks but this should not cause a deadlock.
for w in record:
msg = w.message.args[0]
assert "A worker stopped while some jobs" in msg
joblib-loky-51247f4/tests/utils.py 0000664 0000000 0000000 00000013357 15256713413 0017101 0 ustar 00root root 0000000 0000000 import re
import os
import sys
import time
import pytest
import warnings
import threading
import subprocess
import contextlib
from tempfile import mkstemp, mkdtemp, NamedTemporaryFile, _RandomNameSequence
from _multiprocessing import SemLock as _SemLock
from _multiprocessing import sem_unlink
from loky.backend import resource_tracker
_rand_name = _RandomNameSequence()
def resource_unlink(name, rtype):
resource_tracker._CLEANUP_FUNCS[rtype](name)
def create_resource(rtype):
if rtype == "folder":
return mkdtemp(dir=os.getcwd())
elif rtype == "semlock":
name = f"test-loky-{os.getpid()}-{next(_rand_name)}"
_SemLock(1, 1, 1, name, False)
return name
elif rtype == "file":
tmpfile = NamedTemporaryFile(delete=False)
tmpfile.close()
return tmpfile.name
else:
raise ValueError(f"Resource type {rtype} not understood")
def resource_exists(name, rtype):
if rtype in ["folder", "file"]:
return os.path.exists(name)
elif rtype == "semlock":
try:
_SemLock(1, 1, 1, name, False)
sem_unlink(name)
return False
except OSError:
return True
else:
raise ValueError(f"Resource type {rtype} not understood")
@contextlib.contextmanager
def captured_output(stream_name):
"""Return a context manager used by captured_stdout/stdin/stderr
that temporarily replaces the sys stream *stream_name* with a StringIO."""
import io
orig_stdout = getattr(sys, stream_name)
s = io.StringIO()
setattr(sys, stream_name, s)
try:
yield getattr(sys, stream_name)
finally:
setattr(sys, stream_name, orig_stdout)
def captured_stderr():
"""Capture the output of sys.stderr:
with captured_stderr() as stderr:
print("hello", file=sys.stderr)
self.assertEqual(stderr.getvalue(), "hello\\n")
"""
return captured_output("stderr")
#
# Wrapper
#
class TimingWrapper:
def __init__(self, func):
self.func = func
self.elapsed = None
def __call__(self, *args, **kwds):
t = time.time()
try:
return self.func(*args, **kwds)
finally:
self.elapsed = time.time() - t
def assert_timing_lower_than(self, delay):
msg = (
f"expected duration lower than {delay:.3f}s, "
f"got {self.elapsed:.3f}s"
)
assert self.elapsed < delay, msg
def assert_timing_almost_zero(self):
self.assert_timing_lower_than(0.1)
#
# helper functions
#
def id_sleep(x, delay=0):
"""sleep for delay seconds and return its first argument"""
time.sleep(delay)
return x
def check_subprocess_call(
cmd, timeout=1, stdout_regex=None, stderr_regex=None, env=None
):
"""Runs a command in a subprocess with timeout in seconds.
Also checks returncode is zero, stdout if stdout_regex is set, and
stderr if stderr_regex is set.
"""
if env is not None:
env = {**os.environ, **env}
proc = subprocess.Popen(
cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env, text=True
)
def kill_process():
warnings.warn(f"Timeout running {cmd}")
proc.kill()
timer = threading.Timer(timeout, kill_process)
try:
timer.start()
stdout, stderr = proc.communicate()
if proc.returncode == -9:
message = (
f"Subprocess timeout after {timeout}s.\nStdout:\n{stdout}\n"
f"Stderr:\n{stderr}"
)
raise TimeoutError(message)
elif proc.returncode != 0:
message = (
f"Non-zero return code: {proc.returncode}.\nStdout:\n{stdout}"
f"\nStderr:\n{stderr}"
)
raise ValueError(message)
if stdout_regex is not None and not re.search(stdout_regex, stdout):
raise ValueError(
f"Unexpected stdout: {stdout_regex!r} does not match:"
f"\n{stdout!r}"
)
if stderr_regex is not None and not re.search(stderr_regex, stderr):
raise ValueError(
f"Unexpected stderr: {stderr_regex!r} does not "
f"match:\n{stderr!r}"
)
return stdout, stderr
finally:
timer.cancel()
def skip_func(msg):
def test_func(*args, **kwargs):
pytest.skip(msg)
return test_func
# A decorator to run tests only when numpy is available
try:
import numpy # noqa F401
def with_numpy(func):
"""A decorator to skip tests requiring numpy."""
return func
except ImportError:
def with_numpy(func):
"""A decorator to skip tests requiring numpy."""
return skip_func("Test require numpy")
# A decorator to run tests only when numpy is available
try:
from ._openmp_test_helper.parallel_sum import parallel_sum
def with_parallel_sum(func):
"""A decorator to skip tests if parallel_sum is not compiled."""
return func
def _run_openmp_parallel_sum(*args):
return parallel_sum(*args)
except ImportError:
def with_parallel_sum(func):
"""A decorator to skip tests if parallel_sum is not compiled."""
return skip_func("Test requires parallel_sum to be compiled")
_run_openmp_parallel_sum = None
def check_python_subprocess_call(code, stdout_regex=None, timeout=10):
cmd = [sys.executable]
try:
fid, filename = mkstemp(suffix="_joblib.py")
os.close(fid)
with open(filename, mode="w") as f:
f.write(code)
cmd += [filename]
check_subprocess_call(cmd, stdout_regex=stdout_regex, timeout=timeout)
finally:
os.unlink(filename)
def filter_match(match):
if sys.platform == "win32":
return None
return match
joblib-loky-51247f4/tools/ 0000775 0000000 0000000 00000000000 15256713413 0015354 5 ustar 00root root 0000000 0000000 joblib-loky-51247f4/tools/README.md 0000664 0000000 0000000 00000004003 15256713413 0016630 0 ustar 00root root 0000000 0000000 # Maintaining the stdlib vendored resource tracker
For easier maintainability, the idea is that Loky resource tracker in
`loky/backend/resource_tracker.py` derives from a minimally modified version of
the stdlib resource tracker for a fixed CPython version in
`loky/backend/stdlib_py314_resource_tracker.py`. Loky-specific features (like
Windows support, logging, refcount functionality for shared folders and files,
etc ...) should go as much as possible in `loky/backend/resource_tracker.py`.
`vendor_stdlib_resource_tracker.sh` downloads the `resource_tracker.py` file
from a fixed CPython version. Loky-specific minimal changes to that file should
be stored as a patch in `tools/stdlib_py314_resource_tracker.patch`.
As a minimal patch, it has more chances to still apply when re-vendorig a newer
versions of the CPython resource tracker module with some code changes, e.g. to
benefit from upstream bug fixes.
## Running the vendoring script
Typical use case: update `loky/backend/stdlib_py314_resource_tracker.py` to
mirror the changes in CPython from 3.14.7 to 3.15.x. If the patch still applies
the vendoring script will update
`loky/backend/stdlib_py314_resource_tracker.py`.
- update the `curl` command to fetch `resource_tracker.py` from a more recent CPython version
- run the vendoring script from the repository root:
```bash
bash tools/vendor_stdlib_resource_tracker.sh
```
## Regenerating the patch
- modify `loky/backend/stdlib_py314_resource_tracker.py` if really needed
(remember changes should be minimal to avoid diverging from the reference
CPython code)
- download the Python reference file:
```bash
curl -L \
https://raw.githubusercontent.com/python/cpython/refs/tags/v3.14.7/Lib/multiprocessing/resource_tracker.py \
-o loky/backend/stdlib_py314_resource_tracker.py.ref
```
- generate the patch
```bash
diff -u --minimal \
--label stdlib-ref --label loky-vendored \
loky/backend/stdlib_py314_resource_tracker.py{.ref,} \
>| tools/stdlib_py314_resource_tracker.patch
```
joblib-loky-51247f4/tools/stdlib_py314_resource_tracker.patch 0000664 0000000 0000000 00000005350 15256713413 0024243 0 ustar 00root root 0000000 0000000 --- stdlib-ref
+++ loky-vendored
@@ -23,11 +23,13 @@
import time
import warnings
from collections import deque
+from types import SimpleNamespace
import json
-from . import spawn
-from . import util
+# loky: fix imports
+from multiprocessing import spawn
+from multiprocessing import util
__all__ = ['ensure_running', 'register', 'unregister']
@@ -375,23 +377,37 @@
self._ensure_running_and_write(msg)
-# gh-146313: Per-thread flag set by .popen_fork.Popen._launch() just before
-# os.fork(), telling _after_fork_in_child() to keep the inherited pipe fd so
-# the child can reuse this tracker (gh-80849). Unset for raw os.fork() calls,
-# where the child instead closes the fd so the parent's __del__ can reap the
-# tracker. Using threading.local() keeps multiple threads calling
-# popen_fork.Popen._launch() at once from clobbering eachothers intent.
-_fork_intent = threading.local()
-_resource_tracker = ResourceTracker()
-ensure_running = _resource_tracker.ensure_running
-register = _resource_tracker.register
-unregister = _resource_tracker.unregister
-getfd = _resource_tracker.getfd
+# loky: We want to use multiprocessing.resource_tracker._fork_intent rather
+# than create a separate variable.
+# multiprocessing.resource_tracker._fork_intent is used by
+# multiprocessing.popen_fork to distinguish raw forks from multiprocessing
+# forks.
+try:
+ from multiprocessing.resource_tracker import _fork_intent
+except ImportError:
+ # Older Python versions cannot distinguish multiprocessing forks from raw
+ # os.fork() calls. Preserve the inherited tracker fd in both cases.
+ # TODO remove when Python 3.15 is the minimum supported version. For more details about preserve_fd see
+ # https://github.com/python/cpython/pull/146316
+ # which has been merged in Python 3.15 and backported in 3.14.5 and 3.13.14
+ _fork_intent = SimpleNamespace(preserve_fd=True)
-# gh-146313: See _after_fork_in_child docstring.
-if hasattr(os, 'register_at_fork'):
- os.register_at_fork(after_in_child=_resource_tracker._after_fork_in_child)
+# loky: begin commented out section
+# Remove module-level globals and logic which
+# are unused and can cause confusing errors at interpreter exit because of
+# discrepancies in the code from vendored Python version and used Python
+# version
+# _resource_tracker = ResourceTracker()
+# ensure_running = _resource_tracker.ensure_running
+# register = _resource_tracker.register
+# unregister = _resource_tracker.unregister
+# getfd = _resource_tracker.getfd
+
+# # gh-146313: See _after_fork_in_child docstring.
+# if hasattr(os, 'register_at_fork'):
+# os.register_at_fork(after_in_child=_resource_tracker._after_fork_in_child)
+# loky: end commented out section
def _decode_message(line):
joblib-loky-51247f4/tools/vendor_stdlib_resource_tracker.sh 0000775 0000000 0000000 00000001233 15256713413 0024172 0 ustar 00root root 0000000 0000000 #!/bin/sh
# Download the CPython stdlib reference file. The CPython version can be
# updated from time to time
curl -L \
https://raw.githubusercontent.com/python/cpython/refs/tags/v3.14.7/Lib/multiprocessing/resource_tracker.py \
-o loky/backend/stdlib_py314_resource_tracker.py.ref
# Overwrite the stdlib_py314_resource_tracker.py by applying our custom patch
# to the stdlib reference file
patch \
loky/backend/stdlib_py314_resource_tracker.py.ref \
-i tools/stdlib_py314_resource_tracker.patch \
-o loky/backend/stdlib_py314_resource_tracker.py
# Delete the stdlib reference file
rm -f loky/backend/stdlib_py314_resource_tracker.py.ref
Reusable Process Pool Executor
==============================
|azurepipelines| |codecov|
Goal
~~~~
The aim of this project is to provide a robust, cross-platform and
cross-version implementation of the :class:`~concurrent.futures.ProcessPoolExecutor` class of
:mod:`concurrent.futures`. It notably features:
- **Deadlock free implementation**: one of the major concern in standard :class:`multiprocessing.pool.Pool` and in :class:`concurrent.futures.ProcessPoolExecutor` is their ability to handle crashes of worker processes. This library intends to fix those possible deadlocks and send back meaningful errors. Note that several fixes in ``loky`` have been ported to :class:`concurrent.futures.ProcessPoolExecutor` since python3.7+, which now be as ``loky``'s implementation.
- **Consistent spawn behavior**: All processes are started using fork/exec on POSIX systems. This ensures safer interactions with third party libraries.
- **Reusable executor**: strategy to avoid re-spawning a complete executor every time. A singleton executor instance can be reused (and dynamically resized if necessary) across consecutive calls to limit spawning and shutdown overhead. The worker processes can be shutdown automatically after a configurable idling timeout to free system resources.
- **Transparent** |cloudpickle| **integration**: to call interactively defined functions and lambda expressions in parallel. It is also possible to register a custom pickler implementation to handle inter-process communications.
- **No need for** :code:`if __name__ == "__main__":` **in scripts**: thanks to the use of |cloudpickle| to call functions defined in the :mod:`__main__` module, it is not required to protect the code calling parallel functions under Windows.
Installation
~~~~~~~~~~~~
The recommended way to install :mod:`loky` is with :mod:`pip`,
.. code:: bash
pip install loky
:mod:`loky` can also be installed from sources using
.. code:: bash
git clone https://github.com/joblib/loky
cd loky
python setup.py install
Note that `loky` has an optional dependency on |psutil| to allow early memory leak detections.
Usage
~~~~~
The basic usage of :mod:`loky` relies on the :func:`~loky.get_reusable_executor`, which internally manages a custom :class:`~concurrent.futures.ProcessPoolExecutor` object, which is reused or re-spawned depending on the context.
.. code:: python
import os
from time import sleep
from loky import get_reusable_executor
def say_hello(k):
pid = os.getpid()
print(f"Hello from {pid} with arg {k}")
sleep(.01)
return pid
# Create an executor with 4 worker processes, that will
# automatically shutdown after idling for 2s. This executor
# can be reused, if get_reusable_executor() is called again
# from the same thread before 2 seconds are up.
executor = get_reusable_executor(max_workers=4, timeout=2)
res = executor.submit(say_hello, 1)
print("Got results:", res.result())
results = executor.map(say_hello, range(50))
n_workers = len(set(results))
print("Number of used processes:", n_workers)
assert n_workers == 4
For more advance usage, see our documentation_.
Workflow to contribute
~~~~~~~~~~~~~~~~~~~~~~
To contribute to :mod:`loky`, first create an account on github_. Once this is done, fork the `loky repository`_ to have your own repository, clone it using 'git clone' on the computers where you want to work. Make your changes in your clone, push them to your github account, test them on several computers, and when you are happy with them, send a pull request to the main repository.
Running the test suite
~~~~~~~~~~~~~~~~~~~~~~
To run the test suite, you need the |pytest| (version >= 3) and |psutil|
modules. From the root of the project, run the test suite using:
.. code:: bash
pip install -e .
pytest .
Why was the project named `loky`?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
While developping :mod:`loky`, we had some bad experiences trying to debug deadlocks when using :class:`multiprocessing.pool.Pool` and :class:`concurrent.futures.ProcessPoolExecutor`, especially when calling functions with non-picklable arguments or returned values at the beginning of the project. When we had to chose a name, we had dealt with so many deadlocks that we wanted some kind of invocation to repel them! Hence :mod:`loky`: a mix of a god, locks and the `y` that make it somehow cooler and nicer :) (and also less likely to result in name conflict in google results ^^).
Fixes to avoid those deadlocks in :mod:`concurrent.futures` were also contributed upstream in Python 3.7+, as a less mystical way to repel the deadlocks :D
Acknowledgement
~~~~~~~~~~~~~~~
This work is supported by the Center for Data Science, funded by the
IDEX Paris-Saclay, ANR-11-IDEX-0003-02
.. |azurepipelines| image:: https://dev.azure.com/joblib/loky/_apis/build/status/joblib.loky?branchName=master
:target: https://dev.azure.com/joblib/loky/_build?definitionId=2&_a=summary&repositoryFilter=2&branchFilter=38
.. |codecov| image:: https://codecov.io/gh/joblib/loky/branch/master/graph/badge.svg
:target: https://codecov.io/gh/joblib/loky
.. |cloudpickle| raw:: html