17.4. `concurrent.futures` — 发起并行任务 ¶

3.2 版新增。

源代码： Lib/concurrent/futures/thread.py and Lib/concurrent/futures/process.py

The concurrent.futures 模块为异步执行可调用提供高级接口。

异步执行可以采用线程履行使用 ThreadPoolExecutor ，或单独进程使用 ProcessPoolExecutor 。两者实现相同接口，定义通过抽象 Executor 类。

17.4.1. 执行器对象 ¶

class concurrent.futures. Executor ¶

submit ( fn , *args , **kwargs ) ¶

map ( func , *iterables , timeout=None , chunksize=1 ) ¶

shutdown ( wait=True ) ¶

17.4.2. ThreadPoolExecutor ¶

ThreadPoolExecutor 是 Executor 子类，使用线程池异步执行调用。

死锁会发生当可调用关联 Future waits on the results of another Future 。例如：

import time
def wait_on_b():
    time.sleep(5)
    print(b.result())  # b will never complete because it is waiting on a.
    return 5
def wait_on_a():
    time.sleep(5)
    print(a.result())  # a will never complete because it is waiting on b.
    return 6
executor = ThreadPoolExecutor(max_workers=2)
a = executor.submit(wait_on_b)
b = executor.submit(wait_on_a)

和：

def wait_on_future():
    f = executor.submit(pow, 5, 2)
    # This will never complete because there is only one worker thread and
    # it is executing this function.
    print(f.result())
executor = ThreadPoolExecutor(max_workers=1)
executor.submit(wait_on_future)

class concurrent.futures. ThreadPoolExecutor ( max_workers=None ) ¶

17.4.2.1. ThreadPoolExecutor 范例 ¶

import concurrent.futures
import urllib.request
URLS = ['http://www.foxnews.com/',
        'http://www.cnn.com/',
        'http://europe.wsj.com/',
        'http://www.bbc.co.uk/',
        'http://some-made-up-domain.com/']
# Retrieve a single page and report the URL and contents
def load_url(url, timeout):
    with urllib.request.urlopen(url, timeout=timeout) as conn:
        return conn.read()
# We can use a with statement to ensure threads are cleaned up promptly
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
    # Start the load operations and mark each future with its URL
    future_to_url = {executor.submit(load_url, url, 60): url for url in URLS}
    for future in concurrent.futures.as_completed(future_to_url):
        url = future_to_url[future]
        try:
            data = future.result()
        except Exception as exc:
            print('%r generated an exception: %s' % (url, exc))
        else:
            print('%r page is %d bytes' % (url, len(data)))

17.4.3. ProcessPoolExecutor ¶

The ProcessPoolExecutor class is an Executor subclass that uses a pool of processes to execute calls asynchronously. ProcessPoolExecutor 使用 multiprocessing module, which allows it to side-step the 全局解释器锁 but also means that only picklable objects can be executed and returned.

The __main__ module must be importable by worker subprocesses. This means that ProcessPoolExecutor will not work in the interactive interpreter.

调用 Executor or Future methods from a callable submitted to a ProcessPoolExecutor will result in deadlock.

class concurrent.futures. ProcessPoolExecutor ( max_workers=None ) ¶

17.4.3.1. ProcessPoolExecutor 范例 ¶

import concurrent.futures
import math
PRIMES = [
    112272535095293,
    112582705942171,
    112272535095293,
    115280095190773,
    115797848077099,
    1099726899285419]
def is_prime(n):
    if n % 2 == 0:
        return False
    sqrt_n = int(math.floor(math.sqrt(n)))
    for i in range(3, sqrt_n + 1, 2):
        if n % i == 0:
            return False
    return True
def main():
    with concurrent.futures.ProcessPoolExecutor() as executor:
        for number, prime in zip(PRIMES, executor.map(is_prime, PRIMES)):
            print('%d is prime: %s' % (number, prime))
if __name__ == '__main__':
    main()

17.4.4. Future 对象 ¶

The Future class encapsulates the asynchronous execution of a callable. Future instances are created by Executor.submit() .

class concurrent.futures. Future ¶

cancel ( ) ¶

cancelled ( ) ¶

running ( ) ¶

done ( ) ¶

result ( timeout=None ) ¶

exception ( timeout=None ) ¶

add_done_callback ( fn ) ¶

下列 Future methods are meant for use in unit tests and Executor implementations.

set_running_or_notify_cancel ( ) ¶

set_result ( result ) ¶

set_exception ( exception ) ¶

17.4.5. 模块函数 ¶

concurrent.futures. wait ( fs , timeout=None , return_when=ALL_COMPLETED ) ¶

常量	描述
`FIRST_COMPLETED`	The function will return when any future finishes or is cancelled.
`FIRST_EXCEPTION`	The function will return when any future finishes by raising an exception. If no future raises an exception then it is equivalent to `ALL_COMPLETED` .
`ALL_COMPLETED`	The function will return when all futures finish or are cancelled.

concurrent.futures. as_completed ( fs , timeout=None ) ¶

另请参阅

PEP 3148 – futures - execute computations asynchronously: The proposal which described this feature for inclusion in the Python standard library.

17.4.6. 异常类 ¶

exception concurrent.futures. CancelledError ¶

exception concurrent.futures. TimeoutError ¶

exception concurrent.futures.process. BrokenProcessPool ¶

17.4. concurrent.futures — 发起并行任务 ¶