We're back after a server migration that caused effbot.org to fall over a bit harder than expected. Expect some glitches.

How do I parcel out work among a bunch of worker threads?

Use the Queue module to create a queue containing a list of jobs. The Queue class maintains a list of objects with put(obj) to add an item to the queue and get() to return an item. The class will take care of the locking necessary to ensure that each job is handed out exactly once.

Here’s a trivial example:

import threading, Queue, time

# The worker thread gets jobs off the queue.  When the queue is empty, it 
# assumes there will be no more work and exits.  
# (Realistically workers will run until terminated.)
def worker ():
    print 'Running worker'
    time.sleep(0.1)
    while True:
        try:
            arg = q.get(block=False)
        except Queue.Empty:
            print 'Worker', threading.currentThread(),
            print 'queue empty'
            break
        else:
            print 'Worker', threading.currentThread(),
            print 'running with argument', arg
            time.sleep(0.5)

# Create queue
q = Queue.Queue()

# Start a pool of 5 workers
for i in range(5):
    t = threading.Thread(target=worker, name='worker %i' % (i+1))
    t.start()

# Begin adding work to the queue
for i in range(50):
    q.put(i)

# Give threads time to run
print 'Main thread sleeping'
time.sleep(5)

When run, this will produce the following output:

Running worker Running worker Running worker Running worker Running worker Main thread sleeping Worker <Thread(worker 1, started)> running with argument 0 Worker <Thread(worker 2, started)> running with argument 1 Worker <Thread(worker 3, started)> running with argument 2 Worker <Thread(worker 4, started)> running with argument 3 Worker <Thread(worker 5, started)> running with argument 4 Worker <Thread(worker 1, started)> running with argument 5 …

Consult the module’s documentation for more details; the Queue class provides a featureful interface.

CATEGORY: library