1
1

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

More than 5 years have passed since last update.

Python - Simple multi-thread sample

Posted at

Ref:
[1] - Stackoverflow - Multiple Threads in Python

This pattern is usually used in Http requesting. Since computing tasks might cause a lot of context switches, which might decrease the speed.

import threading

urlTaskList = [
# your tasks here
]

parallelism = 4

# here is the task you want to run
def subTask(urlList):
  import urllib2

  for url in urlList:
    data = urllib2.urlopen(url).read()
  
    # process the data here
    # ...


# this method is used to seperate task chunk
def splitChunk(tasks):
  """ Yield successive n-sized chunks from l.
  """
  for i in xrange(0, len(l), n):
    yield l[i:i+n]


threadList = list()
chunkList = list(splitChunk(urlTaskList))


for subChunk in chunkList:
  worker = threading.Thread(target=subTask, args=(subChunk))
  worker.start()
  threadList.append(worker)
  

# wait for every thread finish its work
for worker in threadList:
  worker.join()

The method splitChunk(taskList) above is from this post.

1
1
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
1
1

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?