- Stackoverflow - Multiple Threads in Python
This pattern is usually used in Http requesting. Since computing tasks might cause a lot of context switches, which might decrease the speed.
import threading urlTaskList = [ # your tasks here ] parallelism = 4 # here is the task you want to run def subTask(urlList): import urllib2 for url in urlList: data = urllib2.urlopen(url).read() # process the data here # ... # this method is used to seperate task chunk def splitChunk(tasks): """ Yield successive n-sized chunks from l. """ for i in xrange(0, len(l), n): yield l[i:i+n] threadList = list() chunkList = list(splitChunk(urlTaskList)) for subChunk in chunkList: worker = threading.Thread(target=subTask, args=(subChunk)) worker.start() threadList.append(worker) # wait for every thread finish its work for worker in threadList: worker.join()
splitChunk(taskList) above is from this post.