Earlier I described a method to find hot or stuck pixels by determining the variance of (sub)pixels over a large set of photos. In this article we take a look at a way to farm out this work to multiple processes.
The parallel algorithm for calculating variance
allows us to combine calculated variances from separate sets. With the help of Python's
multiprocessing module it is straight forward to implement our hot pixel finding algorithm to one that takes advantage of multiple cores:
import Image import numpy as np import sys from glob import glob from multiprocessing import Pool,current_process from time import time def process_pixels(shape,*args): first = True for filename in args: pic = Image.open(filename) if pix.shape != shape: print("shapes don't match") continue if first: first = False firstpix = pix n = np.zeros(firstpix.shape) mean = np.zeros(firstpix.shape) M2 = np.zeros(firstpix.shape) delta = np.zeros(firstpix.shape) n += 1 delta = pix - mean mean += delta/n M2 += delta*(pix - mean) return M2 if __name__ == '__main__': global pool n=int(sys.argv) pool = Pool(n) filenames =  for a in sys.argv[2:]: filenames.extend(glob(a)) shape = np.array(Image.open(filenames)).shape s=time() results =  for i in range(n): results.append(pool.apply_async(process_pixels,tuple([shape]+filenames[i::n]))) for i in range(n): results[i]=results[i].get() M2 = sum(results) print(time()-s) mini = np.unravel_index(M2.argmin(),M2.shape) maxi = np.unravel_index(M2.argmax(),M2.shape) print('min',M2[mini],mini) print('max',M2[maxi],maxi) print('mean',np.mean(M2)) sorti = M2.argsort(axis=None) print(sep="\n",*[(i,M2[i]) for i in [np.unravel_index(i,M2.shape) for i in sorti[:10]]]) print(time()-s)
Note that because we return complete arrays (line 26) we gain almost nothing for small data sets because of the overhead of shuttling these large arrays (> 30 MByte) between processes. This is illustrated in the following graph with shows the elapsed time as a function of the number of processes, both for a small number of pictures (50) and a large number (480).
Some other notable issues: in the code we do not actually implement the parallel algorithm but we simple add together the variances. Because we're looking for a minimum variance we gain nothing by adding a constant value.
Memory usage is another thing to be aware of (and the reason that there is no entry for six cores in the graph. The algorithm we have implemented uses 5 arrays (the pixel data itself included). That makes for 10 megapixel X 3 colors X 5 arrays X 8 bytes data (because we use 64 bit floats by default) which makes for a whopping 1.2 Gigabyte of data per process or more than 6 Gig with 5 processes. With some other applications open a sixth process wouldn't fit on my test machine. Because we're adding pixel values in the range 0 - 255 we could probably gain a lot by using 32 bit floats or even 16 bit floats here.