Intro to Python Image Processing in Computational Photography
前言
原文:Intro to Python Image Processing in Computational Photography 作者:RADU BALABAN
正文
Computational photography is about enhancing the photographic process with computation. While we normally tend to think that this applies only to post-processing the end result (similar to photo editing), the possibilities are much richer since computation can be enabled at every step of the photographic process—starting with the scene illumination, continuing with the lens, and eventually even at the display of the captured image. This is important because it allows for doing much more and in different ways than what can be achieved with a normal camera. It is also important because the most prevalent type of camera nowadays—the mobile camera—is not particularly powerful compared to its larger sibling (the DSLR), yet it manages to do a good job by harnessing the computing power it has available on the device. We’ll take a look at two examples where computation can enhance photography—more precisely, we’ll see how simply taking more shots and using a bit of Python to combine them can create nice results in two situations where mobile camera hardware doesn’t really shine—low light and high dynamic range.Low-light Photography
Let’s say we want to take a low-light photograph of a scene, but the camera has a small aperture (lens) and limited exposure time. This a typical situation for mobile phone cameras which, given a low light scene, could produce an image like this (taken with an iPhone 6 camera):


import os import numpy as np import cv2 folder = 'source_folder' # We get all the image files from the source folder files = list([os.path.join(folder, f) for f in os.listdir(folder)]) # We compute the average by adding up the images # Start from an explicitly set as floating point, in order to force the # conversion of the 8-bit values from the images, which would otherwise overflow average = cv2.imread(files[0]).astype(np.float) for file in files[1:]: image = cv2.imread(file) # NumPy adds two images element wise, so pixel by pixel / channel by channel average += image # Divide by count (again each pixel/channel is divided) average /= len(files) # Normalize the image, to spread the pixel intensities across 0..255 # This will brighten the image without losing information output = cv2.normalize(average, None, 0, 255, cv2.NORM_MINMAX) # Save the output cv2.imwrite('output.png', output)The result (with auto-contrast applied) shows that the noise is gone, a very large improvement from the original image.



average_noise
variable), we can simply subtract it from our shot so far, before normalizing:
average -= average_noise output = cv2.normalize(average, None, 0, 255, cv2.NORM_MINMAX) cv2.imwrite('output.png', output)Here is our final photo:

High Dynamic Range
Another limitation that a small (mobile) camera has is its small dynamic range, meaning the range of light intensities at which it can capture details is rather small. In other words, the camera is able to capture only a narrow band of the light intensities from a scene; the intensities below that band appear as pure black, while the intensities above it appear as pure white, and any details are lost from those regions. However, there is a trick that the camera (or photographer) can use—and that is adjusting the exposure time (the time the sensor is exposed to the light) in order to control the total amount of light that gets to the sensor, effectively shifting the range up or down in order to capture the most appropriate range for a given scene. But this is a compromise. Many details fail to make it into the final photo. In the two images below, we see the same scene captured with different exposure times: a very short exposure (1/1000 sec), a medium exposure (1/50 sec) and a long exposure (1/4 sec).
# Read all the files with OpenCV files = ['1.jpg', '2.jpg', '3.jpg', '4.jpg', '5.jpg'] images = list([cv2.imread(f) for f in files]) # Compute the exposure times in seconds exposures = np.float32([1. / t for t in [1000, 500, 100, 50, 10]]) # Compute the response curve calibration = cv2.createCalibrateDebevec() response = calibration.process(images, exposures)The response curve looks something like this:

# Compute the HDR image merge = cv2.createMergeDebevec() hdr = merge.process(images, exposures, response) # Save it to disk cv2.imwrite('hdr_image.hdr', hdr)For those of us lucky enough to have an HDR display (which is getting more and more common), it may be possible to visualize this image directly in all its glory. Unfortunately, the HDR standards are still in their infancy, so the process to do that may be somewhat different for different displays. For the rest of us, the good news is that we can still take advantage of this data, although a normal display requires the image to have byte value (0-255) channels. While we need to give up some of the richness of the irradiance map, at least we have the control over how to do it. This process is called tone-mapping and it involves converting the floating point irradiance map (with a high range of values) to a standard byte value image. There are techniques to do that so that many of the extra details are preserved. Just to give you an example of how this can work, imagine that before we squeeze the floating point range into byte values, we enhance (sharpen) the edges that are present in the HDR image. Enhancing these edges will help preserve them (and implicitly the detail they provide) also in the low dynamic range image. OpenCV provides a set of these tone-mapping operators, such as Drago, Durand, Mantiuk or Reinhardt. Here is an example of how one of these operators (Durand) can be used and of the result it produces.
durand = cv2.createTonemapDurand(gamma=2.5) ldr = durand.process(hdr) # Tonemap operators create floating point images with values in the 0..1 range # This is why we multiply the image with 255 before saving cv2.imwrite('durand_image.png', ldr * 255)


def countTonemap(hdr, min_fraction=0.0005): counts, ranges = np.histogram(hdr, 256) min_count = min_fraction * hdr.size delta_range = ranges[1] - ranges[0] image = hdr.copy() for i in range(len(counts)): if counts[i] < min_count: image[image >= ranges[i + 1]] -= delta_range ranges -= delta_range return cv2.normalize(image, None, 0, 1, cv2.NORM_MINMAX)
Conclusion
We’ve seen how with a bit of Python and a couple supporting libraries, we can push the limits of the physical camera in order to improve the end result. Both examples we’ve discussed use multiple low-quality shots to create something better, but there are many other approaches for different problems and limitations. While many camera phones have store or built-in apps that address these particular examples, it is clearly not difficult at all to program these by hand and to enjoy the higher level of control and understanding that can be gained. If you’re interested in image computations on a mobile device, check out OpenCV Tutorial: Real-time Object Detection Using MSER in iOS by fellow Toptaler and elite OpenCV developer Altaibayar Tseveenbayar.UNDERSTANDING THE BASICS
How does the sensor in a camera work?
The sensor translates the incoming light into electrical signals, which are then amplified and digitized. Its surface is divided into pixels, which are further divided into channels. Therefore, the sensor produces three numbers corresponding to red, green and blue light intensities for every point in the scene.
What is the meaning of image processing?
Image processing refers to any computational steps taken after a raw image has been captured by the sensor, in order to either enhance or modify it.
How does image processing work?
Image processing is done in software by applying numerical operations on the image data. Most common image processing techniques have a solid mathematical background.
What is Python, NumPy and OpenCV?
Python is a programming language well suited for scientific computing. NumPy is a Python library that simplifies doing numerical operations on arrays. OpenCV is a specialized library, focused on image processing and computer vision.
Why are very low-light photographs noisy?
In low light and with limited exposure, there is a very small amount of light energy that falls on the sensor. The sensor tries to compensate by amplifying the signal, but ends up also amplifying its own electrical noise.
What is HDR photography?
High dynamic range (HDR) photography refers to capturing more accurately the physical light intensity in a scene. Conventional digital photography uses only a small number of intensity levels (typically 256).
What is tone mapping?
Tone mapping is the technique that transforms an HDR image into a conventional image, preserving most details so that the image can be shown on a non-HDR display.
评论已关闭