How Does a Phone Take a Photo?

An earlier article, How does radio carry sound and data to your phone?, said: “Light and radio deliver energy in packets, each called a photon.” Your phone’s camera does something with exactly those packets: it counts photons, one by one.

A quick guess

Two things are often said: “A phone camera sees colors directly, like an eye,” and “More pixels means a sharper photo.”

Neither is quite right. The chip in the camera can’t tell colors apart at all; the colors in a photo are calculated. And whether a photo looks clean often depends less on how many pixels there are than on how many photons each pixel collects.

Taking a photo is counting photons

At the heart of a phone camera is a silicon chip about the size of a fingernail, covered with tens of millions of tiny “buckets,” each called a pixel and only about 1 micrometer (a thousandth of a millimeter) across. A photon entering the silicon can knock an electron loose from its atom, and the electron drops into the bucket. When the exposure ends, the electrons in each bucket are counted, and that tells how bright the light was there. This is the photoelectric effect: Einstein explained it in 1905 with the idea that light comes in packets, and later won the Nobel Prize for it.

But not every photon can free an electron. An electron in silicon needs at least 1.12 electron volts of energy to break loose; that is silicon’s “threshold.”

A photon of visible light carries 1.6 to 3.3 electron volts, enough to clear the threshold. A photon from a phone signal or a microwave oven carries only about a hundred-thousandth of an electron volt; no number of them can free a single electron, so cameras can’t photograph radio waves. The threshold of 1.12 eV corresponds to a wavelength of about 1100 nm, longer than the reddest light our eyes can see (about 750 nm).

Your Eyes Only Count to Three: How Do We See Color? said: “A TV remote uses infrared light around 940 nm. None of the cones respond to it at all, which is why we see nothing when we press a button.” But a 940 nm photon carries 1.32 eV, over silicon’s threshold: point your phone’s camera at a remote, hold down a button, and you’ll see it flashing on the screen. Phone cameras usually have an infrared-blocking filter, and how much it blocks differs from camera to camera, so the front and rear cameras may show it differently; try both.

A photo, built up one photon at a time

Which pixel a photon lands in, and when, is random. Even under perfectly even light, one pixel might count 103 photons while its neighbor counts only 95.

This randomness follows a simple rule: if a pixel gets $N$ photons on average, the count is off by about $\sqrt{N}$. With 100 photons, it’s off by about 10, that is 10%, and the photo looks clearly grainy; with 10,000, it’s off by about 100, only 1%, and you can’t see it. This is called shot noise. It isn’t a flaw in the chip: it comes from light itself arriving in grains.

The chip can’t tell colors apart

Silicon only counts electrons, and an electron freed by a red photon is exactly like one freed by a blue photon. So where does color come from?

The trick is to put a tiny color filter over each pixel, letting through only red, green or blue light. The most common arrangement was invented in 1976 by Bryce Bayer at Kodak: in every 4 pixels, 2 green, 1 red and 1 blue. There are twice as many green ones because our eyes are most sensitive to green, and we see most of the fine detail of light and dark through green. It’s the same idea as the eye seeing color with three kinds of cones.

So each pixel knows only one of the three colors, and has to “guess” the other two from its neighbors. Large patches of color come out right; but if stripes are about as fine as the pixels, the guess goes wrong and colorful patterns appear that weren’t there. You often see them when photographing a computer screen or a fine-checked shirt. Today’s phones guess far more cleverly than the figure, but the idea is the same.

The dimmer the light, the coarser the grain

Grain comes from too few photons. The dimmer the light, the fewer photons land in each pixel in the same time, and the more the grain shows. Take the same phone and the same person under three kinds of light, and work out how many photons each pixel gets:

How brightly light falls on the face is measured in lux, the unit of illuminance: a candle 1 m away gives about 1 lux, and a sunny day outdoors gives tens of thousands up to a hundred thousand lux. By a window in the daytime, the face gets about 1,000 lux, and with an exposure of 1/120 s each face pixel collects about 1,160 photons, a spread of only 3%. Under a living-room lamp in the evening, about 100 lux and 1/30 s, it collects about 460, a spread of 5%. With only a small night light on, about 3 lux and 1/15 s, it gets only about 28, a spread of nearly 20%, and the face is covered in grain.

A phone has two ways out. One is a longer exposure, but then any shake blurs the photo. The other is to take many shots in a row and stack them: that’s “night mode.” Each shot has only a few dozen photons per pixel, but stacking a dozen of them gives a dozen times as many. As for a phone’s “noise reduction,” it smooths the grain away, but it smooths away some detail too, so faces can look waxy. In the end, the best fix is simply more light.

This is also why “more pixels means a sharper photo” isn’t always true: the chip is only so big, so the more pixels it has, the smaller each one is and the fewer photons it collects; pixel by pixel, the grain gets coarser.

Where the light comes from

More light means finer grain; but whether a photo looks good also depends on where the light comes from. Where the light comes from decides where the shadows fall:

A small lamp sends light from almost one direction, so wherever it’s blocked goes completely dark, with a hard edge. A big window sends light from a whole range of directions; where some is blocked, the rest still gets through, so shadows are faint and soft. In the same way, standing farther from the wall makes the shadow behind you fainter.

So for a portrait, stand by a window, facing it: daylight by a window is about ten times brighter than lamplight in the evening, so more photons and finer grain, and the window is big, so the shadows are soft. A flash is the opposite: it’s a small lamp, so its shadows are hard; and it sits right next to the lens, so its light goes into the eyes and bounces off the blood-rich retina straight back into the lens, giving “red eye.”

The same goes for ID photos, which must have no shadows on the face or the background. I built a small tool earlier, SmartIDPhoto: take a phone photo by a window, upload it, and it quickly turns it into an ID photo and a layout ready to print. Still, no software can conjure up photons that never arrived, so the first step is to shoot in good light.

In one sentence: taking a photo is counting photons. Photons fall one by one, at random; count $N$ and you’re off by about $\sqrt{N}$, so the dimmer the light, the coarser the grain. The chip can’t tell colors apart; color is “guessed” after covering the pixels with red, green and blue filters. For a good photo, give it more light, and softer light.

How do we know?

• See the infrared your phone can see: Point your phone’s front camera at a TV remote, hold a button, and watch whether the light flashes on the screen. Then try the rear camera and compare.
• See shot noise for yourself: Take a photo in a very dark room and zoom in: the grain is what too few photons leave behind. Turn on the light and take the same shot again, and the grain shrinks.
• A bit of history: In 1969, Willard Boyle and George Smith at Bell Labs invented the charge-coupled device (CCD), the first chip that could pass the electrons in its pixels along, row by row, to be counted; they shared the 2009 Nobel Prize in Physics for it. Phones today use a different kind of chip (CMOS), in which every pixel has its own amplifier and can be read out directly, but “counting the electrons that photons free” hasn’t changed.

What we still don’t know

Can the human eye see a single photon? The rod cells in our eyes really can be triggered by one photon. In a 2016 experiment, people in the dark were asked to guess whether a single photon had just arrived; they were right about 51.6% of the time, a little better than the 50% of pure guessing, and statistically solid. But whether people can truly “feel” a single photon is still being debated. Meanwhile, engineers are building “single-photon cameras” that register every photon, to take pictures in extremely dark places.

For the curious: formulas, simulations and data

Photon energy. $E = hc/\lambda$, or $E$ (eV) $\approx 1239.84 / \lambda$ (nm). 380 nm is 3.26 eV, 530 nm 2.34, 650 nm 1.91, 750 nm 1.65, 940 nm 1.32, and 1107 nm 1.12 (silicon’s band gap at room temperature). 100 MHz FM radio is $4.1 \times 10^{-7}$ eV, a 2.45 GHz microwave oven $1.0 \times 10^{-5}$ eV; 10 μm infrared is 0.124 eV.

Shot noise. The number of photons a pixel collects follows a Poisson distribution: mean $N$, standard deviation $\sqrt{N}$, relative spread $1/\sqrt{N}$. In figure 1, each frame adds new photons to every pixel, drawn from a Poisson distribution, on top of the previous frame, as the mean rises from 0.01 to 10,000 per pixel. Real chips also have read noise (about 1 to a few electrons per readout) and dark current, which matter only in very dim light; figures 1 and 4 include shot noise only.

How many photons a pixel gets. A lux is 1 lumen per square meter. 1 lux of D65 daylight (380 to 780 nm) delivers about $1.39 \times 10^{16}$ photons per second per square meter (from the CIE 1931 photopic function and 683 lumens per watt). The illuminance on the chip is $E_{\text{chip}} = E_{\text{face}} \, R \, T / (4 F^2)$: $R$ is the face’s reflectance, 0.35; $T$ the lens transmission, 0.9; $F$ the f-number, 1.8. Multiply by the pixel area (1.4 μm square), the exposure time, the quantum efficiency (0.6) and the share passed by the color filter (0.35). The three cases in figure 4: 1,000 lux and 1/120 s, about 1,160; 100 lux and 1/30 s, about 460; 3 lux and 1/15 s, about 28. These are only typical values; phones differ a lot, and the illuminances are only rough orders of magnitude.

The Bayer pattern and color guessing. Each 2×2 block is “red green / green blue” (Bayer, U.S. Patent 3,971,065, 1976). Figure 3 uses the same cartoon as figure 1, with a striped shirt, and the simplest bilinear interpolation: green from the average of its 4 neighbors above, below, left and right; red and blue from the average of the 2 or 4 nearest pixels of the same color. The shirt’s stripes are about 2.3 pixels apart, close to the finest the chip can resolve (2 pixels), which is why false color appears.

Lighting. Lambert’s cosine law: a small source produces an illuminance $E = I \cos\theta / d^2$ at a point, where $\theta$ is the angle between the light and the surface normal and $d$ the distance. Figure 5 splits each light into 500 small pieces and adds them up, checking for each point whether the ball blocks that piece; a little uniform ambient light stands for light bounced off the room’s walls. The ball’s radius stands for 12 cm, with the wall 25 cm behind; the lamp is about 10 cm across and 1.5 m away; the window is 1.6 m × 1.4 m, 1.8 m away.

Other. Photoelectric effect: Einstein, 1905; Nobel Prize in Physics, 1921. CCD: Boyle and Smith, 1969; Nobel Prize in Physics, 2009. The single-photon experiment: Tinsley and others, Nature Communications 7, 12172 (2016).