Most leading smartphones are designed with very capable camera systems. They often feature dual or triple lenses in a configuration of focal lengths, coupled with high-performance, high-resolution image sensors prominent inside the device. This design evolution has been driven largely by demand; people expect their most accessible imaging tool to have all the capabilities of traditional standalone cameras.
The resulting mobile imaging arms race is at a point where there are few visible hardware differences from one design to another. But it’s in this seeming homogeneity that true imaging prowess can be distinguished — specifically in the image signal processor (ISP) embedded in a system-on-chip (SoC).
Good Images Need Better Image Processing
The ISP is a critical subsystem in the core SoC, which serves as the primary processing unit for image data from camera sensors. The faster the data can be captured and pumped into the ISP, the faster this can manipulate the image data, pixel by pixel, into a usable image or video. This sounds simple enough, but as camera designs adapt to rising numbers of camera lenses and geometric growth in sensor pixels, the challenge for manufacturers is to keep ISPs updated with enough processing headroom to handle all this growth.
The latest mobile SoC designs on the market feature multiple ultrafast ISP cores that can reach aggregate data speeds measured in gigapixels per second. For example, Qualcomm’s Spectra ISP in its latest Snapdragon 8 series can support a 200-megapixel camera configuration with speeds of up to 3.2 gigapixels per second. Although these speeds sound fantastical — and perhaps excessive — smartphone-makers are finding plenty of ways to use all the ISP capabilities available.
One common imaging feature that takes advantage of the fast ISP pipeline is high dynamic range (HDR), in which multiple frames of a photo or video are taken at different exposures and combined into a final image with improved highlights and shadows. Capturing HDR photos and videos can be extremely taxing on the SoC; by designing a fast ISP pipeline, all the relevant image processing is addressed in the ISP core before the SoC manipulates the final image. Probably the most challenging use for HDR image processing is capturing 8K HDR videos. The camera ISP must not only process multiple HDR images per frame, but also process 30 different frames each second.
Advanced ISP Designs Support Camera Customization
The ISP pipeline can have many more tricks up its sleeve. Aside from the typical ISP tuning techniques — such as automatic white balance and colour correction — advanced customization in the ISP core and pipeline can create unique camera experiences. One example of this is the Leica Looks filter, designed for the Leitz Phone 1 featuring a large one-inch image sensor. This filter is a mix of traditional image processing algorithms and artificial intelligence neural networks that are processed by different parts of the Snapdragon platform. The algorithms are processed by the ISP, while the neural networks are handled by Qualcomm’s AI Engine. This approach produces images with some of the iconic lens characteristics of Leica cameras. The results mimic those produced by Leica’s Noctilux and Summilux lenses, with image depth maps and neural networks used to calculate out-of-focus background blurring.
ISPs also enable manufacturers to customize the design and configuration of cameras and image sensors. For example, an ultrawide panoramic camera can be created by using a pair of periscopic lenses, imaging two halves of an entire scene with two separate sensors. The resulting image is produced by merging these two disparate sensor outputs together, stitching the “seam” between them in the ISP pipeline.
Another novel approach to mobile camera design uses an anamorphic lens, wide aspect image sensors and an advanced ISP to improve the physically available image resolution. This design essentially stretches the width of the optical image through the anamorphic lens, using the additional image pixels in that dimension to produce a higher-resolution image. The ISP corrects stretching from the anamorphic lens and decompresses the enhanced image to a normal aspect ratio. Again, the crucial piece of this camera evolution is the ever-increasing capability of the ISP.
Advanced ISP Design and Post-Capture Computational Photography
As the camera ISP pipeline has evolved to keep up with growing mobile image demands, much of that role has focussed on real-time image processing capabilities. But further enhancements can be achieved after the camera shutter has been depressed; here, the camera ISP cores work with artificial intelligence capabilities in the SoC to produce amazing outputs after the image has been captured. Manipulating images with machine learning produces truly computational photos and videos.
One of the more widely used versions of computational photography is low-light or night photography. This works by capturing dozens of frames of a low-light scene in short succession. The camera ISP then combines illumination data of all the frames to remove noise from the image and recompile the scene, applying automatic white balance and colour correction. The resulting image can be further enhanced with additional computation to correct for movement in the image caused by shaky hands, through processing the final image pixel by pixel. This additional step takes a bit longer to process after the image has been captured, because it’s not performed in real time — but the final picture is nothing short of magical.
Conclusion
Imaging capabilities are a huge part of why people choose a particular phone over another. As the device industry continues to evolve and photo and video capabilities improve further, many of these signals are getting lost; physical camera designs are converging, making the hardware distinction harder to discern.
The industry is entering a phase in mobile imaging where the conversation needs to move away from what capabilities lie in the lens and image sensors, to what processors can achieve. Future mobile phone cameras will be judged not just on the quality of the camera systems but on the computational capabilities of the ISP pipeline — and the magical photographic and videographic experiences they produce for users.