Background Removal Using Shutter Stream and a Background Image
by Darian Muresan, Ph.D.
In three earlier articles, Practical Algorithms for Background Removal, Background Removal Using Shutter Stream and Mobile Devices, and Image Background Removal Using Shutter Stream and Magic Wand, I discussed several approaches our teams at Iconasys have taken to remove backgrounds in images, and showed how the Iconasys Shutter Stream software can be used to remove backgrounds. Shutter Stream offers three types of background removal algorithms: (1) magic wand, (2) removal with a background image, and (3) chroma keying (sometimes known as green screen removal). For pure white (not transparent) and when objects are shot against close to white backgrounds, Shutter Stream also offers a colors levels-adjustment option. The article Image Background Removal Using Shutter Stream and Magic Wand focused on the first option. This article takes a closer look at the second option, removing backgrounds using a background image, and discusses how this tool can be used to maximize image quality by minimizing camera vibrations.
A compatible camera or mobile device is connected to Shutter Stream, which allows full camera customization options. The different camera settings and their impact on image quality is discussed in details in the article Image Background Removal Using Shutter Stream and Magic Wand. The camera settings and configurations are the same for this type of background removal and therefore I’ll bypass the discussion on the camera settings and focus on the issues that make background removal with a background image different than using the magic wand tool.
Background Removal with a Background Image
The algorithm for removing the background of an image is straight forward conceptually but a little tricky in how the images need to be captured. The algorithm takes as input two images: (1) the input image containing the object and (2) the background image with the object removed. Ideally, the only difference between the two images is the object. Comparing the two images pixel by pixel allows us to determine the pixels that are part of the object and those that are part of the background. The rough block diagram of the algorithm is shown in Figure 1.
Figure 1: Background removal algorithm using an input image and a background image.
From a practical point of view, the algorithm works well when there are no shadows generated by the object in the input image. To keep the algorithm fast, the algorithm is based on pixel color information alone and shadows tend to cause problems. (To help with shadows, texture analysis can be used, in addition to color information, but this will slow the algorithm.)
To minimize shadows during image capture, we recommend the following setup: (1) if shooting with a backdrop, move the backdrop as far away from the object as possible; (2) if available, use acrylic risers with light flooding the riser from underneath, as discussed in the article Image Background Removal Using Shutter Stream and Magic Wand; (3) if possible, suspend the object in the air as shown in the setups of Figure 2.
Figure 2: Suggested ways of suspending objects using fishing line, to capture shadow-less images. The second row shows how objects are suspended using our Platinum Series Medium turntable, for a 360 capture.
The final aspect to consider is that the background image should have colors that are distinct, in terms of RGB values, from the colors in the object. If the colors are close in values, then holes in the object may appear for the pixels that have similar colors. For image holes, there are different adjustments that can be used, such as the ability to automatically fill them in or to add in the missing pixels using the lasso tool.
Measuring Camera Vibrations and Inconsistencies
Next, we tested the consistency of images across multiple shots. We set up the camera and a wireframe rack as shown in Figure 3. Camera was shooting the wire frame at an angle, which means that the captured image contained increasing frequencies from bottom to the top. (We expected errors in the high frequency areas.)
Figure 3: Camera and rack setup to test camera consistency.
As part of the test, we shot the same image twice and then applied the background removal algorithm by varying only the threshold value. If the camera is consistent from shot to shot the background removal algorithm should have output a completely transparent image (in our case we set transparency to white, for better visualization) at small threshold values.
Figure 4: Original and difference at 2, 5 and 13 threshold values.
Figure 4 shows the output at threshold values of 2, 5 and 13. As it can be seen from the figure, the two images were fairly inconsistent. Shooting the same image twice generated images that were quite different, requiring thresholds of up to 13 or higher.
We experimented further with different camera settings (ISO, denoise settings, etc.), and in the end determined that mirror slap and switching the live view on/off was causing additional vibration in the camera sensor, amplifying the noise in the system.
Figure 5: Enabling live view minimizes camera vibration.
After setting the live view to always on and running the same experiment, the consistency across images improved, and the threshold value for which the two images were almost identical dropped from 13 to 7, as shown in Figure 6. This particular observation about enabling live view is critical when shooting micro-photography, such as diamonds and jewelry. In micro-photography the smallest vibration errors can cause blur.
Figure 6: Original and differences at 2,5, and 7 pixels after enabling live view in view finder.
This observation is consistent with our observation in an earlier blog, Mirrorless vs. DSLR Cameras for Still & 360 Product Photography, where we explained that a DSLR camera can be made to behave like a mirrorless camera by enabling the live view setting. By enabling live view on the camera, the mirror is moved in the upward position, covering up the viewfinder and thus avoiding the mirror slap during shooting.
Applying the Background Removal
The process begins by capturing the image and its background, as shown in Figure 7.
Figure 7: Shadow-less object and background are captured as two images.
The background image is set as the background image for the algorithm, and the algorithm is applied to the main image. Besides the threshold values, there are a few other parameters that control the level of denoising, hole filling, speck removal, mask growth, and edge blending, as shown in Figure 8.
Figure 8: Applying the background removal algorithm.
A final cleanup to cut the supporting wires is done with a manual lasso tool, as shown in Figure 9.
Figure 9: Quick lasso tool cuts the suspending wires.
The final result is shown in Figure 10.
Figure 10: Final result, with background removed.
In conclusion, this article described how to use Shutter Stream’s background removal tool when a background image is available. It also discussed how this tool can be used to measure camera consistency from shot to shot, and how we experimentally determined that for our particular camera, Canon 70D, enabling live view minimizes noise and vibration errors by 50 percent.
About the authors: Darian Muresan manages the software and hardware development at Iconasys and is a key contributor to Iconasys’ image processing algorithms. Darian has undergraduate degrees in Electrical Engineering and Mathematics from University of Washington (Seattle, WA.); and a Masters and Ph.D. degree in Electrical and Computer Engineering from Cornell University (Ithaca, NY.).