Figure 1.0 Layers of a typical oil painting (A), Support; (B), Sizing; (C), Ground; (D), Paint or pigment layer; (E), Varnish; (F), Accumulated dirt, soot, dust. etc.
Multi-Dimensional Segmentation of an Oil Painting
To identify artifacts in an oil painting, one must understand the painting and its properties. An artifact of a painting can range from an under painting that was not intended to be seen by an observer to a touch-up of a painting that can be physically seen. The art conservator uses different types of techniques to enhance discontinuities in an oil painting in order to detect artifacts. The artist uses discontinuities of lightness, color, and textures to form his design. Sometimes discontinuities indicate the presence of artifacts. An artifact is anything in the painting the artist did not intend for us to see. For example, by simply looking at the color of the varnish, the conservator often can determine if the painting has been cleaned. Other times a conservator has to further investigate the painting. An example of this is a painting that looks fine under regular gallery illumination, but when using raking illumination shows that the painting has been over painted and re-varnished in a certain area. In general, art conservators have been able to determine these discontinuities using various illuminations and image capturing techniques[1].
To understand the purpose, design
and interpretation of this experiment one must first understand the compositions of a
painting. Many paintings, such as watercolors and pastels, consist of
only a few layers; oil paintings can be composed of up to six layer. Figure
1.0 is an illustration of the six layer.
|
Common Imaging Techniques Used in Painting Analysis
Infrared Reflectography
An infrared reflectography system using a silicon CCD camera was first developed in the 1990's and is still currently used. Solid state CCD systems can range in sensitivity (typically anywhere in the range of 0.6 to 5.7nm). A solid state CCD system is better than the vidicon tube system in terms of geometric distortion, signal stability, linearity and MTF. All of these characteristics improve the overall image quality [2]. Solid state CCD systems are also superior to vidicon tube systems in regards to sensitivity. This is most likely due to a CCD's high light sensitivity, low noise and high contrast that exists at any resolution [3]. Because a video system is restricted to 640x480 pixels, sampling areas of the painting and piecing them together into a "mosaic" will obtain the best results. Twenty percent has proved to be a sufficient amount of overlap to recognize similar features while still keeping the number of images to a minimum [2]. Unfortunately, this need for a "mosaic of images" created difficulties. Because edges can appear within the result, there is a need for near-uniform illumination of the image space.
To eliminate the need for a "mosaic," D. Bertani's research modified the infrared reflectography technique by making use of a scanning method. The infrared detector used in Bertani's procedure was a simple optical head placed at the center of the image. A pinhole with a diameter of 0.4 mm was then placed in front of the detector so an object area of 0.2 mm could be sampled when the optical magnification was 2:1. A translation system was then used to move the optical head and a small illuminator along the X-Y plane. A personal computer made use of a 12 bit analog-to-digital converter to sample and digitize the output signal of the detector. An ATVISTA image-processing board then displayed the final result[4]. The use of a scanner removed the need for a "mosaic" and the result was a smoother image that requires less manipulation. It was also found that the image exhibited greater definition and legibility by using the infrared scanning method [4].
Despite improvements, there are still limitations to the infrared reflectography method. The most difficult of these problems is that this method is still unable to penetrate blue areas of the painting. A related problem is that this method is unable to expose parts of an underdrawing covered by a medium that absorbs and reflects infrared radiation. The use of filters requires the illumination level of the imaged area to be increased. It is necessary for the X-Y translation of the object on a plane that is perpendicular to the camera. Images can still appear grayish and may therefore have poor contrast. Optical and geometrical distortions can be caused by the camera and must therefore be taken into account by the software when analyzing the data. However, the interpolation utilized by the software to correct these problems can also create errors and artifacts in the resulting image [4].
For art historical purposes,
examination by IR radiation is valuable in the study of an artist's creative process, and
in some cases the workshop tradition from which an artwork originated[5]. Underdrawings can be found in many works of art from any
time period. An underdrawing is a preparatory drawing for a painting sketched directly on
a ground. Underdrawings are typically sketched using charcoal, but artists have been known
to also use chalk, pencil, or paint and brush and other media. These underdrawings are
later covered with the artist's medium. Using infrared analysis to image an
underdrawing also becomes impossible if the pigments have large amounts of energy
absorbing carbon. As the thickness of the overlaying paint layer increases, the ability to
successfully image the underdrawing is diminished [6].
Ultraviolet Photography
Many substances fluoresce in ultra-violet. Fluorescence is the absorption of light of a certain wavelength and immediately re-emittted as visible light. . Examination of fluorescence in the ultraviolet light provides a mean of discerning the heterogeneity of an object. Fluorescence analysis has been achieved with high-pressure mercury-discharge tubes, in combination with the Wood's glass filter. This tube emits a number of bright lines at definite wave-lengths in the visible and ultraviolet region.
Ultraviolet illumination is used to determine in-painting. In-painting is used in painting restoration to emphasize that the original paint is sacrosanct and must not be covered with modern additions of color. The only exception is when the painting has been damaged or an area has been rubbed away. In case it should be ever necessary to remove the modern paint, it must be applied over an isolating varnish of dammar in turpentine.[8]. Dammar in turpentine is one of the traditional varnishes used by European and American painters. Over the past several decades synthetic resin varnishes have been used. One main difference between the traditional dammar in turpentine and synthetics is that the synthetic does not leave a yellow film as it ages. The yellow film can be seen with a fluorescent black light. Revarnished areas of a painting fluoresce less than others due to the yellow film that is left by the aging varnish[9].
Radiographic Photography
Radiographic imaging is a
technique that is nondestructive in nature, and is are particularly appropriate for
certain types of analysis of paintings. X-rays are produced whenever electrons
traveling at high velocities are stopped by collisions with an object. X-rays for
radiology are generated in a Coolidge tube. This tube electrons are emitted from a
heated tungsten wire, known as a cathode, and these electrons, being negatively charged,
pass to the positive anode at a velocity proportional to the potential applied between the
cathode and anode. When the electron collides with the anode, they give their
energy, about 99% of which is converted into heat, the remaining 1 % is passed out of the
tube as X-rays. If an object is placed in the X-ray beam, it will cast a
shadow. This shadow can be made visible by allowing the transmitted X-ray beam to
fall on a fluorescent screen or photographic film[10].
Histogram Analysis
Many properties of an image can be
explained by looking at a histogram. The histogram of an image normally refers to a
histogram of the pixel intensity values. Each of the pixels that represents an image
stored inside a computer has a pixel value which describes how bright that pixel is,
and/or what color it should be. In the simplest case of binary images, the pixel value is
a 1-bit number indicating either foreground or background. For a grayscale images, the
pixel value is a single number that represents the brightness of the pixel. Often this
number is stored as an 8-bit integer giving a range of possible values from 0 to 255.
Typically zero is taken to be black, and 255 is taken to be white. Values in between make
up the different shades of gray. This histogram is a graph showing the number of pixels in
an image at each different intensity value found in that image. For an 8-bit grayscale
image there are 256 different possible intensities, and so the histogram will graphically
display 256 numbers showing the distribution of pixels amongst those grayscale
values. In a 2-dimensional histogram analysis, we are interested in how often a
pixel value is present in the first image versus how often the same pixel is present in
the second image.
Calibration of Camera
Before images can be taken under these various lighting conditions, the camera must be calibrated to ensure that the voltage that is produced by the sensors is directly proportional to the amount of light striking the CCD. The camera uses an array of CCD's in a single electronic chip. The array of voltages is read in from the frame grabber. The frame grabber translates the voltages into integer pixel values from 0(no voltage, no light) to 255(max voltage, max light). To ensure that the camera is calibrated linearly, a simple procedure is done. This is done by capturing an image of a 21 Step Kodak Reference Step Wedge. By taking the pixel value and plotting it versus the reflectance value of each step, a resulting linear plot should be obtained. This linear plot ensures that the voltage produced by the sensor is directly proportional to intensity of the light and that the image is a true indication of the illuminated object being imaged. If using a nonlinear camera a different procedure must be done to linearize the camera. First take an image of a 21 Step Kodak Reference Step Wedge. Before converting the pixel values into information characteristic to the object, it is necessary to calculate the Tone Transfer Function of the camera. From the values obtained from the image a third degree polynomial regression function can be derived. Then a function, written in IDL, can convert the measured pixels into linear pixels.
Flat Fielding
These illuminating techniques interact differently with the object. The difference is due to the different intensities of light between the three methods of illumination. This is obvious when looking at the effect of illuminating an object with using two different light sources. An example is illuminating an object with an ultraviolet black light compared to a tungsten source. The resulting two images will not be uniformly illuminated. The same is true when illuminating in the visible and in the infrared. In this case the visible image will seem brighter than the infrared image. These changes in illumination must be considered, especially when these images will be analyzed together. The method that is used to normalize the illumination techniques is called flat fielding[14]
Flat fielding is a process that corrects for non-uniform illumination and cosine lens falloff.. Flat fielding is a process that takes an image that is non-uniformly illuminated and processes it against a reference white to produce a corrected image as it would appear under uniform illumination. To flat field an image you need a reference image of a uniform reflectance and of the actual image of the object. To obtain a flat fielded image, the following calculation is done.
First, cover the camera lens and capture the dark image. This dark image will have a mean pixel value, Pd. Using the dark image will take the system noise into account. This is vital when examining artifacts, because we want to ensure that the artifacts are not due to the noise in the system.
Next, illuminate a reference target with a known reflectance of Rref. This can be achieved by using a simple white piece of paper and illuminating the right and left side of the paper to produce an evenly illuminated object. After the paper is evenly illuminated capture the image. By inspecting the images histogram, the maximum pixel value can be obtained. For an 8 bit image, the maximum pixel value should be between 200 and 240. This can be adjusted by changing the f-number of the camera and recapturing the image. This image is now the reference image, with a mean pixel value, Pref. With this known information, the slope of the Tone Transfer Function (TTF) of the system can be obtained.
TTF of the CCD camera can be characterized by
|
The intercept of this linear equation is Pd, the pixel value of the dark image. the slope, a, that depends on the power setting of the light sources used to illuminate the reference image and R is the reflectance value of the sample. For the reference sample, equation (1) can be solved for the slope, a.
|
Pref is the mean pixel value of the reference image.
By combining equations (1) and (2) we can obtain an equation for each individual pixel in the image, at location (x, y).
|
R(x,y) refers to the reflectance at each location in the image. P(x,y) and Pref(x,y) are the pixel values in the image and in the reference image at each pixel location, respectively. However, when an image is captured, it exists in an array of pixel values, P(x,y). By applying equation (3) to each pixel value, one can convert the image into an array of reflectance values, R(x,y). This would result in pixel values ranging from 0 to 1.
|
Equation (4) is called the Flat
Fielding equation. This equation is used every time an image is taken under
different illuminating conditions. Figure 2.0 is an object that has been illuminated
with a tungsten light source(white light). As you can see the outside edges are
brighter than the middle. Figure 3.0 illustrates how flat fielding corrects for the
non uniformity.
|
|
Image Acquisition
For this research, a Cohu CCD
camera was connected to a frame grabber in a Intel Pentium 300 Mhz. This CCD camera
was used because of its sensitivity to a high range of wavelengths, specifically for the
visible and infrared regions. Two large tungsten light sources were placed 45 degrees to
the camera. The software program Asymetrix DVP 4.0 was used to capture and
save the image.
This apparatus is to be used with little or no alteration to any components of the apparatus. The sequence of the illumination procedures are visible(white light), and infrared. When capturing the infrared image a Kodak Wratten filter 87C was placed directly in front of the camera lens. This sequence is done first with a white reference sample and then with the oil painting. This procedure only involves turning on the lights and adding the infrared filter when needed. This method insures that the images will be in registration.
After the images are captured and correctly flat fielded they are
Illustration of 1-Dimensional Segmentation
Let's consider an
infrared image and an image captured using tungsten light illumination.. When
viewing each histogram separately, (Figure 4.0. and Figure 5.0) they have a regular
distribution due to the variety of gray levels in the image.
|
|
We are interested in segmenting areas that are
spatially contiguous. Spatially contiguous areas are regions of the histogram that
correspond to a single identifiable feature of the painting. Examples of this are
shown below. Figure 6.0 and Figure 7.0 illustrated how 1-D segmentation does not
reveal any contiguous spatial artifacts.
|
|
As you can see in 1-D segmentation it is very hard
to segment a certain area that may lead to a contiguous area. However, in some cases
we are able to segment an area that does lead to a contiguous spatial area in 1-D. .
|
|
Looking at Figure 9.0 we see that we are able
to segment an area that is spatially contiguous. The reason that we were able to segment
an area is due to the contrast difference between the letters and the rest of the image in
Figure 8.0 However, when imaging an oil painting there are areas with slight
differences in gray levels, making the 1-D segmentation very hard, as illustrated in
Figure 7.0
Figure 10.0 is a painting that is believed to be from the Renaissance Era. There was no documentation presented with the painting. The owner(anonymous) suspects the painting is from the European Renaissance and asked us to examine the painting. The boxed area was used for this experiment due to the evidence that it was cleaned in that area. Figure 11.0 and Figure 12.0 are a raking angle image and an infrared image taken of the boxed area in Figure 10.0, respectively. Figure 13.0 and Figure 14.0 are a visible and an infrared image with artificial color added to reveal the intensity of the images. The intensity ranges from red to blue, where red is the darkest and blue is the lightest.
|
|
|
|
|
|
Figure 15.0 is a 2-D histogram of the tungsten illuminated image vs. infrared image. Looking at the histogram we can see a common area where the two images have like areas. We can now do segmentation to the common area of the histogram to view the resulting image. The boxed area of the histogram is used for the segmentation. Figure 16.0 is the resulting image after segmentation see a common area between the visible and infrared image. When 2-dimensional segmentation is performed it reveals one area where the group of pixels are at the same location. This implies that a discontinuity exists in the object.
This discontinuity may be one of
a different color palette. A color pallet is the initial colors of paint used by the
artist. However, due to color difference in the object it could be possible that the
halo is part of a later color palette that was used for restoration. Note the color that
we see may be identical to the nose of the painting , but when
examined in the infrared shows to be different.
|
Figure 18.0 illustrates what happens when 2-d
segmentation is performed in a different area of the histogram, Figure 17.0.
As you can see when segmenting a different section of the histogram we do not
get an area that is spatially contiguous.
|
|
The 2 dimensional segmentation procedure proved to be very effective in segmenting an area with a discontinuity whereas the 1-D segmentation( Figure 7.0) was unable to reveal a discontinuity. To verify this discontinuity, an art conservator was brought in to explain the probable history. Using his own knowledge of Renaissance paintings and the results of this research, he confirmed that a discontinuity exists in the painting.
Looking at Figure 12.0, he acknowledged that a halo is present, but something was
underneath that could not be identified. However, when looking at Figure
16.0 (with 2 dimensional segmentation), the underlying mark was revealed.
With this information, he explained that in the past, halos were painted with gold, and
when they were cleaned, the cleaning solvent would remove the gold. This would leave
a mark on the painting. Then, the halo would be repainted. Therefore, this procedure
revealed that the area examined is not original to the painting.
(1) 
(2) 
(3) 
(4) 
