Applying Hyperspectral Reflectance Imaging to Investigate the Palettes and the Techniques of Painters

Summary

Hyperspectral Reflectance Imaging hypercubes include remarkable information into a large amount of data. Therefore, the request for automated protocols to manage and study the datasets is widely justified. The combination of Spectral Angle Mapper, data manipulation, and a user-adjustable analysis method constitutes a key-turn for exploring the experimental results.

Abstract

Reflectance Spectroscopy (RS) and Fiber Optics Reflectance Spectroscopy (FORS) are well-established techniques for the investigation of works of art with particular attention to paintings. Most modern museums put at the disposal of their research groups portable equipment that, together with the intrinsic non-invasiveness of RS and FORS, makes possible the in situ collection of reflectance spectra from the surface of artefacts. The comparison, performed by experts in pigments and painting materials, of the experimental data with databases of reference spectra drives the characterization of the palettes and of the techniques used by the artists. However, this approach requires specific skills and it is time consuming especially if the number of the spectra to be investigated becomes large as is the case of Hyperspectral Reflectance Imaging (HRI) datasets. The HRI experimental setups are multi-dimensional cameras that associate the spectral information, given by the reflectance spectra, with the spatial localization of the spectra over the painted surface. The resulting datasets are 3D-cubes (called hypercubes or data-cubes) where the first two dimensions locate the spectrum over the painting and the third is the spectrum itself (i.e., the reflectance of that point of the painted surface versus the wavelength in the operative range of the detector). The capability of the detector to simultaneously collect a great number of spectra (typically much more than 10,000 for each hypercube) makes the HRI datasets large reservoirs of information and justifies the need for the development of robust and, possibly, automated protocols to analyze the data. After the description of the procedure designed for the data acquisition, we present an analysis method that systematically exploits the potential of the hypercubes. Based on Spectral Angle Mapper (SAM) and on the manipulation of the collected spectra, the algorithm handles and analyzes thousands of spectra while at the same time it supports the user to unveil the features of the samples under investigation. The power of the approach is illustrated by applying it to Quarto Stato, the iconic masterpiece by Giuseppe Pellizza da Volpedo, held in the Museo del Novecento in Milan (Italy).

Introduction

Reflectance Spectroscopy (RS) and Fiber Optics Reflectance Spectroscopy (FORS) are based on the detection of the light reflected by surfaces once illuminated by a light source, typically a tungsten-halogen lamp. The output of the acquisition system is constituted by spectra where the reflectance is monitored as a function of the wavelength in a range that depends on the characteristics of the employed experimental setup^1,2,3. Introduced during the last four decades^4,5, RS and FORS are typically used in combination with X-ray fluorescence and other spectroscopies to describe the materials and the techniques used by artists to realize their masterpieces^6,7,8,9. The study of the reflectance spectra is usually performed by comparing the data from the sample with a group of reference spectra selected by the user in personal or public databases. Once the reference spectra that comply with the realization period of the sample and with the modus operandi of the artist have been identified, the user recognizes the main features of the reflectance spectra (i.e., transition, absorption, and reflection bands^1,2,10,11) and then, with the help of other techniques^6,7,8 they distinguish the pigments that have been used in the paintings. Finally they discusses the slight differences that there exist between the references and the experimental spectra^7,9.

In most cases, the experimental datasets are composed of a few spectra, collected from areas chosen by art experts and assumed to be significant for the characterization of the painting^{6, 12,13}. Despite the skills and the experience of the user, a few spectra cannot fully exhaust the characteristics of the whole painted surface. Moreover, the result of the analysis will always be strongly dependent on the expertise of the performer. In this scenario, Hyperspectral Reflectance Imaging (HRI^3,14,15) could be a useful resource. Instead of a few isolated spectra, the experimental setups return the reflectance properties of extended portions or even of the whole artefact under investigation¹⁶. The two main advantages with respect to the acquisition of the isolated spectra are evident. On one hand, the availability of the spatial distribution of the reflectance properties allows the identification of areas that hide interesting features, even though they may not seem peculiar¹⁷. On the other hand, the hypercubes guarantee a number of spectra high enough to enable the statistical analysis of the data. These facts support the comprehension of the distribution of pigments within the painted surface^18,19.

With HRI, the comparison of the experimental data with the references could be hard to handle¹⁵. A typical detector returns hypercubes of at least 256 x 256 spectra. This would require the user to evaluate more than 65,000 reflectance spectra against each reference, a task almost impossible to be carried out manually in a reasonable time. Therefore, the request for robust and, possibly, automated protocols to manage and analyze HRI datasets is more than justified^15,17. The proposed method answers this need by handling the whole analytic procedure with the minimum involvement and the maximum flexibility.

An algorithm comprising a set of home-made codes (Table of Materials) reads, manages, and organizes the files returned by the experimental setup. It allows the fine selection of the portions of the Fields of View (FOVs, one field of view is the area of the painting monitored by a single hypercube) to be studied and performs the analysis of the data based on the Spectral Angle Mapper (SAM) method^20,21 and on the manipulation of the original spectra. SAM returns false color gray-scale images called similarity maps. The values of the pixels of these maps correspond to the spectral angles that are the angles between the spectra stored in the hypercubes and the so-called End Members (EMs, a group of reference spectra that should describe the features of the surface monitored by the hypercubes)²². In the case of RS applied to paintings, the EMs are the reflectance spectra of pigments that should match the palette of the Master. They are chosen based on the available information about the artist, the realization period of the painting, and the expertise of the user. Therefore, the output of the SAM is a set of maps that describes the spatial distributions of these pigments over the painting surface and that supports the user to infer the materials used by the artist and their organization in the artefact. The algorithm offers the possibility of employing all kind of references independently from their origin. The references can be specific spectra selected within the hypercubes, come from databases, be acquired by a different instrument on a different surface (such as samples of pigments or the palette of the artist, for instance), or be obtained employing any kind of reflectance spectroscopy, FORS included.

SAM has been preferred among the available classification methods because it has been demonstrated to be effective for characterizing pigments (refer to the book by Richard²³ to have an overview of the main available classification methods). Instead, the idea of developing a home-made protocol rather than adopting one of the many tools freely available on the net^24,25 relies on a practical consideration. Despite the effectiveness and scientific foundation of the existing GUIs and software, a single tool hardly satisfies all the needs of the user. There could be an Input/Output (I/O) issue because a tool does not manage the file containing the raw data. There could be an issue regarding the analysis of the data because another tool does not provide the desired approach. There could be a limitation in the handling of the data because the simultaneous analysis of multiple datasets is not supported. In any case, a perfect tool does not exist. Each method must be adjusted to the data or vice versa. Therefore, the development of a home-made protocol has been preferred.

The presented approach offers neither a complete set of analytical methods (see, for comparison, the tool proposed by Mobaraki and Amigo²⁴) nor an easy-to-manage user-interface (see, for comparison, the software employed by Zhu and co-workers²⁵), but, in exchange, it focuses on a still underrated aspect of hyperspectral data analysis: the opportunity to manipulate the detected spectra. The power of the approach is illustrated by applying it to the painting Quarto Stato by Giuseppe Pellizza da Volpedo (Figure 1), an iconic oil on canvas held in the Museo del Novecento in Milan, Italy. Note that, since the approach requires running home-made codes, the developer arbitrarily chose the names of the codes and both the input and output variables used in the description of the protocol. The names of the variables can be changed by the user but they must be provided as follow: the input and out variables must be written respectively within brackets and eventually separated by comma and within square brackets and eventually separated by a white space. On the contrary the names of the codes cannot be altered.

Protocol

1. Set the spatial resolution of the hypercubes

1. Perform a preliminary inspection of the painted surface (Figure 1) supported by art experts to identify the main features of the painting.
   1. Recognize the pictorial techniques employed by the artist to create the painting.
   2. Identify the different brush strokes of paint on the canvas.
   3. Estimate, qualitatively, the characteristics of the brush strokes with particular attention to their size.
2. Mimic the pictorial technique used by the artist by creating ad-hoc test samples where the brush strokes show characteristics similar to those applied by the artist.
   NOTE: Pellizza da Volpedo was a Divisionist painter. A restorer was asked to prepare some test samples that qualitatively reproduce the brush strokes of the canvas of interest (Figure 2, column A).
3. Set the distance between the surface under investigation and the acquisition equipment.
   NOTE: The distance determines the spatial resolution of the hypercubes²⁶ and therefore the possibility to distinguish the brush strokes on the images and SAM maps of the painted surface.
   1. Evaluate the distance between the surface of the sample and the acquisition equipment taking into account the characteristics of the hyperspectral camera²⁶ (Table of Materials) and the size of the brush strokes drawn in the test samples.
   2. Put the acquisition stage and the hyperspectral camera at the distance evaluated in the previous step. Arrange the test samples on the stage and ensure uniform illumination of the surface of the samples.
   3. Perform a white calibration using the white standard reference supplied with the hyperspectral camera. Acquire the hypercubes.
      NOTE: For each FOV, the hyperspectral camera returns both raw and calibrated images. The latter have been used for the analysis.
   4. Download the files returned by the instrument and save them in a dedicated folder.
4. Check whether the spatial resolution of the hypercubes can distinguish the different brush strokes on the images and SAM maps of the painted surface.
   1. Inspect the RGB pictures returned by the hyperspectral camera to ensure that the brush strokes used to realize the test samples can be recognized (Figure 2, column A). If so, move to the next steps; otherwise go back to step 1.3.1 and restart.
   2. List the files containing the hyperspectral data and the RGB images of the FOVs by running the data reading code, HS_FileLister. Type the following command line (semicolon included) in the terminal window of the language used to develop the codes (Table of Materials) and press Enter to run the code:
      [HS_DataList HS_ImageList] = HS_FileLister;
      1. No input is required and there are two outputs: the list of the files containing the hypercubes, HS_DataList, and the list of the images returned by the hyperspectral camera, HS_ImageList.
         NOTE: The size of each hypercube is 512 x 512 x 204 voxels where 204 is the number of channels used to monitor the reflectance signal. The channels span the wavelength range between 400 and 1,000 nm with a spectral resolution of 7 nm at FWHM²⁶.
   3. Define the 3D portion of the hypercubes that must be analyzed by running the cropping code, HS_Crop_png. Define the desired portion of each data-cube by selecting an area over an interactive window that shows the 2D, RGB image of the FOV monitored by each hypercube. Type the following command line (semicolon included) in the terminal window and press Enter to run the code:
      [HS_ImageList] = HS_Crop_png(HS_ImageList);
      1. There is one input (the list of the images returned by the hyperspectral camera, HS_ImageList) and one output (the input list added with the spatial coordinates to eventually crop the hypercubes).
   4. Apply the D65 illuminant and 1931 observer from CIE (International Commission on Illumination) standards to the hypercubes to retrieve the RGB images of the FOV(s) from the reflectance spectra by running the re-building code, HS_RGB_rebuild. Type the following command line (semicolon included) in the terminal window and press Enter to run the code:
      [HS_ImageList] = HS_RGB_rebuild(HS_ImageList, HS_DataList);
      1. There are two inputs (the list containing the images returned by the hyperspectral camera, HS_ImageList, and the list of the files containing the hypercubes, HS_DataList) and one output (the input list containing the images returned by the hyperspectral camera added with the RGB images of the surfaces of the hypercubes retrieved from the reflectance spectra).
         NOTE: HS_RGB_Rebuild exploits the functions developed by Jeff Mather²⁷ to apply the D65 illuminant and 1931 observer from CIE to the data.
   5. Manually select some reference spectra on the surfaces of the test samples (White Circles and Numbers in Figure 2, column A) by running the isolated measuring points selection code, PointSel. Select the measurement points by clicking an interactive window that shows, one by one, the 2D, RGB images of the FOV(s). Type the following command line (semicolon included) in the terminal window and press Enter to run the code:
      [References] = PointSel(HS_DataList, HS_ImageList);
      1. There are two inputs (the list containing the images returned by the hyperspectral camera, HS_ImageList, and the list of the files containing the hypercubes, HS_DataList) and one output (a variable, References, containing the spectra selected as references within the FOV(s)).
   6. If desired, store the position of the references over the surface of the samples into a set of pictures by running the dedicated code, SaveImPoint. Type the following command line (semicolon included) in the terminal window and press Enter to run the code:
      ​SaveImPoint(References, HS_ImageList);
      1. There are two inputs (the variable containing the reference spectra, References, and the list containing the images returned by the hyperspectral camera, HS_ImageList) and no outputs (the code saves .png images in the current work folder).
   7. Organize the references into a matrix by running the conversion code, RefListToMatrix. Type the following command line (semicolon included) in the terminal window and press Enter to run the code:
      [References_Matrix] = RefListToMatrix(References, HS_ImageList(1).WaveL);
      1. There are two inputs (the variable containing the reference spectra, References, and the list of the wavelengths at which the detector counts the photons during the data acquisition of the spectra, HS_ImageList(1).WaveL) and one output (the same reference spectra organized into a matrix, References_Matrix).
         NOTE: This step is mandatory because the code that evaluates the SAM maps requires the reference spectra to be organized into a matrix. The syntax of the second input, HS_ImageList(1).WaveL, is required to recall the variable WaveL from the list HS_ImageList. The number 1 within brackets refers to the first element of the list named as HS_ImageList; however, since all the hypercubes have the same wavelength range, it can be substituted by each number minor or equal to the total number of listed images.
   8. Extract the SAM maps using the whole spectra by running the standard SAM maps evaluation code, SAM_Standard. Type the following command line (semicolon included) in the terminal window and press Enter to run the code:
      SAM_Standard(HS_ImageList, HS_DataList, References_Matrix);
      1. There are three inputs (the list containing the images returned by the hyperspectral camera, HS_ImageList; the list of the files containing the hypercubes, HS_DataList; and the matrix of the reference spectra, References_Matrix) and no output: the code saves the SAM maps as .png images in the current work folder.
   9. Check whether the obtained similarity maps (Figure 2, columns B - E) display the details of the brush strokes used to realize the test samples. If this is the case, move to the next step of the protocol; otherwise go back to step 1.3.1 and restart.

2. Adjust the experimental parameters to the painting

1. Identify the Region(s) of Interest, ROI(s), of the painting to be studied (red rectangles in Figure 3A).
   NOTE: It is common that more than one FOV is necessary to cover a single ROI.
2. Arrange the acquisition setup and the painting at the distance defined in the previous steps and perform the white calibration employing the white standard reference supplied with the hyperspectral camera.
   NOTE: If the users must do an in situ acquisition (i.e., they must study a painting exposed in a museum or at an exhibition), they can only manage the camera. This is the case of Quarto Stato, which is permanently exposed in a dedicated space at the Museo del Novecento in Milan, Italy.
3. Acquire the hyperspectral data from at least one FOV within the edge of each ROI(s) (unshaded areas within the red rectangles in Figure 3A).
4. Download the files returned by the instrument and save them into a dedicated folder.
5. Check whether the illumination of the surface of the painting has been uniformly set by looking at the RGB images returned by the hyperspectral camera. If this is the case, move to the next steps, otherwise go back to step 2.2 and restart.
   NOTE: Figure 4 illustrates the importance of this check (see the Discussion section for the details).
6. Repeat the sub-steps of step 1.4.
7. Check whether the data have a spatial resolution high enough to distinguish the brush strokes by observing the RGB pictures of the FOVs (Figure 3B) and the SAM maps (Figure 3C) related to the reference spectra selected within the FOVs (green circles in Figure 3B).
8. If the illumination and the spatial resolution have been properly set, complete the collection of the data acquiring the other FOVs necessary to cover the ROI(s); otherwise go back to step 2.2 and restart.
   ​NOTE: When a ROI requires more than one FOV to be covered, ensure a certain degree of superposition between adjacent FOVs to easily stitch the resulting maps^3,15. The extent of the overlapping depends on the distance between the hyperspectral camera and the sample, on the translation, and the horizontal angle of view of the detector²⁸. In the case of the experimental campaign conduced on Quarto Stato, the overlap has been set to be at least the 40% of the FOVs.

3. Hypercubes and the reference spectra management

1. Perform the I/O of the raw data: organize, read, and manage the hypercubes.
   1. Run the HS_FileLister code to store the list of the files containing the hypercubes and the related information into two variables at the disposal of the algorithm (see step 1.4.2 for the practical details).
      NOTE: The hyperspectral camera returns hdr (high dynamic range) files that the code manages exploiting a revisited version of the script developed by Jarek Tuszinsky²⁹.
   2. Run the HS_Crop_png code to select the portion of each FOV to be used in the analysis of the data (see step 1.4.3 for the practical details).
   3. Run the HS_RGB_Rebuild code to retrieve the RGB images of the FOVs from the reflectance spectra (see step 1.4.4 for the practical details).
2. Organize, read (if required), and manage the reference spectra.
   NOTE: The reference spectra will play the role of the end members within the SAM method^20,21. This part of the algorithm is not univocally determined but depends on the selection mode and on the origin of the reference spectra.
   1. Run the PointSel code and click within the displayed interactive window to identify the reference spectra as isolated measuring points over the surface of the monitored area(s) (Figure 5A) (see step 1.4.5 for the practical details).
   2. Automatically select the reference spectra as a regular reticulum of measuring points superimposed to the surface of the monitored area(s) by running the reticular selection code, ReticularSel (Figure 5B). Type the following command line (semicolon included) in the terminal window and press Enter to run the code:
      [References] = ReticularSel(HS_DataList, HS_ImageList, n_pixel);
      1. There are three inputs (the list containing the images returned by the hyperspectral camera, HS_ImageList; the list of the files containing the hypercubes, HS_DataList; and the spacing of the reticulum, n_pixel, expressed in number of pixels) and one output: a variable containing the spectra selected as references within the FOVs, References.
   3. Run the external references importer code, Spectra_Importer, to create a variable containing references from datasets and databases independent from the hypercubes acquired on Quarto Stato. Type the following command line (semicolon included) in the terminal window and press Enter to run the code:
      [ExtReferences] = Spectra_Importer(file_extension);
      1. There is one input (the extension of the file containing the independent reference spectra, file_extension, written between apices) and one output (a variable containing the external references, ExtReferences).
         NOTE: The external reference importer code has been optimized for importing tmr files but, if necessary, it can be easily modified to deal with any kind of text file.
   4. Run the RefListToMatrix code to put the references into a matrix, References_Matrix or ExtReferences_Matrix, as required by the code that evaluates the SAM maps (see step 1.4.7 for the practical details).
   5. Wait for the RefListToMatrix code to equalize both the wavelength range and the spectral resolution (i.e., the number of components) of the hypercubes and the references.
      ​NOTE: The code identifies the wavelength ranges of both the hypercubes and the references. The code compares the wavelength ranges and cuts off the wavelength interval(s) that are not monitored by both the hypercubes and the references. The code identifies the group of hyper-vectors (the hypercubes or the references) constituted by the lower number of components (i.e., characterized by the lower spectral resolution) in the equalized wavelength range. The code reduces the number of components of the longer hyper-vectors (the references or the hypercubes) to that of the shorter ones (the hypercubes or the references). This is done by keeping, for each wavelength of the shorter hyper-vectors, only the values of the longer hyper-vectors that correspond to the nearest wavelength to that of the shorter hyper-vectors.
      1. The code automatically performs the equalization. If the references have been selected within the hypercubes, the wavelength range and the spectral resolution do not need to be equalized and they remain unchanged.
   6. If desired, store the position of the references over the surface of the samples into a set of pictures by running the dedicated code (see step 1.4.7 for the practical details).
      ​NOTE: This option is available only if the references have been selected within the hypercubes (steps 3.2.1 and 3.2.2).

4. SAM analysis

1. Run the SAM_Complete code to evaluate the similarity maps. Type the following command line (semicolon included) in the terminal window and press Enter to run the code:
   SAM_Complete(HS_ImageList, HS_DataList, References_Matrix);
   1. There are three inputs (the list containing the images returned by the hyperspectral camera, HS_ImageList; the list of the files containing the hypercubes, HS_DataList; and the reference matrix, References_Matrix or ExtReferences_Matrix) and no outputs (the code saves the SAM maps as .png files in the current work folder).
      NOTE: Other than the three described input variables, the SAM_complete code must be fed with few additional parameters to tailor the analysis protocol according to the preferences of the user (see the next steps).
2. When required, feed the code with the pre-processing option by typing the number 0 or 1 in the dialog box depending on the desired pre-processing operation and press Enter to continue.
   1. Pre-processing option set to 0: the area subtended by each reflectance spectrum is normalized to 1.
   2. Pre-processing option set to 1: the area subtended by each reflectance spectrum is normalized to 1 and then the normalized spectrum is derived one time.
      NOTE: Both the hypercubes and the references undergo the same pre-processing option.
3. Select the end members to be used for the SAM analysis among the reference matrix by feeding the code with the numbers of the columns that correspond to the desired spectra. When required, enter into the dialog box the sequence of numbers corresponding to the desired columns by typing the numbers separated by a white space. Press Enter to continue.
   NOTE: The sequence [1 2 3] corresponds to the selection of the first three columns of the reference matrix; an empty vector corresponds to the selection of all the columns of the reference matrix.
4. Feed the code with a string containing the first part of the name that will identify the sets of maps to be saved (i.e., the common part of the name of the .png files returned by SAM_Complete). When required, insert the string in the dialog box. Press Enter to continue.
   NOTE: If the user types test, then the name of all the output .png images will start with test.
5. When required, feed the code with the method selected to handle the data by typing the number 0, 1, or 2 in the dialog box depending on the desired handle method and press Enter to continue.
   1. Set the method to 0 for no manipulation of the data.
   2. Set the method to 1 to require manual selection of the wavelength range(s) of the spectra to be considered before starting the analysis (Figure 6).
   3. Set the method to 2 to require the algorithm to order the data on the basis of a specific criterion before the evaluation of the SAM maps (Figure 7).
6. Wait for the protocol to process the data and to save the SAM maps in the current work folder as .png files.
   NOTE: If the handle method has been set to 0 or 2, the user must just wait. If it has been set to 1, the user must select the portion(s) of the spectra to be employed for evaluating the SAM maps by clicking on an interactive window (Figure 6).

Results

The proposed protocol offers a set of interesting features for the management and the analysis of HRI data. The I/O (step 3.1) of the raw data is always the first problem that must be solved before applying any analysis method and it can become a critical issue when dealing with large amounts of data. In the present case, the only task regarding the raw data is to store the experimental results into a dedicated folder and select it by browsing the hard disk when running the reading code (step 3.1.1). Thereafter, the crop...

Discussion

Hyperspectral reflectance imaging datasets are large reservoirs of information; therefore, the development of robust and, possibly, automated protocols to analyze the data is a key turn to exploit their potential^15,17. The proposed algorithm answers this need in the field of cultural heritage with particular attention to the characterization of the pigments of paintings. Based on SAM^20,21, the algorithm s...

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
ImageJ/Fiji	Specim (Oulo, Finlad)	N/A	Portable reflectance hyperspectral camera used to acquire the hypercubes
MATLAB 2019b	StellarNet Inc (Tampa, Florida, USA)	N/A	Portable reflectance spectrometer used to acquire independent reflectance spectra
Specim IQ Hyperspectral Camera	National Institutes of Health (Bethesda, Maryland, USA)	N/A	Open source Java image processing program
StellarNet BLUE-wave Miniature Spectrometer	MathWorks (Natick, Massachusset, USA)	N/A	Program Language and numerical computing environment