Comparison of Pigment Classification Algorithms on Non-Flat Surfaces using Hyperspectral Imaging

Dipendra J. Mandal; Marius Pedersen; Sony George; Clotilde Boust

doi:10.2352/J.ImagingSci.Technol.2023.67.5.050405

Abstract

Cultural heritage objects, such as paintings, provide valuable insights into the history and culture of human societies. Preserving these objects is of utmost importance, and developing new technologies for their analysis and conservation is crucial. Hyperspectral imaging is a technology with a wide range of applications in cultural heritage, including documentation, material identification, visualization and pigment classification. Pigment classification is crucial for conservators and curators in preserving works of art and acquiring valuable insights into the historical and cultural contexts associated with their origin. Various supervised algorithms, including machine learning, are used to classify pigments based on their spectral signatures. Since many artists employ impasto techniques in their artworks that produce a relief on the surface, i.e., transforming it from a flat object to a 2.5D or 3D, this further makes the classification task difficult. To our knowledge, no previous research has been conducted on pigment classification using hyperspectral imaging concerning an elevated surface. Therefore, this study compares different spectral classification techniques that employ deterministic and stochastic methods, their hybrid combinations, and machine learning models for an elevated mockup to determine whether such topographical variation affects classification accuracy. In cultural heritage, the lack of adequate data is also a significant challenge for using machine learning, particularly in domains where data collection is expensive, time-consuming, or impractical. Data augmentation can help mitigate this challenge by generating new samples similar to the original. We also analyzed the impact of data augmentation techniques on the effectiveness of machine learning models for cultural heritage applications.

jist

JIMTE6

Journal of Imaging Science and Technology

J. Imaging Sci. Technol.

1062-3701

1943-3522

Society for Imaging Science and Technology

050405

10.2352/J.ImagingSci.Technol.2023.67.5.050405

1494

Work Presented at CIC31: Color and Imaging 2023

Comparison of Pigment Classification Algorithms on Non-Flat Surfaces using Hyperspectral Imaging

Comparison of pigment classification algorithms on non-flat surfaces using hyperspectral imaging

MandalDipendra J.

▴

PedersenMarius

▴

GeorgeSony

▴

Department of Computer Science, Norwegian University of Science and Technology (NTNU), Norway

dipendra.mandal@ntnu.no

BoustClotilde

Center for Research and Restoration of Museums of France (C2RMF), France

Mandal et al.

▴

IS&T Members.

092023

252023

1082023

2023

Abstract

ccc

1062-3701/2023/67(5)/050405/25/$25.00

printed

Printed in the USA

Introduction

In Cultural Heritage (CH), paintings are an essential tangible component that provides valuable insights into our history, social norms, and beliefs. Therefore, the preservation and restoration of paintings are crucial and poses numerous challenges, including the removal of dirt and old varnish without damaging the paint layer [1] and the selection of appropriate materials for retouching [2]. To address these problems, it is essential to accurately identify the pigment used by the artist in an artwork. Scientific analysis sometimes require the physical samples, however, due to the nature of CH objects, it is not recommended to take samples from the object which in fact destroy the object even at a microscale and so very often, it is not permitted. Consequently, non-invasive or non-contact imaging techniques [3–6] are necessary.

Hyperspectral Imaging (HSI) is a technology that has gained increasing attention in recent years due to its wide range of applications in various fields, including remote sensing [7], agriculture [8], medical sciences [9], forensics [10], biomedical engineering [11], and CH [12]. Additionally, in CH, spectral systems have also been used for measuring special appearance properties such as materials that are goniochromatic [13] or by integrating 3D information with spectral data [14]. An important aspect of CH is pigment classification [15], and HSI can facilitate it by using spectral information about pigments and different classification algorithms. In this work we focus on pigment classification using HSI. Artwork is not confined to two-dimensional canvases or boards, which means they are not always flat. The addition of relief, which introduces 2.5D or 3D to artwork, is also an important consideration [16, 17]. Several factors can contribute to this third dimension [18–20]; For example, morphological textures of brushstrokes on the painted surface [21], a thick layer of pigments applied by many renowned artists to their artwork for creating depth (impasto technique) [22–25]. The geometry of relief raised from a brush painting, impasto techniques, or any other factors may affect how light interacts with the surface, affecting the spectral signature captured by the hyperspectral sensor for a given pixel. Considering the importance of pigment identification or classification of an artwork, the influence of such factors must be explored.

Most of the research conducted so far has explored the effectiveness of various traditional supervised algorithms and machine learning models for pigment classification using HSI [26, 27]. Pigment classification has been done for single pigments or mixtures of pigments. In the latter case, unmixing is required to identify the different pigments. When the materials are known pigment mapping is used to find the right combination of the materials and their relative concentrations [28]. This can be done by using the optical propeties of the material and a mixing model. For this, several models exist, for example based on Kubelka-Munk theory [28–31]. Recently, Fukomoto et al. [32, 33] used an encoder-decoder neural network to estimate pigment concentrations. In our work we deal with a single pigment, not mixtures of pigments. The existing studies have primarily focused on flat surfaces and, to date, no research has investigated the same for artwork with an elevation. This paper aims to investigate how surface elevation in artworks affects the accuracy of pigment classification using HSI, with the underlying hypothesis that surface elevation impacts this accuracy. In CH, the lack of sufficient training datasets is also a considerable challenge for using machine learning, particularly in domains where collecting data is expensive, time-consuming, or impractical [34, 35]. Data augmentation is a technique that can help mitigate this challenge by generating new samples similar to the original data. Therefore, in this paper, we have also compared and analyzed the impact of data augmentation techniques on the effectiveness of machine learning algorithms for pigment classification. This study is primarily concerned with answering the following research questions:

∙

To what extent does elevation of a surface in artworks affect the accuracy of pigment classification using hyperspectral imaging?

∙

What is the influence of data augmentation techniques on the efficacy of machine learning models for material classification?

The rest of the paper is organized as follows. Section 2 briefly reviews the classification algorithms used in the CH field, focusing on pigment classification. Section 3 provides a brief overview of the classification algorithms used in this study. Section 4 describes the materials and methods used in the present study. Section 5 presents the results of the experiments and discusses the findings in detail. Finally, Section 6 concludes the paper and highlights directions for future research.

Background

Over the last few years, HSI, a non-invasive technique, has been widely employed for pigment classification in artwork [5, 26, 36]. It has resulted in significant advances in the study of spectral signatures and matching, broadening the scope of HSI technology in the CH domain. Molecules in the materials have unique vibration frequencies, which can be detected by analyzing how they absorb or reflect light at specific wavelengths. These characteristics of materials are known as spectral signatures and help to identify and distinguish pigment based on how it interacts with electromagnetic radiation. Many spectral matching algorithms have evolved in hyperspectral image processing, ranging from traditional clustering techniques to more recent automated matching models. The approaches used for matching spectra can be classified as deterministic, i.e., based on geometrical and physical aspects, or stochastic, which is based on the distributions [37]. These algorithms are essential for accurate and efficient pigment identification and analysis, making them a critical component of any HSI workflow. This section will briefly overview the classification algorithms employed in CH, specifically for pigment classification.

The Euclidean Distance (ED) metric is widely used to measure spectral similarity in HSI, and it works well when a data set has distinct or isolated clusters [38–40]. Mandal et al. [27] implemented ED and other supervised classification techniques to classify pigments on a flat surface and observed that classification accuracy declines for some pigments with similar spectral characteristics.

Deborah et al. [5] explored the application of HSI in mapping the pigments of Edvard Munch’s painting, The Scream. They used two methods for spectral image classification, namely Spectral Angle Mapper (SAM) and Spectral Correlation Mapper (SCM). They observed that SCM performed better than SAM as it accounted for both positive and negative correlations between the spectra. Adjusting the threshold value for SAM reduced false detection, but it also decreased the accuracy of pigment classification, which varied depending on the type of pigment. They further suggested that different classes of pigments would require the use of distinct threshold values. In Ref. [41], the authors used the SAM technique to classify traditional Chinese pigments and recommended a similar suggestion of using different thresholds. The SAM algorithm measures the angle between two vectors, independent of the vector length, and therefore, insensitive to gain. As a result, this algorithm does not account for magnitude shifts in the spectrum. Please refer to Osmar et al. [42] for more information.

George and Hardeberg [43] demonstrate the usefulness of HSI to separate inks using SAM and SID (spectral information divergence) as classification algorithms. They found that the SID algorithm performs better than SAM in cases, where two distinct inks were overlaid. However, they also posited that misclassification might arise from noise and non-uniformity in spectral signatures resulting from ink-paper blends. Mishra et al. [44] used HSI to evaluate hybrid spectral similarity measures to classify paper samples used in forensic investigations. The findings indicate that the hybrid similarity measures of SIDSCM demonstrate better classification accuracy than conventional spectral similarity measures. Furthermore, the classification accuracies of SIDSCM and SIDSAM are comparable.

The authors [45] discuss the use of SAM and machine learning (ML) models to classify mineral pigments used in ancient Chinese paintings using HSI. The results show that, for similar colors and spectra, SAM is unable to classify; however, combining it with a decision tree can effectively improve the accuracy. The authors of [46] discuss the effectiveness of HSI technology in archaeological research for identifying and classifying materials in ancient tombs. They found that combining HSI data with Principal Component Analysis (PCA) transformation and Support Vector Machine (SVM) classification was an effective method for accurately classifying and identifying materials. The SVM classification based on feature bands improved classification accuracy and reduced data processing time. Kleynhans et al. [47] discussed using reflectance HSI and Neural Networks (NNs) to create labeled pigment maps of paintings. The authors reported that a one-dimensional convolutional neural network (1D-CNN) model could accurately label the pigments in most of the paintings studied. However, this finding highlights the importance of having comprehensive training data for the model to perform well, the need for further studies to expand the training dataset, and the possibility of augmenting existing training datasets to develop a more robust solution.

Lie et al. [48] explored the potential of using NNs to analyze HSI data in the CH field. They presented a thorough overview of the different applications and constraints of NNs models. Their findings indicate that NNs offer a promising alternative to conventional statistical and multivariate analysis techniques for pigment identification and classification. The authors in Ref. [49] present a method for identifying pure pigments in CH using a combination of CNNs and SCM. The HSI data, collected within the range of 400nm to 720nm and at a resolution of 10nm, was pre-processed by smoothing and computing the first derivative before being fed to the network. The study emphasizes the significance of employing deep learning NNs for this application and the requirement for a comprehensive training dataset. A recent study by Mandal et al. [27] investigated the performance of various traditional supervised algorithms, their hybrid combinations, and ML models for pigment classification on flat surfaces. A research gap exists as the efficacy of these algorithms on non-flat objects has not been explored by any of the authors. Thus, further investigation is necessary for this area.

Classification Algorithms

This section presents the fundamental theory and mathematical expressions for the classification algorithms used in the study.

3.1

Euclidean Distance

Euclidean Distance (ED) is a distance metric that measures the distance between two points in an N-dimensional space [50]. It is calculated as the square root of the sum of the squared differences between the corresponding elements of the two points. In spectral analysis, the ED can compare the similarity between two spectra by measuring the difference between their respective pixel or spectral values. The formula for ED between the image spectrum ti and a reference spectrum ri, each with n elements, is defined using Eq. (1).

(1)

E D = \sqrt{\sum_{i = 1}^{n b} {(t_{i} - r_{i})}^{2}},

where, nb is the number of spectral bands.

3.2

Spectral Angle Mapper

Spectral Angle Mapper (SAM), introduced by Boardman in 1992 [51], is a method for measuring the spectral similarity between two spectra, i.e., a test (also referred to as a target) and reference spectrum. This technique treats the spectra as N-dimensional vectors in space, where N is the number of spectral bands and calculates the arccosine angles between them. The spectral angle, α, between the two spectra is computed using Eq. (2). A smaller angle indicates a better match between the spectra.

(2)

α = {cos}^{- 1} (\frac{\sum_{i = 1}^{n b} t_{i} r_{i}}{\sqrt{\sum_{i = 1}^{n b} {t_{i}}^{2}} \sqrt{\sum_{i = 1}^{n b} {r_{i}}^{2}}}),

where ti represents the image spectrum, ri denotes the reference spectrum, and nb is the total number of bands.

3.3

Spectral Correlation Mapper

Spectral Correlation Mapper (SCM) is one of several algorithms used in spectral similarity analysis for classification and feature extraction. It measures the Pearson correlation coefficient between two spectra by standardizing the data and centering them around the mean of the test and reference spectra. The result is then expressed as an angle using the arccosine function. This algorithm excludes negative correlation and retains the shading effect minimization characteristics similar to SAM, resulting in better classification results [42, 52]. SCM is computed using Eq. (3).

(3)

α = {cos}^{- 1} (\frac{\sum_{i = 1}^{n b} (t_{i} - {\bar{t}}_{i}) (r_{i} - {\bar{r}}_{i})}{\sqrt{\sum_{i = 1}^{n b} {(t_{i} - {\bar{t}}_{i})}^{2} \sum_{i = 1}^{n b} {(r_{i} - {\bar{r}}_{i})}^{2}}}),

where α is the arccosine of the spectral correlation measure in radians, ti and

{\bar{t}}_{i}

are the image spectrum and its sample mean, similarly ri and

{\bar{r}}_{i}

are the reference spectrum and its sample mean; and nb is the total number of bands.

3.4

Spectral Information Divergence

In terms of spectral similarity, Spectral Information Divergence (SID) measures the dissimilarity between two spectra by comparing their spectral information content. It is based on the concept of Kullback-Leibler (KL) divergence, a measure of the difference between two probability distributions. In SID, the spectral information content of each pixel is modeled as a probability distribution, and the divergence between the two distributions is calculated. If two pixels have similar spectral information, their probability distributions will be similar, and the SID value will be low, and vice versa. The probability distribution of the test and reference spectra is expressed as Eq. (4) and Eq. (5), respectively [53].

(4)

\begin{matrix} p_{i} = \frac{t_{i}}{\sum_{i = 1}^{n b} t_{i}} \end{matrix}

(5)

\begin{matrix} q_{i} = \frac{r_{i}}{\sum_{i = 1}^{n b} r_{i}}, \end{matrix}

where, ti is the image spectrum, ri is the reference spectrum, and nb is the total number of bands. Using these two probability distributions, SID can be calculated with Eq. (6).

(6)

S I D = \sum_{i = 1}^{n b} p_{i} log (\frac{p_{i}}{q_{i}}) + \sum_{i = 1}^{n b} q_{i} log (\frac{q_{i}}{p_{i}})

3.5

Spectral Similarity Scale

The Spectral Similarity Scale (SSS) provides a quantitative measure of the similarity between two spectra. This algorithm uses the ED metric for magnitude and correlation to compare the shape of the spectra. This method combines both calculations, giving each equal weighting [54]. An SSS value of 0 indicates that the two spectra are identical, while a value of 1 indicates that the two spectra are entirely dissimilar. SSS is computed using Eq. (7).

(7)

S S S = \sqrt{{(d_{e})}^{2} + {(\hat{r})}^{2}},

where, de is the ED between two spectra and is computed using Eq. (8) and its value ranges from 0 to 1 due to the factor 1∕nb.

(8)

d_{e} = \sqrt{\frac{1}{n b} \sum_{i = 1}^{n b} {(t_{i} - r_{i})}^{2}}

Equation (9) computes the value for

\hat{r}

, where r is the correlation coefficient between the two spectra and is computed using Eq. (10).

(9)

\begin{matrix} \hat{r} & = & (1 - r^{2}) \end{matrix}

(10)

\begin{matrix} r^{2} & = & {(\frac{\sum_{i = 1}^{n b} (t_{i} - {\bar{t}}_{i}) (r_{i} - {\bar{r}}_{i})}{\sqrt{\sum_{i = 1}^{n b} {(t_{i} - {\bar{t}}_{i})}^{2} \sum_{i = 1}^{n b} {(r_{i} - {\bar{r}}_{i})}^{2}}})}^{2} \end{matrix}

3.6

Spectral Information Divergence Spectral Angle Mapper

The Spectral Information Divergence Spectral Angle Mapper (SIDSAM) is a hybrid approach that incorporates quantitative and qualitative matching measures. It utilizes the SID algorithm to assess the dissimilarity between two spectra and the SAM algorithm to evaluate their geometric similarity. This hybrid computation enhances the comparability of similar spectra and makes dissimilar spectra more distinctive, thus improving spectral discriminability. SIDSAM is computed by multiplying the SID by the tangent or sine function of the SAM, which calculates the perpendicular distance between the test and reference vectors. Both measures yield similar results, as reported in previous studies [55]. SIDSAM can be calculated using either Eq. (11) or (12).

(11)

\begin{matrix} SIDSAM = SID * tan (SAM) \end{matrix}

(12)

\begin{matrix} SIDSAM = SID * sin (SAM), \end{matrix}

where, SID and SAM are calculated using Eqs. (6) and (2), respectively.

3.7

Spectral Information Divergence Spectral Correlation Mapper

The Spectral Information Divergence Spectral Correlation Mapper (SIDSCM) is another hybrid approach that combines qualitative and quantitative metrics to increase spectral discriminability. It combines SID and SCM algorithms, similar to SIDSAM, where SID measures the difference between two spectra and SCM determines the Pearson correlation coefficient. To integrate the two measures, the product of SID and either the tangent or sine function of the correlation coefficient between two spectrais used [56]. The resultant method may be calculated using either Eq. (13) or (14).

(13)

\begin{matrix} SIDSCM = SID * tan (SCM) \end{matrix}

(14)

\begin{matrix} SIDSCM = SID * sin (SCM), \end{matrix}

where SID and SCM are computed using Eqs. (6) and (3) respectively.

3.8

Jeffries Matusita-Spectral Angle Mapper

The Jeffries Matusita (JM) distance is a statistical metric considering the covariance of two spectral vectors. The SAM method computes the angle between two spectral vectors to determine their spectral similarity. Jeffries Matusita-Spectral Angle Mapper (JMSAM)is calculated by first calculating the JM distance and then converting it to an angle with the inverse cosine method. The angle obtained is then compared to a threshold value to assess whether the two spectra belong to the same class. It can increase spectral classification accuracy by considering the spectral similarity and statistical distance [57]. It can be computed using either Eq. (15) or (16).

(15)

\begin{matrix} JMSAM = JMD * tan (SAM) \end{matrix}

(16)

\begin{matrix} JMSAM = JMD * sin (SAM), \end{matrix}

where, JMD is JM distance and is computed using Eq. (17)

(17)

J M D = 2 (1 - e^{- B}) .

Here B is the Bhattacharyya distance and is computed using Eq. (18).

(18)

\begin{matrix} B & = & \frac{1}{8} {(μ_{t} - μ_{r})}^{T} {[\frac{σ_{t} + σ_{r}}{2}]}^{- 1} (μ_{t} - μ_{r}) \\ + \frac{1}{2} l n [\frac{| \frac{σ_{t} + σ_{r}}{2} |}{\sqrt{| σ_{t} | | σ_{r} |}}], \end{matrix}

where, μt and μr are the mean of the test and reference spectra, respectively; σt and σr are the covariance of the test and reference spectra, respectively.

3.9

Support Vector Machine

Support Vector Machine (SVM) is a ML algorithm used for classification and regression analysis [58]. SVM classification aims to find a hyperplane that separates the data into two classes with maximum margin. It can handle non-linearly separable data using a kernel trick that maps it into a higher-dimensional space. This algorithm involves data preprocessing, such as normalization, to ensure that the features are on the same scale. It selects the most relevant features for the classification task. After that, it trains the model by finding the optimal hyperplane that maximizes the margin using a cost function. The cost function penalizes misclassified data points and encourages the SVM to find the hyperplane that separates the data with the largest margin. Finally, the model is tested on a validation set or test data set to evaluate its performance [59, 60].

3.10

1D-Convolutional Neural Network

A NN is a machine learning algorithm inspired by the structure and function of the human brain. The basic building block of a neural network is the artificial neuron, which takes inputs and applies a transformation to produce an output [61]. The architecture of a NN can vary widely depending on the task and the data being used. A 1D Convolutional Neural Network (1D-CNN) is a type of NN commonly used for processing one-dimensional data such as time series, audio signals, and text data [62–64]. It consists of one or more convolutional layers, a pooling layer, and fully connected layers. The convolutional layer applies convolution operations to the input sequence using a set of learnable filters or kernels. This generates a set of feature maps representing the convolutional layer’s output. The pooling layer is typically used to reduce the dimensionality of feature maps while maintaining the most important information. It applies an aggregation function such as max or average pooling to extract the most relevant features from each feature map. This helps reduce the number of parameters in the model and prevent overfitting. The fully connected layer takes the output of the pooling layer, flattens it into a one-dimensional vector, and passes it through a set of fully connected neurons. The output of the fully connected layer is often fed into a softmax function to generate class probabilities [65]. Overall, the 1D-CNN architecture is designed to extract and learn discriminative features from one-dimensional data sequences, making it suitable for various applications. Figure 1 illustrates the general architecture of a 1D-CNN for use with HSI data.

Figure 1.

The architecture of 1D-CNN; typically comprises three fundamental layers: convolutional, pooling, and fully connected layer. Input from the HSI datacube (pixel value across wavelength) is fed to the convolutional layers, which apply a sliding window over given input data to perform feature extraction. The pooling layers reduce the size of the extracted features, and the fully connected layers classify the input based on the features obtained from the previous layers.

Materials and Methods

This section describes the mockup used in the study and the HSI acquisition laboratory setup. Additionally, we will describe the steps used to process the HSI data for the classification task.

4.1

Mockup

As shown in Figure 2(a), a pigment mockup was prepared and used in a laboratory environment. The mockup’s base was printed using 3D printing. It consisted of different elevation levels, including a flat surface and regions raised to different levels 2.5 mm, 5 mm, and 10 mm from the base. A linen fabric was glued to the surface of the base. Three layers of white gesso were applied evenly across the entire surface of the canvas. The surface was carefully sanded between each layer using sandpaper to create a smooth and even coat. Fig. 2(b) illustrates the grayscale depth map of the pigment mockup to visualize the relative distances of different elevations from the camera’s perspective.

Figure 2.

The pigment mockup used in the study. (a) The mockup consisted of ten pigments labelled P1 to P10. These pigments were applied to a surface that included a flat region and three different elevation levels, namely 2.5 mm, 5 mm, and 10 mm. For better visualization of the elevation, the left part of the image is shown in grayscale, captured at 998 nm in the IR region, while the right side is a colour image produced using bands at 640 nm, 551 nm, and 458 nm. (b) A grayscale depth map of the mockup, this map is inverted, so black represents the foreground or the regions that has highest elevation (10 mm height).

The selection of pigments for the research work was a crucial step to ensure that the mockup accurately reflected the properties of pigments commonly used in historical artworks. The pigments were carefully chosen on the basis of their spectral characteristics and frequent appearance in CH research articles. The selection process also involved consultations with CH experts. Furthermore, web-based research was conducted on 164 known paintings from different centuries to determine the final pigments selection. We selected the pigments that were used most frequently in those paintings. This approach ensured that the pigments used in the mockup represented those used in real paintings, making the research findings more applicable to real-world scenarios. The final selection of pigments included Veridian (V), Cerulean Blue (CB), Green Earth (GE), Yellow Ochre Light (YOL), Blue Cobalt (BC), Ultramarine Blue Deep (UBD), Lead White Hue (LWH), Genuine Vermilion (GV), Burnt Umber (BU), and Ivory Black (IB). Each pigment was applied to the mockup surface with a width of 6 mm, and a 3 mm gap between each pigment. Each pigment was applied to the mockup surface with a width of 6 mm and a 3 mm gap between each pigment. We tried to achieve a uniform thickness for all pigments. Additionally, the pigments utilized were mostly opaque, effectively concealing the underlying substrate. The tubes were purchased from Zecchi, a supplier of art materials [66]. Safflower oil was used for the whites, while linseed oil was used for all other pigment tubes.

4.2

HSI Acquisition Setup

Hyperspectral images were acquired in a laboratory using HySpex VNIR-1800, a line scanner camera developed by Norsk Elektro Optikk [67], and a translation stage setup, the schematic illustration of the HSI system used in this study is illustrated in Figure 3. The detector of the HySpex camera consists of an actively cooled and stabilized complementary metal-oxide-semiconductor (CMOS). It has a spectral range of 400–1000 nm with 186 spectral bands having a spectral resolution of 3.26 nm. A computer with HySpex-GROUND software controlled the HSI acquisition system. This software synchronizes the scanning speed for the integration time set by the user. We have used a close-range 30 cm cylindrical lens for the acquisition. This allows to capture 1800 spatial pixels across a line with a field of view of approximately 86 mm.

During the experiment, we positioned the SpectralonⓇ [68], a ColorChecker [69], and a pigment mockup on a movable part of the translation stage, as shown in Fig. 3. These were placed at the same horizontal level and perpendicular to the camera’s focal axis [70]. The SpectralonⓇ, a multi-step reference target with four adjacent panels with reflectance levels of 99%, 50%, 25%, and 12% was used to calculate the normalized reflectance at the pixel level. To verify the obtained spectral data, we used a ColorChecker.

Figure 3.

Schematic illustration of the HSI system used in the experiment. A 500 Watt halogen-based floodlight was used for illumination, and the illumination geometry was set to 45

^{\circ}

-0

^{\circ}

-45

^{\circ}

, where 0

^{\circ}

is the camera angle to normal.

4.3

Data Processing

The raw hyperspectral data are preprocessed for dark current factor, sensor correction, and radiometric calibration using HySpex RAD software provided by the camera manufacturer. The preprocessed data (converted to sensor-level absolute radiance value) are then converted to normalized reflectance using the known reflectance value of the reference target used in the experiment. The reference target surface might have some variation in the pixel value, so we averaged the values from 100 pixels for each line scan and calculated the reference target radiance value. Due to the shorter distance between the sensor and the object, we assumed that the path radiance effect was negligible. The spectral data were then cropped to exclude the ColorChecker and the reference target. The data processing steps were computed using the open-source software Python 3.9 [71]. Equation (19) provides the mathematical formulation used for conversion. Further data processing steps for supervised and ML-based classifications are discussed in the following sections.

(19)

R_{Obj} (λ) = R_{Ref_t} (λ) \frac{r_{Obj} (λ)}{r_{Ref_t} (λ)},

where RObj(λ) is the reflectance of an object, RRef_t(λ) is the reflectance of reference target, rObj(λ) and rRef_t are the absolute sensor radiance values for the object and the reference target, respectively.

4.3.1

Data Processing Steps for Supervised Classification Algorithms

To conduct supervised classification, reference or ground truth spectra are required to compare similarity. For this purpose, we selected a flat region with dimensions of around 10 × 10 pixels to establish a spectral library. We then saved the mean spectra for each pigment based on these regions, ensuring that the number of pixels was consistent. The plot for the spectrum of ten pigments and a substrate is included in Appendix D. Our approach to constructing this library involved considering three different elevations for each pigment, as well as calculating an average spectrum that accounted for both elevated and flat surfaces. Data processing steps are shown with a block diagram in Figure 4.

Figure 4.

Workflow diagram illustrating the data processing steps for supervised classification algorithms used in the present study.

Selecting a threshold value is an important step in spectral matching [5, 41, 72], which involves identifying specific spectral ranges as belonging to one of several given classes. To achieve optimal classification results, the threshold value must balance minimizing misclassification rates and maximizing the number of correctly classified pigments. Mandal et al. [27] employed an empirical approach to determine an optimal threshold value, where the authors chose a small section of the HSI dataset from a mockup and extracted the reference spectrum by averaging 11 × 11 pixels from a flat region. They tested a range of values, computed the classification accuracy for various algorithms, and evaluated their accuracy using a confusion matrix. The threshold value used in our study was directly taken from this research and is mentioned in Table I

Table I.

The selected threshold value for eight different classification algorithms [27].

Algorithms	Threshold value
ED	0.9
SAM	0.1
SCM	0.8
SID	0.03
SSS	1.1
SIDSAM	0.003
SIDSCM	0.005
JMSAM	0.09

4.3.2

Data Processing Steps for ML-based Classification Algorithms

Before feeding the normalized reflectance HSI data to the ML model, the data were labeled for distinct classes using a label encoder. We used one hot encoder for our dataset, where each class has one hot value (1), and the rest are cold (0). The data was then split into training and testing sets using an 80-20 split and was further standardized. Subsequently, the model was built and implemented. The training dataset was used to train the model by updating the weights and biases of neurons with each epoch until a considerably low Mean Square Error (MSE) and high accuracy were achieved. Once the model was trained, the test dataset was used to validate its performance. The overall workflow is illustrated in a block diagram in Figure 5.

Figure 5.

Workflow diagram illustrating the data processing steps for ML.

Hyperparameter tuning is crucial in building robust and better generalized SVM models [73]. We tuned our SVM model for three key hyperparameters: kernel function, regularization (C), and gamma [59]. The kernel function transforms the input space into a higher-dimensional feature space, allowing the SVM to find a hyperplane (decision boundary) that can separate the classes. Several kernel functions are available, including linear, polynomial, and radial basis functions (RBF). The C parameter in SVM introduces a penalty for each misclassified data point. A smaller value of C results in a low penalty for misclassifications, leading to a decision boundary with a larger margin but more misclassifications. On the other hand, a larger value of C results in a higher penalty for misclassifications, leading to a decision boundary with a smaller margin and fewer misclassifications. The gamma parameter of RBF controls the distance of influence of a single training point. Low gamma values indicate a large similarity radius, resulting in more points being grouped together. For high gamma values, the points must be very close to each other to be considered in the same class. Therefore, models with very large gamma values tend to overfit. If the gamma is large, the effect of C becomes negligible. GridSearch cross-validation was used to optimize the hyperparameters of the SVM model [74]. This involved generating and testing the model for every possible combination of algorithm parameters specified in a grid. Table II shows the details of the hyperparameters.

Table II.

SVM hyperparameters, tuning range, and the optimal value selected for classification.

Hyperparameter	Range used	Optimum value selected
Kernel	‘Polynomial’, ‘RBF’, ’Sigmoid’, ‘Linear’	RBF
C	0.1, 1, 10, 100, 1000	100
Gamma	1, 0.1, 0.01, 0.001,0.0001	1
k-fold	5	5

To optimize the hyperparameters of the 1D-CNN model, we used KerasTuner [75]. The tuning process involved adjusting the number of convolutional layers, filter size, dropout rate, dense layer filter size, learning rate, and epoch. The resulting optimized model is illustrated in Figure 6, along with the specific hyperparameters used. The Adam optimizer with a learning rate of 0.001 and categorical cross-entropy loss function was used in the training process.

Figure 6.

The architecture of a tuned 1D-CNN model with optimized hyperparameters.

4.4

Data Augmentation

In ML, data augmentation is a method employed to expand the size of the training dataset by implementing various transformations on the available training data samples. The fundamental idea behind data augmentation is that alterations made to the labeled data should not modify the semantic interpretation of the labels [76, 77]. McFee et al. [78] suggested using deformation techniques that preserve the semantics of audio signals, improving the model’s accuracy for the music classification task. Bjerrum et al. [79] used data augmentation techniques on spectral data to employ deep learning algorithms to predict drug composition in tablets using near-infrared regions. The results showed that data augmentation improves overall performance.

The data augmentation technique should be chosen based on the specific characteristics of the analyzed signal. Some techniques might be more appropriate than others, depending on the context. In our study, we augmented the datasets by introducing four attributes to the spectrum: offset, multiplication, Gaussian noise [80], and speckle noise [81]. An example of the implementation of these attributes is shown in Figure 7. Offset was varied (0.0001 to 0.1 with a step size of 0.001) times the standard deviation of the training set. Multiplication was done with 1(0.0001 to 0.1 with a step size of 0.001) times the standard deviation of the training set, and the two different noises, Gaussian distributed additive noise and speckle, a multiplicative noise, were added ten times with variation of 0.00001 and 0.000001 respectively. Using augmentation techniques, we produced two different training datasets. The first training dataset was generated by considering only a single spectrum from the spectral library, whereas for the second training dataset, we augmented each spectrum within the training dataset.

Figure 7.

An example of data augmentation; created with an addition of offset and multiplication to data by a factor of 0.2; Gaussian noise and speckle noise are added to the data with variations of 0.00002 and 0.0002, respectively.

4.5

Accuracy Assessment

Accuracy assessment is an important step in evaluating the performance of classification algorithms. The most common and widely accepted method to express classification accuracy is confusion matrices. It helps to visualize the cross-tabulation of classified pigments; the matrix’s main diagonal represents the correctly classified values, while the other elements indicate how many pixels in one category are incorrectly classified into other categories. For additional information on the confusion matrix, we refer to the work of Congalton [82]. For each algorithm, we calculate the accuracy for the predefined region illustrated in Figure 8.

Figure 8.

The ground truth image of the mockup showing regions of interest (RoIs) used for accuracy assessment corresponds to different elevation labels. R7 represents the flat region, while S represents the region for a substrate.

Results and Discussion

This section presents the classification results obtained, along with an evaluation of the overall accuracy of the algorithms used, considering various reference spectra (ground truth). Furthermore, the outcomes of the SVM model employing data augmentation will also be elaborated.

Figure 9.

The classification accuracy from various algorithms used for evaluating ten different pigments on a flat surface and three elevated surfaces; the reference spectrum from the flat area was used for supervised and ML models; the color gradient utilized in this figure employs green to represent higher accuracy, red to indicate lower accuracy, and white represents an accuracy of 50%.

Figure 9 illustrates the classification accuracy of ten different algorithms, including two machine learning models, for each pigment on both elevated surfaces and a flat surface. Algorithms, 1D-CNN and SVM perform better than all eight supervised algorithms, with the least accurate being algorithm ED. SAM and SCM performed better than the other algorithms after the machine learning models. On the other hand, algorithms with hybrid approaches did not perform well overall. The images showing the classification results for each algorithm are attached in Appendix A.

Although there are variations in accuracy among the different algorithms, we can discern a pattern in the obtained classification results; the classification accuracy decreases with increasing elevation. An interesting observation is that SVM has lower accuracy for a region with an elevation of 10 mm than most other algorithms for that same region. Figure 10(a) also presents the confusion matrix for SVM, revealing that the accuracy is particularly low for regions R3, R13, and R10, all located on one particular side of the elevation in the mockup (Fig. 8). This side of the mockup has a shadow (Fig. 2a), with the lowest accuracy being on the shadow side of the highest elevated regions (Fig. 2b), which could be a significant factor in misclassifying these regions. Abed [31] stated in his work that surface changes become problematic when it comes to material aspects, which our results confirm.

Appendix B contains the confusion matrix for R3 and R13, illustrating the misclassification of pigments for SVM. The SAM algorithm determines the angle between two vectors, irrespective of their length, and thus its classification accuracy is less affected by any changes in the spectrum’s magnitude. On the other hand, the SCM algorithm eliminates negative correlations while preserving the SAM characteristics. Therefore, these algorithms provide greater accuracy in the shadow area, as evidenced by the confusion matrix depicted in Fig. 10(b) and Fig. 10(c) for SAM and SCM, respectively.

Figure 10.

Classification accuracy for each pigment across all Regions of Interest (RoIs); For algorithms, (a) SVM, (b) SAM, and (c) SCM.

The accuracy of all supervised algorithms is lower for pigment P6 and P9 in flat regions, but this is only due to the exact location chosen for accuracy assessment (R7), which includes most of the unclassified areas. This trend is not necessarily representative of all flat regions on average; also, elevated surfaces do not exhibit the same level of lower accuracy as flat regions. The classification results are presented in Appendix A. However, the ED, JMSAM, and SSS algorithms have lower accuracy for pigments P6 and P9 due to their similar spectra, which are also discussed by authors in Ref. [27]. This similarity is evident in Figure 11(a), where the Pearson correlation coefficient between these two pigments is almost 1. Similarly, Fig. 11(b) also indicates a very low dissimilarity measure between these two pigments.

Figure 11.

The matrices display the correlation and dissimilarity between the pigments; (a) Pearson’s correlation coefficient [85], a coefficient of 1 indicates a high correlation, while a value of zero represents no correlation; (b) SID calculated for dissimilarity, where a value of 0 implies a high degree of similarity between the spectra, while 1 indicates maximum dissimilarity.

The accuracy of the algorithms SID and its hybrid combination with SAM and SCM for pigment P1 has decreased as shown in Fig. 9, and this decrease was further observed with an increase in elevation. It is important to note that not all pigments are affected in the same way. In the case of P1, the pixels were either classified as P1 or remained unclassified, as indicated by the confusion matrix for SID in Appendix C. Mandal et al. [27] suggested that changing the threshold value could improve classification accuracy, but this approach could lead to misclassification of other pigments, which is generally undesirable in CH. The stochastic algorithm, SID, depends on the probability distributions of spectra, and alterations in data distribution can influence the overall entropy value. Moreover, the changes in the dataset can produce a varying impact on the entropy value for normal distribution and skewed distribution. In other words, if both distributions are shifted equally, the symmetric distribution would experience less change in entropy than the skewed distribution, owing to its higher predictability and lower uncertainty compared to the skewed distribution.

Figure 12.

Statistical measures of mean, median, standard deviation (SD), and Spectral flatness measure (SFM), computed for all ten pigments.

As illustrated in Figure 12, pigment P1 exhibits a notable difference between its mean and median values, resulting in a skewed distribution. Additionally, the standard deviation of this distribution is greater than its mean, indicating that the data points are more widely dispersed. Given the same amount of shift in datasets (identical in absolute terms), this shift may have a more significant impact on the dataset with a larger spread than the one with data points being more tightly clustered around the mean, leading to a larger relative entropy between the two datasets. This is likely why some pigments are affected more than others. This also explains why we need to set different threshold values for different pigments.

The accuracy of Pigment P10 is higher for all the algorithms used on a flat surface, but it decreases significantly for supervised algorithms with elevation changes. On the other hand, ML models, SVM, and 1D-CNN show consistent classification accuracy for P10, regardless of elevation. When the reference spectrum is obtained from a flat region, it is likely to have minimal variation with test datasets within the same region, resulting in a higher number of correctly classified pixels. However, if the reference spectrum is taken from an elevated region, changes in reflectance values at higher elevations could cause higher variation and lead to more unclassified pixels. Based on this, one can hypothesize that using a reference spectrum taken from an elevated region is more likely to result in correct pixel classification on the same elevated surface region.

Low spectral intensity makes distinguishing between different land cover classes or features in an image difficult, leading to misclassification, especially for classes with similar low reflectance values [83]. We observed that, in most cases, the P10 and P5 pigments were misclassified as each other or as the substrate, or they remained unclassified in most of the algorithms used. The Spectral Flatness Measure (SFM), also known as Wiener Entropy, is a metric that can be used to quantify the degree of flatness or peakiness of a spectrum by computing the geometric mean ratio to the arithmetic mean of the power spectrum [84]. We computed the SFM for the pigments, as shown in Fig. 12. The SFM values for P7, P10, and P5 were higher, indicating a flat spectrum with nearly identical reflectance values across different wavelengths. This can make it difficult for a classifier algorithm to differentiate between different classes, resulting in lower classification accuracy.

Using a reference target with a surface height equivalent to the flat surface of the mockup, the normalized reflectance value was calculated for the HSI datacube. Ideally, the reflectance value of an elevated surface should be higher than that of a flat surface. However, shadows caused by the surface elevation lead to a decrease in the obtained reflectance value as the surface height increases. Generally, the substrate spectrum has a lower value than the P7 reference spectrum (the reference spectrum for ten pigments used and the substrate is provided in Appendix D). With an increase in surface elevation, the distance between the P7 and its reference spectrum increases, while the distance between the P7 and substrate reference spectrum decreases. As a result, most regions for P7 are misclassified as substrate.

Figure 13.

Overall classification accuracy for four different surface elevations (flat, 2.5 mm, 5 mm, and 10 mm) using reference spectra from the flat surface.

In Figure 13, we can see the overall classification accuracy obtained by averaging all pigments for four different surface elevations: flat, 2.5 mm, 5 mm, and 10 mm, using all ten algorithms in the study. The reference spectrum used for building the spectral library and training the ML models was taken from the flat surface. For almost all algorithms, the classification accuracy for an elevated surface is lower compared to the flat region. Surfaces with a 2.5 mm elevation have accuracy similar to or less than the flat surface, followed by 5 mm and 10 mm, respectively. This pattern is consistent for ED, SSS, SVM, and 1D-CNN algorithms. However, for the other six algorithms, we see that the accuracy at an elevation of 10 mm is slightly greater than that of the surface with a 5 mm elevation. The increase in overall classification accuracy can be attributed to the higher accuracy obtained for pigments P5 and P10. This higher accuracy for P5 and P10 might be due to the shadow effect; decreased reflectance value in the shadow region might have reduced the distance between the reference and measured pixels.

Earlier in this paper, we hypothesized that using a reference spectrum from an elevated surface would result in more accurate pixel classification for that same elevated surface region. To test this hypothesis, we built spectral libraries and training datasets using the reference spectrum from each elevated surface, i.e., 2.5 mm, 5 mm, and 10 mm and then computed the classification accuracy. In addition, we built spectral libraries and training datasets that represent an average spectrum of flat and varying elevation surfaces. To obtain these spectra, we averaged the pixel values over a spatial region taken as in a straight line (for example, in Fig. 8, it is pixels values of a line from right to left) for each band. The result from classifaction accuracy is summarized in Figure 14.

Figure 14.

Classifaction accuracy obtained using ten different algorithms for flat (F), 2.5 mm elevation (E1), 5 mm elevation (E2), and 10 mm eleavation (E3), each computed for different spectral library and training datasets, built using reference spectrum from the flat region, three different elevated region and an average of all these.

Fig. 14 shows that classification accuracy for the flat region (F) is higher when using reference spectra from the flat surface than at different elevations. Similarly, when using reference spectra from a 2.5 mm elevated region, the classification accuracy is higher for regions with 2.5 mm elevation (E1) for all algorithms. However, for regions with 5 mm elevation (E2), the classification accuracy is almost identical to E1 and very close to other regions. In contrast, for regions with 10 mm elevation (E3), the classification accuracy is greater than other surfaces, mainly for stochastic algorithms (SID, SIDSAM, SIDSCM), SVM, and 1D-CNN. However, it is the lowest for ED and SSS. The accuracy of SAM, SCM, and JMSAM is similar to E1 and E2 elevations but lower than the flat surface. Using average reference spectra improved accuracy for almost all flat and elevated regions compared to the accuracy obtained when using four different conditions of reference spectra. A detailed result displaying the classification accuracy for individual pigments at each elevation and for different reference spectra conditions can be found in Appendix E.

Result for average classification accuracy using ten algorithms and five reference spectra conditions (i.e., from the regions F, E1, E2, E3, and the average of these regions) is shown in Figure 15. When the reference spectrum was taken from the flat region, the classification accuracy was lower, or comparable (in the case of E1), for all algorithms compared to the accuracy obtained when the reference spectra were taken from regions E2, E3, or the average spectrum. The classification accuracy for most algorithms was almost the same when considering reference spectra from E2 or E3. Notably, using an average reference spectrum improved classification accuracy for almost all algorithms. The 1D-CNN algorithm had the highest classification accuracy among all the algorithms used, with slightly lower accuracy when the reference spectrum was taken from the flat region and almost similar results for all other reference spectra conditions. The classified images produced by the SVM and 1D-CNN algorithms using an average reference spectrum are provided in Appendix F.

Figure 15.

Overall classification accuracy for four surface elevations (flat, 2.5 mm, 5 mm, and 10 mm) obtained by using reference spectra taken from an elevated region of 2.5 mm.

Data augmentation was performed to create additional training datasets for SVM, as shown in the flow diagram illustrated in Figure 16. Before computing classification accuracy, hyperparameter tuning was performed using these augmented datasets. Finally, the SVM model was executed with the optimal hyperparameters of a polynomial kernel function and a regularization value of 0.1. Figure 17 shows the classification accuracy obtained for each pigment across all regions using the augmented training dataset where training dataset from a flat region was selected. We observed that overall classification accuracy was higher than the SVM without data augmentation (Fig. 10a). We also augmented the data from a single spectrum taken from the flat region first and then from the average spectrum; results for classification accuracy are included in Appendix G. Figure 18 shows the overall classification accuracy for each condition for flat and three different elevations.

Figure 16.

A workflow diagram illustrating the generation of training datasets using data augmentation for SVM; classification results from a, b, c and d are illustrated in Figure 19.

Figure 17.

Classification accuracy for each pigment across all Regions of Interest (RoIs) for SVM using augmented dataset.

Figure 18.

Overall classification accuracy for SVM at four different surface elevations (flat, 2.5 mm, 5 mm, and 10 mm) using different conditions of data augmentation, in comparison with SCM, employing reference spectra obtained from the flat region.

Classification accuracy for an augmented data set is higher when compared with all other three conditions, i.e., without data augmentation, data augmentation considering a single spectrum from a flat region and considering data augmentation considering a single average spectrum. However, using a single spectrum for augmentation yielded lower performance, except for an elevated region of 10 mm, where the accuracy was significantly improved compared to the non-augmented condition. Comparable accuracy was achieved using data augmentation with a flat region and augmenting a single averaged spectrum. As illustrated in Fig. 9, SAM and SCM are the algorithms which performed better after ML models. Fig. 18 shows that data augmentation using a single spectrum from a flat region is still better than SCM, highlighting that even with the single spectrum available, one can obtain better classification accuracy than supervised-based algorithms. The results suggest that data augmentation can improve classification accuracy, particularly when multiple spectra are augmented or when an average spectrum is used for augmentation. The classified images for each of these conditions are presented in Fig. 19.

Figure 19.

The classification results for SVM; (a): without data augmentation, (b): with data augmentation, (c): with data augmentation using single spectrum from flat region, and (d): with data augmentation using single spectrum from averaged region.

Conclusion

This study investigated the performance of different supervised algorithms and machine learning models for pigment classification using HSI on an elevated mockup. We have observed that the elevation itself does not significantly impact the classification accuracy; however, the elevation can result in the formation of shadows, which can have a significant effect on the classification accuracy of the algorithms used and varies for different algorithms. It was also observed that the choice of reference spectra plays a significant role in the accuracy of pigment classification. An average reference spectrum from different elevated regions yields better results than individual spectra. Among the ten algorithms tested, the 1D-CNN algorithm showed the highest classification accuracy, followed by SVM, SAM and SCM. Furthermore, results also indicated that data augmentation could significantly improve classification accuracy, particularly when multiple spectra are augmented or when an average spectrum is used for augmentation. This study provides valuable insights for analysing paintings in the CH domain. It could be beneficial in selecting appropriate classification algorithms when artworks have elevation or data that have shadows. In addition to paintings, it could also be useful to other museum’s low elevations objects, such as glazed plates [86] and bas-reliefs [87]. In future work, removing shadows using image-processing techniques could be explored as a possible direction to further improve the accuracy of pigment classification. Future work will also include mixtures of pigments as well as the influence of the thickness of the pigment.

Appendix A.

Classification Results

Figure A1.

The classification results from ten algorithms using the reference spectrum from the flat region.

Appendix B.

Confusion Matrix for Regions R3 and R13, Illustrating the Misclassification of Pigments for SVM

Figure B.1.

Confusion matrix for SVM; (a): RoI : 3, (b): RoI : 13.

Appendix C.

Confusion Matrix Illustrating Classification Accuracy Obtained on a Flat Surface using SID Algorithm

Figure C.1.

Confusion matrix obtained by utilizing the SID Algorithm on a flat surface for ten pigments and substrate.

Appendix D.

Normalized Reflectance Spectrum of 10 Pigments and Substrate

Figure D.1.

Spectrum for ten pigments (P1 to P10) and substrate (S) measured at the flat surface by taking an average of 10 × 10 pixels.

Appendix E.

Accuracy of Pigment Classification at Various Elevations Using Different Reference Spectra Conditions

Figure E.1.

The classification accuracy from various algorithms used for evaluating ten different pigments on a flat surface and three elevated surfaces; spectral library and training datasets built using reference spectrum from an elevated region of 2.5 mm; the color gradient utilized in this figure employs green to represent higher accuracy, red to indicate lower accuracy, and white represents an accuracy of 50%.

Figure E.2.

The classification accuracy from various algorithms used for evaluating ten different pigments on a flat surface and three elevated surfaces; spectral library and training datasets built using reference spectrum from an elevated region of 5 mm; the color gradient utilized in this figure employs green to represent higher accuracy, red to indicate lower accuracy, and white represents an accuracy of 50%.

Figure E.3.

The classification accuracy from various algorithms used for evaluating ten different pigments on a flat surface and three elevated surfaces; spectral library and training datasets built using reference spectrum from an elevated region of 10 mm; the color gradient utilized in this figure employs green to represent higher accuracy, red to indicate lower accuracy, and white represents an accuracy of 50%.

Figure E.4.

The classification accuracy from various algorithms used for evaluating ten different pigments on a flat surface and three elevated surfaces; using average reference spectra for supervised and ML models; the color gradient utilized in this figure employs green to represent higher accuracy, red to indicate lower accuracy, and white represents an accuracy of 50%.

Figure E.5.

Overall classification accuracy for four surface elevations (flat, 2.5 mm, 5 mm, and 10 mm) obtained by using reference spectra taken from an elevated region of 2.5 mm.

Figure E.6.

Overall classification accuracy for four surface elevations (flat, 2.5 mm, 5 mm, and 10 mm) obtained by using reference spectra taken from an elevated region of 5 mm.

Figure E.7.

Overall classification accuracy for four surface elevations (flat, 2.5 mm, 5 mm, and 10 mm) obtained by using reference spectra taken from an elevated region of 10 mm.

Figure E.8.

Overall classification accuracy for four surface elevations (flat, 2.5 mm, 5 mm, and 10 mm) obtained using average reference spectra; the averages were computed by considering a pixels values of a line drawn from right to left of the mockup for each pigment.

Appendix F.

Classification results for SVM and 1D-CNN

Figure F.1.

The classification results for SVM and 1D-CNN using the averaged reference spectrum.

Appendix G.

Classification Accuracy for SVM using Data Augmentation

Figure G.1.

Classification accuracy for each pigment across all Regions of Interest (ROIs) for SVM, (a): With data augmentation using single spectrum from flat region, and (b): With data augmentation using averaged single spectrum.

References

1StulikD.MillerD.KhanjianH.KhandekarN.CarlsonJ.WolbersR.PetersenW. C.Solvent Gels for the Cleaning of Works of Art: The Residue Question2004Getty PublicationsLos Angeles, CA

2Digney-PeerS.ThomasK.PerryR.TownsendJ.GrittS.The imitative retouching of easel paintingsConservation of Easel Paintings2020RoutledgeOxfordshire626653626–53

3ShresthaR.HardebergJ. Y.Evaluation and comparison of multispectral imaging systemsProc. IS&T CIC22: Twenty-Second Color and Imaging Conf.2014IS&TSpringfield, VA107112107–12

4CucciC.CasiniA.Hyperspectral imaging for artworks investigationData Handling in Science and Technology2019Vol. 32ElsevierAmsterdam583604583–604

5DeborahH.GeorgeS.HardebergJ. Y.Pigment mapping of the Scream (1893) based on hyperspectral imagingImage and Signal Processing: 6th Int’l. Conf., ICISP 2014, Cherbourg, France, June 30–July 22014SpringerCham247256247–56

6JanssensK.Van der SnicktG.VanmeertF.LegrandS.NuytsG.AlfeldM.MonicoL.AnafW.De NolfW.VermeulenM.VerbeeckJ.2017Non-invasive and non-destructive examination of artistic pigments, paints, and paintings by means of X-ray methodsAnalytical Chemistry for Cultural Heritage7712877–128Springer10.1007/s41061-016-0079-2

7van der MeerF. D.van der WerffH. M.A.van RuitenbeekF. J. A.HeckerC. A.BakkerW. H.NoomenM. F.van der MeijdeM.CarranzaE. J. M.Boudewijn de SmethJ.WoldaiT.2012Multi- and hyperspectral geologic remote sensing: A reviewInt. J. Appl. Earth Obs. Geoinformation14112128112–2810.1016/j.jag.2011.08.002

8DaleL. M.ThewisA.BoudryC.RotarI.DardenneP.BaetenV.Fernández PiernaJ. A.2013Hyperspectral imaging applications in agriculture and agro-food product quality and safety control: A reviewAppl. Spectrosc. Rev.48142159142–5910.1080/05704928.2012.705800

9LuG.FeiB.2014Medical hyperspectral imaging: A reviewJ. Biomed. Opt.191241–2410.1117/1.JBO.19.9.096013

10EdelmanG.GastonE.van LeeuwenT.CullenP. J.AaldersM. C. G.2012Hyperspectral imaging for non-contact analysis of forensic tracesForensic Sci. Int.223283928–3910.1016/j.forsciint.2012.09.012

11LiQ.HeX.WangY.LiuH.XuD.GuoF.2013Review of spectral imaging technology in biomedical engineering: Achievements and challengesJ. Biomed. Opt.181291–29

12FischerC.KakoulliI.2006Multispectral and hyperspectral imaging technologies in conservation: Current research and potential applicationsStud. Conservation513163–1610.1179/sic.2006.51.Supplement-1.3

13ArteagaY.HardebergJ. Y.BoustC.Hdr multispectral imaging-based brdf measurement using flexible robot arm systemProc. IS&T CIC30: Thirtieth Color and Imaging Conf.2022IS&TSpringfield, VA758075–8010.2352/CIC.2022.30.1.15

14ChaneC. S.MansouriA.MarzaniF. S.BoochsF.2013Integration of 3d and multispectral data for cultural heritage applications: Survey and perspectivesImage Vision Comput.319110291–10210.1016/j.imavis.2012.10.006

15TaftW. S.MayerJ. W.2001The science of paintingsMeas. Sci. Technol.12653653653–

16RubensP. P.The Virgin as the Woman of the ApocalypseJ. Paul Getty MuseumLos Angeles, CA162316241623–4

17TownsendJ. H.Painting techniques and materials of Turner and other British artists 1775–1875Historical Painting Techniques, Materials, and Studio Practice: Preprints of a Symposium1995University of LeidenThe Netherlands176185176–85

18BarrettS.StulikD. CAn integrated approach for the study of painting techniquesHistorical Painting Techniques, Materials, and Studio Practice: Preprints of a Symposium1995University of LeidenThe Netherlands6116–11

19SinclairE.The polychromy of Exeter and Salisbury Cathedrals: a preliminary comparisonHistorical Painting Techniques, Materials, and Studio Practice: Preprints of a Symposium1995University of LeidenThe Netherlands105110105–10Getty Conservation Institute Malibu

20BaxterW.WendtJ.LinM. C.IMPaSTo: A realistic, interactive model for paintProc. 3rd Int’l. Symposium on Non-photorealistic Animation and Rendering2004ACMNew York, NY4514845–148

21FuY.YuH.YehC.-K.ZhangJ.LeeT.-Y.2018High relief from brush paintingIEEE Trans. Vis. Comput. Graphics25276327762763–7610.1109/TVCG.2018.2860004

22PlissonJ. S.de ViguerieL.TahrouchtL.MenuM.DucouretG.2014Rheology of white paints: How Van Gogh achieved his famous impastoColloids Surf. A458134141134–4110.1016/j.colsurfa.2014.02.055

23GroenK.1997Investigation of the use of the binding medium by RembrandtZ. Kunsttechnologie und Konservierung2208211208–11

24ElkhuizenW. S.Dore-CallewaertT. W. J.LeonhardtE.VandivereA.SongY.PontS. C.GeraedtsJ. M.P.DikJ.2019Comparison of three 3D scanning techniques for paintings, as applied to Vermeer’s ’Girl with a Pearl EarringHeritage Sci.71221–2210.1186/s40494-019-0331-5

25GonzalezV.CotteM.WallezG.van LoonA.De NolfW.EvenoM.KeuneK.NobleP.DikJ.2019Unraveling the Composition of Rembrandt’s Impasto through the Identification of Unusual Plumbonacrite by Multimodal X-ray Diffraction AnalysisAngew. Chem.131567556785675–810.1002/ange.201813105

26MandalD. J.GeorgeS.PedersenM.BoustC.2021Influence of acquisition parameters on pigment classification using hyperspectral imagingJ. Imaging Sci. Technol.6505040610.2352/J.ImagingSci.Technol.2021.65.5.050406

27MandalD. J.PedersenM.GeorgeS.DeborahH.BoustC.2023An experiment-based comparative analysis of pigment classification algorithms using hyperspectral imagingJ. Imaging Sci. Technol.6703040310.2352/J.ImagingSci.Technol.2023.67.3.030403

28YonghuiZ.BernsR. S.TaplinL. A.CoddingtonJ.2008An investigation of multispectral imaging for the mapping of pigments in paintingsProc. SPIE681068100710.1117/12.765711

29BernsR. S.ImaiF. H.The use of multi-channel visible spectrum imaging for pigment identificationProc. 13th Triennal ICOM-CC Meeting2002CiteseerUniversity Park, PA217222217–22

30BernsR. S.ImaiF. H.Pigment identification of artist materials via multi-spectral imagingProc. IS&T/SID CIC9: Ninth Color Imaging Conf.2001IS&TSpringfield, VA859085–90

31AbedF. M.Pigment Identification of Paintings based on Kubelka-Munk Theory and Spectral Images2014Rochester Institute of TechnologyRochester, NY

32FukumotoK.TsumuraN.BernsR.Estimating concentrations of pigments using encoder-decoder type of neural networkProc. IS&T CIC27: Twenty-seven Color and Imaging Conf.2019IS&TSpringfield, VA149152149–5210.2352/issn.2169-2629.2019.27.28

33FukumotoK.TsumuraN.BernsR.2020Estimating pigment concentrations from spectral images using an encoder decoder neural networkJ. Imaging Sci. Technol.6403050210.2352/J.ImagingSci.Technol.2020.64.3.030502

34FiorucciM.KhoroshiltsevaM.PontilM.TravigliaA.BueA. D.JamesS.2020Machine learning for cultural heritage: A surveyPattern Recognit. Lett.133102108102–810.1016/j.patrec.2020.02.017

35SmirnovS.EguizabalA.Deep learning for object detection in fine-art paintings2018 Metrology for Archaeology and Cultural Heritage (Metro Archaeo)2018IEEEPiscataway, NJ454945–910.1109/MetroArchaeo43810.2018.9089828

36BalasC.EpitropouG.TsaprasA.HadjinicolaouN.2018Hyperspectral imaging and spectral classification for pigment identification and mapping in paintings by El Greco and his workshopMultimedia Tools Appl.77973797519737–5110.1007/s11042-017-5564-2

37VishnuS.NidamanuriR. R.BremananthR.2013Spectral material mapping using hyperspectral imagery: a review of spectral matching and library search methodsGeocarto Int.28171190171–9010.1080/10106049.2012.665498

38KeshavaN.2004Distance metrics and band selection in hyperspectral processing with applications to material identification and spectral librariesIEEE Trans. Geosci. Remote Sens.42155215651552–6510.1109/TGRS.2004.830549

39LiJ.HibbertD. B.FullerS.CattleJ.WayC. P.2005Comparison of spectra using a Bayesian approach. An argument using oil spills as an exampleAnal. Chem.77639644639–4410.1021/ac048894j

40CerraD.BieniarzJ.AvbeljJ.MüllerR.ReinartzP.Spectral matching through data compressionISPRS Workshop on High-Resolution Earth Imaging for Geospatial Information2011Vol. XXXVIII-4/W19CopernicusGöttingen141–410.5194/isprsarchives-XXXVIII-4-W19-75-2011

41MengtingG.PingliF.2014Preliminary study on the application of hyperspectral imaging in the classification of and identification Chinese traditional pigments classification—a case study of spectral angle mapperSci. Conservation Archaeology4768376–83

42De CarvalhoO. A.MenesesP. R.Spectral correlation mapper (SCM): an improvement on the spectral angle mapper (SAM)Summaries of the 9th JPL Airborne Earth Science Workshop2000Vol. 9JPL PublicationPasadena, CA

43GeorgeS.HardebergJ. Y.Ink classification and visualisation of historical manuscripts: Application of hyperspectral imaging2015 13th Int’l. Conf. on Document Analysis and Recognition (ICDAR)2015IEEEPiscataway, NJ113111351131–510.1109/ICDAR.2015.7333937

44DeepthiD.DevassyB. M.GeorgeS.NussbaumP.ThomasT.2022Classification of forensic hyperspectral paper data using hybrid spectral similarity algorithmsJ. Chemometr.36e338710.1002/cem.3387

45ChengF.ZhangP.WangS.HuB.2018A study on classification of mineral pigments based on spectral angle mapper and decision treeProc. SPIE10806108065Z10.1117/12.2503088

46MingyanG.ShuqiangL.MiaoleH.MaS.GaoZ.BaiS.ZhouP.Classification and recognition of tomb information in hyperspectral imageInt’l. Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences-ISPRS Archives2018Vol. 42411416411–6

47KleynhansT.Schmidt PattersonCatherine M.DooleyKathryn AMessingerDavid WDelaneyJohn K2020An alternative approach to mapping pigments in paintings with hyperspectral reflectance image cubes using artificial intelligenceHeritage Sci.81161–1610.1186/s40494-020-00427-7

48LiuL.MitevaT.DelnevoG.MirriS.WalterP.de ViguerieL.PouyetE.2023Neural Networks for Hyperspectral Imaging of Historical Paintings: A Practical ReviewSensors23241910.3390/s23052419

49ChenA.JesusR.VilariguesM.Convolutional neural network-based pure paint pigment identification using hyperspectral imagesACM Multimedia Asia2021Association for Computing Machinery171–710.1145/3469877.3495641

50GowerJ. C.1985Properties of Euclidean and non-Euclidean distance matricesLinear Algebra and its Applications67819781–9710.1016/0024-3795(85)90187-9

51KruseF. A.LefkoffA. B.BoardmanJ. W.HeidebrechtK. B.ShapiroA. T.BarloonP. J.GoetzA. F. H.The spectral image processing system (SIPS)—interactive visualization and analysis of imaging spectrometer dataRemote Sensing of Environment1993Vol. 44145163145–63Airbone Imaging Spectrometry

52de Carvalho JúniorO.GuimarãesR.GillespieA.SilvaN.GomesR.2011A new approach to change vector analysis using distance and similarity measuresRemote Sens.3247324932473–9310.3390/rs3112473

53ChangC.-I.2000An information-theoretic approach to spectral variability, similarity, and discrimination for hyperspectral image analysisIEEE Trans. Information Theory46192719321927–3210.1109/18.857802

54KerekesJ. P.CiszA. P.SimmonsR. E.2005A comparative evaluation of spectral quality metrics for hyperspectral imageryProc. SPIE580610.1117/12.605916

55DuY.ChangC.-I.RenH.ChangC.-C.JensenJ. O.D’AmicoF. M.2004New hyperspectral discrimination measure for spectral characterizationOpt. Eng.43177717861777–8610.1117/1.1805563

56Naresh KumarM.SeshasaiM. V. R.Vara PrasadK. S.KamalaV.RamanaK. V.DwivediR. S.RoyP. S.2011A new hybrid spectral similarity measure for discrimination among Vigna speciesInt. J. Remote Sens.32404140534041–5310.1080/01431161.2010.484431

57PadmaS.SanjeeviS.2014Jeffries matusita-spectral angle mapper (JM-SAM) spectral matching for species level mapping at Bhitarkanika, Muthupet and Pichavaram mangrovesInt. Arch. Photogramm. Remote Sens. Spatial Information Sciences40140314111403–1110.5194/isprsarchives-XL-8-1403-2014

58WangL.Support Vector Machines: Theory and Applications2005Vol. 177SpringerCham

59BoserB. E.GuyonI. M.VapnikV. N.A training algorithm for optimal margin classifiersProc. Fifth Annual Workshop on Computational Learning Theory1992ACMNew York, NY144152144–5210.1145/130385.130401

60NobleW. S.2006What is a support vector machine?Nature Biotech.24156515671565–710.1038/nbt1206-1565

61AbiodunO. I.JantanA.OmolaraA. E.DadaK. V.MohamedN. A.ArshadH.2018State-of-the-art in artificial neural network applications: A surveyHeliyon4e0093810.1016/j.heliyon.2018.e00938

62RonnebergerO.FischerP.BroxT.U-net: Convolutional networks for biomedical image segmentationInt’l. Conf. on Medical Image Computing and Computer-assisted Intervention2015SpringerCham234241234–4110.1007/978-3-319-24574-4_28

63YuS.JiaS.XuC.2017Convolutional neural networks for hyperspectral image classificationNeurocomputing219889888–9810.1016/j.neucom.2016.09.010

64HongD.GaoL.YaoJ.ZhangB.PlazaA.ChanussotJ.2021Graph convolutional networks for hyperspectral image classificationIEEE Trans. Geosci. Remote Sens.59596659785966–7810.1109/TGRS.2020.3015157

65HuW.HuangY.LiW.ZhangF.LiH.2015Deep convolutional neural networks for hyperspectral image classificationJ. Sensors20151121–1210.1155/2015/258619

66ZECCHI, Soluzioni per artisti, materiale per restauro. Accessed: 01 May 2023

67Norsk Elektro Optikk. http://www.hyspex.no/. Accessed: 01 May 2023

68Spectralon multi-step targets. Accessed: 1 May 2023

69X-Rite Color Checker Classic. Accessed: 01 May 2023

70SaundersD.CupittJ.1993Image Processing at the National Gallery: The VASARI ProjectNational Gallery technical bulletin14728572–85

71Python Software Foundation. Python Language Reference, Version 3.9, 2021. [Accessed 05 December 2022]

72TanL.HouM.A study on the application of SAM classification algorithm in seal of calligraphy and painting based on hyperspectral technology2016 4th Int’l. Workshop on Earth Observation and Remote Sensing Applications (EORSA)2016IEEEPiscataway, NJ415418415–810.1109/EORSA.2016.7552841

73WeertsH. J. P.MuellerA. C.VanschorenJ.Importance of tuning hyperparameters of machine learning algorithms. arXiv preprint arXiv:2007.07588, (2020)

74MüllerA. C.GuidoS.Introduction to Machine Learning with Python: A Guide for Data Scientists2016O’Reilly Media, Inc.Sebastopol, CA

75O’MalleyT.BurszteinE.LongJ.CholletF.JinH.InvernizziL.KerasTuner. https://github.com/keras-team/keras-tuner, 2019. Accessed: 20 January 2023

76ShortenC.KhoshgoftaarT. M.2019A survey on image data augmentation for deep learningJ. Big Data61481–4810.1186/s40537-019-0197-0

77WenQ.SunL.YangF.SongX.GaoJ.WangX.XuH.Time series data augmentation for deep learning: A survey. arXiv preprint arXiv:2002.12478 (2020)

78McfeeB.HumphreyE. J.BelloJ. P.A software framework for musical data augmentationISMIR2015Vol. 2015CiteseerUniversity Park, PA248254248–54

79BjerrumE. J.GlahderM.SkovT.Data augmentation of spectral data for convolutional neural network (CNN) based deep chemometrics. arXiv preprint arXiv:1710.01927, (2017)

80DavenportW. B.RootW. L.An Introduction to the Theory of Random Signals and Noise1958Vol. 159McGraw-HillNew York

81BoyatA. K.JoshiB. K.A review paper: noise models in digital image processing. arXiv preprint arXiv:1505.03489, (2015)

82CongaltonR. G.1991A review of assessing the accuracy of classifications of remotely sensed dataRemote Sens. Environ.37354635–4610.1016/0034-4257(91)90048-B

83JiL.GongP.GengX.ZhaoY.2015Improving the accuracy of the water surface cover type in the 30 m FROM-GLC productRemote Sens.7135071352713507–2710.3390/rs71013507

84GrayA.MarkelJ.1974A spectral-flatness measure for studying the autocorrelation method of linear prediction of speech analysisIEEE Trans. Acoustics, Speech, Signal Process.22207217207–1710.1109/TASSP.1974.1162572

85BenestyJ.ChenJ.HuangY.CohenI.2009Pearson correlation coefficientNoise Reduction in Speech Processing141–4SpringerBerlin, Heidelberg10.1007/978-3-642-00296-0_5

86BouquillonA.LigovichG.RoisineG.2019De la couleur du bestiaire “esmaillé” de PalissyTechnè. La science au service de l’histoire de l’art et de la préservation des biens culturels47516051–60

87MuseumL.Relief of Seti I and Hathor, 2021, https://collections.louvre.fr/ark:/53355/cl010009693. [Accessed 05 August 2023]