jist JIMTE6 Journal of Imaging Science and Technology J. Imaging Sci. Technol. J. Imaging Sci. Technol. 1062-3701 1943-3522 Society for Imaging Science and Technology 060404 10.2352/J.ImagingSci.Technol.2022.66.6.060404 1366 Papers based on Work Presented at Electronic Imaging 2023 Structure-Aware Color Halftoning with Adaptive Sharpness Control Structure-aware color halftoning with adaptive sharpness control Abedini Fereshteh ▴ Gooran Sasan ▴ Linköping University, Media and Information Technology Division, Department of Science and Technology, Norra Grytsgatan 10, Norrköping, Sweden, 60233 fereshteh.abedini@liu.se

Abedini and Gooran

IS&T Members.

 11 2022 66 6 060404-1 060404-11 30 6 2022 10 11 2022 © Society for Imaging Science and Technology 2022 Abstract

Structure-aware halftoning algorithms aim at improving their non-structure-aware version by preserving high-frequency details, structures, and tones and by employing additional information from the input image content. The recently proposed achromatic structure-aware Iterative Method Controlling the Dot Placement (IMCDP) halftoning algorithm uses the angle of the dominant line in each pixel’s neighborhood as supplementary information to align halftone structures with the dominant orientation in each region and results in sharper halftones, gives a more three-dimensional impression, and improves the structural similarity and tone preservation. However, this method is developed only for monochrome halftoning, the degree of sharpness enhancement is constant for the entire image, and the algorithm is prohibitively expensive for large images. In this paper, we present a faster and more flexible approach for representing the image structure using a Gabor-based orientation extraction technique which improves the computational performance of the structure-aware IMCDP by an order of magnitude while improving the visual qualities. In addition, we extended the method to color halftoning and studied the impact of orientation information in different color channels on improving sharpness enhancement, preserving structural similarity, and decreasing color reproduction error. Furthermore, we propose a dynamic sharpness enhancement approach, which adaptively varies the local sharpness of the halftone image based on different textures across the image. Our contributions in the present work enable the algorithm to adaptively work on large images with multiple regions and different textures.

 ccc 1062-3701/2022/66(6)/060404/11/$25.00 printed Printed in the USA Introduction

Image reproduction devices such as printers are binary tools that can only choose to place a dot at a particular position of a substrate using only a limited number of inks [1]. Digital halftoning is the process of converting a continuous-tone image into a binary image for printing purposes. Halftoning algorithms are divided into three main categories: thresholding and table halftoning [2], error diffusion [3–7], and iterative [8–11] methods. Thresholding and non-modified error diffusion are simple techniques, but they introduce different types of artefacts that worsen the representation of the original image. However, iterative methods such as the Direct Binary Search (DBS) [8] and the Iterative Method Controlling the Dot Placement (IMCDP) [9] create higher halftone quality at the cost of computational complexity.

Color printers use four primary colorants, typically cyan, magenta, yellow, and black (CMYK). In addition to the four primary colors, multi-channel or High-Fidelity (HiFi) printing uses extra inks to improve the quality of color printing. Color halftoning converts a continuous-tone color image into binary images for each ink. One simple, yet efficient, approach for color halftoning is to apply achromatic halftoning algorithms to different color channels independently. The final color halftoned image is then retrieved by superposition of all the monochrome halftoned images.

Halftoning techniques have gone through different improvements by supplementing the halftoning algorithm with supplemental information about the original image content to better preserve tonal and structural resemblance to the original continuous-tone input image [12–21]. As these techniques are aware of details and structures of the original image, they are referred to as structure-aware or structure-based halftoning.

In this paper, we introduce an adaptive structure-aware color halftoning algorithm based on the CMY colorant channels. It is an extension of the IMCDP halftoning algorithm, which reproduces well-formed halftone structures with the possibility of changing the halftone’s shape and structure. The achromatic structure-aware IMCDP was recently proposed as a structure-aware halftoning algorithm that improves the sharpness of the halftone image and simultaneously better preserves the tone and structure similarity compared to the original IMCDP [21]. However, the preprocessing step of the algorithm was slow, and it has been applied only to monochrome images. In this paper, (1) we review the approach in the preprocessing step and propose a faster and more flexible technique for obtaining the local orientation in structure-aware IMCDP, (2) extend the algorithm to color halftoning, and (3) improve its performance by introducing an adaptive approach to adjust the degree of sharpness enhancement in a spatial varying manner across the image.

 Related Work

A high-quality halftone should present faithful reproduction of the tonal and structural content of the original image. As structures may be lost during halftoning, the halftone image sometimes fails to truly represent the structures and tones of the continuous-tone original image. Structure-aware (structure-based) halftoning methods have been developed with the aim of better preserving tonal and structural details by feeding the important content of the original image as an additional input to the halftoning process. For instance, Eschbach and Knox used the inverse of input image as a threshold in the halftoning process and enhanced edge reproduction in error diffusion [13]. Chang et al. used the main component of local frequency in the original image as lookup table indices to modify the error-diffusion coefficients and improved the visual quality of halftones [15]. Liu et al. proposed an entropy-based method that measures the intensity’s impact and adaptively modifies the threshold’s constraints in the halftoning algorithm [16]. Pang et al. defined an objective function, measuring tone and structure similarity between the original image and the halftone image. By optimizing the objective metric, the algorithm generates images, which preserve texture details and local tone faithfully [14]. Li and Mould employed the contrast information of the original image and generated a contrast-aware mask for diffusing the error to neighborhood’s pixels in error diffusion halftoning [12]. Li et al. also used the error diffusion halftoning algorithm as the underlying method and employed the intensity of edges in the original image to calculate the texture information. They used the texture information as a mask to better diffuse the error to pixels [20]. Abedini et al. used the local orientations of the original image to align the halftone structure with the dominant line in each neighborhood. Their proposed structure-aware halftoning reproduces sharper halftones, while better preserves tonal and structural similarities compared to its non-structure-aware version [21]. Not all the proposed structure-aware halftoning methods have been tested on color images. Some researchers have extended their proposed method to color halftoning by applying their achromatic halftoning algorithm independently on different colorant channels; however, they have not studied the effect of separate monochromatic halftoning on final results [12, 14, 16, 20].

In this paper, we aim to improve the recently proposed monochromatic structure-aware IMCDP [21] by introducing a faster and more stable approach for extracting the local orientations, extend it to color halftoning by investigating the effect of running an independent or dependent halftoning over each colorant channel, and improve its flexibility by proposing an adaptive sharpness enhancement across the image.

 Method

In our previous work, we used the local orientation of pixels to develop a structure-aware halftoning method based on IMCDP [21]. It has been shown that the structure-aware version of the IMCDP reproduces sharper halftones and better preserves the tone and structure similarity of the input image. However, the method presented in Ref. [21] is only developed for monochrome halftoning and the degree of sharpness enhancement is constant for the entire image.

The contributions of this paper are: (1) using a faster and more flexible approach for obtaining the orientation in the preprocessing step, (2) extending the structure-aware IMCDP to color halftoning, and (3) using adaptive sharpness enhancement across the image. These contributions enable the algorithm to work on large and diverse images that have multiple regions with different textures. In Section 3.1, we briefly review the monochrome structure-aware IMCDP, and we elaborate on our proposed method for improving the structure-aware IMCDP in Section 3.2.

Monochrome Structure-Aware IMCDP

The key idea of the monochrome structure-aware IMCDP is to align the dot placement along edges and textures of the image. Our goal was to improve the reproduction of high-frequency details while preserving the tonal and structural resemblance to the original image. We developed the method based on the IMCDP halftoning algorithm [9] and used the Hough transform [22] to extract local orientation information.

IMCDP is an iterative halftoning algorithm, which starts with a blank image the same size as the original image (to be the halftone image). The algorithm searches for the pixel holding the maximum value in the continuous-tone image, and the first dot is placed at its corresponding position in the blank halftone image. To consider the effect of this quantization, the low-pass filtered version of the halftone image is subtracted from the low-pass filtered version of the original image. This process is called feed-back process, and, accordingly, the filter is referred to as feed-back filter. The algorithm continues with finding the next pixel holding the highest value after subtracting the filtered halftone image. Because the average tone value in different gray-tone regions in the halftone image should be the same as the original image, the total number of black dots to be placed in each tonal region is known in advance, so the algorithm terminates when the predetermined number of black dots is placed in the halftone image [9].

One important point about IMCDP is the shape of the feed-back filter. In the original IMCDP, a symmetrical Gaussian kernel, as defined in Eq. (1) is used in the feed-back process. As a result, the dots are placed symmetrically in all directions. However, using a nonsymmetrical Gaussian kernel, as in Eq. (2), makes the distribution of dots nonsymmetrical [23]. f(x,y)=Ke−x2+y22σ2 f(x,y)=Ke−(Ax2+2Bxy+Cy2), where, (x, y) are the spatial coordinates of pixels, K is a normalization factor to ensure that the filter elements sum to 1, σ is the standard deviation of the Gaussian kernel, and constants A, B, and C are calculated as: A=cos2φ2k1σ2+sin2φ2k2σ2, B=−sin2φ4k1σ2+sin2φ4k2σ2, C=sin2φ2k1σ2+cos2φ2k2σ2. Parameters k1 and k2 in the nonsymmetrical kernel are used to adjust the shape of the halftone structure. Setting k1 = k2 creates symmetrical halftones, while k1 > k2 makes halftones grow faster in vertical direction, resulting in vertical line-halftones, and conversely, k1 < k2 generates halftones with horizontal alignments, resulting in horizontal line-halftones. Figure 1 shows a visual example of flexibility of IMCDP in generating different halftone structures by using different Gaussian kernels in the feed-back process.

Grayscale image being halftoned using; (a) a symmetrical filter as in Eq. (1) in the feed-back process; (b) a nonsymmetrical filter of the form as in Eq. (2) in the feed-back process with vertical alignment k1=3,k2=1,φ=0∘; (c) a nonsymmetrical filter of the form as in Eq. (2) in the feed-back process with horizontal alignment k1=1,k2=3,φ=0∘; (d) a nonsymmetrical filter of the form as in Eq. (2) in the feed-back process with horizontal alignment k1 = 1, k2 = 3, rotated by φ=30∘. Halftone images are displayed at 150 dpi. This figure is best viewed in the electronic version of this paper.

Different halftone structures, produced by the IMCDP, could be combined in the same image with smooth transition [24]. The flexibility of IMCDP in generating line-halftone structures makes it a powerful algorithm to adaptively change the halftone structure according to the image content. In the monochrome structure-aware IMCDP, we used the Hough transform to detect the dominant line in the neighborhood of each pixel [21]. As a result, the input image is divided into structured (union of pixels with a dominant line or sharp edge in their neighborhood) and structureless regions (union of pixels with no line in their neighborhood). Then, different halftone structures have been used based on the orientation of the dominant line. To be more specific, at pixels lying in the structureless region, a symmetrical Gaussian filter with the form of Eq. (1) is used as the feedback filter, while at pixels with a line in their neighborhood a nonsymmetrical Gaussian filter with the form of Eq. (2) is applied.

To align halftones with the dominant line in the neighborhood of a pixel, we made horizontal halftone (k1 < k2) and rotated it by setting the angle (φ) to the angle of the dominant line. This means that the halftone structure is aligned with the dominant line in the pixel’s neighborhood. Therefore, the edge information is more emphasized, and the halftone image preserves more details at edges. Figure 2 illustrates the workflow of developing the recently proposed structure-aware IMCDP [21]. It was found that generating horizontal halftone line by setting parameters k1 and k2 to 1 and 1.8, respectively, and setting the angle φ to the angle of the dominant line in the structure-aware IMCDP results in sharper halftones while better preserves the tonal and structural similarity compared to the original IMCDP.

Workflow of developing the monochrome structure-aware IMCDP.

 Color Structure-Aware IMCDP

In Ref. [21], the Hough transform was used to detect the dominant line in the neighborhood of each pixel in the input image, and partition the image into structured and structureless regions. The angle of the dominant line was then calculated, and a nonsymmetrical Gaussian kernel was applied based on its orientation in the feedback process in structured regions and a symmetrical Gaussian kernel in structureless regions. This approach has two main limitations:

The Hough transform based preprocessing can be prohibitively expensive as it has cubic complexity.

Relative sharpness enhancement is static across the entire structured regions of the image.

In this paper, we address both these limitations, and extend the method to color halftoning.

A Faster preprocessing Approach to Calculate the Local Orientation

In the preprocessing step, a fast and stable approach to estimate the local orientations of an image is needed. We require the method to have at most quadratic complexity since it is the best-case complexity imposed by the nature of IMCDP. We reviewed approaches based on local image gradient and Gabor responses at different scales. They both have the same time complexity; however, we observed that the Gabor-based approach is more robust.

Gabor filter captures local frequency and direction of an image and is extensively used in texture analysis [25]. The image is convoluted with a set of filters called Gabor filter bank. Each filter is characterized by its wavelength (λ), orientation (θ), bandwidth (σ), phase offset (ψ), and aspect ratio (γ). Intuitively, each Gabor filter is a Gaussian kernel function modulated by a sinusoidal plane wave, where the Gaussian component provides the weights, and the sinusoidal component provides the directionality. One constructs a Gabor filter bank with several parameters and infers the local texture characteristics of the image from its response to the filters. Refer to [26] for a detailed explanation of the use of Gabor filter. In this work, we used a bank of 36 Gabor filters at equally distanced orientation in θ=[0∘,180∘) with constant wavelength λ = 20, σ = 1, ψ = 0, and γ = 0.5. We calculated the response of Gabor filters at those 36 directions and the direction with the largest magnitude of response is considered the dominant orientation at each pixel.

 Extending to Color Halftoning

Color printers use four different colorants, typically CMYK. In addition to these four primary colors, some printers, referred to as multi-channel or HiFi printers, use more inks to reproduce more of the color spectrum. One simple, yet efficient, approach for color halftoning is to apply the monochrome halftoning algorithms to different color channels independently and retrieve the final color halftone image by the superposition of all the monochrome halftone images.

In this work, to apply the proposed method to color images, we convert the RGB continuous-tone image to three colorant channels: C = 1 − R, M = 1 − G, and Y = 1 − B, we extract the orientation information in each channel independently and run our monochrome structure-aware IMCDP over C, M, and Y channels, separately. To consider four-color printers (C, M, Y, K), we assume that full under-color removal is employed. It means that at a given position, a black dot is placed, if and only if all three channels are present [27]. Note that the conversion from RGB to CMY depends on the physical process being used to lay down the CMY ink; therefore, there are several methods to transform an RGB image to CMY color channels. For instance, in Ref. [28], Shaked et al. proposed a different approach to initialize CMYK for color error diffusion halftoning.

In this paper, we illustrate our approach to extend the monochromatic structure-aware IMCDP to color halftoning based on C, M, Y channels; however our proposed structure-aware color halftoning is not limited to CMYK printers. Because the input color image is first converted to different channels and the method is applied on each channel separately, our approach could also be employed by printers which use extra inks.

Extracting orientation information for three colorant channels separately is computationally expensive. One possible approach is to derive the orientation information from the luminance channel and use the same angle information for three colorant channels C, M, and Y. To study if the luminance orientation is a good approximation of color channels’ orientation, we derived the luminance component of each RGB input image using Eq. (6) and used its orientation information for all three colorant channels. The coefficients used in Eq. (6) are identical to those used to calculate luminance in Ref. [29]. The impact of this simplification on the algorithm is discussed in Section 5. Luminance component =0.289R+0.587G+0.114B.

 Adjusting Sharpness Enhancement

In the original structure-aware IMCDP, the sharpness enhancement was static throughout the structured regions with fixed values of k1 = 1 and k2 = 1.8. Although a fixed sharpness enhancement produces acceptable results, it may not produce optimal results, especially for images that have multiple objects or areas with different textures. We propose an approach where the relative degree of sharpness enhancement is defined for each pixel of the image with variable k values. Without loss of generality, we assume k1 = 1 and calculate a variable k2 for each pixel of the image. We use the Gabor filters described in Section 3.2.1 to analyze the texture of the image to infer the degree of local image sharpness. Intuitively, in a sharp area of an image, such as at an edge, we expect a high response from the Gabor filter that is aligned with the edge and low response in other directions. On the other hand, in an area with no edge or other significant feature, we expect relatively similar responses from Gabor filters of all orientations.

We use the ratio between the maximum Gabor response to the average of the responses for each point as a representation of local image sharpness. For a bank of n Gabor filters, and G = [g1, g2, …, gn] responses to those n filters, we define local sharpness at each pixel as: s=max(G)mean (G)for G=[g1,g2,…,gn]. Calculated ratio of local sharpness, s, needs to be transferred to an appropriate value for k2 in IMCDP. Since the calculated local sharpness changes with color, brightness, and other features for different images, we calculate the transformation from local sharpness, s, to k2 empirically for each input image. For each input image, we generate two reference images, one random white noise (reference image W) and one sinusoidal image with wavelength similar to the Gabor filters (reference image S), both with the same dynamic range as the original image. The n Gabor filters are applied to the images S and W to produce G(s) and G(W) at each pixel and then using Eq. (7), the local sharpness value for two reference images is calculated as s(S) and s(W). The random noise should produce the local sharpness values (s) corresponding to k2 = 1, and the sinusoidal image should produce local sharpness values corresponding to k2 = maxk2. The inferred k2 for each pixel is then calculated as a linear interpolation between these two values and capped at maxk2 using Eq. (8): k2=maxs−S(W)S(S)−S(W)maxk2,maxk2, where s is the sharpness value at each pixel of the input image (calculated by Eq. (7)), maxk2 is a hyperparameter of the proposed algorithm, S(S) and S(W) are the average of s(S) and s(W), respectively. The effect of the variable k2 on halftoning is studied in Section 5.

 Evaluation Approach

The goal of this paper is to improve the recently proposed monochrome structure-aware IMCDP halftoning algorithm to use a faster and more flexible approach for obtaining the angle in the preprocessing step, extend it to color halftoning, and apply a dynamic sharpness enhancement ratio across the image. We study the impact of the proposed improvements on the performance of the structure-aware color halftoning by assessing sharpness, structure similarity, color consistency, and computation time.

Perceptual Aspects

To evaluate a halftoning algorithm and compare the halftone image with the continuous-tone color version of it, it is important to consider the human visual system (HVS) characteristics. Human visual system creates an illusion of a continuous-tone image when the halftone image is viewed from a sufficiently large distance. The decrease in sensitivity (blurring effect) that occurs in the HVS increases as a function of cycles-per-degree of visual angle. Visual angle is a function of resolution and viewing distance, and it is calculated as in Eq. (9): V isual Angle =R180πtan−11Dcycles ∕degree . In Eq. (9), R defines the resolution in dpi or ppi and D is the viewing distance in inches.

Different methods have been proposed to model the blurring effect of the HVS and the contrast sensitivity function (CSF) [2, 30–33]. The CSF describes the sensitivity of the HVS to different spatial and temporal frequencies that are present in the visual stimulus.

Because the halftone and the continuous-tone image are viewed at different resolutions, we use two different visual angles to properly simulate the function of the HVS. According to Eq. (9), to simulate a continuous-tone image, being viewed at a typical display of 100 ppi from a distance of 11.8 inches (30 cm), the visual angle would be 20.6 cycles/degree. However, to simulate the contrast sensitivity function of HVS for a halftone image, printed with a typical high-quality printer, at a resolution of 600 dpi and being viewed from distance of 13 inches (33 cm), the visual angle is 136.4 cycles/degree. In all the evaluations in this paper, to account for the HVS functionality, the halftone and the original image have been processed by visual angles of 136.4 and 20.6 cycles/degree, respectively. We perform the proposed color structure-aware IMCDP halftoning algorithm in CMY color space, but we evaluate the results in RGB space. The reason is that our HVS is modeled in RGB space [14].

 Evaluation Metrics

We evaluate our proposed method by studying three important aspects in image reproduction process: improving generating high-frequency contents, better preserving the structures and details, and faithful color reproduction of the original image.

Sharpness

To quantify the performance of the proposed algorithm in generating details, we find the no-reference perceptual blur metric proposed by Crete et al. practical for our study [34]. Their blur metric is estimated only on the luminance component and based on the discrimination between different levels of perceptible blur. Crete et al. quantified the final blur metric value in the range of 0 to 1, corresponding to the sharpest and blurriest perception, respectively. A lower blur metric corresponds to better reproduction of the high-frequency details in a halftone image; thus, the halftone structure has less blur annoyance and is considered as more pleasant to the human eye.

 Structure Similarity

The halftone image should appear visually similar to the original (input) image being viewed from an appropriate distance. We use the structural similarity index measure (SSIM) proposed by Wang et al. to evaluate the structural similarity between images [35]. Here, we compute the mean SSIMs (MSSIM) over blocks of size L × L (L = 11) for evaluating the structure similarity of results. The valid range for the MSSIM is from 0 to 1, a higher value corresponds to a higher similarity. The original quality assessment metric MSSIM has been applied only to the luminance component of the image. According to Wang et al., even though their experiments showed that the use of the other color components did not significantly change the performance of the metric, this should not be considered generally true for color image quality assessment. Therefore, we used the color version of MSSIM as Eq. (10) by running it over R, G, B channels separately, and compared the results to the MSSIM values using only the luminance component. MSSIM=13(MSSIMR+MSSIMG+MSSIMB). As with Wang et al.., we did not find a significant difference in the performance of the metric using the other color components of images therefore, we use the luminance component of images to compute the MSSIM reported in Section 5.

 Color Consistency

To measure the perceived color difference between the halftone and continuous-tone image, we use the Spatial CIELAB (S-CIELAB) metric proposed by Zhang et al. [33]. S-CIELAB is a spatial extension of CIELAB, based on an opponent-channel-representation-based HVS model to predict the perceived quality of color halftone images. S-CIELAB computes the perceived color difference as follows. First, the CIEXYZ representation of a color image is transformed to the opponent color space (Luminance, Red–Green, and Blue–Yellow). Next, the data in each color channel is convolved by a two-dimensional spatial kernel of the form of a series of Gaussian functions, simulating the visual spatial sensitivity of the HVS to the corresponding color dimension, as: f(x,y)=k∑iwiEi(x,y), where, Ei(x,y)=kiexp−x2+y2σi. The parameters k and ki normalize the filters such that they sum to one. The parameters wi and σi represent the weight and standard deviation of the Gaussian kernel, respectively. Then, the filtered version of each color channel is transformed back to CIEXYZ representation and thereafter to CIELAB. As a result, the representation of the image is called S-CIELAB, which includes both the spatial filtering and the CIELAB processing. Finally, the color difference between the S-CIELAB representation of two images is computed using the standard CIELAB color difference formula: ΔEab∗=(ΔL∗)2+(Δa∗)2+(Δb∗)2, where ΔL∗ = Lo∗− Lh∗, Δa∗ = ao∗− ah∗, and Δb∗ = bo∗− bh∗, and Lo∗, ao∗, and bo∗ are the original continuous-tone images and Lh∗, ah∗, and bh∗ are the halftone images in CIELAB (after going through S-CIELAB process and being converted back to CIELAB). Computation of ΔEab∗ is a pixel-by-pixel operation and the color difference between two images is usually reported as the average ΔEab∗ over all pixels. However, relying only on the average of S-CIELAB color difference values might not give us a complete insight to color reproduction error over images; therefore, we consider the maximum S-CIELAB color difference and S-CIELAB color difference greater than 5 units together with the average S-CIELAB in this work.

 Results

In this section, we study the impact of using a Gabor filter bank instead of Hough transform in improving the sharpness, structural similarity, color difference, and computation time. In our previous work [21], we used Hough transform in the preprocessing step, and found that by setting k1 = 1 and k2 = 1.8, halftone output achieves a desirable degree of sharpness enhancement and simultaneously improves the preservation of tonal and structural similarity. We used a test set including 20 images with different texture characteristics. Some of them are natural photographs and some are photographs of artworks such as oil paintings and sculptures. Links to access the test images are provided in Ref. [36]. The proposed algorithm and evaluations are implemented using Matlab.

As discussed in Section 3.2.2, to extend the monochrome structure-aware IMCDP to color halftoning, we used two approaches. In this section, we use the notation “3CH” to indicate that the orientation information of three channels have been processed separately, and “1CH” to indicate that orientation information has been extracted from the luminance component. We then evaluate the performance of the Gabor filter using fixed (k2 = 1.8) and variable k2 (1.6 ≤ k2 ≤ 2) for one channel (1CH) and three channels (3CH) and compare them to the results obtained from the original IMCDP and Hough-based structure-aware IMCDP. In all the evaluations, we use Hough 1CH (the approach in our previous work [21]) as a baseline, and the reported blur, MSSIM, and S-CIELAB metrics for all images are normalized to those reference values for each image. This means that the normalized value of blur, MSSIM, and S-CIELAB of Hough 1CH is by definition 1 and it is illustrated as a dashed horizontal line in all reported graphs in this section.

Sharpness

As discussed in Section 4.2.1, we used the blur metric to study the sharpness of the results. Figure 3(a) illustrates the normalized blur metric for different approaches. The dashed line at 1 represents the reference method, which is using the Hough transform 1CH in the preprocessing step. The solid line inside each box shows the median of results for each approach. According to Fig. 3(a), regardless of using Hough transform or Gabor filter in preprocessing step, the structure-aware versions of IMCDP overall result in lower blur metric, indicating sharper halftones, than the original IMCDP. Moreover, all the box plots corresponding to approaches using Gabor filter bank in preprocessing step lie under the Hough 3CH’s box and the reference line, showing that Gabor filter banks approach outperforms Hough transform in extracting orientation information. Comparing boxes for 1CH and 3CH versions of each approach shows a similar median value (1CH have a slightly better increase in sharpness). However, bigger box length and whiskers for 1CH versions show that the results are slightly more dispersed compared to 3CH versions. It means that using the orientation information extracted from the luminance component for all three channels is an acceptable simplification. This could also be inferred by the shape of the box for Hough 3CH. Hough 3CH’s box is a very small box with median close to the reference line, which represents the Hough 1CH results. The small box-length in Hough 3CH indicates that the overall results in Hough 3CH have a high agreement with that in Hough 1CH. In other words, using the orientation information extracted from the luminance component for all three channels is a reasonable approximation, but not a precise alternative solution.

(a) Sharpness evaluation for different approaches according to normalized blur metrics. Lower blur metric indicates sharper halftones, (b) MSSIM evaluation for different approaches according to normalized MSSIMs. Higher MSSIM indicates more structural similarity. In both panels, dashed line at 1 represents the reference method (Hough 1CH).

 Structure Similarity

As discussed in Section 4.2.2, MSSIM was used to evaluate the performance of the proposed method in preserving the structural similarity in halftone images. Fig. 3(b) represents the normalized MSSIM for different approaches. The dashed line at 1 represents the reference method, which is using the Hough transform 1CH in the preprocessing step. According to this figure, regardless of using Hough transform or Gabor filter banks in preprocessing step, the structure-aware IMCDP results in higher MSSIM, indicating that the structure-aware versions of IMCDP reproduce details and structures better than the original IMCDP. Moreover, all the box plots corresponding to approaches using Gabor filter bank in preprocessing step lie above the Hough 3CH’s box and the reference line, showing that Gabor filter banks outperforms Hough transform in preserving structural similarity. Comparing boxes for 1CH and 3CH versions of each approach shows a slightly higher median value for 1CH approaches. However, the sections of 3CH boxes are more even compared to those of 1CH boxes.

Bigger box-length and whiskers for 1CH versions show that the results are slightly more dispersed compared to 3CH versions. The results for MSSIM also show that using the orientation information extracted from the luminance component for all three channels is a reasonable approximation, but not a precise alternative solution. Looking at boxes for Gabor 3CH with fixed and variable k2 (i.e., Gabor 3CH k2 = 1.8 and Gabor 3CH 1.6 ≤ k2 ≤ 2, respectively), one can observe that the box plot in Gabor 3CH has a higher median value, resulting in halftone with better preservation of structures.

We also did a further investigation regarding the outlier datapoints in Gabor 3CH k2 = 1.8, Gabor 3CH 1.6 ≤ k2 ≤ 2, and Gabor 1CH k2 = 1.8 in Fig. 3(a). The outlier datapoints in these box plots correspond to the same image. However, we did not observe the same image resulting in an outlier of MSSIM value in Fig. 3(b). This suggests that this outlier point could be derived from the nature of the image, and not the method.

 Color Difference

As discussed in Section 4.2.3, we used S-CIELAB to evaluate the color reproduction error in our proposed method. Figure 4(a), (b), and (c) illustrates the results for average S-CIELAB, maximum S-CIELAB, and S-CIELAB >5, respectively. The dashed line at 1 represents the reference method, which is using the Hough transform 1CH in the preprocessing step. According to Fig. 4(a), the average color reproduction error in the original IMCDP is lower than the structure-aware versions of it, indicating better color reproduction in halftone outputs. However, relying only on the average of S-CIELAB values might not give us a complete insight to color reproduction error over images, therefore, we consider the maximum S-CIELAB and S-CIELAB >5 together with the average S-CIELAB. Comparing the maximum values of S-CIELAB in Fig. 4(b) indicates that the structure-aware versions of IMCDP result in smaller maximum values of S-CIELAB. When studying the color reproduction error, color differences exceeding 5 units are easily perceptible by our HVS, and according to Fig. 4(c), the original IMCDP reproduces more visible errors than the structure- aware versions of it. In addition, results in Fig. 4(b) and (c) show that the Gabor filter outperforms Hough transform in reproducing halftones with faithful colors, resulting in smaller maximum S-CIELAB and fewer color difference >5.

Color difference evaluation according to normalized S-CIELAB metrics, using the Hough 1CH’s S-CIELAB metric as a measure of scale. (a) Average S-CIELAB (b) maximum S-CIELAB (c) S-CIELAB >5. The dashed line at 1 represents the reference method (Hough 1CH).

 Computation Time

As discussed in Section 3, one of the contributions of this paper is using a faster and more flexible approach for obtaining the angle in the preprocessing step. The Hough-transform-based preprocessing can be prohibitively expensive as it has cubic complexity however, as IMCDP has a quadratic complexity, using a preprocessing approach with at most quadratic complexity makes the structure-aware IMCDP as fast as original IMCDP. We used a Gabor-based approach to obtain the local orientation information in preprocessing step. G Gabor filters of size S × S are generated only once and thereafter applied to images. As Gabor filters are applied by convolving the images with the filters, the complexity of calculating the response to one Gabor filter of size S × S at one pixel is O(S2). For an image with size N × N, applying G Gabor filters on all pixels will be of O(N2 × G × S2), which is quadratic in terms of the image size or linear in terms of number of pixels.

Figure 5 shows the result for preprocessing computation time. As can be seen in Fig. 5(a), using a Gabor-based approach in preprocessing step considerably decreases the computation time. Because the order of magnitude in computation time is different for Hough-based and Gabor-based approaches, we use Fig. 5(b) to clearly illustrate the computation time in Gabor-based approaches. The reported time is the average running time for three trials, and it is based on the current implementation of the proposed algorithm in Matlab on a MacBook Pro equipped with 2.6 GHz Intel Core i7 CPU and 16 GB 2400 MHz DDR4 memory.

preprocessing computation time. (a) Comparison of all approaches (b) comparison of only Gabor-based approaches.

Figures 6 and 7 represent visual examples of images in our dataset. As can be seen in Fig. 6, images being halftoned by the structure-aware IMCDP, Fig. 6(c) and (d), are perceived to be sharper than the image halftoned by the original IMCDP in Fig. 6(b). The structure-aware IMCDP reproduces both the details and structures and the color better than the original IMCDP. Structure-aware IMCDP performs better than the original IMCDP in displaying highly-detailed appearance and conveying feeling of 2.5D structural features present in a 2D image.

(a) Original continuous-tone image: top row “In the wind” [37], and bottom row “Relief for the chapel” [38]. (b) Halftoned image with original IMCDP, (c) Halftoned image with proposed color structure-aware IMCDP, Gabor 3CH with fixed k1 = 1, k2 = 1.8, (d) Halftoned image with proposed color structure-aware IMCDP, Gabor 3CH with k1 = 1 and variable k2 (1.6 ≤ k2 ≤ 2). In (c) and (d), the orientation information is extracted from three different colorant channels C, M, Y, independently. Halftone images are displayed at resolution 200 dpi. The color version of this figure is best viewed in the electronic version of this paper.

(a) IMCDP, (b) Gabor 3CH with variable k2 (1.6 ≤ k2 ≤ 2). The orientation information is extracted from three different colorant channels C, M, Y, independently. (c) Gabor 1CH with variable k2 (1.6 ≤ k2 ≤ 2). The orientation information is extracted from only the luminance channel. Halftone images are displayed at resolution 100 dpi. The color version of this figure is best viewed in the electronic version of this paper.

Fig. 7 illustrates a cropped region of “Relief for the chapel” at low resolution of 100 dpi to clearly demonstrate the individual dots and halftone structure in our proposed method. The original IMCDP and its structure-aware versions present pleasant and homogeneous halftone structures. Gabor 3CH with variable k2 (1.6 ≤ k2 ≤ 2) in Fig. 7(b) and Gabor 1CH with variable k2 (1.6 ≤ k2 ≤ 2) in Fig. 7(c) create sharper halftones compared to the original IMCDP in Fig. 7(a). Moreover, according to the evaluation metrics presented in Figs. 3 and 4, using the orientation information extracted from the luminance component for all three channels is acceptable approximation, but not a precise alternative. Visual comparison of Gabor 3CH in Fig. 7(b) and Gabor 1CH in Fig. 7(c) presents similar halftone structures and agrees with objective results that using only the luminance component to extract orientation information could be considered as a reasonable approximation.

 Conclusion and Future Work

The recently proposed structure-aware IMCDP improves the original IMCDP halftoning algorithm in terms of the reproduction of high-frequency details, structures, and tones by aligning the halftone structures with the dominant orientation of pixels in a neighborhood. However, Hough-transform-based preprocessing is computationally costly (cubic complexity), and sharpness enhancement is applied statically across the entire structured regions of the image. These limitations made the structure-aware version not suitable for larger images as larger images often contain variable sharpness and texture, and the size of the image can make the computation challenging. In this paper, we address both these limitations, and extend the method to color halftoning. We use a Gabor-based approach in preprocessing step that not only decreases the computation time by orders of magnitude (from cubic to linear) but also improves sharpness enhancement, structure similarity, and color consistency. Using the magnitude of Gabor responses to characterize texture and sharpness allows the method to have control over the degree of sharpness enhancement in different parts of an image. The adaptive adjustment of sharpness enhancement gives more flexibility to the user in improving the visual quality of image reproduction in our proposed structure-aware halftoning algorithm. In addition, we extend the method to color halftoning. We studied two approaches: extracting the structure information separately from each channel and applying them to color channels independently and using only the luminance channel structure information for all color channels. According to our investigation, halftoning each color channel based on its structure information produces more stable improvement for all image types; however, the objective evaluation and visual comparison of halftone structures show that the results are on average similar. In other words, the luminance orientation information is an acceptable approximation for orientation in different color channels, but it is not a precise alternative.

In this work, the performance of the structure-aware color halftoning has been validated using digital images. However, in printing color halftoned images, print settings such as print resolution, type of the printer, paper quality, etc. affect the print quality and appearance. To study the impact of print parameters on reproduction of image structures and details, we used a custom printer (screen printing) and validated the performance of the presented method [39].

Moreover, the proposed efficient adaptive structure-aware color halftoning can be extended to 3D halftoning to improve the reproduction of 3D printed surfaces. The original IMCDP algorithm has been extended to 3D domain in Ref. [40] and the effect of applying variable halftone structures based on the spatial information has shown that different halftone structures could be used to control the reproduction of details [24, 41]. As a next step, we are planning to develop an adaptive structure-aware 3D color halftoning algorithm, which has all the reproduction qualities of IMCDP while being aware of the color and geometry of the 3D object.

 Acknowledgment

This project was funded by ApPEARS (Appearance Printing European Advanced Research School), the European Union’s Horizon 2020 programme under the Marie Skłodowska-Curie grant agreement No. 814158.

 References Lau D. L. Arce G. R. Modern Digital Halftoning 2018 CRC Press Boca Raton, FL Ulichney R. Digital Halftoning 1987 MIT Press Cambridge, MA Floyd R. W. Steinberg L. 1976 An adaptive algorithm for spatial gray-scale Proc. Soc. Inf. Disp 17 75 77 75–7 Ostromoukhov V. 2001 A simple and efficient error-diffusion algorithm Proc. 28th Annual Conf. on Computer Graphics and Interactive Techniques 567 572 567–72 ACM New York City, NY 10.1145/383259.383326 Zhou B. Fang X. 2003 Improving mid-tone quality of variable-coefficient error diffusion using threshold modulation ACM Trans. Graph. 22 437 444 437–44 10.1145/882262.882289 Jarvis J. F. Judice C. N. Ninke W. H. 1976 A survey of techniques for the display of continuous tone pictures on bilevel displays Comput. Graph. Image Process. 5 13 40 13–40 10.1016/S0146-664X(7680003-2 Stucki P. 1981 MECCA-a multiple-error correction computation algorithm for bi-level image hardcopy reproduction, IBM Thomas J Watson Research Division IBM Thomas J. Watson Research Center New York City, NY Analoui M. Allebach J. P. 1992 Model-based halftoning using direct binary search Human Vision, Visual Processing, and Digital Display III 1666 96 108 96–108 10.1117/12.135959 Gooran S. 2004 Dependent color halftoning: Better quality with less ink J. Imaging Sci. Technol. 48 354 362 354–62 Lieberman D. J. Allebach J. P. 1997 Efficient model based halftoning using direct binary search Proc. Int’l. Conf. on Image Processing vol. 1 775 778 775–8 IEEE Piscataway, NJ 10.1109/ICIP.1997.648077 Lieberman D. J. Allebach J. P. 2000 A dual interpretation for direct binary search and its implications for tone reproduction and texture quality IEEE Trans. Image Process. 9 1950 1963 1950–63 10.1109/83.877215 Li H. Mould D. 2010 Contrast-aware halftoning Comput. Graph. Forum 29 273 280 273–80 10.1111/j.1467-8659.2009.01596.x Eschbach R. Knox K. T. 1991 Error-diffusion algorithm with edge enhancement JOSA A 8 1844 1850 1844–50 10.1364/JOSAA.8.001844 Pang W. M. Qu Y. Wong T. T. Cohen-Or D. Heng P.-A. 2008 Structure-aware halftoning ACM SIGGRAPH 2008 Papers 1 8 1–8 ACM New York City, NY 10.1145/1360612.1360688 Chang J. Alain B. Ostromoukhov V. 2009 Structure-Aware Error Diffusion TOG 28 1 8 1–8 10.1145/1618452.1618508 Liu L. Chen W. Zheng W. Geng W. 2014 Structure-aware error-diffusion approach using entropy-constrained threshold modulation Vis. Comput. 30 1145 1156 1145–56 10.1007/s00371-013-0895-0 Kwak N. J. Ryu S. P. Ahn J. H. Edge-enhanced error diffusion halftoning using human visual properties Proc. Int’l. Conf. on Hybrid Information Technology 2006 Vol. 1 IEEE Piscataway, NJ 499 504 499–504 10.1109/ICHIT.2006.253533 Li X. 2006 Edge-directed error diffusion halftoning IEEE Signal Process. Lett. 13 688 690 688–90 10.1109/LSP.2006.879465 Lee H. S. Kong K. K. Hong K. S. “Laplacian based structure-aware error diffusion Int’l. Conf. on Image Processing 2010 IEEE Piscataway, NJ 525 528 525–8 10.1109/ICIP.2010.5651243 Li D. Kiyotomo T. Horiuchi T. Tanaka M. Shigeta K. 2020 Texture-aware error diffusion algorithm for multi-level digital halftoning J. Imaging Sci. Technol. 64 50410 50411 50410–1 10.2352/J.ImagingSci.Technol.2020.64.5.050410 Abedini F. Gooran S. Kitanovski V. Nyström D. 2021 Structure-aware halftoning using the iterative method controlling the dot placement J. Imaging Sci. Technol. 65 60404-1 10.2352/J.ImagingSci.Technol.2021.65.6.060404 Hough P. V. “Method and means for recognizing complex patterns,” U.S. Patent 3,069,654” (1962) Gooran S. Kruse B. 2015 High-speed first- and second-order frequency modulated halftoning J. Electron. Imag. 24 1 19 1–19 10.1117/1.JEI.24.2.023016 Gooran S. Abedini F. 2020 3D surface structures and 3D halftoning Proc. IS&T Printing for Fabrication 2020 69 74 69–74 IS&T Springfield, VA Campbell F. W. Robson J. G. 1968 Application of Fourier analysis to the visibility of gratings The J. Phys. 197 551 Jain A. K. Farrokhnia F. 1991 Unsupervised texture segmentation using Gabor filters Pattern Recognit. 24 1167 1186 1167–86 10.1016/0031-3203(91)90143-S Sharma G. Bala R. Digital Color Imaging Handbook 2017 CRC Press Boca Raton, FL Shaked D. Arad N. Fitzhugh A. E. Sobel I. E. 1998 Color diffusion: error diffusion for color halftones Proc. SPIE 3648 10.1117/12.334589 “Recommendation ITU-R BT.601-7, Studio encoding parameters of digital television for standard 4:3 and wide-screen 16:9 aspect ratios,” (2011) Pappas T. N. Neuhoff D. L. 1999 Least-squares model-based halftoning IEEE Trans. Image Process. 8 1102 1116 1102–16 10.1109/83.777090 Näsänen R. 1984 Visibility of halftone dot textures IEEE Trans. Syst. Man Cybern. SMC-14 920 924 920–4 10.1109/TSMC.1984.6313320 Daly S. J. Rabbani M. Chen C. T. “Digital image compression and transmission system visually weighted transform coefficients,” US Patent 4,780,761, (1988) Zhang X. Wandell B. A. 1997 A spatial extension of CIELAB for digital color image reproduction J. Soc. Inf. Disp. 5 61 63 61–3 10.1889/1.1985127 Crete F. Dolmiere T. Ladret P. Nicolas M. 2007 The blur effect: perception and estimation with a new no-reference perceptual blur metric Proc. SPIE 6492 649201 10.1117/12.702790 Wang Z. Bovik A. C. Sheikh H. R. Simoncelli E. P. 2004 Image quality assessment: from error visibility to structural similarity IEEE Trans. Image Process. 13 600 612 600–12 10.1109/TIP.2003.819861 Links to test images. Accessed: Oct. 17, 2022 Haren H. In The Wind. Accessed: Jun. 14, 2022 Kotomi, Relief for the Chapel. Accessed: Jun. 21, 2021 Abedini F. Trujillo Vazquez A. Gooran S. Klein S. 2023 Effect of halftones on printing iridescent colors Accepted for Presentation at the Color Imaging: Displaying, Processing, Hardcopy, and Applications Conf., at IS&T Electronic Imaging IS&T Springfield, VA Abedini F. Gooran S. Nyström D. 2020 3D halftoning based on iterative method controlling dot placement Proc. IS&T Printing for Fabrication 69 74 69–74 IS&T Springfield, VA Abedini F. Gooran S. Nyström D. 2021 The effect of halftoning on the appearance of 3D printed surfaces 47th Annual Conf. of Iarigai Iarigai Darmstadt, Germany