Improved Partial Differential Equation and Fast Approximation Algorithm for Hazy/Underwater/Dust Storm Image Enhancement

This paper presents an improved and modified partial differential equation (PDE)-based de-hazing algorithm. The proposed method combines logarithmic image processing models in a PDE formulation refined with linear filter-based operators in either spatial or frequency domain. Additionally, a fast, simplified de-hazing function approximation of the hazy image formation model is developed in combination with fuzzy homomorphic refinement. The proposed algorithm solves the problem of image darkening and over-enhancement of edges in addition to enhancement of dark image regions encountered in previous formulations. This is in addition to avoiding enhancement of sky regions in de-hazed images while avoiding halo effect. Furthermore, the proposed algorithm is utilized for underwater and dust storm image enhancement with the incorporation of a modified global contrast enhancement algorithm. Experimental comparisons indicate that the proposed approach surpasses a majority of the algorithms from the literature based on quantitative image quality metrics.


Introduction
Image scenes taken in outdoor environments can be affected by weather resulting in hazy appearance [1]. This leads to poor visibility as a result of scattering and absorption of light by suspended particles in the air [1]. Thus, there is the need for haze removal to improve image visibility and contrast in computer vision applications such as surveillance, intelligent transport systems, etc. Image de-hazing is a current and active area of research in image processing [2]. There are numerous approaches which are classified as single-or multi-image-based schemes [2]. Furthermore, interest has shifted to single image based methods due to feasibility and cost concerns inherent in multiple image-based methods.
The single image-based methods can be classified under enhancement, restoration, fusion-and deeplearning-based domains [2]. Previously, restoration-based approaches were the preferred route for image de-hazing with the dark channel prior (DCP)-based method [3] being the most popular and incorporated in numerous algorithms [1]. For instance, Li et al proposed an image de-hazing method involving content-adaptive dark channel employing an associative filter, with structure transfer ability for efficient dark channel computation [4]. This is in addition to post enhancement of the luminance of the de-hazed image and local contrast preservation [4]. Also, due to the excessive computation time of is manifested by edge enhancement or increase in gradients across the image scene. The same explanation is applicable to dusty and underwater images, which also exhibit haze and/or uneven illumination. Based on this realization, we can describe the example categories of dark, hazy and underwater images depicted in Fig. 1 as shown in Table 1. What all these image categories have in common is that edges or details are less pronounced and that they exhibit low visibility and contrast. The generalized enhancement models for processing these type of images is as shown in Fig. 1(b). The enhancement operator can be based on tonal correction, exponential, power law or statistical enhancement operators & previous works utilized these configurations for processing the aforementioned images types.   Can suffer from both poor illumination and haze.
Can also suffer from both poor illumination and haze.
In summary, illumination and hazy conditions are similar in nature and underwater images may suffer from additional illumination or foggy conditions (or a combination of both) in addition to colour channel distortion. These issues are handled best with Retinex-based operators without modification. However, in the absence of Retinex, statistics-based contrast enhancement methods also yield reasonable results. The Retinex works for o Dark images (due to logarithm function) o Hazy images due to local and global (multiscale) surround functions for enhancement o Underwater images due to local and global enhancement properties. o Contributions of proposed approach include: o Repurposing of a log-less logarithmic image processing (LIP) algorithm for de-hazing in a PDE framework. o Modification of algorithm for underwater and sandstorm image enhancement. o Post refinement of de-hazing results using filter-and fuzzy logic-based enhancement. o Development of a fast de-hazing algorithm based on approximation of hazy formation model.

Proposed algorithm and modifications
The proposed approach seeks to solve the problems and shortcomings of the previously proposed PDEbased approaches [19] [18]. The former method performed remarkably well for illumination correction and de-hazing, but still required some improvements in terms of reduction of darkening of image regions in de-hazed images.

Revised PDE-based formulation
The PDE-based method performed relatively well in illumination correction but led to images with colour fading and darkened regions with over-sharpened edges when used for de-hazing applications [18]. This was mitigated by using frequency emphasis filters to increase brightness and preserving low frequency components, reducing noise due to over-sharpening [18]. Thus, based on research and experiments, gradient increase is a reliable indicator of image enhancement. Thus, the average gradient (AG) [20] was chosen as a suitable and reliable metric for the stopping criterion of the PDE and this modification was also adapted to the de-hazing process [18]. Though results were improved, the absence of local contrast enhancement and darkening of image regions still persisted in varying degrees.

PDE-based formulation using Patrascu LIP for de-hazing
One of the drawbacks encountered in contrast enhancement of hazy images is the darkening of processed images. This was also true of the revised PDE formulation, leading to the exploration of other algorithms to address this problem. This was coupled with the challenge to maintain the relatively lowcomplexity of the previous algorithm and its advantages while improving its performance. Thus, in order to avoid complex and complicated algorithms, we selected the logarithmic image processing (LIP) formulation by Patrascu et al [21], which does not involve actual logarithmic calculation. In this section, we modify the proposed approach to utilize this algorithm, which does not affect edges. It should be noted that the hardware architecture of the modified LIP algorithm was also developed and verified in previous work [22]. This makes it a practical alternative for use in the formulation.
Positive attributes of the LIP included dynamic range compression without colour fading in addition to preservation of image highlights in bright images. The algorithm was subsequently adapted to image de-hazing but with initially mixed results. Thus it was reformulated into a PDE flow, incorporating aspects such as adaptive computation of the regularization parameter, and AG-based optimization. Initially, the exponent, , was fixed for the LIP. However, we re-imagined this parameter as determined by the amount of dark/black area (BA) and light/white area (WA) in the images. By adaptively computing , we account for the differences in each hazy image. The LIP guides the evolution of the image in the PDE formulation and the results were better than the previous formulation [18]. However, some hazy images were still darkened or had over-enhanced sky regions, though the adaptive computation of reduced the degree of the effect. Furthermore, the addition of the illuminationreflectance contrast enhancement scheme (IRCES) as a post-processing operation brightened the images. The PDE-based expression is given as; In (1) Using the finite difference method (FDM) yields the discrete form as; After the processing, we now utilize the IRCES to brighten the image and enhance the edges without amplifying the noise as; ( , ) = ( ( , )). This complete process is the proposed algorithm (PA) and it solves the problem of dark images after de-hazing operations in most cases but may also lighten other ones. The approach is still amenable to hardware implementation and just requires the inversion of the image before and after processing.

Underwater image enhancement
For the case of underwater image enhancement, we utilized a modified function known as gain offset correction contrast stretching (GOC-CS) [10] to perform colour correction for the effect of water prior to enhancement. This configuration was compared with PDE-based formulations utilizing GOC1, GOC2, piecewise linear transform (PWL) and histogram specification (HS) [15].

Application to dust/sand-storm image enhancement
For dusty image enhancement, we utilize the same setup for underwater images since the images are degraded in a similar manner. The flowchart of the PDE-based formulation with the schemes for hazy/dust storm and underwater images is shown in Fig. 3. The colour enhancement version of the hazy image enhancement can be performed for faded images using a previously proposed algorithm called red-green-blue-intensity-value (RGB-IV) [23].

Fast de-hazing algorithm and modifications
In this section, we also propose a fast de-hazing algorithm using a simple approximation of the hazy image formation model. In the model a hazy image [24] is defined as shown; In (4) ( , ) is the hazy image, ( , ) is the haze-free image or radiance, is the sky light [25] and ( , ) is the transmission map [25] expressed as; In (5), (. ) is the scattering coefficient, , is the light ray at pixel location, , [25]. The objective of de-hazing process is to obtain ( , ) from ( , ) [1] as shown in (6); However, by noting from previous experiments that ( , ) is similar to the inverted image ( , ) or illumination component, ( , ) , gives ( , ) ≡ 255 − ( , ) or ( , ) ≡ 1 − ( , ) , if normalized. Additionally, is equivalent to an array of ones or reflectance value and can be converted to a scalar value such that = 1. The radiance, ( , ) becomes the de-hazed or enhanced output image. Thus we can rewrite this expression in (6) as; Comparing (4) Thus, the radiance, ( , ) is similar to the reflectance, ( , ), while ( , ) is the inverse of the illumination component as before and finding ( , ) is simpler in this case. We derive the equation for the alternate enhancement-based de-hazing algorithm as; In eqn. (9), { ( , )} is a local-global or multi-scale contrast enhancement function and ( , ) can be computed as; The global function used for { ( , )} is the LIP algorithm by Patrascu, while the local contrast function is the contrast limited adaptive histogram equalization (CLAHE).
The fast de-hazing algorithm initially resulted in clearly visible halos due to the CLAHE algorithm, which necessitated a global contrast function prior to the CLAHE (clip limit is set to 0.002 with tile size of 32). Reducing or increasing the tile size beyond this baseline leads to greater colour distortion and fading, which is worse around tile size of 8. By performing the LIP algorithm first, then followed by CLAHE, there is a smoother transition from low to high frequency regions as observed in the refined transmission maps of the de-hazed images. Increasing tile size above 32 to 64 leads to increase in highpass filtering action, yielding more detail enhancement but distorted colours, halos and noise artifacts. However, in images with non-uniform or thick haze the algorithm leads to distorted colours. This occurs when fixing the clip limit at a particular value and adaptive computation does not lead to consistency.
Additionally, operating in the RGB colour space appears better than HSI or HSV space as the colours are seemed more consistent. Other colour space transformations lead to considerable colour fading. Based on experiments, the CLAHE introduces uncertainties and additional parameters to adjust. This makes this scheme unreliable and only suited to images with uniform and thin haze. Additional experiments were performed to fully render the algorithm adaptively; however, results were generally unsatisfactory. Thus, we reformulated the algorithm in the following form; ̅ ( , ) = 20 10 | ̅ ( , )| In (11) bilateral filtering ( ( , )) result. The former is chosen for speed. Then, the radiance or de-hazed image is obtained as; This is then incorporated into the existing PDE-based formulation to gradually improve the contrast. The results are shown in Fig. 4 and are better than the initial formulation with results more balanced than the previous modified PDE. This version does not require the CLAHE, thus maintains results free from halos and colour distortion and does not require tuning of multiple parameters. However, it also suffers from minimal local contrast enhancement and its run-time is similar to the modified PDE, which is slightly slower than the initial formulation.

Hazy images
In Fig. 5, we present sample visual results from [6] amended with the results of PA for comparison. Based on visual assessment, PA yields comparable and brighter de-hazed results compared to several of the other algorithms. We verify this by utilizing the Fog Aware Density Evaluator (FADE) devised by Choi et al [30] and compare with other algorithms tested in work by Li and Zheng [6] as shown in Table  3. Lower FADE values indicate improved visibility and considerably reduced fog or haze density in the de-hazed images. Results show that PA yields comparable or best results for certain images despite being relatively less complex compared to the other approaches which involve the DCP or its variant.  We also compare the methods by Ancuti and the DEnsity of Fog Assessment-based Defogger (DEFADE) algorithm proposed by Choi et al [30] to results of PA in Fig. 6. Visual results show that PA yields the sharpest most detailed image; however gradient reversal artifacts and darkened regions are observed for the building image, while the other images are more balanced and brighter than the results of Ancuti et al and the DEFADE algorithm. We also present FADE results from [30] amended with those obtained from images processed with PA for quantitative comparison in Table 3. The numerical values confirm the improved results of PA, which has the lowest haze density values with DEFADE yielding the second best results (red bolded).
We also compare with other algorithms such as those by Tan [33], Fattal [34], Kopf et al [35], He et al [3], Tarel et al [36] in addition to the method by Ancuti et al [37], DEFADE [30] and PA. The visual comparison in Fig. 6 is from Choi et al [30] amended with the image results of PA. The results of PA are shown without the brightening filter and are darker than the other results, though with increased contrast. The fog density values of the images processed with PA are compared with those of the various algorithms presented in [30] and shown in Table 5. Results indicate that PA yields images with the lowest perceptual fog density in most cases apart from the method by Tan [33].   Figure 7 Fattal [34] Kopf et al [35] He et al [3] Tarel et al [36] Ancuti et al [

Underwater images
We also compare the results of PA for underwater images with results of Khan et al [38] and Galdran et al [39]. Fig. 8 depicts the visual results from Khan et al [38] amended with PA for visual comparison. Visual observation indicates that the results of PA are the most vivid and colourful in most cases. However, some images are quite dark and in one particular image, distorted (Ancuti2 image in penultimate image column, bottom row).
We also compare underwater image enhancement results of PA with the algorithms by He et al [3], Ancuti & Ancuti [37], Drews-Jr et al [40], Galdran et al [39], Emberton et al [41], Ancuti et al1 [7] and Ancuti et al2 [11]. The visual results are shown in Fig. 9, which is from [11] amended with results of PA. Once more, PA shows the most vivid results, though some images are dark and some colour distortion observed in the last image result of penultimate image row in Fig.9.

Figure 8: Image results from [38] amended with results of PA (a) Original underwater images (b) Galdran et al [39] (c) Khan et al [38] (d) PA
We also quantitatively compare the algorithms using the Underwater Color Image Quality Evaluation (UCIQE) metric developed by Yang and Sowmya [42] in Table 5. Generally, the higher the numerical value of the UCIQE, the better the enhancement outcome. Numerical results show that PA consistently has the highest UCIQE values for almost all the processed images and the highest average UCIQE value. This is consistent with the considerable contrast and colour enhancement observed in most of the images processed with PA. Ancuti & Ancuti [37] Drews-Jr [40] Galdran et al [39] Emberton et al [41] Ancuti et al1 [7] Ancuti et al 2 [11] Khan et al [38] PA

Sand/dust-storm images
For the sandstorm images, we use images from [43] and compare results with the proposed approach in Fig. 10 and Table 6. Results indicate that there is more contrast and colour enhancement using the proposed algorithm compared with the method by [43]. However, in some cases, the proposed approach leads to discolouration of processed images. These images have little overlap between red, green blue channel histograms, making colour correction much more difficult. However, such problems can be mitigated using previous approaches for such images, though this implies additional modifications. Based on results for hazy, underwater and dust storm images, PA is relatively versatile in handling these image categories with minimal modification, while problems of previous methods have been improved upon. However, the proposed approach still darkens certain images with some colour distortion. This is a shortcoming of the PA, which we address in the next subsection by substitution of the filter-based post enhancement operator with an edge-agnostic version.

Rectification of darkening of image results using fuzzy homomorphic enhancements
We wish to resolve the darkening of the processed images for hazy, dust and underwater images and require a low-complexity, edge-agnostic and effective tonal correction algorithm. Thus, we utilize the fuzzy homomorphic enhancement (FHE) algorithm from previous work [16] to replace the IRCES as a post processing step in PA. The improvements are shown (and compared with previous results) in Fig. 11 for hazy, underwater and dust storm images. The modification of the algorithm does not affect the other processed images, which are not darkened by the de-hazing process. Thus, we have rectified the issue of darkened images, while maintaining good de-hazing outcomes. Also, this modification reduces the runtime of PA when the FHE is utilized in the PDE formulation, eliminating the need for a post-processing. This also results in faster convergence of the PDE and a more balanced enhanced output free from edge artifacts.

Conclusion
This paper has presented the theoretical formulation and experimental validation of modified PDEbased image enhancement algorithms for hazy and underwater images. Problems such as darkening, sky region, excessive edge and noise enhancement were addressed by utilizing a soft edge-agnostic LIP algorithm and utilizing the spatial filter as post-processing enhancement operation. Thus, absence of halo effect and sky region enhancement is maintained in the result of the modified PDE formulations in addition to brightening of dark regions. A fuzzy homomorphic enhancement algorithm, which rectified the darkening of certain outlier images was utilized as a post-processing alternative to the IRCES. Moreover, the proposed modified algorithms have been employed in underwater and dust storm image enhancement with mostly improved results compared to several well-known and more complex algorithms from the literature.