Robust Real-World Image Super-Resolution against
Adversarial AttacksConference: Proceedings of the 29th ACM Int’l Conference on Multimedia; October 20–24, 2021; Virtual Event, China.Proceedings of the 29th ACM Int’l Conference on Multimedia (MM ’21), Oct. 20–24, 2021, Virtual Event, ChinaPrice: 15.00ISBN: 978-1-4503-8651-7/21/10DOI: 10.1145/XXXXXX.XXXXXXCCS: Computing methodologies Image processing

Single-image super-resolution (SISR) requires to synthesize high-resolution contents from a single low-resolution image. SISR algorithms have been studied for a long time, and can be grouped into two categories: traditional and learning-based methods. Traditional SR methods are mainly based on edge priors (Fattal 2007; Jian Sun et al. 2008), image registration (Irani and Peleg 1991) and statistics (Jinggang Huang and Mumford 1999; Shan et al. 2008; Aly and Dubois 2005). Among learning-based SR methods (Kim and Kwon 2010; He and Siu 2011; Ni and Nguyen 2007; Yang et al. 2010), DNN (Dong et al. 2016; Zhang et al. 2017; Lai et al. 2017) is one of the most popular and effective models. According to the source of degraded images, SISR tasks could be divided into three groups: non-blind synthetic SR, blind image SR and real-world SR. Non-blind synthetic SR (Dai et al. 2019; Guo et al. 2020) simply adopts bicubic or Gaussian downsampling to simulate the image degradation on both training and testing set, but the SR models trained with the synthetic dataset may fail to adapt to unknown blur kernels in practice. Blind image SR (Gu et al. 2019; Huang and Xia 2021) is based on a more realistic setting that the blur kernels during the testing are unavailable when training a SR model. Blind image SR algorithms could be evaluated with synthetic or real image sets, but these real datasets lack HR ground truths.

To fill the gap, real-world image SR task (Chen et al. 2019; Zhang et al. 2019a; Cai et al. 2019; Wei et al. 2020) has been proposed, upsampling real degraded images by collecting many pairs of real LR sample and its HR version. Cai et al. (Cai et al. 2019) employ two types of digital cameras to collect real SR pairs as a new dataset, RealSR. Wei et. al. (Wei et al. 2020) further provide a larger real-world SR benchmark DRealSR, which consists of well-aligned SR pairs captured by up to five different DSLR cameras. Particularly, Cai et al. (Cai et al. 2019) reveal that existing SR networks trained on a simulated dataset (Timofte et al. 2017) do not show advantages with real-world samples. It is because real degradation includes lots of factors, e.g., anisotropic blur, signal-dependent noise and cross-camera degradation processes. Thus real-world image SR is a more challenging and meaningful problem.

To solve this task, Cai et. al. (Cai et al. 2019) develop a laplacian pyramid kernel prediction network LP-KPN to explicitly learn a specific restoration kernel for each pixel. Wei et. al. (Wei et al. 2020) introduce a component divide-and-conquer model CDC that constructs three Component-Attentive Blocks (CAB) associated with flatten regions, edges, and corners. CDC infers super-resolved contents with the outputs of three CABs, and obtains the state-of-the-art results in real-world image super-resolution. However, these real-world SR networks may still be sensitive to adversarial noises, even though they are trained with complicated blur kernels. In this paper we focus on implementing a robust neural network for real-world SISR.

Previous studies have shown that DNNs are vulnerable to adversarial attacks (Szegedy et al. 2014; Moosavi-Dezfooli et al. 2017; Rahmati et al. 2020) which cheat a neural model by applying inconspicuous changes to an input image. Adversarial attacks can be categorized into two groups: black-box and white-box attacks, according to the knowledge acquired by attackers. Black-box attackers (Ilyas et al. 2018; Papernot et al. 2017) are allowed to access only limited knowledge of the data and the targeted network, such as the output of querying the targeted model. In this paper we only consider white-box attacks (Szegedy et al. 2014; He et al. 2019a) in which all parameters of the target model are exposed to the attacker. As for white-box attacks, Szegedy et al. (Szegedy et al. 2014) for the first time propose an adversarial attack for deep learning models, by maximizing the classification loss and minimizing the magnitude of the adversarial perturbation with a box-constrained L-BFGS. Goodfellow et al. (Goodfellow et al. 2015) introduce an attack, fast gradient sign method (FGSM), which computes backward propagated gradients to maximize the classification loss and takes the sign of gradients to update an adversarial sample. Kurakin et al.(Kurakin et al. 2017) further extend FGSM to an iterative variant, I-FGSM. To attack an image SR network, Choi et al. (Choi et al. 2019) respectively implement basic, universal and partial attacks based on I-FGSM. The basic and the universal attacks can affect existing CNN-based image super-resolution models considerably. Following Choi et al. (Choi et al. 2019), we adopt the basic attack in this paper since it presents higher success rate of attack than the universal one.

Many defense methods and robust models (Prakash et al. 2018; Choi et al. 2020; Li et al. 2020; Zhou et al. 2021) which have been developed to resist adversarial attacks, attempt to remove, destroy or adapt to adversarial noises. Adversarial training (Tramer et al. 2018; Shafahi et al. 2019; Zheng et al. 2020) is a large group of defense methods that trains the target network with adversarial samples and are effective against white-box attacks. Some early defenses that apply image transformations (Guo et al. 2018; Xie et al. 2018; Li et al. 2020) to disrupt or eliminate adversarial noises, are limited to gray-box setting (Athalye et al. 2018) where the defense is unknown for attackers. Other improved methods that propose new neural modules to denoise or smooth image features (Xie et al. 2019; He et al. 2019b; Li et al. 2020), do show their robustness. Zhang et al. (Zhang et al. 2019b) propose to suppress high frequency in the discrete Fourier transform (DFT) of an input image, which is mostly related to our proposed method. Zhang et al. (Zhang et al. 2019b) utilize a fixed radius to erase noises, which discarded all the high-frequency elements including natural textures in an image. Differently, we attempt to maintain some parts of high-frequency components by randomly masking the perturbations, so that our proposed method has the chance to reconstruct the original high-frequency textures.

We develop a novel random frequency mask module to mitigate adversarial attacks, since most of adversarial perturbations are encoded as high-frequency components in a frequency domain. Figure 2 illustrates the architecture of the proposed mask module that consists of three steps: transforming an input to the DCT frequency domain, masking the DCT representations with a sampled map, and converting the masked DCT back to the image or feature space.

Our proposed mask module could take an image or a feature map as input. Without loss of generality, the module input is denoted as a $H\times W\times C$ tensor. Let $X\in\mathcal{R}^{H\times W}$ denote one of the $C$ channel slices in the $H\times W\times C$ input tensor. We adopt discrete cosine transform to compute the frequency representations of $X$. Let $\hat{X}\in\mathcal{R}^{H\times W}$ stand for the DCT result of $X$. $\hat{X}$ can be calculated as:

where $c(u)$ is a compensation coefficient. $c(u)$ is set as $\sqrt{1/H}$ for $u=0$ and $\sqrt{2/H}$ for $u\neq 0$, and the definition of $c(v)$ is the same as $c(u)$.

To understand DCT representations, we normalize and take average of different channels in the DCT results that are computed from a clean or attacked input. The averaged DCT maps are visualized in Figure 1 where the first row is for clean inputs and the second is for the attacked ones. In a DCT map, low-frequency coefficients are located nearby the upper-left corner. Figure 1 shows that the attacked DCT maps have larger high-frequency components, and smaller middle-frequency coefficients than the clean ones. We argue that adversarial attacks harm the original middle-frequency details to some degree, and add more high-frequency noises to the attacked image. Thus we propose to reduce the adversarial perturbations by multiplying the DCT ${\hat{X}}$ with a binary mask $\mathcal{M}\in\mathcal{R}^{H\times W}$: $\hat{X}_{m}={\hat{X}}\odot{\mathcal{M}}$, where $\odot$ is element-wise multiplication. Then the masked DCT $\hat{X}_{m}$ is translated to the same image or feature space as the module input $X$ via inverse discrete cosine transform. The whole process of our proposed mask module is formulated as: $X_{m}=\mathcal{F}^{-1}(\mathcal{M}\odot\mathcal{F}(X))$, where $\mathcal{F}(\cdot)$ stands for DCT, and $\mathcal{F}^{-1}$ denotes the Inverse-DCT. The module output $X_{m}$ has the same shape as the input $X$.

We discuss how to determine the binary mask $\mathcal{M}$. Considering that the distance from the lowest-frequency component corresponds to the frequency degree of a component. For each coefficient of position $(u,v)$, We set its corresponding weight in the binary mask according to its normalized distance from $(0,0)$:

where $r_{max}$ equals to $\sqrt{(H-1)^{2}+(W-1)^{2}}$ and denotes the maximum radius for a DCT map of size $H\times W$. For a DCT component whose $r_{(u,v)}$ is smaller than a threshold $r_{t}$, the component probably contains the information of mean colors, smooth regions or sharp edges, which are image elements of low-to-middle frequency. To preserve the original image contents, we keep such a DCT coefficient unchanged by setting $\mathcal{M}(u,v)$ as 1. Since the boundary between middle-frequency details and high-frequency noises is uncertain, we uniformly sample $r_{t}$ from $[r_{l},r_{u}]$. $r_{l}$ and $r_{u}$ are manually set lower and upper bounds respectively by visualizing the difference of DCT maps between clean and attacked samples. If the $r_{(u,v)}$ value of a DCT coefficient is larger than the threshold, the coefficient might still encode normal contents such as highly repetitive textures. Since a higher-frequency coefficient more possibly contains adversarial noises, we adopt a Bernoulli distribution with the probability $p=r_{(u,v)}$ to decide if masking the current component. The distribution returns 1 with probability p and 0 with probability 1 - p. The binary mask $\mathcal{M}(u,v)=Bernoulli(p=r_{(u,v)})$ is formally defined as:

The proposed mask module has two strengths. First, it supports backward propagation and could be placed at arbitrary positions of a SR network. Second, the mask module has no learnable parameters and is a lightweight module.

Although we have developed a stochastic strategy to eliminate adversarial noises and preserve original contents at the same time, the proposed mask module still inevitably discards some fine details and degrades the performance on clean images. Thus we devise a two-class classifier shown in Figure 3 to predict if the input image is adversarial. The frequency mask modules are skipped for a clean input so that the high-frequency elements of clean images are not affected. Considering the pattern difference between clean DCT maps and the attacked ones, we take DCT results as the classifier input.

For an input image $X\in\mathcal{R}^{H\times W\times C}$ that could be an adversarial sample or a clean one, we first compute its DCT of the same shape of $H\times W\times C$. Then we take average of different channel slices to obtain a single-channel DCT map of size $H\times W$, which corresponds to the visualization in Figure 1. The DCT map is further processed by a $\gamma\times\gamma$ 2D average pooling layer with stride 3 and padding 0. The shape of the pooled map becomes $H/\gamma\times W/\gamma$. Afterwards, we flatten the pooled map as a vector of size $1\times HW/\gamma^{2}$ and feed the vector into 3 consecutive fully-connected (FC) layers. The parameters of these FC layers are denoted as ${\theta}=\left\{\left(\mathcal{W}_{i},b_{i}\right),i=1,2,3\right\}$. The overall classifier could be denoted as a mapping function: $\mathcal{G}:X\to\mathcal{G}(X)$ and computed as:

where $\phi{(\cdot)}$ stands for the combination of the channel averaging, the 2D spatial average pooling and the flattening operation. $\sigma$ is an activation function LeakyReLU and $\hat{\sigma}$ is the Softmax function. $\mathcal{G}(X)\in\{False,True\}$, $True$ indicates that the input image is an adversarial sample while $False$ means a clean input. The prediction of classifier determines whether we conduct the subsequent mask operations. If $False$, we skip all the random mask modules. It is feasible because the input and the output of the mask module have the same shape, and are in the same image or feature space. Otherwise, we go through each mask module to mitigate the adversarial perturbations. Note that the input shape in the inference stage might be inconsistent with that in the training stage. To align the inconsistent shapes, we simply insert a spatial adaptive pooling layer after the channel averaging step in the testing stage.

In this section we first introduce a super-resolution baseline network, and then describe how to incorporate our proposed mask module and classifier into the baseline. Inspired by Newell et al. (Newell et al. 2016), we implement an image SR backbone by stacking six hourglass (HG) modules in a sequential manner. For simplicity, Figure 4 shows a network architecture with only three HG modules. The hourglass modules have the same architecture but do not share their weights. Two Residual Inception Blocks (RIB) are located in between each two adjacent HG modules. Following Wei et al. (Wei et al. 2020), we deploy three component-attentive blocks (CAB) at the end of the 2nd, 4th and 6th hourglass modules respectively. An input image is decomposed of three components: smooth regions, edges and corners. Each CAB only focuses on one of the three components by weighting its SR output with a 2D attentive map. The intermediate output of the three CABs are aggregated to yield the final super-resolved image.

In our proposed method, each hourglass module contains two random frequency mask modules: one at the beginning of the HG module, and the other before the residual blocks. Our proposed adversarial classifier is placed at the very beginning to predict if the input is adversarial. The classification result is sent to all mask modules and decides if passing through the random mask modules.

We employ a stagewise adversarial training strategy to tune our proposed SR network. First, we train the SR backbone network without the mask modules or the classifier. Second, we train the proposed classifier using clean images and adversarial samples that are synthesized with the basic attack (Choi et al. 2019) and the SR backbone trained at the first step. Third, we equip the SR backbone with the trained classifier and all the frequency mask modules to construct our complete SR network. The overall SR network is trained after freezing the classifier and randomly initializing the backbone. Following Wei et al. (Wei et al. 2020), We adopt two loss functions, an intermediate one and a gradient-weighted one. To achieve adversarial training, we maximize these loss functions to yield adversarial samples for training the overall proposed SR network. Note that the adversarial sample classifier is not attacked in the adversarial training.

We adopt the real-world single image SR dataset RealSR (Cai et al. 2019) for evaluation. The dataset contains 595 pairs of LR and HR images. These image pairs are captured by adjusting the focal length of digital cameras, and have been well aligned. Following Cai et al. (Cai et al. 2019), 495 pairs are selected for training and 100 pairs are used for testing. Their image sizes are in the range of [700, 3100] and [600, 3500]. $48\times 48$ image patches are cropped for training SR models. We use the Adam optimizer, exponential decay rate of 0.9, batch size of 16, the initial learning rate of 2e-4. The learning rate is reduced by half after every 1e5 iterations. The maximum number of iterations is 4e5. For the proposed mask module, we set the lower and upper bounds $[r_{l},r_{u}]$ as $[0.43,0.5]$. For the proposed adversarial classifier, $\gamma$ is set as 3. For adversarial training, we use the basic attack (Choi et al. 2019) with 2 iterations and the attack intensity as 6/255. For evaluation, we use the basic attack with 10 iterations, and the intensity $\alpha\in\{1,2,4,6,8\}/255$. Peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are used as the evaluation metrics. Different from Choi et al. (Choi et al. 2019), PSNR and SSIM are calculated between the HR ground truths and the super-resolved results of adversarial images.

We compare our proposed method with existing SR models based on deep CNNs, including SRResNet (Ledig et al. 2017), EDSR (Lim et al. 2017), RRDB (Wang et al. 2018), RCAN (Zhang et al. 2018), CARN (Ahn et al. 2018), LP-KPN (Cai et al. 2019), CDC (Wei et al. 2020). CARN is a model designed for lightweight image SR, LP-KPN and CDC are developed for real-world image SR. We train these methods on the RealSR dataset and evaluate them from two perspectives below.

Qualitative Evaluation Figure 5 visually compares the super-solved results of our proposed model with those of existing models. The upper row contains the cropped patches of input images, including a clean LR image on the left and the adversarial LR sample on the right. These adversarial samples are synthesized with the intensity $\alpha=8/255$ and the iteration number $T=10$. The middle row is the cropped super-resolved results of clean inputs, while the lower one is the outputs with adversarial samples. Figure 5 shows that our method is able to reconstruct sharp textures and maintain high-level clarity with clean samples. Under the adversarial attack, existing SR models usually produce unreasonable and structured artifacts, which severely affect the quality of super-resolved images. However, such highly repetitive artifacts are not seen in the results produced by our proposed method. This confirms to a certain extent that the proposed mask modules can effectively reduce high-frequency noises.

Quantitative Evaluation Table 1 shows the quantitative results in terms of PSNR & SSIM. Under the attack intensity of $8/255$, the proposed network achieves the best value of PSNR and SSIM: 26.88 dB and 0.753 respectively. Compared with our baseline CDC (Wei et al. 2020), our proposed network obtains an improvement of 7.9 dB in PSNR and 0.37 in SSIM. For the clean images, the proposed model shows a small performance gap of 0.03 dB in PSNR and 0.002 in SSIM by comparing with the state-of-the-art CDC. Besides, it can be seen in Figure 6 that, as the intensity of adversarial perturbations increases, our proposed model presents smaller decreases than other state-of-the-art methods. Therefore, the above results suggest that our proposed network not only achieves more robust defense than existing real-world SR methods when resisting the white-box attack, but also maintains a competitive performance with clean images.

We further verify the effectiveness of our proposed modules by comparing with other existing defenses, including image compression (JPEG) (Dziugaite et al. 2016), image Random Resizing (RR) (Xie et al. 2018), a High-Frequency components Suppressing method (HFS) (Zhang et al. 2019b) and a Feature Denoising method (FD) (Xie et al. 2019; Li et al. 2020). For the fairness, we adopt CDC (Wei et al. 2020) as the baseline that is combined with each of the above defenses respectively in this subsection. We place the JPEG Compression module, the Random Resizing and the High-Frequency Suppressing method in the front of the SR network, following their original setting. We insert four feature denoising blocks at the head of network, the tail of the 2nd, 4th and 6th HG module. Adversarial training strategy is utilized for High-Frequency Suppressing and Feature Denoising, which corresponds to their most effective settings (Zhang et al. 2019b; Xie et al. 2019). Figure 7 shows the robustness of all the above defense methods with CDC. It could be observed that our proposed defense obtains the most significant performance at almost every adversarial intensity. With the highest intensity $\alpha=8/255$, the proposed defense exceeds the second best method by 1.05 dB. The baseline and Feature Denoising could obtains approximate results with our proposed method on clean images. But these two models are far worse than our defense on adversarial samples.

We study the effectiveness of the proposed random frequency mask module, the proposed adversarial sample classifier and the adversarial training in our model. Our proposed network is compared with the baseline (described in Section 3.3), the baseline with the Random frequency Mask (+RM), the baseline with the Fixed frequency Mask (+FM), the baseline with Adversarial Training (+AT), the baseline with Fixed frequency Mask and Adversarial Training (+FM+AT), and the one with Random frequency Mask and Adversarial Training (+RM+AT). The line charts of the above comparison with different attack strengths are plotted in Figure 8. The +RM setting outperforms the +FM on all values of the attack intensity, which means that the proposed random mask module is more robust than the one with a fixed pre-defined mask. Note that the +RM+AT model is equivalent to removing the proposed classifier from our finally proposed method (denoted as ‘Ours’ in Figure 8). By comparing the line charts of the +RM+AT model with our method, we find that our proposed model is superior to the +RM+AT model without classifier on clean images or low-intensity adversarial samples. It indicates that the proposed classifier could detect adversarial samples and avoid unnecessary masking to preserve the original image details.

Table 2 shows the numerical results of the ablation study. As the table displays, the +RM model surpasses the +FM model by 2.17 dB in PSNR and 0.104 in SSIM against the attack of intensity 8/255. It suggests that the proposed stochastic strategy in our random mask module is effective to improve the super-resolved results, in comparison to the fixed mask module. We verify that adversarial training could enhance the PSNR of the baseline by 3.15 dB with the adversarial samples of intensity 8/255. Besides, our proposed mask module could further boost the baseline with adversarial training by 4.6 dB in PSNR, by comparing the +RM+AT model with the +AT model. Figure 9 is the qualitative results of the ablation study, which shows that our proposed method successfully reduces the highly repetitive artifacts in the SR result of the baseline.

We investigate if our proposed defenses including the random frequency mask and the adversarial classifier are compatible with other super-resolution models. Three state-of-the-art SR networks (including LP-KPN, SRResNet and EDSR) are selected to combine with our proposed modules. For these SR networks, we incorporate a random frequency mask module at their head, and insert a mask module for every four residual blocks. The proposed classifier predicts if the input is adversarial, and determines whether to apply the mask modules. In Figure 10, the results of three SR networks are shown as solid lines, while the results of adding our proposed modules are dotted lines. As the figure shows, our proposed defenses significantly enhance the performance against adversarial attacks for the three SR networks. Even with the increase of the attack intensity, the networks with our method still present a stable performance. It indicates that our proposed modules form a general defense framework that has the potential to work with arbitrary SR neural networks.