Skip to content

Commit

Permalink
init
Browse files Browse the repository at this point in the history
  • Loading branch information
Martin committed Oct 9, 2022
1 parent 89ddef0 commit c6eacd5
Show file tree
Hide file tree
Showing 48 changed files with 5,184 additions and 1 deletion.
157 changes: 156 additions & 1 deletion README.md
View file
Original file line number Diff line number Diff line change
@@ -1 +1,156 @@
# FMNet
<p align="center">
<img src="./figures/acmmm.jpg" width="150">
</p>

This is the official PyTorch implementation of FMNet in the ACM MM 2022 paper "[FMNet: Frequency-Aware Modulation Network for SDR-to-HDR Translation](https://github.com/MCG-NKU/FMNet.git)".

# Updates
- 2022.10.09 Init the repositories.

# Contents
0. [Introduction](#Introduction)
0. [Requirement](#Requirement)
0. [Dataset](#Dataset)
0. [Demo](#Demo)
0. [Train](#Train)
0. [Test](#Test)
0. [Results](#Results)
0. [Citation](#Citation)
0. [Acknowledgment](#Acknowledgment)
0. [Contact](#Contact)
0. [License](#License)

# Introduction

High-dynamic-range (HDR) media resources that preserve high contrast and more details in shadow and highlight areas in television are becoming increasingly popular for modern display technology compared to the widely available standard-dynamic-range
(SDR) media resources. However, due to the exorbitant price of HDR cameras, researchers have attempted to develop the SDR-to-HDR techniques to convert the abundant SDR media resources to the HDR versions for cost-saving. Recent SDR-to-HDR meth-
ods mostly apply the image-adaptive modulation scheme to dynamically modulate the local contrast. However, these methods often fail to properly capture the low-frequency cues, resulting in artifacts in the low-frequency regions and low visual quality.
Motivated by the Discrete Cosine Transform (DCT), in this paper, we propose a Frequency-aware Modulation Network (FMNet) to enhance the contrast in a frequency-adaptive way for SDR-to-HDR translation. Specifically, we design a frequency-aware modulation block that can dynamically modulate the features according to its frequency-domain responses. This allows us to reduce the structural distortions and artifacts in the translated low-frequency regions and reconstruct high-quality HDR content in the translated results. Experimental results on the HDRTV1K dataset show that our FMNet outperforms previous methods and the perceptual quality of the generated HDR images can be largely improved.

<p align="center">
<img src="./figures/method_pipeline.png" width="1000">
</p>

<p align="center">
<img src="./figures/method_block.png" width="1000">
</p>

# Requirement

1. Python 3.6.10

2. NVIDIA GPU

3. NVIDIA CUDA 10.0.130

4. Python Package:
```
torchvision==0.9.1
opencv_contrib_python==4.5.1.48
numpy==1.18.1
torch==1.8.1
scipy==1.5.4
matplotlib==3.3.3
PyYAML==5.3.1
tensorboardX==2.2
```

# Dataset
We use the [HDRTV1K](https://github.com/chxy95/HDRTVNet#configuration) dataset for training and evaluation.
You can obtain this dataset from [HDRTVNet](https://github.com/chxy95/HDRTVNet#configuration).
Once you have downloaded the HDRTV1K dataset, you can reorganize the dataset to get a folder with the following structure:
```
hdrtv1k
├── training_set
| ├── train_hdr
| | └── *.png
| └── train_sdr
| └── *.png
└── test_set
├── test_hdr
| └── *.png
└── test_sdr
└── *.png
```
You should move this dataset into ```$ROOT/data/``` folder and then prepare the training set through:
```
cd $ROOT/codes
python preprocessing.py --input_folder ../data/hdrtv1k/training_set/train_sdr --save_folder ../data/hdrtv1k/training_set/train_sdr_sub_c096_s240 --n_thread 20 --crop_sz 96 --step 240 --thres_sz 48 --compression_level 0
python preprocessing.py --input_folder ../data/hdrtv1k/training_set/train_hdr --save_folder ../data/hdrtv1k/training_set/train_hdr_sub_c096_s240 --n_thread 20 --crop_sz 96 --step 240 --thres_sz 48 --compression_level 0
```

# Demo
### 1. Get pretrained model
Our pretrained model ```fmnet_final.pth``` can be downloaded via [Google Drive](https://drive.google.com/file/d/1VJB8hrHim7BP0V3Q9lw2EjK7sVTEi__9/view?usp=sharing) or [Baidu Netdisk](https://pan.baidu.com/s/1QhCtAbJoG2GwcZ9DiCbQJw) (access code: 5kgw).
Once you have downloaded the pretrained model, please put it into the ```$ROOT/checkpoints``` folder.

### 2. Inference
You can generate the HDR results through:
```
cd $ROOT/codes
python inference.py --test_hdr ../data/hdrtv1k/test_set/test_hdr --test_sdr ../data/hdrtv1k/test_set/test_sdr --config ./options/train/fmnet_final.yml --parameter ../checkpoints/fmnet_final.pth
```
The HDR results will be placed in the ```$ROOT/experiments/fmnet_final/val_images``` folder.

# Train
### 1. Set up configuration
Our training settings can be found at ```$ROOT/codes/options/train/fmnet_final.yml```.
Please carefully check the hyperparameters inside this config file.

### 2. Train the FMNet
You can train the FMNet following the configuration in ```$ROOT/codes/options/train/fmnet_final.yml``` through:
```
cd $ROOT/codes
python train.py -opt options/train/fmnet_final.yml
```
You can find the saved checkpoints in the folder ```$ROOT/experiments/fmnet_final/models```.

# Test
You should first download the ```hdr_toolbox``` via [HDR_Toolbox](https://github.com/banterle/HDR_Toolbox) and ```hdrvdp-3.0.6``` via [HDRVDP3](https://sourceforge.net/projects/hdrvdp/files/hdrvdp/3.0.6/), and place these two folder (```hdr_toolbox``` and ```hdrvdp-3.0.6```) into the ```$ROOT\codes``` folder.
Then you can evaluate the performance of FMNet with specific checkpoint ```*.pth``` using the [HDRTV1K](https://github.com/chxy95/HDRTVNet#configuration) dataset:
```
cd $ROOT/codes
python inference.py --test_hdr ../data/hdrtv1k/test_set/test_hdr --test_sdr ../data/hdrtv1k/test_set/test_sdr --config ./options/train/fmnet_final.yml --parameter ../experiments/fmnet_final/models/*.pth
```
Once the generated HDR results are placed in ```$ROOT/experiments/fmnet_final/val_images``` folder, you can evaluate the quantitative performance through :
```
cd $ROOT/codes
python evaluation.py --pd_folder ../experiments/fmnet_final/val_images --gt_folder ../data/hdrtv1k/test_set/test_hdr --norm 65535
```
Besides, you can also use Matlab to obtain the HDRVDP3 and SR-SIM scores through ```evaluation.m``` (remember to configure the input arguments in the Matlab file).

# Results
## Quantitative Results
Quantitative comparison of previous methods and ours on the HDRTV1K dataset:
<p align="center">
<img src="./figures/table_sota.png" width="1200">
</p>

## Visual Results
Qualitative comparison of visual quality by different methods on the HDRTV1K dataset:
<p align="center">
<img src="./figures/experiment_sota.png" width="1200">
</p>

# Citation
If you find the code helpful in your research or work, please cite our paper:
```BibTeX
@InProceedings{xu2022fmnet,
author = {Gang Xu and Qibin Hou and Zhang Le and Ming-Ming Cheng},
title = {FMNet: Frequency-Aware Modulation Network for SDR-to-HDR Translation},
booktitle = {Proceedings of the 30th ACM International Conference on Multimedia},
month = {October},
year = {2022}
}
```

# Acknowledgment
Our code is built on [HDRTVNet](https://github.com/chxy95/HDRTVNet).
We thank the authors for sharing their codes.

# Contact
If you have any questions, feel free to E-mail me with [[email protected]]([email protected]).

# License
The code is released under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Public License for NonCommercial use only.
Any commercial use should get formal permission first.
3 changes: 3 additions & 0 deletions checkpoints/.gitignore
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Ignore everything in this foldr except the .gitignore

!.gitignore
13 changes: 13 additions & 0 deletions codes/PQEOTF.m
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
function linRGB = PQEOTF(E)

m1 = 0.1593017578125;
m2 = 78.84375;
c1 = 0.8359375;
c2 = 18.8515625;
c3 = 18.6875;

linRGB = 10000 * ((max((E.^(1/m2))-c1, 0))./(c2-c3*(E.^(1/m2)))).^(1/m1);
linRGB = ClampImg(linRGB, 0, 10000);

end

15 changes: 15 additions & 0 deletions codes/calculate_hdrvdp3.m
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
function output = calculate_hdrvdp3(HQ_img_path, GT_img_path)

HQ_img = imread(HQ_img_path);
GT_img = imread(GT_img_path);
HQ_img = im2double(HQ_img);
GT_img = im2double(GT_img);
HQ_img = PQEOTF(HQ_img);
GT_img = PQEOTF(GT_img);

ppd = 50;
res = hdrvdp3('side-by-side', HQ_img, GT_img, 'rgb-bt.2020', ppd, {'rgb_display', 'led-lcd-wcg', 'disable_lowvals_warning', true});

output = res.Q;

end
123 changes: 123 additions & 0 deletions codes/calculate_srsim.m
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
function sim = calculate_srsim(HQ_img_path, GT_img_path)
% ========================================================================
% SR_SIM Index with automatic downsampling, Version 1.0
% Copyright(c) 2011 Lin ZHANG
% All Rights Reserved.
%
% ----------------------------------------------------------------------
% Permission to use, copy, or modify this software and its documentation
% for educational and research purposes only and without fee is hereQ
% granted, provided that this copyright notice and the original authors'
% names appear on all copies and supporting documentation. This program
% shall not be used, rewritten, or adapted as the basis of a commercial
% software or hardware product without first obtaining permission of the
% authors. The authors make no representations about the suitability of
% this software for any purpose. It is provided "as is" without express
% or implied warranty.
%----------------------------------------------------------------------
%
% This is an implementation of the algorithm for calculating the
% Spectral Residual based Similarity (SR-SIM) index between two images. For
% more details, please refer to our paper:
% Lin Zhang and Hongyu Li, "SR-SIM: A fast and high performance IQA index based on spectral residual", in: Proc. ICIP 2012.
%
%----------------------------------------------------------------------
%
%Input : (1) image1: the first image being compared
% (2) image2: the second image being compared
%
%Output: sim: the similarity score between two images, a real number
%
%-----------------------------------------------------------------------


% image1 = imread('/home/ubuntu/Project/eccv2022/evaluation/result/HDRTVNET_AGCM_LE_HG_HDRTV1K/001.png');
% image2 = imread('/home/ubuntu/Project/eccv2022/evaluation/result/GT_HDRTV1K/001.png');

image1 = imread(HQ_img_path);
image2 = imread(GT_img_path);

image1 = im2double(image1) * 255;
image2 = im2double(image2) * 255;

[rows, cols, junk] = size(image1);
if junk == 3
Y1 = 0.299 * double(image1(:,:,1)) + 0.587 * double(image1(:,:,2)) + 0.114 * double(image1(:,:,3));
Y2 = 0.299 * double(image2(:,:,1)) + 0.587 * double(image2(:,:,2)) + 0.114 * double(image2(:,:,3));
else
Y1 = double(image1);
Y2 = double(image2);
end

%%%%%%%%%%%%%%%%%%%%%%%%%
% Download the image
%%%%%%%%%%%%%%%%%%%%%%%%%
minDimension = min(rows,cols);
F = max(1,round(minDimension / 256));
aveKernel = fspecial('average',F);

aveY1 = conv2(Y1, aveKernel,'same');
aveY2 = conv2(Y2, aveKernel,'same');
Y1 = aveY1(1:F:rows,1:F:cols);
Y2 = aveY2(1:F:rows,1:F:cols);

%%%%%%%%%%%%%%%%%%%%%%%%%
% Calculate the visual saliency maps
%%%%%%%%%%%%%%%%%%%%%%%%%
saliencyMap1 = spectralResidueSaliency(Y1);
saliencyMap2 = spectralResidueSaliency(Y2);
%%%%%%%%%%%%%%%%%%%%%%%%%
% Calculate the gradient map
%%%%%%%%%%%%%%%%%%%%%%%%%
dx = [3 0 -3; 10 0 -10; 3 0 -3]/16;
dy = [3 10 3; 0 0 0; -3 -10 -3]/16;
IxY1 = conv2(Y1, dx, 'same');
IyY1 = conv2(Y1, dy, 'same');
gradientMap1 = sqrt(IxY1.^2 + IyY1.^2);

IxY2 = conv2(Y2, dx, 'same');
IyY2 = conv2(Y2, dy, 'same');
gradientMap2 = sqrt(IxY2.^2 + IyY2.^2);

%%%%%%%%%%%%%%%%%%%%%%%%%
% Calculate the SR-SIM
%%%%%%%%%%%%%%%%%%%%%%%%%
C1 = 0.40; %fixed
C2 = 225;
alpha = 0.50;%fixed

GBVSSimMatrix = (2 * saliencyMap1 .* saliencyMap2 + C1) ./ (saliencyMap1.^2 + saliencyMap2.^2 + C1);
gradientSimMatrix = (2*gradientMap1.*gradientMap2 + C2) ./(gradientMap1.^2 + gradientMap2.^2 + C2);

weight = max(saliencyMap1, saliencyMap2);
SimMatrix = GBVSSimMatrix .* (gradientSimMatrix .^ alpha) .* weight;
sim = sum(sum(SimMatrix)) / sum(weight(:));

return;

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
function saliencyMap = spectralResidueSaliency(image)
%this function is used to calculate the visual saliency map for the given
%image using the spectral residue method proposed by Xiaodi Hou and Liqing
%Zhang. For more details about this method, you can refer to the paper:
%Saliency detection: a spectral residual approach.

%there are some parameters needed to be adjusted
scale = 0.25; %fixed
aveKernelSize = 3; %fixed
gauSigma = 3.8; %fixed
gauSize = 10; %fixed

inImg = imresize(image, scale);

%%%% Spectral Residual
myFFT = fft2(inImg);
myLogAmplitude = log(abs(myFFT));
myPhase = angle(myFFT);

mySpectralResidual = myLogAmplitude - imfilter(myLogAmplitude, fspecial('average', aveKernelSize), 'replicate');
saliencyMap = abs(ifft2(exp(mySpectralResidual + 1i*myPhase))).^2;

%%%% After Effect
saliencyMap = mat2gray(imfilter(saliencyMap, fspecial('gaussian', [gauSize, gauSize], gauSigma)));
saliencyMap = imresize(saliencyMap,[size(image,1) size(image,2)]);
Loading

0 comments on commit c6eacd5

Please sign in to comment.