Code base for 2023 Summer Research Internship Programme both in Department of Computer Science at The University of Hong Kong assigned to the AI, Robotics and Visual Computing and School of Computer Science and Engineering at Beihang University assigned to the IRIP Lab.
We found the sample distribution proposed in PTQ4DM does not match the experiment results. The code in PTQ4DM shows that they only quantize the 2D convolutional, linear layer and the activation after these layers. However, U-Net used in improved-diffusion has different structure from traditional ResNet, which indicates function from QDrop can not be applied directly. Therefore, we quantized the diffusion model again, instead of using the partly quantized model. Then, we compared the performance of Normally Distributed Time-step Calibration (NDTC, proposed in PTQ4DM) and Uniformly Distributed Time-step Calibration on both ImageNet64 and CIFAR10. We found normally calibration even worse than uniform calibration, which is same to the result in Q-Diffusion.
We observed that IS and sFID in 8-bit PTQ4DM can reach or even exceed those in the full-precision model. However, there is always a significant loss on FID after quantization. Therefore, we propose some methods (D2IN) to improve the performance on FID and achieve a lower FID than that of DDIM.
Method | CIFAR10 | IS↑ | FID↓ | sFID↓ | ImageNet64 | IS↑ | FID↓ | sFID↓ |
---|---|---|---|---|---|---|---|---|
FP32 | DDIM | 9.25 | 10.60 | 7.41 | DDIM | 15.20 | 19.59 | 9.45 |
PTQ4DM | Normal | 9.38 | 12.85 | 7.53 | Normal | 15.59 | 22.02 | 6.62 |
DI2N (Ours) | Uniform | 9.25 | 10.71 | 7.38 | Uniform | 15.30 | 19.27 | 6.63 |
IS loss is caused by the extent of quantization. However, it still outperforms that of DDIM.
Xiuyu-Li/q-diffusion: [ICCV 2023] Q-Diffusion: Quantizing Diffusion Models. (github.com)
CompVis/latent-diffusion: High-Resolution Image Synthesis with Latent Diffusion Models (github.com)
What are Diffusion Models? | Lil'Log (lilianweng.github.io)
Generative Modeling by Estimating Gradients of the Data Distribution | Yang Song (yang-song.net)
openai/guided-diffusion (github.com)
Generative Modeling by Estimating Gradients of the Data Distribution | Yang Song (yang-song.net)
CompVis/stable-diffusion: A latent text-to-image diffusion model (github.com)
yhhhli/BRECQ: Pytorch implementation of BRECQ, ICLR 2021 (github.com)
[2305.10657] PTQD: Accurate Post-Training Quantization for Diffusion Models (arxiv.org)
[1911.07190] Loss Aware Post-training Quantization (arxiv.org)
[2006.11239] Denoising Diffusion Probabilistic Models (arxiv.org)
[2102.09672] Improved Denoising Diffusion Probabilistic Models (arxiv.org)
[2010.02502] Denoising Diffusion Implicit Models (arxiv.org)
[2105.05233] Diffusion Models Beat GANs on Image Synthesis (arxiv.org)
[1505.04597] U-Net: Convolutional Networks for Biomedical Image Segmentation (arxiv.org)
[2302.04304] Q-Diffusion: Quantizing Diffusion Models (arxiv.org)