Vqvae results - images from stage 1 Intermediate results - images from stage 2 Left column - no blending is used (output = PSNR + VQ) Right column - blending is used on inference ( output = PSNR + (1-SEG)*VQ) 2 bag things: 1. segmentation for low resolution images is still bad 2. psnr + vq is better for both graphical content and photo naturalistic content, so no need to have blending

Different PDE architectures for stage 2. The Best one is Density estimation Dilate (DED)

PCA visualization of encoder output and codebook words. Blue dots - encoder output Orange dots - words Left - Standard upscale Right - Upscale with word regularization on sphere

1 stage - vq_loss 2 stage - focal loss There are 3 models: standard one, local attention in vq, local attention in pde.

from left to right: 1. gan loss + rec loss + lpips loss 2. rec loss + lpips loss 3. rec loss

vqvae-results - Stage 1 images intermediate_results - Stage 2 images

left - with pixel unshuffle right - without pixel unshuffle

1st image - ground truth 2nd image - image with noise 3rd image - put noised image into vq-vae 4th image - predict words for noised image via PDE, process with vq decoder

1st column - VQGan training with our framework 2nd and 3rd column - our vqvae training with our framework

To choose image, click шестеренка, in the left right corner of image panel, choose image idx from left to right: frozed_disc_2 - loss at stage 2 = focal_loss - 0.5*logsoftmax(D(reconstructed_images)), where D - discriminator, frozed_disc - loss at stage 2 = focal_loss - D(reconstructed_images), where D - discriminator, vq_lpips - VQ-VAE training with LPIPS loss at stage 1 and stage 2 curr_model - current model we use

from left to right: a) 1 stage - reconstruction loss + lpips loss, 2 stage - gumble with lpips and rec loss b) 1 stage - reconstruction loss + lpips loss, 2 stage - focal loss + gumble with lpips and rec loss c) 1 stage - reconstruction loss + lpips loss + gan loss, 2 stage - focal loss d) 1 stage - reconstruction loss + lpips loss, 2 stage - focal loss e) 1 stage - reconstruction loss, 2 stage - focal loss

From left to right: GAN Upscale (without VQ) Density trained only with l1 loss on high frequencies of output image Density trained with l1 loss + GAN loss on high frequencies of output image Density trained with GAN loss on high frequencies of output image

pix2pix fashion. 6->64->128->256->512 intermediate results - images during training of model with gan loss + l1 loss examples - images of model trained without gan loss (for comparison)

Trained with Hinge Loss Discriminator number of channels 3->64->128->256->512 (spatial dimensions at the end = 16x16) No multiiterations for Discriminator Same learning rate

from left to right: without lr skip connection and conv1x1, 1 stack without lr skip connection and conv1x1, 4 stack without lr skip connection and conv3x3, 4 stack with lr skip connection and conv3x3, 4 stack with lr skip connection and conv1x1, 4 stack

2 runs with gumble softmax. The only difference is having PDE. Output images wiht UNET PDE are worse.

2 runs with gumble softmax. The only difference is multiplying loss by (1+G), where G is spatial gradient. Output images look the same, though loss and metrics plots are very different.