Skip to main content
wandb
Projects
retail-tutorial
Reports
Comparative Analysis of Virtual Try-On Methods

Comparative Analysis of Virtual Try-On Methods

An exploration of different techniques transforming the online shopping experience
Kevin Stofan, Adrish Dey, Ramit Goolry
Created on July 27|Last edited on September 9

Table of Contents (Click to Expand)

IntroductionBackgroundClothing Agnostic RepresentationTPS TransformationCP-VTONGeometric Matching ModuleTry-On ModuleACGPNSemantic Generation ModuleClothes Warping ModuleContent Fusion ModuleM3D-VTONMonocular Prediction ModuleDepth Refinement ModuleTexture Fusion ModuleQualitative Comparison


Introduction

Virtual try-on is a machine learning task offering subjects the ability to 'try on' items of their choice, such as clothing. Doing so creates a virtual trial room — a useful feature, particularly for retailers.
Historically, the virtual try-on task was done using methods of computational physics. However, these methods were difficult to scale due to the high costs of installing hardware, and collecting 3D annotated data inhibited their large-scale deployment.
Several new virtual try-on methods were proposed to address those limitations. We'll be comparing three popular techniques: CP-VTON, ACGPN, and M3D-VTON.

Background

The concept of machine learning based virtual try-on can be traced back to VITON, where they introduced the implementation of two central ideas that have carried over in almost every other implementation of virtual try-on methods: Clothing Agnostic Representation and TPS Transformation.

Clothing Agnostic Representation

One of the main technical challenges of the virtual try-on task is to collect a dataset of triples - (P,c,Pt)(P, c, P_t) where PP is the image of a person, cc is the image of the desired clothing item and PtP_t is the target image, where person PP is wearing the clothing item cc.
To overcome this, the authors of VITON proposed a "Clothing Agnostic Representation" where the input image is transformed into a 22-channel representation consisting of the following components:
  • Pose Heatmap: an 18-channel feature map with each channel corresponding to one human pose keypoint, drawn as an 11x11 rectangle.
  • Body Shape: a 1-channel feature map of a blurred binary mask that roughly covers different parts of the human body, such as arms and legs, in order to retain the body shape of the person PP.
  • Reserved Regions: the representation also consists of an RGB image containing identifying information about the person which don't have to be changed, such as face and hair.
Using this representation, one can use triples as (P,c,P)(P, c, P). Since all the information regarding the clothes is processed out, the system is "agnostic" to the clothing and learns to generate clothing for a person in any pose without requiring a new target image to train over.
A Clothing Agnostic Representation for an image consists of identifying information like body shape and pose keypoints.

TPS Transformation

A Thin Plate Spline (TPS) is the 2D equivalent of a cubic spline in one dimension.
Given a set of data points, a weighted combination of the TPS centered around each data point gives the interpolation function that passes through the points exactly while minimizing the "bending energy", which is defined as the integral over R2\mathbb{R}^2:
I[f(x,y)]=∫∫R2(fxx2+2fxy2+fyy2)dx dyI[f(x, y)] = \int\int_{\mathbb{R}^2} (f_{xx}^2 + 2f_{xy}^2 + f_{yy}^2) dx \ dy

The parameters to generate this transformation for a given pose are learned and this transformation is then used to warp the clothing into the desired shape before being superimposed on the target.
Since a TPS minimizes bending energy, it simulates the effect a set of clothes would have in the real world while being completely differentiable, allowing for back propagation.
A TPS Transformation is used to warp the clothing to fit a target's shape.

CP-VTON

While VITON was able to get a realistic representation of how the clothes would fit on a user, they were unable to retain the defining characteristics of clothes, due to the Coarse-to-Fine structure of the Model.
The authors of CP-VTON aimed to improve this by introducing a new Geometric Matching Module (GMM) and Try-On Module (TOM).

Geometric Matching Module

The loss for the Geometric Matching Module is calculated as:
LGMM=∥c^−ct∥1=∥Tθ(c)−ct∥1\mathcal{L}_{GMM} = \|\hat{c} - c_t \|_1 = \| T_{\theta}(c) - c_t\|_1

where TθT_\theta is a TPS transformation with parameters θ\theta and ctc_t is the target shape.
The training metrics for this can be seen below:


Run: GMM


Try-On Module

The loss for the Try-On Module is calculated as:
LTOM=λL1∥I^−I∥1+λVGGLVGG(I^,I)+λmask∥1−M∥1\mathcal{L}_{TOM} = \lambda_{L_1} \|\hat{I} - I\|_1 + \lambda_{VGG} \mathcal{L}_{VGG}(\hat{I}, I) + \lambda_{mask}\|1-M\|_1

LVGG(Io,It)=∑i=15λi∥ϕi(Io)−ϕi(It)∥1\mathcal{L}_{VGG}(I_o, I_t) = \sum_{i=1}^{5} \lambda_i \| \phi_i(I_o) - \phi_i(I_t) \|_1

where ϕi\phi_i is a feature map of the i-th layer in the visual perception network ϕ\phi, which is VGG trained on ImageNet.
The training metrics for this can be seen below:

Run: TOM


ACGPN

The goal of ACGPN was to preserve the characteristics of images that would be left out in VITON and CP-VTON, such as hands and arms, which can occlude the clothing in certain poses.
It consists of three modules: Semantic Generation Module, Clothes Warping Module, and Content Fusion Module.

Semantic Generation Module

The Semantic Generation Module works to separate the target clothing region and preserve the person's body parts. The loss of the module is computed as:
Lm=λ1L1+λ2L2\mathcal{L}_m = \lambda_1 \mathcal{L}_1 + \lambda_2 \mathcal{L}_2

where L1=Ex,y[log⁡(D(x,y))]+Ex,z[log⁡(1−D(x,G(x,z)))]\mathcal{L}_1 = \mathbb{E}_{x, y}[\log(\mathcal{D}(x, y))] + \mathbb{E}_{x, z}[\log(1 - \mathcal{D}(x, \mathcal{G}(x, z)))] (Conditional GAN loss) and L2\mathcal{L}_2 is the pixel-wise cross entropy.

Clothes Warping Module

The Clothes Warping Module works to warp the clothing images according to the generated semantic layout that uses a second order difference constraint introduced to stabilize the TPS warp during training. The loss for this module is computed as:
Lw=L3+L4\mathcal{L}_w = \mathcal{L}_3 + \mathcal{L}_4

where L3=∑p∈Pλr∣∥pp0∥2−∥pp1∥2+∥pp2∥2−∥pp3∥2∣+λs(∣S(p,p0)−S(p,p1)∣+∣S(p,p2)−S(p,p3)∣)\mathcal{L}_3 = \sum_{p \in \textbf{P}} \lambda_r | \|pp_0 \|_2 - \|pp_1 \|_2 + \|pp_2\|_2 - \|pp_3\|_2| + \lambda_s (|S(p, p_0) - S(p, p_1)| + |S(p, p_2) - S(p, p_3)|) and L4=∥TcW−Ic∥1\mathcal{L}_4 = \|\mathcal{T}^W_c - \mathcal{I}_c \|_1.

Content Fusion Module

The Content Fusion Module works to retain the fine-scale details of the target human and generate the final try-on image.
The results for training with ACGPN are visualized below:

Run set



M3D-VTON

M3D-VTON takes a novel approach, where it attempts to generate a 3D try-on result from 2D input. The authors of the paper state that their method tries to take the benefits of both 2D and 3D virtual try-on since a 3D result is more realistic and gives a better sense of the product, but 2D data is easier to capture and process on the consumer end.
The 3D training data was synthetically produced using PIFu-HD.
M3D-VTON consists of three modules: Monocular Prediction Module, Depth Refinement Module, and Texture Fusion Module.

Monocular Prediction Module

The Monocular Prediction Module plays a preparatory role in the model. It provides constructive guidance for the other two modules by warping the clothing, predicting a person segmentation, and estimating a base 3D shape using a multi-target network.
It consists of three branches:
  • Clothing Warping Branch: this branch learns the parameters for the TPS warp on the clothing and aligns the clothes to the target person. Its loss is computed as:
Lw=∥Cw−Ic∥1\mathcal{L}_w = \|C^w - I^c\|_1

  • Conditional Segmentation Estimation Branch: this branch predicts a Person Segmentation for the rest of the model to follow. It uses a pixel-level cross-entropy Ls\mathcal{L}_s to optimize this branch.
  • Depth Estimation Branch: this branch generates a base 3D Shape of the target person. The 3D shape is represented in a "Double Depth" form, i.e., a front and back depth map corresponding to the respective sides of the 3D human representation. The loss for this module is computed as:
Lz=∥Dfi−Dfgt∥1+∥Dbi−Dbgt∥1\mathcal{L}_z = \|D^i_f - D^{gt}_f\|_1 + \|D^i_b - D^{gt}_b\|_1

The module combines all these three losses as:
LMPM=Lw+Ls+Lz\mathcal{L}_{MPM} = \mathcal{L}_w +\mathcal{L}_s + \mathcal{L}_z

The training results for this module are visualized below:

Run: devout-haze-12
1


Depth Refinement Module

The depth map from the MPM fails to capture geometric details such as clothing details, and facial characteristics. This is largely due to the fact that the L1 loss used in the MPM tends to penalize low-frequency differences in the depth maps. In order to compensate for this, the authors of M3D-VTON use a Depth Refinement Module that aims to add high frequency details to the depth map. Its loss is calculated as:
LDRM=λdepthLdepth+λgradLgrad\mathcal{L}_{DRM} = \lambda_\text{depth} \mathcal{L}_\text{depth} + \lambda_\text{grad} \mathcal{L}_\text{grad}

where
Ldepth=1n∑i=1nln⁡(ϵi+1)\mathcal{L}_\text{depth} = \frac{1}{n} \sum_{i=1}^{n} \ln(\epsilon_i + 1)

Lgrad=1n∑i=1n(ln⁡(∇x(ϵi)+1)+ln⁡(∇y(ϵi)+1))\mathcal{L}_\text{grad} = \frac{1}{n} \sum_{i=1}^{n}(\ln(\nabla_x(\epsilon_i) + 1) + \ln(\nabla_y(\epsilon_i) + 1))

where ϵi\epsilon_i denotes the L1 loss at the i-th depth point, and ∇\nabla denotes the Sobel Edge detector.
The training results for this module are as follows:

Run: glad-sound-20
1


Texture Fusion Module

Finally, the Texture Fusion Module is responsible for synthesizing a photo-realistic body texture for the 3D mesh generated by the DRM, which it achieves by fusing the warped clothing with the original image of the person.
Specifically, given the unchanged person part IpI^p, the warped clothing CwC^w, the Segmentation mask SS and the frontal body depth map DfiD^i_f, the TFM considers the 2D information provided by Ip,Cwand SI^p, C^w \text{and} \ S along with the depth information from DfiD^i_f to generate a coarse try-on result I~c\tilde{I}^c and a fusion mask M~\tilde{M}.
The final try-on result is then generated as:
It=Cw⊙M~+I~c⊙(1−M~)I^t = C^w \odot \tilde{M} + \tilde{I}^c \odot (1 - \tilde{M})

The TFM is trained using a perceptual loss Lperc\mathcal{L}_\text{perc} between the try on result ItI^t and II, the L1 loss Ltry-on\mathcal{L}_\text{try-on} between ItI^t and II, as well as the L1 loss Lmask\mathcal{L}_\text{mask} between the estimated fusion mask M~\tilde{M} and real clothing mask MM as:
LTFM=Lperc+Ltry-on+Lmask\mathcal{L}_\text{TFM} = \mathcal{L}_\text{perc} + \mathcal{L}_\text{try-on} + \mathcal{L}_\text{mask}


The results for this training step are as follows:

Run: dashing-voice-22
1


Qualitative Comparison

As you can tell, each virtual try-on method has its advantages and differs from one another. Using Weights & Biases, it's extremely easy to compare models qualitatively by logging results into our own Tables.

Run: fluent-smoke-1
1

All the 3D models in the last column can be expanded and inspected by going full screen.
💡


Created with ❤️ on Weights & Biases.

https://wandb.ai/wandb/retail-tutorial/reports/Comparative-Analysis-of-Virtual-Try-On-Methods--VmlldzoyMzg1ODQ3
Made with Weights & Biases. Sign up or log in to create reports like this one.