Pouring, Ablations

(1) No skip connections, (2) pointwise before SVD.
Created on June 7|Last edited on June 9
Comment
﻿
Overview(06/06/2022) Launching 5X no skip connections.(06/08/2022) Launching 5X Pointwise Before SVD(06/09/2022) Results look bad.ResultsGIFs, No SkipGIFs, Pointwise before SVD
﻿
Overview
(06/06/2022) Launching 5X no skip connections.This is weird, why is performance pretty much zero here?  For all methods? If you notice we weren't getting that here for the sphere env:
﻿https://wandb.ai/mooey5775/mixed_media/reports/BC03-1-Sphere-Ablations--VmlldzoyMTE4Njc5﻿
The MSEs are doing down but why is the performance so bad? Compare that with the proposed method: https://wandb.ai/mooey5775/mixed_media/reports/Pouring-SVD-Methods--VmlldzoyMTAzNDU1 . 
Here, train/MSE_loss at ~150 epochs is ~0.006, eval/MSE_loss at ~150 is ~0.007.
There, train/MSE_loss at ~150 epochs is ~0.002, eval/MSE_loss at ~150 is ~0.003.
OK so there is definitely something to do with the MSEs here. Also in the scooping env, this ablation isn't that bad...  
OK after inspecting I think the reason is that without skip connections it cannot easily learn rotations, see:
﻿
Whereas in the scooping env, no rotation can still lead to some success, here that is no longer the case (we need rotations to succeed in the pouring task). We could clarify that distinction with this env and the other one in the paper (e.g., see Implicit Behavioral Cloning style reporting).
(06/08/2022) Launching 5X Pointwise Before SVDThis ablation is also done for the scooping.
# 6D flow, but using pointwise loss before SVD.
SVD_PRE_POINTWISE_6D_EE2FLOW_SEP_RT_SVD = dict﻿(
    obs_type=﻿'point_cloud'﻿,
    act_type=﻿'ee2flow_sep_rt'﻿,
    encoder_type=﻿'pointnet_svd_pointwise_6d_flow'﻿,
    method_flow2act=﻿'svd'﻿,
    use_consistency_loss=﻿False﻿,
    scale_pcl_flow=﻿True﻿,
    scale_pcl_val=﻿250﻿,
    separate_MLPs_R_t=﻿False﻿,
    gaussian_noise_PCL=﻿0.0﻿,  # normally 0
)
(06/09/2022) Results look bad.Well these are ablations, but results do look bad. So I think there has be something fundamental to doing a loss after the SVD. Also we could still do the consistency loss here because that's now something that Brian and Chuer (and us) are doing, we're also calling it a "dense loss" which is like what we have here.
Results﻿
No Skip Connections5
Pointwise before SVD5
﻿
GIFs, No SkipYeah I've tried locally and it's not learning the rotations here, even if losses are going down. That's strange but I don't think we should spend too much time dissecting. I am sure there is no change in the action space, this is literally just 1 argument to our method that removes part of skip connections.
﻿
﻿
﻿
﻿
﻿
﻿
GIFs, Pointwise before SVDHmm  these don't look awful but seems like it's raising too high? If you look at say the 500 demo case with pointwise after SVD: https://wandb.ai/mooey5775/mixed_media/reports/Pouring-Compare-of-Demos--VmlldzoyMTA4NDg5﻿
it seems like here, we will be raising too much and getting more water dispersed?
﻿
﻿
﻿
﻿
﻿
﻿
Add a comment