Skip to main content
mooey5775
Projects
mixed_media
Reports
Physical Experiments, June 22+, Translation-Only

Physical Experiments, June 22+, Translation-Only

Trying to get the flow methods to work the best on translation-only data for scooping. So far we have something that shows some promise. However, it still feels unreliable and a reviewer who is watching the videos may not be impressed.
Daniel Seita
Created on June 22|Last edited on June 23
Contents:
Experiments LogFirst, 77 Train DemosSecond, 123 Train Demos, Fix Rotation Data AugmentationThird, Just use PN++ with Averaging?Fourth, Use Re-Balanced Point Cloud? ("Denser" data) (06/23/2022)Results, Flow Methods (but maybe don't use)? [123 train demos]Results, PN++ Averaging Layer [123 Train Demos]Results, PN++ Averaging Layer vs Naive Methods, DENSER data [133 train demos]Discussion


Experiments Log

First, 77 Train Demos

Second, 123 Train Demos, Fix Rotation Data Augmentation

Update 06/22/2022: fixed rotation data augmentation, they now look like this for select degrees about the z-axis (if we use right hand rule, points downwards from the ladle's tip) and then in the code we now randomize this value, and only for training:

9:20pm: with that fixed, I now also have more training demonstrations for the same dataset: v03_physicalTranslations_pkls_combined_k_step_4 has 123 training demos and 25 validation (by the way we really should change this so each time we increase the data we are fixing on the same set of validation data, but never mind). This is the run, which trains on that data, for 100 epochs, and uses Gaussian (0.0001) and rotation data (-90,90) augmentation (rotation happens first):
https://wandb.ai/mooey5775/mixed_media/runs/2byu6l5v?workspace=user-danieltakeshi
and with just Gaussian data augm:
https://wandb.ai/mooey5775/mixed_media/runs/1g4mh12e?workspace=user-danieltakeshi
But we have not yet updated the point clouds. The point cloud composition still looks like this (this is the SVD with 3d flow, for translation only data):
Done loading BC data. Buffer full? False
  idx: 8628 out of cap: 1000000
  scaling? False, using None
  scaling targets? False
  Train idxs for (s,a) pairs (incl, excl): (0,7053)
  Valid idxs for (s,a) pairs (incl, excl): (7053,8628)
  #obs: 8628, each is a PCL of possibly different size
    data.x shape: 1200.00 +/- 0.0
    #pts min max: 1200, 1200
    # tool pts: 1103.02 +/- 17.1
    # ball pts: 96.85 +/- 17.1
    min max (tool): 1074, 1200
    min max (ball): 0, 100
  #act: 8628, type <class 'list'>
  max flow: [0.05178635 0.03591739 0.09390617]
  min flow: [-0.04398376 -0.07601072 -0.07498637]
Sarthak is actively addressing this to change the point clouds, but in the meantime we'll assume about 1100 tool points and about 100 ball points.
Also, this has a total of 7053 training data points, which is at least getting close to the 100x75=7500 that we have in simulation.
I'm not exactly sure how well this will work if only because in simulation the flow and SVD isn't necessarily the best, we might as well try with the averaging layer?

Third, Just use PN++ with Averaging?

# Segm PN++ to per-point preds, then average. Regress to MSE. This is ONLY
# applicable with 3DoF translation data. In sim this was giving good results,
# though it was similar to the classification PN++ so I don't think we will
# end up testing this a lot. (Edit: actually let's see what the new sim results
# look like with translation only, we might consider using this.)
NAIVE_SEGM_PN2_TO_FLOW_AVG_3DoF = dict(
    obs_type='point_cloud',
    act_type='ee',
    encoder_type='pointnet_avg',
    scale_targets=True,
)

But I need to check if scale_targets will affect things. Should we just set that to be off? But I feel like we might want that, maybe just have it on for now?
This should compare with and without the rotation data augmentation.
NOTE: see the next panel grid for these, as mentioned elsewhere we can't compare MSE results.

Fourth, Use Re-Balanced Point Cloud? ("Denser" data) (06/23/2022)

Sarthak adjusted the point clouds, need to copy this over to lambda-dual2 machine.
/data/seita/softgym_mm/data_demo/v03_physicalTranslations_pkls_combined_k_step_4_denser/
These have a max of 1400 points, with 1100 reserved for the tool, and up to 300 for the ball. (Check the print-out to be safe)
These results are in a separate panel grid below.

Results, Flow Methods (but maybe don't use)? [123 train demos]

Results, PN++ Averaging Layer [123 Train Demos]

Results, PN++ Averaging Layer vs Naive Methods, DENSER data [133 train demos]

These now use v03_physicalTranslations_pkls_combined_k_step_4_denser as the data.
See plots below, for eval MSE and train MSE (with and without log scales).

PN++ avg, rot + Gauss 0.0001, dense data
1
Class. PN++, rot + Gauss 0.0001, dense data
1
PN++ avg, Gauss 0.0001, dense data
1
Class. PN++, Gauss 0.0001, dense data
1


Discussion

What conclusions can we make?

Created with ❤️ on Weights & Biases.

https://wandb.ai/mooey5775/mixed_media/reports/Physical-Experiments-June-22-Translation-Only--VmlldzoyMjExMDcx
Made with Weights & Biases. Sign up or log in to create reports like this one.