BC03 4DoF, 1 Sphere, Baseline Methods (PN++ and CNN)
These are baseline methods that we hope do not do well. :). UPDATE: also adding the naive PN++ baseline but where we use pointwise loss! That will tell us if pointwise is sufficient, or if we need more. (UPDATE 06/02/2022, running again, see notes here)
Created on May 20|Last edited on June 4
Comment
ResultsQualitative ResultsResults with PN++ Naive Methods (no data augmentation) with MSEResults with Naive PN++ with Pointwise LossResults with CNN Naive Methods (no data augmentation)What if we use data augmentation?GIFs of CNN with Data AugmentationPointwise with Clearing DoFs
Notes:
- Compare with https://wandb.ai/mooey5775/mixed_media/reports/BC03-4DoF-1-Sphere-PN-SVD-Pointwise-6D-flow-ee2flow-100-demos--VmlldzoyMDM5ODgy for the SVD with 6D flow (using ee2flow) on this data.
- Great news, looks like the baseline methods (BOTH images and with point clouds) are not doing nearly as well here w.r.t. eval/info_done_final.
- These are not flow-based methods so we can't visualize flow.
- Update 05/21/2022: see below where I also test with data augmentation for image baseline.
- Update 05/27/2022: adding 5X more runs where we do the naive PN++ with classification, to produce a single 6D pose, but where we now do pointwise loss.
- Update: results are VERY bad. We might want to check if they will be as good with 3DoF demos? Though for that I guess it would just be predicting the 3D, averaging, applying that 'transform' with the identity, etc.?
- (05/31/2022) After inspecting GIFs, I actually do think there might be a bug, but what could it be?
- See results folders below:
(base) dseita@seuss:/data/dseita/softagent_mm$ ls -lh BC03_MMOneSphere_v01_ntrain_0100_PCL_PNet2_classif_6D_pointwise_ee2flow_4DoF_ar_8_hor_100_scalePCL_noScaleTarg/total 260Kdrwxrwxr-x 5 dseita dseita 12 May 27 20:47 BC03_MMOneSphere_v01_ntrain_0100_PCL_PNet2_classif_6D_pointwise_ee2flow_4DoF_ar_8_hor_100_scalePCL_noScaleTarg_2022_05_27_20_43_00_0001drwxrwxr-x 5 dseita dseita 12 May 27 20:51 BC03_MMOneSphere_v01_ntrain_0100_PCL_PNet2_classif_6D_pointwise_ee2flow_4DoF_ar_8_hor_100_scalePCL_noScaleTarg_2022_05_27_20_43_00_0002drwxrwxr-x 5 dseita dseita 12 May 27 20:51 BC03_MMOneSphere_v01_ntrain_0100_PCL_PNet2_classif_6D_pointwise_ee2flow_4DoF_ar_8_hor_100_scalePCL_noScaleTarg_2022_05_27_20_43_00_0003drwxrwxr-x 5 dseita dseita 12 May 27 20:51 BC03_MMOneSphere_v01_ntrain_0100_PCL_PNet2_classif_6D_pointwise_ee2flow_4DoF_ar_8_hor_100_scalePCL_noScaleTarg_2022_05_27_20_43_00_0004drwxrwxr-x 5 dseita dseita 12 May 28 02:38 BC03_MMOneSphere_v01_ntrain_0100_PCL_PNet2_classif_6D_pointwise_ee2flow_4DoF_ar_8_hor_100_scalePCL_noScaleTarg_2022_05_27_20_43_00_0005(base) dseita@seuss:/data/dseita/softagent_mm$
(base) dseita@seuss:~/softagent_rpad_MM/data/local$ ls -lh BC03_MMOneSphere_v01_ntrain_0100_PCL_PNet2_classif_6D_pointwise_ee2flow_4DoF_ar_8_hor_100_scalePCL_noScaleTarg/drwxrwxr-x 4 dseita dseita 4.0K May 31 13:27 BC03_MMOneSphere_v01_ntrain_0100_PCL_PNet2_classif_6D_pointwise_ee2flow_4DoF_ar_8_hor_100_scalePCL_noScaleTarg_2022_05_31_13_23_36_0001drwxrwxr-x 2 dseita dseita 4.0K May 31 13:24 BC03_MMOneSphere_v01_ntrain_0100_PCL_PNet2_classif_6D_pointwise_ee2flow_4DoF_ar_8_hor_100_scalePCL_noScaleTarg_2022_05_31_13_23_36_0002drwxrwxr-x 2 dseita dseita 4.0K May 31 13:24 BC03_MMOneSphere_v01_ntrain_0100_PCL_PNet2_classif_6D_pointwise_ee2flow_4DoF_ar_8_hor_100_scalePCL_noScaleTarg_2022_05_31_13_23_36_0003drwxrwxr-x 2 dseita dseita 4.0K May 31 13:24 BC03_MMOneSphere_v01_ntrain_0100_PCL_PNet2_classif_6D_pointwise_ee2flow_4DoF_ar_8_hor_100_scalePCL_noScaleTarg_2022_05_31_13_23_36_0004drwxrwxr-x 2 dseita dseita 4.0K May 31 13:24 BC03_MMOneSphere_v01_ntrain_0100_PCL_PNet2_classif_6D_pointwise_ee2flow_4DoF_ar_8_hor_100_scalePCL_noScaleTarg_2022_05_31_13_23_36_0005(base) dseita@seuss:~/softagent_rpad_MM/data/local$
- Anyway I don't expect this should improve too much but it SHOULD help as the input #s will be much easier for the PN++ to "digest".
Update 06/02/2022 see results at the bottom again where I am now trying to further improve pointwise methods by clearing out unused DoFs. I think this is reasonable.
Results
Naive CNN to 6D
5
Naive Cls PN++ to 6D, MSE
5
Naive Cls PN++ to 6D, Pointwise (w/bad input scale)
5
Naive Cls PN++ to 6D, Pointwise (good input scale)
5
Qualitative Results
These qualitative results should be compared with those from dense flow methods here:
The fact that these results are showing some promising results are good in the sense that it's less likely there's some bug and more likely due to some fundamental limitation with vector-based methods.
Results with PN++ Naive Methods (no data augmentation) with MSE
These are at 300 epochs. A lot worse than the dense flow methods. :)
seita@takeshi:/data/seita/softagent_mm$ ls -lh BC03_MMOneSphere_v01_ntrain_0100_PCL_PNet2_eepose_4DoF_ar_8_hor_100_rawPCL_scaleTarg/drwxrwxr-x 5 seita seita 4.0K May 21 12:01 BC03_MMOneSphere_v01_ntrain_0100_PCL_PNet2_eepose_4DoF_ar_8_hor_100_rawPCL_scaleTarg_2022_05_19_23_02_54_0001drwxrwxr-x 5 seita seita 4.0K May 21 12:02 BC03_MMOneSphere_v01_ntrain_0100_PCL_PNet2_eepose_4DoF_ar_8_hor_100_rawPCL_scaleTarg_2022_05_19_23_02_54_0002drwxrwxr-x 5 seita seita 4.0K May 21 12:03 BC03_MMOneSphere_v01_ntrain_0100_PCL_PNet2_eepose_4DoF_ar_8_hor_100_rawPCL_scaleTarg_2022_05_19_23_02_54_0003drwxrwxr-x 5 seita seita 4.0K May 21 12:03 BC03_MMOneSphere_v01_ntrain_0100_PCL_PNet2_eepose_4DoF_ar_8_hor_100_rawPCL_scaleTarg_2022_05_19_23_02_54_0004drwxrwxr-x 5 seita seita 4.0K May 21 12:02 BC03_MMOneSphere_v01_ntrain_0100_PCL_PNet2_eepose_4DoF_ar_8_hor_100_rawPCL_scaleTarg_2022_05_19_23_02_54_0005seita@takeshi:/data/seita/softagent_mm$
Results with Naive PN++ with Pointwise Loss
These are actually looking awful. Why?? (Not that I'm complaining, of course...)
Update: actually after looking at these, I think we actually should now double check if all this pipeline is correct, why are they like this? Maybe try overfitting to 1 demo?
Results with CNN Naive Methods (no data augmentation)
These are at 300 epochs. A lot worse than the dense flow methods. :)
Look at this directory but be careful about if it has data augmentation (check variant.json) or not. We are showing the non-data augmented version.
BC03_MMOneSphere_v01_ntrain_0100_cam_rgb_pixel_eepose_4DoF_ar_8_hor_100_scaleTarg
seita@takeshi:/data/seita/softagent_mm$ ls -lh BC03_MMOneSphere_v01_ntrain_0100_cam_rgb_pixel_eepose_4DoF_ar_8_hor_100_scaleTarg/drwxrwxr-x 5 seita seita 4.0K May 21 13:24 BC03_MMOneSphere_v01_ntrain_0100_cam_rgb_pixel_eepose_4DoF_ar_8_hor_100_scaleTarg_2022_05_19_22_49_33_0001drwxrwxr-x 5 seita seita 4.0K May 20 22:00 BC03_MMOneSphere_v01_ntrain_0100_cam_rgb_pixel_eepose_4DoF_ar_8_hor_100_scaleTarg_2022_05_20_07_05_49_0001drwxrwxr-x 5 seita seita 4.0K May 20 23:14 BC03_MMOneSphere_v01_ntrain_0100_cam_rgb_pixel_eepose_4DoF_ar_8_hor_100_scaleTarg_2022_05_20_09_53_53_0001drwxrwxr-x 5 seita seita 4.0K May 20 22:43 BC03_MMOneSphere_v01_ntrain_0100_cam_rgb_pixel_eepose_4DoF_ar_8_hor_100_scaleTarg_2022_05_20_13_15_32_0001drwxrwxr-x 5 seita seita 4.0K May 20 20:26 BC03_MMOneSphere_v01_ntrain_0100_cam_rgb_pixel_eepose_4DoF_ar_8_hor_100_scaleTarg_2022_05_20_20_25_18_0001seita@takeshi:/data/seita/softagent_mm$
What if we use data augmentation?
Interesting, looks like it might have actually been on its way upwards, even though it's taking a LONG time!
Naive CNN 6D (data aug)
5
GIFs of CNN with Data Augmentation
Interesting! It's definitely getting there. :) More good news that there's not too much of an issue with bugs, etc.
Pointwise with Clearing DoFs
After:
5X on cluster, 06/02/2022 evening.
Naive Pointwise, Clearing Unused DoFs
5
Huh why is it not working that well? Here are the results. It still seems like there's something weird, why is it moving upwards?!? It is overfitting to 1 demo, right (at least for pouring). Why for some of these is it collapsing? TODO: overfit to 1 demo?
Add a comment