BC03 3DoF, 1 Sphere, PN++ Naive EE
Classification PN++. Testing with and without the older data.
Created on May 13|Last edited on May 27
Comment
The naive method. Interesting, it seems like 3 out of the 5 runs actually did well and were increasing at the end. By the way this is MORE than what we see with the image-based classifier (also a naive baseline) but the image-based one has more consistent performance.
Some of the runs ended early though. We may want to run this again to get more numbers... not sure why the cluster seems to be unreliable. Also this is before May 14, so the eval/MSE_loss isn't evaluated correctly, so we'll just have to look at the eval/info_done_final along with train/MSE_loss. It seems like PN generally requires longer to learn compared to CNNs.
The two runs that failed have the two worse (higher) train/MSE_loss but those are still "reasonable" so I'm not sure why ...
As of 05/19/2022, doing a comparison with the newer data v08 and the older v02 data.
- Old data ladle_algorithmic_v02:
- Started this 05/19 so we can look at the eval/MSE_loss. Those curves look good, they are decreasing and definitely correlating with success...
- Why are the results looking so good (it even seems like this is getting to 0.6...) when earlier I was getting a lot worse performance? The only new thing I changed is that I got rid of zero padding. However even if I redo zero padding, that actually isn't bad amazingly! I have only done 1 with zero padding but amazingly it's doing very well.
- I can definitely see from the variant.json that this is using ladle_algorithmic_v02 dataset (as expected) with all the standard stuff.
- There should be the same amount of 7500 training data (obs, EE delta) for BC, from the first 100 episodes.
- Compare with earlier report here: https://wandb.ai/mooey5775/mixed_media/reports/BC-02-MMOneSphere-Naive-Method-Regress-Straight-to-3DoF--VmlldzoxOTEzNTUz this one seemed to suggest that I needed 500 demos instead of 100 to get good performance. With BC01 I got good results with 100 demos, but the two that I did for BC02 got worse performance. The results are using "bc_data_dir": "/data/dseita/softgym_mm/data_demo/MMOneSphere_v01_BClone_filtered_ladle_algorithmic_v02_nVars_2000_obs_combo_act_translation", which is the SAME as what I am using now!
- This is BC01: https://wandb.ai/mooey5775/mixed_media/reports/MMOneSphere-BC_PCL_PNet2_ee--VmlldzoxODY5MTE1 this is the one that doesn't have the translation in the file name, but that should still be fine?
- New data ladle_algorithmic_v08:
- Why again does this seem to have slower initial performance? It's the same that I see with the SVD methods. There must be something about this property of the data.
So far it seems like the naive method is doing better than I expected.
Section 1
Data v08 (new)
5
Data v02 (old)
5
Data v02 (old) with zero-padding
1
GIFs
One of the runs that failed (the one from 05/12)
Representative failure BC03_MMOneSphere_v01_ntrain_0100_PCL_PNet2_ee_3DoF_ar_8_hor_100_rawPCL_scaleTarg_2022_05_13_10_12_36_0002
I'm not sure why but the above run stopped at epoch 239? Really need to get these settings more consistent.
Code
After:
Add a comment