BC#03 Test Overfitting to 1 Demonstration
Need to see if this can overfit to 1 demonstration. Done before running larger-scale BC#03 experiments, with 10 runs total (5 for 1-sphere, 5 for multi-sphere).
Created on May 6|Last edited on May 6
Comment
(Friday, 05/06/2022) I ran 250 epochs of training on data from one demonstration, and did evaluation on that same demonstration (the starting state for it, that is).
I did 6x runs with 3DoF, 4x runs with 4DoF. The extra 6x is because we can do the PNet2_avg method which does per-point "flow" predictions, and then averages the result to get the EE translation (and to do that for 4D, we'd need something else, though we can do that 6D flow which we have yet to try).
Results
The results suggest that the models are able to overfit for the most part, though the naive PNet2_eepose_4DoF method seems to struggle after 100 epochs, for some reason. I wonder if we should increase model capacity?
In terms of success rates, we do expect to see one-sphere MMOneSphere get higher success rates. Interestingly, in that case the pointwise SVD seems to be doing the best here followed by the naive method. I thought the averaging layer would do the best.
So, are these algorithms correctly overfitting to the demonstrator? I think for 3DoF they are, but is this the case for 4DoF? Not shown here but locally on 05/06/2022 and reported in my Notion, I was able to overfit to the demonstrator with a rotation using RGB images.
One-Sphere
5
Multi-Sphere
5
These plots look interesting! Ah, I wish there was a way we could see this on an episodic basis (i.e., finding the flow predictions at each time step instead of at a random time). The SVD one for 3DoF is looking good when it predicts that the tool should go upwards. Also next time keep the y-axis (really, z here) at zero so we see the full tool!
Multi-Sphere
5
One-Sphere
5
Code, etc.
Add a comment