Physical Datasets (Naive Methods, Translation Data v02)
Focusing on training of baseline methods here. Just do translation data.
Created on June 11|Last edited on June 12
Comment
Contents:
OverviewResults (Not Scaling Targets)Results (Scaling Targets)Histogram of Evaluation PredictionsNaive Classification PN++ to Vector, with Data Augm (No Scaling)Naive Classification PN++ to Vector, no Data Augm (No Scaling)Naive Classification PN++ to Vector, with Data Augm (Scaling Targets)Naive Classification PN++ to Vector, no Data Augm (Scaling Targets)
Overview
(06/11/2022) Trying to test naive classification PN++ on the translation only data v02_physicalTranslations_pkls so the nets output 3D vectors. Also, this has no scaling applied. See the commit after: https://github.com/Xingyu-Lin/softagent_rpad/commit/e419edd305e4a0b6596b33ce30dcc66ce8cae6f6 NOTE: the reason for doing this is that classification PN++ can do reasonably well in my domains if the job is just to predict translation only (in fact it was better than the flow-based methods).
Note: training one run for 300 epochs should be under an hour (roughly, depends on other factors, etc.).
These are using this config:
NAIVE_CLASS_PN2_TO_VECTOR_3DoF = dict(obs_type='point_cloud',act_type='ee',encoder_type='pointnet',data_augm_PCL='None',scale_targets=True,)
With modifications to data_augm_PCL and scale_targets as needed.
Results (Not Scaling Targets)
Naive PN++ to Vec, no aug
2
Naive PN++ to Vec, DataAug
2
Results (Scaling Targets)
Naive PN++ to Vec, no aug
2
Naive PN++ to Vec, DataAug
2
Histogram of Evaluation Predictions
For reference here's what the training data looks like (UNSCALED)
And SCALED which means dividing by 0.004 (or multiplying by 250) just because that was a number we used in simulation:
Naive Classification PN++ to Vector, with Data Augm (No Scaling)
(Gaussian noise 0.0001)
The network seems to be predicting nothing but 0 after 100 or so epochs?
at the start before training
after 25 epochs
after 50 epochs
at 100 epochs
It follows the same pattern as without using data augmentation (see next section). I think for deltax it is fixing at 0.001, and for y and z at 0?
Naive Classification PN++ to Vector, no Data Augm (No Scaling)
This is strange.
The evaluation prediction histogram is concentrated in the center as we would expect and the ranges (which are the same as the training one) look reasonable.
at the start before training
epoch 25
epoch 50
epoch 100
But notice what happens at epoch 100, it looks like training has collapsed? This is what the plot will look like from epoch 100 onwards. The network seems to be predicting nothing but 0?
Here's another seed where I have more digits in the decimal:
FYI this is after 300 epochs, and the titles of the subplots show the min and max predictions, so this is predicting the same thing, and the units are meters so it's predicting down to the same 0.1 millimeter.
Naive Classification PN++ to Vector, with Data Augm (Scaling Targets)
after 25
after 100
After 300
Hmm, it seems to be way more stable now. These are scaled units, so divide by 250 to get the actual units in meters. In fact we should just re-run this with the raw version there at some point.
Naive Classification PN++ to Vector, no Data Augm (Scaling Targets)
Despite no data augmentation, after 300 epochs, seems fairly reasonable:
No performance collapse, and dividing by 250 gives the units in meters.
Also I re-ran this again and decided to now save what the plot looks like when dividing by 250 again. Here's the raw unscaled then the scaled:
Same shape, as expected. Also note how the distribution is able to cover the range of roughly 0.01-ish translation-wise about 0 (though most actions are closer to 0 as expected).
Add a comment