Reproducing Results for Open-Source, PyFlexRobotics
Only ToolFlowNet and Direct Vector MSE (an older version), for both variants of PourWater (3D and 6D action spaces). Uses PyFlexRobotics, NOT PyFlex.
Created on January 31|Last edited on February 4
Comment
Update: I used the old configuration for "Direct Vector MSE" for these experiments, but just look at the newer report here which uses PyFlex.
I did this after merging:
See my Notion for other info to cross reference.
Results, PourWater 3D
Max value for ToolFlowNet is 0.536 vs 0.400 for Direct Vector MSE.
In the paper the raw results (Table S6) show 0.720 for ToolFlowNet vs 0.480 for Direct Vector MSE. That does seem a bit off for ToolFlowNet though it still does better than Direct Vector MSE, just not as much. The ToolFlowNet improvement is much more prominent with 6D pouring (see next plots).
From the variant.json we can see that the file name used for BC is:
"bc_data_dir": "/data/dseita/softgym_mm/data_demo/PourWater_v01_BClone_filtered_wDepth_pw_algo_v02_nVars_1500_obs_combo_act_translation_axis_angle_withWaterFrac"
28
PourWater3D ToolFlowNet
5
PourWater3D DirVectMSE
5
Results, PourWater 6D
Now let's see what happens with 6D variant... here the gap is much larger. Do we see that in the paper?
Here, max value for ToolFlowNet is 0.552 vs 0.328 for Direct Vector MSE.
In the paper the raw results (Table S6) show 0.544 for ToolFlowNet vs 0.216 for Direct Vector MSE. Seems a bit off for Direct Vector MSE, while it's pretty accurate for TFN -- the opposite conclusion from the 3D results, however the overall lesson is TFN > Direct Vector MSE.
From variant.json the training data:
"bc_data_dir": "/data/dseita/softgym_mm/data_demo/PourWater6D_v01_BClone_filtered_wDepth_pw_algo_v02_nVars_2000_obs_combo_act_translation_axis_angle_withWaterFrac",
However I think if we used the data named without "_withWaterFrac" at the end, it should be equivalent.
PourWater6D ToolFlowNet
5
PourWater6D DirVectMSE
5
Run set
28
Add a comment