ppo_fast vs orz ppo + orz verifier

Created on May 8|Last edited on May 8
Comment
Below is a comparison between oi's ppo_fast.py and Open Reasoner Zero's ppo. Note orz's runtime appears slower partly because it runs in-loop eval, whereas our ppo_fast.py saves the checkpoint and launches other jobs for eval﻿﻿
﻿
﻿
objective/verifiable_correct_rate, avg_raw_rewards
objective/verifiable_correct_rate, avg_raw_rewards
50100150Time (hours)0.20.40.6
loss/value, critic_loss
loss/value, critic_loss
50100150Time (hours)0.020.040.060.08
val/sequence_lengths_unsolved, avg_incorrect_num_tokens
val/sequence_lengths_unsolved, avg_incorrect_num_tokens
50100150Time (hours)1000200030004000
val/sequence_lengths_unsolved_hist
val/sequence_lengths_unsolved_hist
2M4M6Mglobal_step02000400060008000
2M4M6Mglobal_step02000400060008000
val/sequence_lengths_solved_hist
val/sequence_lengths_solved_hist
2M4M6Mglobal_step02000400060008000
2M4M6Mglobal_step02000400060008000
val/sequence_lengths, avg_response_length
val/sequence_lengths, avg_response_length
50100150Time (hours)500100015002000250030003500
val/sequence_lengths_solved, avg_correct_num_tokens
val/sequence_lengths_solved, avg_correct_num_tokens
50100150Time (hours)50010001500200025003000
training_step
training_step
20406080Time (hours)200400600800
orz PPO1
ppo_fast1
﻿
﻿
Add a comment