apiche

Apiche's group workspace

Group: debug_actor_test_logging_july_23rd_1_guessing

416

189

4762

1-3

of 3

Timestamps visible

2025-07-23 14:58:52

[finetune]: 07/23/2025 14:58:52.745 - INFO - pipelinerl.finetune_loop - All workers synchronized successfully

2025-07-23 14:58:52

[finetune]: 07/23/2025 14:58:52.902 - INFO - pipelinerl.finetune_loop - Completed steps 17: {'stats/lr': '0.000', 'stats/grad_norm': '0.164', 'stats/samples': '17442.000', 'stats/tokens': '1913874.000', 'stats/samples_too_old_to_queue': '0.000', 'stats/samples_too_old_to_train': '0.000', 'stats/passes': '172.000', 'stats/completed_steps': '17.000', 'stats/epoch': '0.000', 'stats/min_actor_version': '15390.000', 'stats/max_actor_version': '16416.000', 'stats/queue/batches': '1.000', 'stats/time_waiting_for_data': '42.605', 'stats/lag': '1026.000', 'throughput/tokens_perGPU_per_sec': '3036.315', 'throughput/tokens_per_step': '142041.000', 'throughput/micro_batches_per_step': '13.000', 'throughput/min_tokens_per_micro_batch': '184.000', 'throughput/max_tokens_per_micro_batch': '3979.000', 'throughput/tokens_per_micro_batch': '3642.077', 'throughput/tokens_per_sec': '9108.945', 'throughput/real_tokens_per_sec': '3025.862', 'throughput/sec_per_pass': '1.200', 'throughput/steps_per_sec': '0.064', 'throughput/samples_per_sec': '65.796', 'throughput/sec_per_step': '15.647', 'throughput/max_sequences_per_micro_batch': '31.000', 'throughput/min_sequences_per_micro_batch': '1.000', 'throughput/sequences_per_micro_batch': '26.308', 'rl/loss': '0.143', 'rl/max_loss': '0.032', 'rl/min_loss': '-0.030', 'rl/reward': '-0.957', 'rl/max_reward': '1.400', 'rl/min_reward': '-1.800', 'rl/entropy': '0.061', 'rl/old_logprobs': '-0.075', 'rl/new_logprobs': '-0.081', 'rl/ref_logprobs': '-0.075', 'rl/advantage': '-1.078', 'rl/max_advantage': '1.802', 'rl/min_advantage': '-2.327', 'rl/kl': '0.019', 'rl/max_kl': '39.463', 'rl/min_kl': '-0.000', 'rl/policy_loss': '0.083', 'rl/surr1': '0.000', 'rl/surr2': '0.000', 'rl/ratio_new_old': '1.000', 'rl/ratio_new_old_sum': '10352.242', 'rl/ratio_new_old_squared_sum': '10498.188', 'rl/ratio_ref_new': '1.025', 'rl/ratio_ref_old': '1.000', 'rl/clamp_log_ratio_ref_new_indicator': '0.000', 'rl/clamp_log_ratio_new_old_indicator': '0.000', 'rl/num_nans': '0.000', 'rl/token_weight': '0.001', 'rl/max_token_weight': '0.001', 'rl/min_token_weight': '0.001', 'rl/kl_coef': '0.000', 'rl/entropy_bonus_coef': '0.000', 'rl/num_output_tokens_sum': '10325.000', 'rl/input_size': '137.900', 'rl/value_mean': '0.121', 'rl/value_max': '0.715', 'rl/value_min': '-0.848', 'rl/value_loss': '0.008', 'rl/value_mse': '1.610', 'rl/ess': '0.989'}

2025-07-23 14:58:52

[finetune]: 07/23/2025 14:58:52.903 - INFO - pipelinerl.finetune_loop - Start gathering and sending ZeRO Stage 3 weights

2025-07-23 14:58:52

[finetune]: 07/23/2025 14:58:52.907 - INFO - pipelinerl.finetune_loop - Published weight update request for version 17442

2025-07-23 14:58:53

[finetune]: 07/23/2025 14:58:53.089 - INFO - pipelinerl.finetune_loop - Wait for HTTP requests

2025-07-23 14:58:53

[finetune]: 07/23/2025 14:58:53.365 - INFO - pipelinerl.finetune_loop - Finished broadcasting weights

2025-07-23 14:58:53

[finetune]: 07/23/2025 14:58:53.369 - INFO - pipelinerl.finetune_loop - Start step at 1753282733.3694928