apiche

Apiche's group workspace

Group: debug_gspo13

4473

1702

36179

1-3

of 3

Timestamps visible

2025-09-28 04:23:38

[finetune]: 09/28/2025 04:23:38.080 - INFO - pipelinerl.finetune_loop - All workers synchronized successfully

2025-09-28 04:23:38

[finetune]: 09/28/2025 04:23:38.119 - INFO - pipelinerl.finetune_loop - Completed steps 153: {'stats/lr': '1.000e-06', 'stats/grad_norm': "tensor(886.5561, device='cuda:0', dtype=torch.float64)", 'stats/samples': '2448.000', 'stats/tokens': '912440.000', 'stats/samples_too_old_to_queue': '0.000', 'stats/samples_too_old_to_train': '0.000', 'stats/passes': '169.000', 'stats/completed_steps': '153.000', 'stats/epoch': '0.000', 'stats/min_actor_version': '2144.000', 'stats/max_actor_version': '2144.000', 'stats/queue/batches': '1.000', 'stats/time_waiting_for_data': '12.186', 'stats/lag': '288.000', 'throughput/tokens_perGPU_per_sec': '760.950', 'throughput/tokens_per_step': '1788.000', 'throughput/micro_batches_per_step': '1.000', 'throughput/min_tokens_per_micro_batch': '447.000', 'throughput/max_tokens_per_micro_batch': '447.000', 'throughput/tokens_per_micro_batch': '447.000', 'throughput/tokens_per_sec': '3043.802', 'throughput/real_tokens_per_sec': '752.971', 'throughput/sec_per_pass': '0.587', 'throughput/steps_per_sec': '1.684', 'throughput/samples_per_sec': '27.238', 'throughput/sec_per_step': '0.594', 'throughput/max_sequences_per_micro_batch': '16.000', 'throughput/min_sequences_per_micro_batch': '16.000', 'throughput/sequences_per_micro_batch': '16.000', 'rl/loss': '-284.537', 'rl/max_loss': '-71.112', 'rl/min_loss': '-71.184', 'rl/reward': '0.000', 'rl/max_reward': '0.000', 'rl/min_reward': '0.000', 'rl/entropy': '3.820', 'rl/old_logprobs': '-3.758', 'rl/new_logprobs': '-3.798', 'rl/ref_logprobs': '-3.758', 'rl/advantage': '3.811', 'rl/max_advantage': '4.031', 'rl/min_advantage': '2.688', 'rl/kl': '0.514', 'rl/kl_new_old': '0.570', 'rl/max_kl': '8.865', 'rl/min_kl': '3.695e-06', 'rl/ratio_new_old': '1.530', 'rl/ratio_new_old_sum': '56.044', 'rl/ratio_new_old_squared_sum': '298.103', 'rl/ratio_ref_new': '1.554', 'rl/ratio_ref_old': '1.000', 'rl/clamp_log_ratio_ref_new_indicator': '0.000', 'rl/clamp_log_ratio_new_old_indicator': '0.000', 'rl/token_weight': '0.062', 'rl/max_token_weight': '0.062', 'rl/min_token_weight': '0.062', 'rl/kl_coef': '0.000', 'rl/entropy_bonus_coef': '0.000', 'rl/num_output_tokens_sum': '40.000', 'rl/input_size': '111.750', 'rl/value_mean': '-3.811', 'rl/value_max': '-2.688', 'rl/value_min': '-4.031', 'rl/value_loss': '17.464', 'rl/value_mse': '14.603', 'rl/ess': '0.263'}

2025-09-28 04:23:38

[finetune]: 09/28/2025 04:23:38.120 - INFO - pipelinerl.finetune_loop - Start gathering and sending ZeRO Stage 3 weights

2025-09-28 04:23:38

[finetune]: 09/28/2025 04:23:38.123 - INFO - pipelinerl.finetune_loop - Published weight update request for version 2448

2025-09-28 04:23:38

[finetune]: 09/28/2025 04:23:38.204 - INFO - pipelinerl.finetune_loop - Wait for HTTP requests

2025-09-28 04:23:38

[finetune]: 09/28/2025 04:23:38.460 - INFO - pipelinerl.finetune_loop - Finished broadcasting weights

2025-09-28 04:23:38

[finetune]: 09/28/2025 04:23:38.463 - INFO - pipelinerl.finetune_loop - Start step at 1759033418.4635386