apiche

Apiche's group workspace

Group: w_quant_vllm_3

337

114

1-3

of 3

Timestamps visible

2025-09-19 21:57:57

[finetune]: 09/19/2025 21:57:57.082 - INFO - pipelinerl.finetune_loop - All workers synchronized successfully

2025-09-19 21:57:57

[finetune]: 09/19/2025 21:57:57.136 - INFO - pipelinerl.finetune_loop - Completed steps 5: {'stats/lr': '1.000e-06', 'stats/grad_norm': "tensor(327.2159, device='cuda:0', dtype=torch.float64)", 'stats/samples': '80.000', 'stats/tokens': '36894.000', 'stats/samples_too_old_to_queue': '0.000', 'stats/samples_too_old_to_train': '0.000', 'stats/passes': '5.000', 'stats/completed_steps': '5.000', 'stats/epoch': '0.000', 'stats/min_actor_version': '64.000', 'stats/max_actor_version': '64.000', 'stats/queue/batches': '0.000', 'stats/time_waiting_for_data': '23.264', 'stats/lag': '0.000', 'throughput/tokens_perGPU_per_sec': '205.093', 'throughput/tokens_per_step': '8476.000', 'throughput/micro_batches_per_step': '1.000', 'throughput/min_tokens_per_micro_batch': '4238.000', 'throughput/max_tokens_per_micro_batch': '4238.000', 'throughput/tokens_per_micro_batch': '4238.000', 'throughput/tokens_per_sec': '410.186', 'throughput/real_tokens_per_sec': '130.317', 'throughput/sec_per_pass': '20.664', 'throughput/steps_per_sec': '0.031', 'throughput/samples_per_sec': '0.774', 'throughput/sec_per_step': '32.521', 'throughput/max_sequences_per_micro_batch': '8.000', 'throughput/min_sequences_per_micro_batch': '8.000', 'throughput/sequences_per_micro_batch': '8.000', 'rl/loss': '-1869.762', 'rl/max_loss': '-46.848', 'rl/min_loss': '-994.625', 'rl/reward': '0.000', 'rl/max_reward': '0.000', 'rl/min_reward': '0.000', 'rl/entropy': '2.773', 'rl/old_logprobs': '-2.736', 'rl/new_logprobs': '-2.737', 'rl/ref_logprobs': '-2.736', 'rl/advantage': '-0.143', 'rl/max_advantage': '0.730', 'rl/min_advantage': '-0.742', 'rl/kl': '0.001', 'rl/max_kl': '0.144', 'rl/min_kl': '-5.960e-08', 'rl/policy_loss': '9.467', 'rl/surr1': '0.000', 'rl/surr2': '0.000', 'rl/ratio_new_old': '1.000', 'rl/ratio_new_old_sum': '31137.418', 'rl/ratio_new_old_squared_sum': '31209.898', 'rl/ratio_ref_new': '1.002', 'rl/ratio_ref_old': '1.000', 'rl/clamp_log_ratio_ref_new_indicator': '0.000', 'rl/clamp_log_ratio_new_old_indicator': '0.000', 'rl/num_nans': '0.000', 'rl/token_weight': '0.062', 'rl/max_token_weight': '0.062', 'rl/min_token_weight': '0.062', 'rl/kl_coef': '0.000', 'rl/entropy_bonus_coef': '0.000', 'rl/num_output_tokens_sum': '31130.000', 'rl/input_size': '2047.875', 'rl/value_mean': '0.143', 'rl/value_max': '0.742', 'rl/value_min': '-0.730', 'rl/value_loss': '54.297', 'rl/value_mse': '0.051', 'rl/ess': '0.998'}

2025-09-19 21:57:57

[finetune]: 09/19/2025 21:57:57.137 - INFO - pipelinerl.finetune_loop - Start gathering and sending ZeRO Stage 3 weights

2025-09-19 21:57:57

[finetune]: 09/19/2025 21:57:57.140 - INFO - pipelinerl.finetune_loop - Published weight update request for version 80

2025-09-19 21:57:57

[finetune]: 09/19/2025 21:57:57.233 - INFO - pipelinerl.finetune_loop - Wait for HTTP requests

2025-09-19 21:57:57

[finetune]: 09/19/2025 21:57:57.520 - INFO - pipelinerl.finetune_loop - Finished broadcasting weights

2025-09-19 21:57:57

[finetune]: 09/19/2025 21:57:57.524 - INFO - pipelinerl.finetune_loop - Start step at 1758319077.52464

2025-09-19 21:57:57

[finetune]: 09/19/2025 21:57:57.625 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.1

2025-09-19 21:57:57

[finetune]: 09/19/2025 21:57:57.775 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.15000000000000002

2025-09-19 21:57:58

[finetune]: 09/19/2025 21:57:58.001 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.22500000000000003

2025-09-19 21:57:58

[finetune]: 09/19/2025 21:57:58.340 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.3375

2025-09-19 21:57:58

[finetune]: 09/19/2025 21:57:58.847 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.5062500000000001

2025-09-19 21:57:59

[finetune]: 09/19/2025 21:57:59.607 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.7593750000000001