apiche

Apiche's group workspace

Group: w_quant_vllm

444

220

1-3

of 3

Timestamps visible

2025-09-19 21:40:42

[finetune]: 09/19/2025 21:40:42.444 - INFO - pipelinerl.finetune_loop - Completed steps 11: {'stats/lr': '1.000e-06', 'stats/grad_norm': "tensor(340.0209, device='cuda:0', dtype=torch.float64)", 'stats/samples': '176.000', 'stats/tokens': '63322.000', 'stats/samples_too_old_to_queue': '0.000', 'stats/samples_too_old_to_train': '2.000', 'stats/passes': '11.000', 'stats/completed_steps': '11.000', 'stats/epoch': '0.000', 'stats/min_actor_version': '160.000', 'stats/max_actor_version': '160.000', 'stats/queue/batches': '0.000', 'stats/time_waiting_for_data': '49.885', 'stats/lag': '0.000', 'throughput/tokens_perGPU_per_sec': '64.666', 'throughput/tokens_per_step': '5200.000', 'throughput/micro_batches_per_step': '1.000', 'throughput/min_tokens_per_micro_batch': '2600.000', 'throughput/max_tokens_per_micro_batch': '2600.000', 'throughput/tokens_per_micro_batch': '2600.000', 'throughput/tokens_per_sec': '129.331', 'throughput/real_tokens_per_sec': '59.031', 'throughput/sec_per_pass': '40.207', 'throughput/steps_per_sec': '0.023', 'throughput/samples_per_sec': '0.398', 'throughput/sec_per_step': '44.045', 'throughput/max_sequences_per_micro_batch': '8.000', 'throughput/min_sequences_per_micro_batch': '8.000', 'throughput/sequences_per_micro_batch': '8.000', 'rl/loss': '-1467.065', 'rl/max_loss': '-28.683', 'rl/min_loss': '-882.120', 'rl/reward': '0.000', 'rl/max_reward': '0.000', 'rl/min_reward': '0.000', 'rl/entropy': '3.046', 'rl/old_logprobs': '-3.068', 'rl/new_logprobs': '-3.068', 'rl/ref_logprobs': '-3.068', 'rl/advantage': '-0.150', 'rl/max_advantage': '0.543', 'rl/min_advantage': '-0.680', 'rl/kl': '0.001', 'rl/max_kl': '0.084', 'rl/min_kl': '-5.960e-08', 'rl/policy_loss': '8.639', 'rl/surr1': '0.000', 'rl/surr2': '0.000', 'rl/ratio_new_old': '1.001', 'rl/ratio_new_old_sum': '23474.398', 'rl/ratio_new_old_squared_sum': '23532.562', 'rl/ratio_ref_new': '1.002', 'rl/ratio_ref_old': '1.000', 'rl/clamp_log_ratio_ref_new_indicator': '0.000', 'rl/clamp_log_ratio_new_old_indicator': '0.000', 'rl/num_nans': '0.000', 'rl/token_weight': '0.062', 'rl/max_token_weight': '0.062', 'rl/min_token_weight': '0.062', 'rl/kl_coef': '0.000', 'rl/entropy_bonus_coef': '0.000', 'rl/num_output_tokens_sum': '23461.000', 'rl/input_size': '1587.750', 'rl/value_mean': '0.150', 'rl/value_max': '0.680', 'rl/value_min': '-0.543', 'rl/value_loss': '50.540', 'rl/value_mse': '0.048', 'rl/ess': '0.998'}

2025-09-19 21:40:42

[finetune]: 09/19/2025 21:40:42.445 - INFO - pipelinerl.finetune_loop - Start gathering and sending ZeRO Stage 3 weights

2025-09-19 21:40:42

[finetune]: 09/19/2025 21:40:42.448 - INFO - pipelinerl.finetune_loop - Published weight update request for version 176

2025-09-19 21:40:42

[finetune]: 09/19/2025 21:40:42.513 - INFO - pipelinerl.finetune_loop - Wait for HTTP requests

2025-09-19 21:40:42

[finetune]: 09/19/2025 21:40:42.790 - INFO - pipelinerl.finetune_loop - Finished broadcasting weights

2025-09-19 21:40:42

[finetune]: 09/19/2025 21:40:42.793 - INFO - pipelinerl.finetune_loop - Start step at 1758318042.793769

2025-09-19 21:40:42

[finetune]: 09/19/2025 21:40:42.894 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.1

2025-09-19 21:40:43

[finetune]: 09/19/2025 21:40:43.044 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.15000000000000002

2025-09-19 21:40:43

[finetune]: 09/19/2025 21:40:43.270 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.22500000000000003

2025-09-19 21:40:43

[finetune]: 09/19/2025 21:40:43.607 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.3375

2025-09-19 21:40:44

[finetune]: 09/19/2025 21:40:44.114 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.5062500000000001

2025-09-19 21:40:44

[finetune]: 09/19/2025 21:40:44.874 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.7593750000000001

2025-09-19 21:40:46

[finetune]: 09/19/2025 21:40:46.013 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 1.1390625