apiche

Apiche's group workspace

Group: w_quant_vllm_1

299

1-3

of 3

Timestamps visible

2025-09-19 21:49:48

[finetune]: 09/19/2025 21:49:48.470 - INFO - pipelinerl.finetune_loop - All workers synchronized successfully

2025-09-19 21:49:48

[finetune]: 09/19/2025 21:49:48.517 - INFO - pipelinerl.finetune_loop - Completed steps 3: {'stats/lr': '1.000e-06', 'stats/grad_norm': "tensor(119.8033, device='cuda:0', dtype=torch.float64)", 'stats/samples': '48.000', 'stats/tokens': '20422.000', 'stats/samples_too_old_to_queue': '0.000', 'stats/samples_too_old_to_train': '0.000', 'stats/passes': '3.000', 'stats/completed_steps': '3.000', 'stats/epoch': '0.000', 'stats/min_actor_version': '32.000', 'stats/max_actor_version': '32.000', 'stats/queue/batches': '0.000', 'stats/time_waiting_for_data': '17.175', 'stats/lag': '0.000', 'throughput/tokens_perGPU_per_sec': '204.690', 'throughput/tokens_per_step': '4566.000', 'throughput/micro_batches_per_step': '1.000', 'throughput/min_tokens_per_micro_batch': '2283.000', 'throughput/max_tokens_per_micro_batch': '2283.000', 'throughput/tokens_per_micro_batch': '2283.000', 'throughput/tokens_per_sec': '409.380', 'throughput/real_tokens_per_sec': '170.109', 'throughput/sec_per_pass': '11.153', 'throughput/steps_per_sec': '0.075', 'throughput/samples_per_sec': '1.435', 'throughput/sec_per_step': '13.421', 'throughput/max_sequences_per_micro_batch': '8.000', 'throughput/min_sequences_per_micro_batch': '8.000', 'throughput/sequences_per_micro_batch': '8.000', 'rl/loss': '-332.115', 'rl/max_loss': '-37.411', 'rl/min_loss': '-294.704', 'rl/reward': '0.000', 'rl/max_reward': '0.000', 'rl/min_reward': '0.000', 'rl/entropy': '2.730', 'rl/old_logprobs': '-2.769', 'rl/new_logprobs': '-2.771', 'rl/ref_logprobs': '-2.769', 'rl/advantage': '-0.144', 'rl/max_advantage': '0.637', 'rl/min_advantage': '-0.688', 'rl/kl': '0.001', 'rl/max_kl': '0.136', 'rl/min_kl': '-5.960e-08', 'rl/policy_loss': '8.471', 'rl/surr1': '0.000', 'rl/surr2': '0.000', 'rl/ratio_new_old': '0.999', 'rl/ratio_new_old_sum': '10717.483', 'rl/ratio_new_old_squared_sum': '10735.459', 'rl/ratio_ref_new': '1.003', 'rl/ratio_ref_old': '1.000', 'rl/clamp_log_ratio_ref_new_indicator': '0.000', 'rl/clamp_log_ratio_new_old_indicator': '0.000', 'rl/num_nans': '0.000', 'rl/token_weight': '0.062', 'rl/max_token_weight': '0.062', 'rl/min_token_weight': '0.062', 'rl/kl_coef': '0.000', 'rl/entropy_bonus_coef': '0.000', 'rl/num_output_tokens_sum': '10723.000', 'rl/input_size': '775.000', 'rl/value_mean': '0.144', 'rl/value_max': '0.688', 'rl/value_min': '-0.637', 'rl/value_loss': '15.206', 'rl/value_mse': '0.047', 'rl/ess': '0.998'}

2025-09-19 21:49:48

[finetune]: 09/19/2025 21:49:48.518 - INFO - pipelinerl.finetune_loop - Start gathering and sending ZeRO Stage 3 weights

2025-09-19 21:49:48

[finetune]: 09/19/2025 21:49:48.521 - INFO - pipelinerl.finetune_loop - Published weight update request for version 48

2025-09-19 21:49:48

[finetune]: 09/19/2025 21:49:48.618 - INFO - pipelinerl.finetune_loop - Wait for HTTP requests

2025-09-19 21:49:48

[finetune]: 09/19/2025 21:49:48.883 - INFO - pipelinerl.finetune_loop - Finished broadcasting weights

2025-09-19 21:49:48

[finetune]: 09/19/2025 21:49:48.887 - INFO - pipelinerl.finetune_loop - Start step at 1758318588.887365

2025-09-19 21:49:48

[finetune]: 09/19/2025 21:49:48.989 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.1

2025-09-19 21:49:49

[finetune]: 09/19/2025 21:49:49.139 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.15000000000000002

2025-09-19 21:49:49

[finetune]: 09/19/2025 21:49:49.365 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.22500000000000003

2025-09-19 21:49:49

[finetune]: 09/19/2025 21:49:49.703 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.3375

2025-09-19 21:49:50

[finetune]: 09/19/2025 21:49:50.209 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.5062500000000001

2025-09-19 21:49:50

[finetune]: 09/19/2025 21:49:50.969 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.7593750000000001