apiche

Apiche's group workspace

Group: no_quant_vllm_2

353

132

1-3

of 3

Timestamps visible

2025-09-19 21:31:47

[finetune]: 09/19/2025 21:31:47.930 - INFO - pipelinerl.finetune_loop - All workers synchronized successfully

2025-09-19 21:31:47

[finetune]: 09/19/2025 21:31:47.969 - INFO - pipelinerl.finetune_loop - Completed steps 6: {'stats/lr': '1.000e-06', 'stats/grad_norm': '126.717', 'stats/samples': '96.000', 'stats/tokens': '38852.000', 'stats/samples_too_old_to_queue': '0.000', 'stats/samples_too_old_to_train': '0.000', 'stats/passes': '6.000', 'stats/completed_steps': '6.000', 'stats/epoch': '0.000', 'stats/min_actor_version': '80.000', 'stats/max_actor_version': '80.000', 'stats/queue/batches': '0.000', 'stats/time_waiting_for_data': '26.402', 'stats/lag': '0.000', 'throughput/tokens_perGPU_per_sec': '77.558', 'throughput/tokens_per_step': '4172.000', 'throughput/micro_batches_per_step': '1.000', 'throughput/min_tokens_per_micro_batch': '2086.000', 'throughput/max_tokens_per_micro_batch': '2086.000', 'throughput/tokens_per_micro_batch': '2086.000', 'throughput/tokens_per_sec': '155.117', 'throughput/real_tokens_per_sec': '60.462', 'throughput/sec_per_pass': '26.896', 'throughput/steps_per_sec': '0.029', 'throughput/samples_per_sec': '0.595', 'throughput/sec_per_step': '34.501', 'throughput/max_sequences_per_micro_batch': '8.000', 'throughput/min_sequences_per_micro_batch': '8.000', 'throughput/sequences_per_micro_batch': '8.000', 'rl/loss': '-188.681', 'rl/max_loss': '6.562', 'rl/min_loss': '-114.067', 'rl/reward': '0.000', 'rl/max_reward': '0.000', 'rl/min_reward': '0.000', 'rl/entropy': '1.563', 'rl/old_logprobs': '-1.548', 'rl/new_logprobs': '-1.548', 'rl/ref_logprobs': '-1.548', 'rl/advantage': '-0.102', 'rl/max_advantage': '0.602', 'rl/min_advantage': '-0.684', 'rl/kl': '9.267e-04', 'rl/max_kl': '0.377', 'rl/min_kl': '-5.960e-08', 'rl/policy_loss': '3.353', 'rl/surr1': '0.000', 'rl/surr2': '0.000', 'rl/ratio_new_old': '1.001', 'rl/ratio_new_old_sum': '32742.777', 'rl/ratio_new_old_squared_sum': '32775.941', 'rl/ratio_ref_new': '1.001', 'rl/ratio_ref_old': '1.000', 'rl/clamp_log_ratio_ref_new_indicator': '0.000', 'rl/clamp_log_ratio_new_old_indicator': '0.000', 'rl/num_nans': '0.000', 'rl/token_weight': '0.062', 'rl/max_token_weight': '0.062', 'rl/min_token_weight': '0.062', 'rl/kl_coef': '0.000', 'rl/entropy_bonus_coef': '0.000', 'rl/num_output_tokens_sum': '32748.000', 'rl/input_size': '2131.375', 'rl/value_mean': '0.102', 'rl/value_max': '0.684', 'rl/value_min': '-0.602', 'rl/value_loss': '28.193', 'rl/value_mse': '0.037', 'rl/ess': '0.999'}

2025-09-19 21:31:47

[finetune]: 09/19/2025 21:31:47.970 - INFO - pipelinerl.finetune_loop - Start gathering and sending ZeRO Stage 3 weights

2025-09-19 21:31:47

[finetune]: 09/19/2025 21:31:47.973 - INFO - pipelinerl.finetune_loop - Published weight update request for version 96

2025-09-19 21:31:48

[finetune]: 09/19/2025 21:31:48.050 - INFO - pipelinerl.finetune_loop - Wait for HTTP requests

2025-09-19 21:31:48

[finetune]: 09/19/2025 21:31:48.410 - INFO - pipelinerl.finetune_loop - Finished broadcasting weights

2025-09-19 21:31:48

[finetune]: 09/19/2025 21:31:48.413 - INFO - pipelinerl.finetune_loop - Start step at 1758317508.4139087

2025-09-19 21:31:48

[finetune]: 09/19/2025 21:31:48.514 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.1

2025-09-19 21:31:48

[finetune]: 09/19/2025 21:31:48.664 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.15000000000000002

2025-09-19 21:31:48

[finetune]: 09/19/2025 21:31:48.890 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.22500000000000003

2025-09-19 21:31:49

[finetune]: 09/19/2025 21:31:49.228 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.3375

2025-09-19 21:31:49

[finetune]: 09/19/2025 21:31:49.735 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.5062500000000001