apiche

Apiche's group workspace

Group: w_quant_vllm_2

282

1-3

of 3

Timestamps visible

2025-09-19 21:52:40

[finetune]: 09/19/2025 21:52:40.838 - INFO - pipelinerl.finetune_loop - Completed steps 2: {'stats/lr': '1.000e-06', 'stats/grad_norm': '86.437', 'stats/samples': '32.000', 'stats/tokens': '11160.000', 'stats/samples_too_old_to_queue': '0.000', 'stats/samples_too_old_to_train': '0.000', 'stats/passes': '2.000', 'stats/completed_steps': '2.000', 'stats/epoch': '0.000', 'stats/min_actor_version': '16.000', 'stats/max_actor_version': '16.000', 'stats/queue/batches': '0.000', 'stats/time_waiting_for_data': '11.948', 'stats/lag': '0.000', 'throughput/tokens_perGPU_per_sec': '261.823', 'throughput/tokens_per_step': '4998.000', 'throughput/micro_batches_per_step': '1.000', 'throughput/min_tokens_per_micro_batch': '2499.000', 'throughput/max_tokens_per_micro_batch': '2499.000', 'throughput/tokens_per_micro_batch': '2499.000', 'throughput/tokens_per_sec': '523.647', 'throughput/real_tokens_per_sec': '208.737', 'throughput/sec_per_pass': '9.545', 'throughput/steps_per_sec': '0.084', 'throughput/samples_per_sec': '1.676', 'throughput/sec_per_step': '11.972', 'throughput/max_sequences_per_micro_batch': '8.000', 'throughput/min_sequences_per_micro_batch': '8.000', 'throughput/sequences_per_micro_batch': '8.000', 'rl/loss': '-207.993', 'rl/max_loss': '-42.762', 'rl/min_loss': '-165.230', 'rl/reward': '0.000', 'rl/max_reward': '0.000', 'rl/min_reward': '0.000', 'rl/entropy': '2.793', 'rl/old_logprobs': '-2.924', 'rl/new_logprobs': '-2.921', 'rl/ref_logprobs': '-2.924', 'rl/advantage': '-0.136', 'rl/max_advantage': '0.625', 'rl/min_advantage': '-0.734', 'rl/kl': '0.001', 'rl/max_kl': '0.114', 'rl/min_kl': '-5.960e-08', 'rl/policy_loss': '7.879', 'rl/surr1': '0.000', 'rl/surr2': '0.000', 'rl/ratio_new_old': '1.004', 'rl/ratio_new_old_sum': '7386.515', 'rl/ratio_new_old_squared_sum': '7407.947', 'rl/ratio_ref_new': '0.998', 'rl/ratio_ref_old': '1.000', 'rl/clamp_log_ratio_ref_new_indicator': '0.000', 'rl/clamp_log_ratio_new_old_indicator': '0.000', 'rl/num_nans': '0.000', 'rl/token_weight': '0.062', 'rl/max_token_weight': '0.062', 'rl/min_token_weight': '0.062', 'rl/kl_coef': '0.000', 'rl/entropy_bonus_coef': '0.000', 'rl/num_output_tokens_sum': '7382.000', 'rl/input_size': '573.125', 'rl/value_mean': '0.136', 'rl/value_max': '0.734', 'rl/value_min': '-0.625', 'rl/value_loss': '12.309', 'rl/value_mse': '0.053', 'rl/ess': '0.998'}

2025-09-19 21:52:40

[finetune]: 09/19/2025 21:52:40.839 - INFO - pipelinerl.finetune_loop - Start gathering and sending ZeRO Stage 3 weights

2025-09-19 21:52:40

[finetune]: 09/19/2025 21:52:40.842 - INFO - pipelinerl.finetune_loop - Published weight update request for version 32

2025-09-19 21:52:40

[finetune]: 09/19/2025 21:52:40.911 - INFO - pipelinerl.finetune_loop - Wait for HTTP requests

2025-09-19 21:52:41

[finetune]: 09/19/2025 21:52:41.177 - INFO - pipelinerl.finetune_loop - Finished broadcasting weights

2025-09-19 21:52:41

[finetune]: 09/19/2025 21:52:41.180 - INFO - pipelinerl.finetune_loop - Start step at 1758318761.1809225

2025-09-19 21:52:41

[finetune]: 09/19/2025 21:52:41.281 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.1

2025-09-19 21:52:41

[finetune]: 09/19/2025 21:52:41.432 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.15000000000000002

2025-09-19 21:52:41

[finetune]: 09/19/2025 21:52:41.657 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.22500000000000003

2025-09-19 21:52:41

[finetune]: 09/19/2025 21:52:41.995 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.3375

2025-09-19 21:52:42

[finetune]: 09/19/2025 21:52:42.502 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.5062500000000001

2025-09-19 21:52:43

[finetune]: 09/19/2025 21:52:43.262 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.7593750000000001

2025-09-19 21:52:44

[finetune]: 09/19/2025 21:52:44.402 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 1.1390625