apiche

Apiche's group workspace

Group: debug_dataset_buffer_size_again

9476

10433

23969

1-3

of 3

Timestamps visible

2025-07-18 18:53:37

[finetune]: 07/18/2025 18:53:37.133 - INFO - pipelinerl.finetune_loop - All workers synchronized successfully

2025-07-18 18:53:37

[finetune]: 07/18/2025 18:53:37.358 - INFO - pipelinerl.finetune_loop - Completed steps 1: {'stats/lr': '0.000', 'stats/grad_norm': '4.300', 'stats/samples': '256.000', 'stats/tokens': '169100.000', 'stats/samples_too_old_to_queue': '0.000', 'stats/samples_too_old_to_train': '0.000', 'stats/passes': '8.000', 'stats/completed_steps': '1.000', 'stats/epoch': '0.000', 'stats/min_actor_version': '0.000', 'stats/max_actor_version': '0.000', 'stats/queue/batches': '1.000', 'stats/time_waiting_for_data': '78.221', 'stats/lag': '0.000', 'throughput/tokens_perGPU_per_sec': '6178.179', 'throughput/tokens_per_step': '169100.000', 'throughput/micro_batches_per_step': '8.000', 'throughput/min_tokens_per_micro_batch': '2671.000', 'throughput/max_tokens_per_micro_batch': '11960.000', 'throughput/tokens_per_micro_batch': '10568.750', 'throughput/tokens_per_sec': '12356.359', 'throughput/real_tokens_per_sec': '919.601', 'throughput/sec_per_pass': '1.711', 'throughput/steps_per_sec': '0.011', 'throughput/samples_per_sec': '18.706', 'throughput/sec_per_step': '91.942', 'throughput/max_sequences_per_micro_batch': '27.000', 'throughput/min_sequences_per_micro_batch': '3.000', 'throughput/sequences_per_micro_batch': '16.000', 'rl/loss': '64.630', 'rl/max_loss': '7.972', 'rl/min_loss': '0.201', 'rl/reward': '0.191', 'rl/max_reward': '1.000', 'rl/min_reward': '0.000', 'rl/entropy': '0.534', 'rl/old_logprobs': '-0.537', 'rl/new_logprobs': '-0.538', 'rl/ref_logprobs': '-0.537', 'rl/advantage': '0.192', 'rl/max_advantage': '1.508', 'rl/min_advantage': '-0.516', 'rl/kl': '0.000', 'rl/max_kl': '0.107', 'rl/min_kl': '-0.000', 'rl/policy_loss': '-0.109', 'rl/surr1': '0.000', 'rl/surr2': '0.000', 'rl/ratio_new_old': '1.000', 'rl/ratio_new_old_sum': '151275.969', 'rl/ratio_new_old_squared_sum': '151395.375', 'rl/ratio_ref_new': '1.001', 'rl/ratio_ref_old': '1.000', 'rl/clamp_log_ratio_ref_new_indicator': '0.000', 'rl/clamp_log_ratio_new_old_indicator': '0.000', 'rl/num_nans': '0.000', 'rl/token_weight': '0.004', 'rl/max_token_weight': '0.004', 'rl/min_token_weight': '0.004', 'rl/kl_coef': '0.000', 'rl/entropy_bonus_coef': '0.000', 'rl/num_output_tokens_sum': '151283.000', 'rl/input_size': '693.418', 'rl/value_mean': '-0.000', 'rl/value_max': '0.516', 'rl/value_min': '-0.570', 'rl/value_loss': '0.211', 'rl/value_mse': '0.208', 'rl/ess': '0.999'}

2025-07-18 18:53:37

[finetune]: 07/18/2025 18:53:37.360 - INFO - pipelinerl.finetune_loop - Start step at 1752864817.3600051

2025-07-18 18:53:46

wandb: WARNING Tried to log to step 1 that is less than the current step 15. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.