apiche

Apiche's group workspace

Group: debug_gspo18

251

1-3

of 3

Timestamps visible

2025-10-02 18:46:04

[finetune]: 10/02/2025 18:46:04.944 - INFO - pipelinerl.finetune_loop - Starting training with 0 samples, 4 samples per lead trainer, 16 samples per step, 4 gradient accumulation passes per lead trainer

2025-10-02 18:46:04

[finetune]: 10/02/2025 18:46:04.946 - INFO - pipelinerl.finetune_loop - Start step at 1759430764.9467325

2025-10-02 18:46:04

[finetune]: 10/02/2025 18:46:04.947 - WARNING - pipelinerl.streams - Waiting for training_data/0/0 to be created

2025-10-02 18:46:05

[finetune]: 10/02/2025 18:46:05.047 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.1

2025-10-02 18:46:05

[finetune]: 10/02/2025 18:46:05.197 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.15000000000000002

2025-10-02 18:46:05

[finetune]: 10/02/2025 18:46:05.423 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.22500000000000003

2025-10-02 18:46:05

[finetune]: 10/02/2025 18:46:05.761 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.3375

2025-10-02 18:46:06

[finetune]: 10/02/2025 18:46:06.268 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.5062500000000001

2025-10-02 18:46:07

[finetune]: 10/02/2025 18:46:07.028 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.7593750000000001

2025-10-02 18:46:07

[finetune]: 10/02/2025 18:46:07.948 - INFO - pipelinerl.finetune_loop - Starting data loader for stream training_data on partition 0

2025-10-02 18:46:08

[finetune]: 10/02/2025 18:46:08.167 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 1.1390625

2025-10-02 18:46:09

[finetune]: 10/02/2025 18:46:09.876 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 1.7085937500000001