apiche

Apiche's group workspace

Group: debug_gspo17

618

296

1613

1-3

of 3

Timestamps visible

2025-10-02 18:43:28

[finetune]: 10/02/2025 18:43:28.143 - INFO - pipelinerl.finetune_loop - Published weight update request for version 0

2025-10-02 18:43:33

[finetune]: 10/02/2025 18:43:33.590 - INFO - pipelinerl.finetune_loop - Wait for HTTP requests

2025-10-02 18:43:33

[finetune]: 10/02/2025 18:43:33.834 - INFO - pipelinerl.finetune_loop - Finished broadcasting weights

2025-10-02 18:43:33

[finetune]: 10/02/2025 18:43:33.843 - INFO - pipelinerl.finetune_loop - Start training

2025-10-02 18:43:33

[finetune]: 10/02/2025 18:43:33.846 - INFO - pipelinerl.finetune_loop - Starting training with 0 samples, 4 samples per lead trainer, 16 samples per step, 4 gradient accumulation passes per lead trainer

2025-10-02 18:43:33

[finetune]: 10/02/2025 18:43:33.846 - WARNING - pipelinerl.streams - Waiting for training_data/0/0 to be created

2025-10-02 18:43:33

[finetune]: 10/02/2025 18:43:33.849 - INFO - pipelinerl.finetune_loop - Start step at 1759430613.8491042

2025-10-02 18:43:33

[finetune]: 10/02/2025 18:43:33.950 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.1

2025-10-02 18:43:34

[finetune]: 10/02/2025 18:43:34.100 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.15000000000000002

2025-10-02 18:43:34

[finetune]: 10/02/2025 18:43:34.326 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.22500000000000003

2025-10-02 18:43:34

[finetune]: 10/02/2025 18:43:34.664 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.3375

2025-10-02 18:43:35

[finetune]: 10/02/2025 18:43:35.170 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.5062500000000001

2025-10-02 18:43:35

[finetune]: 10/02/2025 18:43:35.930 - INFO - pipelinerl.finetune_loop - Batch queue is empty, retrying with timeout 0.7593750000000001