apiche

Apiche's group workspace

Group: debug_gspo14

2487

46436

1045

1-3

of 3

Timestamps visible

2025-09-28 05:17:54

[finetune]: 09/28/2025 05:17:53.997 - INFO - pipelinerl.finetune_loop - Start gathering and sending ZeRO Stage 3 weights

2025-09-28 05:17:54

[finetune]: 09/28/2025 05:17:54.000 - INFO - pipelinerl.finetune_loop - Published weight update request for version 624

2025-09-28 05:17:54

[finetune]: 09/28/2025 05:17:54.080 - INFO - pipelinerl.finetune_loop - Wait for HTTP requests

2025-09-28 05:17:54

[finetune]: 09/28/2025 05:17:54.366 - INFO - pipelinerl.finetune_loop - Finished broadcasting weights

2025-09-28 05:17:54

[finetune]: 09/28/2025 05:17:54.370 - INFO - pipelinerl.finetune_loop - Start step at 1759036674.37027

2025-09-28 05:17:55

Invalidate trace cache @ step 322: expected module 1, but got module 320

2025-09-28 05:17:55

wandb: WARNING Tried to log to step 33 that is less than the current step 122. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.

2025-09-28 05:17:55

wandb: WARNING Tried to log to step 34 that is less than the current step 122. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.

2025-09-28 05:17:55

wandb: WARNING Tried to log to step 35 that is less than the current step 122. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.

2025-09-28 05:17:55

wandb: WARNING Tried to log to step 36 that is less than the current step 122. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.

2025-09-28 05:17:55

wandb: WARNING Tried to log to step 37 that is less than the current step 122. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.

2025-09-28 05:17:55

wandb: WARNING Tried to log to step 38 that is less than the current step 122. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.

2025-09-28 05:17:55

wandb: WARNING Tried to log to step 39 that is less than the current step 122. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.