kastan

Kastan's group workspace

Group: Aug-05__14:10

1-20

of 32

Timestamps visible

2022-08-05 19:12:14

Wandb run id: 3ho4lznl

2022-08-05 19:12:14

[08/05/22 14:11:54] INFO colossalai - colossalai - INFO:

2022-08-05 19:12:14

/u/kastanday/LLM-Distributed-Quantization/wandb_log

2022-08-05 19:12:14

s/custom_wandb_log_hook.py:122 before_train

2022-08-05 19:12:14

INFO colossalai - colossalai - INFO: training starts

2022-08-05 19:12:14

wandb: WARNING Saving files without folders. If you want to preserve sub directories pass base_path to wandb.save, i.e. wandb.save("/mnt/folder/file.h5", base_path="/mnt")

2022-08-05 19:12:22

wandb: 429 encountered (Filestream rate limit exceeded, retrying in 2.20787936935772 seconds), retrying request

2022-08-05 19:12:26

wandb: 429 encountered (Filestream rate limit exceeded, retrying in 2.37554088843196 seconds), retrying request

2022-08-05 19:12:32

wandb: 429 encountered (Filestream rate limit exceeded, retrying in 4.525697612288115 seconds), retrying request

2022-08-05 19:13:34

❌ Error logging layer norms -- check message: 'NaiveAMPModel' object has no attribute 'embed'

2022-08-05 19:19:19

wandb: 429 encountered (Filestream rate limit exceeded, retrying in 2.0878371505079767 seconds), retrying request

2022-08-05 19:27:20

wandb: 429 encountered (Filestream rate limit exceeded, retrying in 2.4043702007959618 seconds), retrying request