Skip to main content
kastan
Projects
col_ai
Run groups
gpt2_pp_2gpu
Logs
Group
Workspace
Runs
Logs
Files
Artifacts

Kastan's group workspace

Timestamps visible
2022-06-21 19:09:37
made it to before build optimizer
2022-06-21 19:09:37
made it to before engine init
2022-06-21 19:09:37
[06/21/22 15:09:35] WARNING  colossalai - colossalai - WARNING:
2022-06-21 19:09:37
                             /u/kastan/.conda/envs/pytorch_1.11_colossal/lib/python3.8/site-packages/colossalai/initialize.py:304 initialize
2022-06-21 19:09:37
                    WARNING  colossalai - colossalai - WARNING: Initializing an non ZeRO model with optimizer class
2022-06-21 19:09:37
Watching model!
2022-06-21 19:09:37
                    INFO     colossalai - colossalai - INFO: /u/kastan/.conda/envs/pytorch_1.11_colossal/lib/python3.8/site-packages/colossalai/utils/memory.py:91
2022-06-21 19:09:37
                             report_memory_usage
2022-06-21 19:09:37
                    INFO     colossalai - colossalai - INFO: Before-train: GPU: allocated 946.51 MB, max allocated 1020.5 MB, cached: 1068.0 MB, max cached: 1068.0
2022-06-21 19:09:37
                             MB
2022-06-21 19:09:37
                    INFO     colossalai - colossalai - INFO: train_gpt.py:223 before_train
2022-06-21 19:09:37
                    INFO     colossalai - colossalai - INFO: training starts
2022-06-21 19:10:21
Error when collecting samples_per_sec, Tflops... not enough values to unpack (expected 2, got 1)
2022-06-21 19:14:03
Error when collecting samples_per_sec, Tflops... not enough values to unpack (expected 2, got 1)
2022-06-21 19:21:26
Error when collecting samples_per_sec, Tflops... not enough values to unpack (expected 2, got 1)
2022-06-21 19:23:30
Error when collecting samples_per_sec, Tflops... not enough values to unpack (expected 2, got 1)
2022-06-21 19:26:04
Error when collecting samples_per_sec, Tflops... not enough values to unpack (expected 2, got 1)