eleutherai

Igoro's group workspace

Group: final_image_test_0_11lvrrvg

172

1-12

of 12

Timestamps visible

2021-11-02 22:57:05

make: Leaving directory '/home/mchorse/gpt-neox/megatron/data'

2021-11-02 22:57:07

[2021-11-02 22:57:06,807] [WARNING] [config.py:77:_sanity_check] DeepSpeedConfig: cpu_offload is deprecated. Please use offload_optimizer.

2021-11-02 22:57:13

Traceback (most recent call last):

2021-11-02 22:57:13

File "train.py", line 27, in <module>

2021-11-02 22:57:13

pretrain(neox_args=neox_args)

2021-11-02 22:57:13

File "/home/mchorse/gpt-neox/megatron/training.py", line 82, in pretrain

2021-11-02 22:57:13

model, optimizer, lr_scheduler = setup_model_and_optimizer(

2021-11-02 22:57:13

File "/home/mchorse/gpt-neox/megatron/training.py", line 399, in setup_model_and_optimizer

2021-11-02 22:57:13

model, optimizer, _, lr_scheduler = deepspeed.initialize(

2021-11-02 22:57:13

File "/usr/local/lib/python3.8/dist-packages/deepspeed/__init__.py", line 128, in initialize

2021-11-02 22:57:13

engine = PipelineEngine(args=args,

2021-11-02 22:57:13

File "/usr/local/lib/python3.8/dist-packages/deepspeed/runtime/pipe/engine.py", line 60, in __init__

2021-11-02 22:57:13

super().__init__(*super_args, **super_kwargs)

2021-11-02 22:57:13

File "/usr/local/lib/python3.8/dist-packages/deepspeed/runtime/engine.py", line 192, in __init__

2021-11-02 22:57:13

self._configure_optimizer(optimizer, model_parameters)

2021-11-02 22:57:13

File "/usr/local/lib/python3.8/dist-packages/deepspeed/runtime/engine.py", line 674, in _configure_optimizer

2021-11-02 22:57:13

assert self.zero_allow_untested_optimizer(), \

2021-11-02 22:57:13

AssertionError: You are using an untested ZeRO Optimizer. Please add <"zero_allow_untested_optimizer": true> in the configuration file to use it.