eleutherai

Igoro's group workspace

Group: 20B_eval

317

845

231

100862

6546

8156

1-8

of 8

Timestamps visible

2021-11-24 02:30:23

eval exists

2021-11-24 02:30:23

Using existing pod: eval

2021-11-24 02:30:23

* 127.0.0.1:/home/sid/Code/pyfra_scripts$ kubectl exec -it eval-0 -- python -c "import yaml;yaml.dump({'pipe-parallel-size': 4, 'model-parallel-size': 2, 'num-layers': 44, 'hidden-size': 6144, 'num-attention-heads': 64, 'seq-length': 2048, 'max-position-embeddings': 2048, 'norm': 'layernorm', 'pos-emb': 'rotary', 'rotary_pct': 0.25, 'no-weight-tying': True, 'gpt_j_residual': True, 'output_layer_parallelism': 'column', 'scaled-upper-triang-masked-softmax-fusion': True, 'bias-gelu-fusion': True, 'init_method': 'small_init', 'output_layer_init_method': 'wang_init', 'optimizer': {'type': 'Adam', 'params': {'lr': 9.7e-05, 'betas': [0.9, 0.95], 'eps': 1e-08}}, 'min_lr': 9.7e-06, 'zero_optimization': {'stage': 1, 'allgather_partitions': True, 'allgather_bucket_size': 1260000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 1260000000, 'contiguous_gradients': True, 'cpu_offload': False}, 'train_micro_batch_size_per_gpu': 4, 'gradient_accumulation_steps': 32, 'data-impl': 'mmap', 'split': '995,4,1', 'checkpoint-activations': True, 'checkpoint-num-layers': 1, 'partition-activations': False, 'synchronize-each-layer': True, 'gradient_clipping': 1.0, 'weight-decay': 0.01, 'hidden-dropout': 0, 'attention-dropout': 0, 'fp16': {'fp16': True, 'enabled': True, 'loss_scale': 0, 'loss_scale_window': 1000, 'initial_scale_power': 12, 'hysteresis': 2, 'min_loss_scale': 1}, 'train-iters': 150000, 'lr-decay-iters': 150000, 'distributed-backend': 'nccl', 'lr-decay-style': 'cosine', 'warmup': 0.01, 'save-interval': 1000, 'eval-interval': 1000, 'eval-iters': 10, 'log-interval': 2, 'steps_per_print': 2, 'wall_clock_breakdown': False, 'tokenizer_type': 'HFTokenizer', 'vocab-file': '/mnt/ssd-1/data/20B_tokenizer.json', 'data-path': '/mnt/ssd-1/data/pile_20B_tokenizer/pile_20B_tokenizer_text_document', 'save': '/mnt/ssd-1/20B_checkpoints', 'load': '/mnt/ssd-1/20B_checkpoints', 'tensorboard-dir': '/mnt/ssd-1/tensorboard', 'log-dir': '/mnt/ssd-1/logs', 'wandb_team': 'eleutherai', 'wandb_project': 'gpt-thicc', 'eval_results_prefix': '68186e4b-56c2-4d12-a21a-c8619eb8b39e'}, open('/home/mchorse/gpt-neox/configs/tmp.yml', 'w'))"

2021-11-24 02:30:25

./deepy.py evaluate.py -d configs tmp --eval_tasks lambada piqa hellaswag winogrande mathqa pubmedqa