eleutherai

Schoelkopf's group workspace

Group: mamba-tp1-bs16-fusedinner_r7kwr8b2_11cu2i6r

1-4

of 4

Tags

Notes

Author

schoelkopf

State

Crashed

Start time

March 13th, 2024 11:32:11 AM

Runtime

13m 30s

Tracked hours

Run path

eleutherai/mamba-neox-tp-memsavings/ysuqskz9

Linux-5.15.0-1037-aws-x86_64-with-glibc2.31

Python version

3.9.18

Git repository

git clone https://github.com/EleutherAI/gpt-neox

Git state

git checkout -b "ip-10-0-241-47-0" 696454f00c71c702a10f5a0e2f28ebbe065e2704

Command

/weka/hailey/mistral-support-neox/gpt-neox/train.py --deepspeed_config eyJ0cmFpbl9iYXRjaF9zaXplIjogMTAyNCwgInRyYWluX21pY3JvX2JhdGNoX3NpemVfcGVyX2dwdSI6IDE2LCAiZ3JhZGllbnRfYWNjdW11bGF0aW9uX3N0ZXBzIjogMiwgIm9wdGltaXplciI6IHsidHlwZSI6ICJBZGFtIiwgInBhcmFtcyI6IHsibHIiOiAwLjAwMDYsICJiZXRhcyI6IFswLjksIDAuOTVdLCAiZXBzIjogMWUtMDh9fSwgImZwMTYiOiB7ImZwMTYiOiB0cnVlLCAiZW5hYmxlZCI6IHRydWUsICJsb3NzX3NjYWxlIjogMCwgImxvc3Nfc2NhbGVfd2luZG93IjogMTAwMCwgImluaXRpYWxfc2NhbGVfcG93ZXIiOiAxMiwgImh5c3RlcmVzaXMiOiAyLCAibWluX2xvc3Nfc2NhbGUiOiAxfSwgInplcm9fb3B0aW1pemF0aW9uIjogeyJzdGFnZSI6IDEsICJhbGxnYXRoZXJfcGFydGl0aW9ucyI6IHRydWUsICJhbGxnYXRoZXJfYnVja2V0X3NpemUiOiA1MDAwMDAwMDAsICJvdmVybGFwX2NvbW0iOiB0cnVlLCAicmVkdWNlX3NjYXR0ZXIiOiB0cnVlLCAicmVkdWNlX2J1Y2tldF9zaXplIjogNTAwMDAwMDAwLCAiY29udGlndW91c19ncmFkaWVudHMiOiB0cnVlLCAiY3B1X29mZmxvYWQiOiBmYWxzZX0sICJ3YWxsX2Nsb2NrX2JyZWFrZG93biI6IHRydWV9 --megatron_config eyJsYXVuY2hlciI6ICJzbHVybSIsICJub19zc2hfY2hlY2siOiB0cnVlLCAidHJhaW5fYmF0Y2hfc2l6ZSI6IDEwMjQsICJ0cmFpbl9taWNyb19iYXRjaF9zaXplX3Blcl9ncHUiOiAxNiwgImdyYWRpZW50X2FjY3VtdWxhdGlvbl9zdGVwcyI6IDIsICJvcHRpbWl6ZXIiOiB7InR5cGUiOiAiQWRhbSIsICJwYXJhbXMiOiB7ImxyIjogMC4wMDA2LCAiYmV0YXMiOiBbMC45LCAwLjk1XSwgImVwcyI6IDFlLTA4fX0sICJmcDE2IjogeyJmcDE2IjogdHJ1ZSwgImVuYWJsZWQiOiB0cnVlLCAibG9zc19zY2FsZSI6IDAsICJsb3NzX3NjYWxlX3dpbmRvdyI6IDEwMDAsICJpbml0aWFsX3NjYWxlX3Bvd2VyIjogMTIsICJoeXN0ZXJlc2lzIjogMiwgIm1pbl9sb3NzX3NjYWxlIjogMX0sICJ6ZXJvX29wdGltaXphdGlvbiI6IHsic3RhZ2UiOiAxLCAiYWxsZ2F0aGVyX3BhcnRpdGlvbnMiOiB0cnVlLCAiYWxsZ2F0aGVyX2J1Y2tldF9zaXplIjogNTAwMDAwMDAwLCAib3ZlcmxhcF9jb21tIjogdHJ1ZSwgInJlZHVjZV9zY2F0dGVyIjogdHJ1ZSwgInJlZHVjZV9idWNrZXRfc2l6ZSI6IDUwMDAwMDAwMCwgImNvbnRpZ3VvdXNfZ3JhZGllbnRzIjogdHJ1ZSwgImNwdV9vZmZsb2FkIjogZmFsc2V9LCAid2FsbF9jbG9ja19icmVha2Rvd24iOiB0cnVlLCAicHJlY2lzaW9uIjogImZwMTYiLCAibnVtX2xheWVycyI6IDI0LCAiaGlkZGVuX3NpemUiOiA3NjgsICJudW1fYXR0ZW50aW9uX2hlYWRzIjogMTIsICJzZXFfbGVuZ3RoIjogMjA0OCwgIm1heF9wb3NpdGlvbl9lbWJlZGRpbmdzIjogMjA0OCwgIm5vcm0iOiAicm1zbm9ybSIsICJybXNfbm9ybV9lcHNpbG9uIjogMWUtMDUsICJwb3NfZW1iIjogInJvdGFyeSIsICJub193ZWlnaHRfdHlpbmciOiB0cnVlLCAiYXR0ZW50aW9uX2NvbmZpZyI6IFsibWFtYmEiLCAibWFtYmEiLCAibWFtYmEiLCAibWFtYmEiLCAibWFtYmEiLCAibWFtYmEiLCAibWFtYmEiLCAibWFtYmEiLCAibWFtYmEiLCAibWFtYmEiLCAibWFtYmEiLCAibWFtYmEiLCAibWFtYmEiLCAibWFtYmEiLCAibWFtYmEiLCAibWFtYmEiLCAibWFtYmEiLCAibWFtYmEiLCAibWFtYmEiLCAibWFtYmEiLCAibWFtYmEiLCAibWFtYmEiLCAibWFtYmEiLCAibWFtYmEiXSwgInNwYXJzaXR5X2NvbmZpZyI6IHt9LCAiYWN0aXZhdGlvbiI6ICJzaWx1IiwgInJvdGFyeV9wY3QiOiAwLjI1LCAib3V0cHV0X2xheWVyX2luaXRfbWV0aG9kIjogInNpbmdsZV9yZXNpZHVhbF9zY2FsZWRfbm9ybWFsIiwgImdwdF9qX3Jlc2lkdWFsIjogdHJ1ZSwgIm1hbWJhX3NlbGVjdGl2ZV9zY2FuX2Z1c2lvbiI6IHRydWUsICJtYW1iYV9jYXVzYWxfY29udl9mdXNpb24iOiB0cnVlLCAibWFtYmFfaW5uZXJfZnVuY19mdXNpb24iOiB0cnVlLCAibHJfZGVjYXlfc3R5bGUiOiAiY29zaW5lIiwgImxyX2RlY2F5X2l0ZXJzIjogMTQzMDAwLCAibWluX2xyIjogNmUtMDUsICJvcHRpbWl6ZXJfdHlwZSI6ICJBZGFtIiwgInplcm9fc3RhZ2UiOiAxLCAiemVyb19yZWR1Y2Vfc2NhdHRlciI6IHRydWUsICJ6ZXJvX2NvbnRpZ3VvdXNfZ3JhZGllbnRzIjogdHJ1ZSwgInplcm9fcmVkdWNlX2J1Y2tldF9zaXplIjogNTAwMDAwMDAwLCAiemVyb19hbGxnYXRoZXJfYnVja2V0X3NpemUiOiA1MDAwMDAwMDAsICJsciI6IDAuMDAwNiwgInRva2VuaXplcl90eXBlIjogIkhGVG9rZW5pemVyIiwgInRyYWluX2RhdGFfcGF0aHMiOiBbIi93ZWthL3BpbGUvcGlsZV8yMEJfdG9rZW5pemVyX3RleHRfZG9jdW1lbnQiXSwgInRlc3RfZGF0YV9wYXRocyI6IFsiL3dla2EvcGlsZS9waWxlXzIwQl90b2tlbml6ZXJfdGV4dF9kb2N1bWVudCJdLCAidmFsaWRfZGF0YV9wYXRocyI6IFsiL3dla2EvcGlsZS9waWxlXzIwQl90b2tlbml6ZXJfdGV4dF9kb2N1bWVudCJdLCAidHJhaW5fZGF0YV93ZWlnaHRzIjogWzEuMF0sICJ2YWxpZF9kYXRhX3dlaWdodHMiOiBbMS4wXSwgInRlc3RfZGF0YV93ZWlnaHRzIjogWzEuMF0sICJkYXRhX2ltcGwiOiAibW1hcCIsICJjb25maWdfZmlsZXMiOiB7Im1hbWJhLTE2MG0ueW1sIjogIntcbiAgXCJwaXBlX3BhcmFsbGVsX3NpemVcIjogMCxcbiAgXCJtb2RlbF9wYXJhbGxlbF9zaXplXCI6IDEsXG5cbiAgXCJudW1fbGF5ZXJzXCI6IDI0LFxuICBcImhpZGRlbl9zaXplXCI6IDc2OCxcbiAgXCJudW1fYXR0ZW50aW9uX2hlYWRzXCI6IDEyLFxuICBcInNlcV9sZW5ndGhcIjogMjA0OCxcbiAgXCJtYXhfcG9zaXRpb25fZW1iZWRkaW5nc1wiOiAyMDQ4LFxuICBcInBvc19lbWJcIjogXCJyb3RhcnlcIixcbiAgXCJyb3RhcnlfcGN0XCI6IDAuMjUsXG4gIFwibm9fd2VpZ2h0X3R5aW5nXCI6IHRydWUsXG4gIFwiZ3B0X2pfcmVzaWR1YWxcIjogdHJ1ZSxcbiAgXCJvdXRwdXRfbGF5ZXJfcGFyYWxsZWxpc21cIjogXCJjb2x1bW5cIixcblxuICBcImF0dGVudGlvbl9jb25maWdcIjogW1tbXCJtYW1iYVwiXSwgMjRdXSxcblxuICAjIFwic2NhbGVkX3VwcGVyX3RyaWFuZ19tYXNrZWRfc29mdG1heF9mdXNpb25cIjogdHJ1ZSxcbiAgIyBcImJpYXNfZ2VsdV9mdXNpb25cIjogdHJ1ZSxcblxuICBcIm1hbWJhX3NlbGVjdGl2ZV9zY2FuX2Z1c2lvblwiOiB0cnVlLFxuICBcIm1hbWJhX2NhdXNhbF9jb252X2Z1c2lvblwiOiB0cnVlLFxuICBcIm1hbWJhX2lubmVyX2Z1bmNfZnVzaW9uXCI6IHRydWUsXG4gIFwibWFtYmFfc2VsZWN0aXZlX2ZwMzJfcGFyYW1zXCI6IHRydWUsXG5cbiAgXCJhY3RpdmF0aW9uXCI6IFwic2lsdVwiLFxuICBcIm5vcm1cIjogXCJybXNub3JtXCIsXG4gIFwicm1zX25vcm1fZXBzaWxvblwiOiAxLjBlLTUsXG5cbiAgXCJvdXRwdXRfbGF5ZXJfaW5pdF9tZXRob2RcIjogXCJzaW5nbGVfcmVzaWR1YWxfc2NhbGVkX25vcm1hbFwiLFxuXG5cbiAgIyBcImluaXRfbWV0aG9kXCI6IFwic21hbGxfaW5pdFwiLFxuICAjIFwib3V0cHV0X2xheWVyX2luaXRfbWV0aG9kXCI6IFwid2FuZ19pbml0XCIsXG5cbiAgXCJvcHRpbWl6ZXJcIjoge1xuICAgIFwidHlwZVwiOiBcIkFkYW1cIixcbiAgICBcInBhcmFtc1wiOiB7XG4gICAgICBcImxyXCI6IDAuMDAwNixcbiAgICAgIFwiYmV0YXNcIjogWzAuOSwgMC45NV0sXG4gICAgICBcImVwc1wiOiAxLjBlLThcbiAgICB9XG4gIH0sXG4gIFwibWluX2xyXCI6IDAuMDAwMDYsXG5cbiAgXCJ6ZXJvX29wdGltaXphdGlvblwiOiB7XG4gICAgXCJzdGFnZVwiOiAxLFxuICAgIFwiYWxsZ2F0aGVyX3BhcnRpdGlvbnNcIjogdHJ1ZSxcbiAgICBcImFsbGdhdGhlcl9idWNrZXRfc2l6ZVwiOiA1MDAwMDAwMDAsXG4gICAgXCJvdmVybGFwX2NvbW1cIjogdHJ1ZSxcbiAgICBcInJlZHVjZV9zY2F0dGVyXCI6IHRydWUsXG4gICAgXCJyZWR1Y2VfYnVja2V0X3NpemVcIjogNTAwMDAwMDAwLFxuICAgIFwiY29udGlndW91c19ncmFkaWVudHNcIjogdHJ1ZSxcbiAgICBcImNwdV9vZmZsb2FkXCI6IGZhbHNlXG4gIH0sXG5cbiAgXCJ0cmFpbl9taWNyb19iYXRjaF9zaXplX3Blcl9ncHVcIjogMTYsXG4gIFwiZ3JhZGllbnRfYWNjdW11bGF0aW9uX3N0ZXBzXCI6IDIsXG4gIFwiZGF0YV9pbXBsXCI6IFwibW1hcFwiLFxuICBcIm51bV93b3JrZXJzXCI6IDEsXG5cbiAgXCJjaGVja3BvaW50X2FjdGl2YXRpb25zXCI6IHRydWUsXG4gIFwiY2hlY2twb2ludF9udW1fbGF5ZXJzXCI6IDEsXG4gIFwicGFydGl0aW9uX2FjdGl2YXRpb25zXCI6IHRydWUsXG4gIFwic3luY2hyb25pemVfZWFjaF9sYXllclwiOiB0cnVlLFxuXG4gIFwiZ3JhZGllbnRfY2xpcHBpbmdcIjogMS4wLFxuICBcIndlaWdodF9kZWNheVwiOiAwLjEsXG4gIFwiaGlkZGVuX2Ryb3BvdXRcIjogMCxcbiAgXCJhdHRlbnRpb25fZHJvcG91dFwiOiAwLFxuXG4gIFwiZnAxNlwiOiB7XG4gICAgXCJmcDE2XCI6IHRydWUsXG4gICAgXCJlbmFibGVkXCI6IHRydWUsXG4gICAgXCJsb3NzX3NjYWxlXCI6IDAsXG4gICAgXCJsb3NzX3NjYWxlX3dpbmRvd1wiOiAxMDAwLFxuICAgIFwiaW5pdGlhbF9zY2FsZV9wb3dlclwiOiAxMixcbiAgICBcImh5c3RlcmVzaXNcIjogMixcbiAgICBcIm1pbl9sb3NzX3NjYWxlXCI6IDFcbiAgfSxcblxuICBcInRyYWluX2l0ZXJzXCI6IDE0MzAwMSxcbiAgXCJscl9kZWNheV9pdGVyc1wiOiAxNDMwMDAsXG4gIFwiZGlzdHJpYnV0ZWRfYmFja2VuZFwiOiBcIm5jY2xcIixcbiAgXCJscl9kZWNheV9zdHlsZVwiOiBcImNvc2luZVwiLFxuICBcIndhcm11cFwiOiAwLjAxLFxuICBcImNoZWNrcG9pbnRfZmFjdG9yXCI6IDI1MCxcbiAgIyBcImV4dHJhX3NhdmVfaXRlcnNcIjogWzAsMSwyLDQsOCwxNiwzMiw2NCwxMjgsMjU2LDUxMl0sXG4gIFwiZXZhbF9pbnRlcnZhbFwiOiAxNDMwMDAsXG4gIFwiZXZhbF9pdGVyc1wiOiAxMCxcblxuICBcImxvZ19pbnRlcnZhbFwiOiAxMCxcbiAgXCJzdGVwc19wZXJfcHJpbnRcIjogMTAsXG4gIFwid2FsbF9jbG9ja19icmVha2Rvd25cIjogdHJ1ZSxcblxuICBcInRva2VuaXplcl90eXBlXCI6IFwiSEZUb2tlbml6ZXJcIixcbiAgXCJ2b2NhYl9maWxlXCI6IFwiL3dla2EvcGlsZS8yMEJfdG9rZW5pemVyLmpzb25cIixcblxuICAjIFwic2F2ZVwiOiBcIi93ZWthL2hhaWxleS9tYW1iYS1ja3B0cy9tYW1iYS0xNjBtLXB5dGhpYS10ZXN0LWNvbnYtYmlhc1wiLFxuICAjIFwibG9hZFwiOiBcIi93ZWthL2hhaWxleS9tYW1iYS1ja3B0cy9tYW1iYS0xNjBtLXB5dGhpYS10ZXN0LWNvbnYtYmlhc1wiLFxuXG4gICMgXCJzM19wYXRoXCI6IFwiczM6Ly9zLWVhaS1uZW94LXdlc3QvaGFpbGV5L21hbWJhL3Rlc3QtY2twdHMvbWFtYmEtMTYwbS1weXRoaWEtdGVzdC1jb252LWJpYXNcIixcblxuICAjIFwia2VlcF9sYXN0X25fY2hlY2twb2ludHNcIjogMixcblxuICBcInRyYWluX2RhdGFfcGF0aHNcIjogW1wiL3dla2EvcGlsZS9waWxlXzIwQl90b2tlbml6ZXJfdGV4dF9kb2N1bWVudFwiXSxcbiAgXCJ2YWxpZF9kYXRhX3BhdGhzXCI6IFtcIi93ZWthL3BpbGUvcGlsZV8yMEJfdG9rZW5pemVyX3RleHRfZG9jdW1lbnRcIl0sXG4gIFwidGVzdF9kYXRhX3BhdGhzXCI6IFtcIi93ZWthL3BpbGUvcGlsZV8yMEJfdG9rZW5pemVyX3RleHRfZG9jdW1lbnRcIl0sXG5cbiAgXCJsYXVuY2hlclwiOiBcInNsdXJtXCIsIFxuICBcImRlZXBzcGVlZF9zbHVybVwiOiB0cnVlLFxuICAjICBcImFjY291bnRcIjogXCJlbGV1dGhlclwiLFxuICBcIm5vX3NzaF9jaGVja1wiOiB0cnVlLFxuXG4gIFwidXNlX3dhbmRiXCI6IHRydWUsXG4gIFwid2FuZGJfZ3JvdXBcIjogXCJtYW1iYS10cDEtYnMxNi1mdXNlZGlubmVyXCIsXG4gIFwid2FuZGJfdGVhbVwiOiBcImVsZXV0aGVyYWlcIixcbiAgXCJ3YW5kYl9wcm9qZWN0XCI6IFwibWFtYmEtbmVveC10cC1tZW1zYXZpbmdzXCIsXG59XG4ifSwgImNoZWNrcG9pbnRfZmFjdG9yIjogMjUwLCAiYmF0Y2hfc2l6ZSI6IDE2LCAidHJhaW5faXRlcnMiOiAxNDMwMDEsICJldmFsX2l0ZXJzIjogMTAsICJldmFsX2ludGVydmFsIjogMTQzMDAwLCAidm9jYWJfZmlsZSI6ICIvd2VrYS9waWxlLzIwQl90b2tlbml6ZXIuanNvbiIsICJudW1fd29ya2VycyI6IDEsICJjaGVja3BvaW50X2FjdGl2YXRpb25zIjogdHJ1ZSwgInN5bmNocm9uaXplX2VhY2hfbGF5ZXIiOiB0cnVlLCAicGFydGl0aW9uX2FjdGl2YXRpb25zIjogdHJ1ZSwgImR5bmFtaWNfbG9zc19zY2FsZSI6IHRydWUsICJ3b3JsZF9zaXplIjogMzIsICJ1c2Vfd2FuZGIiOiB0cnVlLCAid2FuZGJfZ3JvdXAiOiAibWFtYmEtdHAxLWJzMTYtZnVzZWRpbm5lcl9yN2t3cjhiMl8xMWN1Mmk2ciIsICJ3YW5kYl90ZWFtIjogImVsZXV0aGVyYWkiLCAid2FuZGJfcHJvamVjdCI6ICJtYW1iYS1uZW94LXRwLW1lbXNhdmluZ3MiLCAibG9nX2ludGVydmFsIjogMTAsICJ0ZXh0X2dlbl90eXBlIjogInVuY29uZGl0aW9uYWwiLCAibG9jYWxfcmFuayI6IDAsICJyYW5rIjogMCwgImRlZXBzcGVlZF9zbHVybSI6IHRydWUsICJ1c2VyX3NjcmlwdCI6ICIvd2VrYS9oYWlsZXkvbWlzdHJhbC1zdXBwb3J0LW5lb3gvZ3B0LW5lb3gvdHJhaW4ucHkiLCAic2F2ZV9pdGVycyI6IFsyNTAsIDUwMCwgNzUwLCAxMDAwLCAxMjUwLCAxNTAwLCAxNzUwLCAyMDAwLCAyMjUwLCAyNTAwLCAyNzUwLCAzMDAwLCAzMjUwLCAzNTAwLCAzNzUwLCA0MDAwLCA0MjUwLCA0NTAwLCA0NzUwLCA1MDAwLCA1MjUwLCA1NTAwLCA1NzUwLCA2MDAwLCA2MjUwLCA2NTAwLCA2NzUwLCA3MDAwLCA3MjUwLCA3NTAwLCA3NzUwLCA4MDAwLCA4MjUwLCA4NTAwLCA4NzUwLCA5MDAwLCA5MjUwLCA5NTAwLCA5NzUwLCAxMDAwMCwgMTAyNTAsIDEwNTAwLCAxMDc1MCwgMTEwMDAsIDExMjUwLCAxMTUwMCwgMTE3NTAsIDEyMDAwLCAxMjI1MCwgMTI1MDAsIDEyNzUwLCAxMzAwMCwgMTMyNTAsIDEzNTAwLCAxMzc1MCwgMTQwMDAsIDE0MjUwLCAxNDUwMCwgMTQ3NTAsIDE1MDAwLCAxNTI1MCwgMTU1MDAsIDE1NzUwLCAxNjAwMCwgMTYyNTAsIDE2NTAwLCAxNjc1MCwgMTcwMDAsIDE3MjUwLCAxNzUwMCwgMTc3NTAsIDE4MDAwLCAxODI1MCwgMTg1MDAsIDE4NzUwLCAxOTAwMCwgMTkyNTAsIDE5NTAwLCAxOTc1MCwgMjAwMDAsIDIwMjUwLCAyMDUwMCwgMjA3NTAsIDIxMDAwLCAyMTI1MCwgMjE1MDAsIDIxNzUwLCAyMjAwMCwgMjIyNTAsIDIyNTAwLCAyMjc1MCwgMjMwMDAsIDIzMjUwLCAyMzUwMCwgMjM3NTAsIDI0MDAwLCAyNDI1MCwgMjQ1MDAsIDI0NzUwLCAyNTAwMCwgMjUyNTAsIDI1NTAwLCAyNTc1MCwgMjYwMDAsIDI2MjUwLCAyNjUwMCwgMjY3NTAsIDI3MDAwLCAyNzI1MCwgMjc1MDAsIDI3NzUwLCAyODAwMCwgMjgyNTAsIDI4NTAwLCAyODc1MCwgMjkwMDAsIDI5MjUwLCAyOTUwMCwgMjk3NTAsIDMwMDAwLCAzMDI1MCwgMzA1MDAsIDMwNzUwLCAzMTAwMCwgMzEyNTAsIDMxNTAwLCAzMTc1MCwgMzIwMDAsIDMyMjUwLCAzMjUwMCwgMzI3NTAsIDMzMDAwLCAzMzI1MCwgMzM1MDAsIDMzNzUwLCAzNDAwMCwgMzQyNTAsIDM0NTAwLCAzNDc1MCwgMzUwMDAsIDM1MjUwLCAzNTUwMCwgMzU3NTAsIDM2MDAwLCAzNjI1MCwgMzY1MDAsIDM2NzUwLCAzNzAwMCwgMzcyNTAsIDM3NTAwLCAzNzc1MCwgMzgwMDAsIDM4MjUwLCAzODUwMCwgMzg3NTAsIDM5MDAwLCAzOTI1MCwgMzk1MDAsIDM5NzUwLCA0MDAwMCwgNDAyNTAsIDQwNTAwLCA0MDc1MCwgNDEwMDAsIDQxMjUwLCA0MTUwMCwgNDE3NTAsIDQyMDAwLCA0MjI1MCwgNDI1MDAsIDQyNzUwLCA0MzAwMCwgNDMyNTAsIDQzNTAwLCA0Mzc1MCwgNDQwMDAsIDQ0MjUwLCA0NDUwMCwgNDQ3NTAsIDQ1MDAwLCA0NTI1MCwgNDU1MDAsIDQ1NzUwLCA0NjAwMCwgNDYyNTAsIDQ2NTAwLCA0Njc1MCwgNDcwMDAsIDQ3MjUwLCA0NzUwMCwgNDc3NTAsIDQ4MDAwLCA0ODI1MCwgNDg1MDAsIDQ4NzUwLCA0OTAwMCwgNDkyNTAsIDQ5NTAwLCA0OTc1MCwgNTAwMDAsIDUwMjUwLCA1MDUwMCwgNTA3NTAsIDUxMDAwLCA1MTI1MCwgNTE1MDAsIDUxNzUwLCA1MjAwMCwgNTIyNTAsIDUyNTAwLCA1Mjc1MCwgNTMwMDAsIDUzMjUwLCA1MzUwMCwgNTM3NTAsIDU0MDAwLCA1NDI1MCwgNTQ1MDAsIDU0NzUwLCA1NTAwMCwgNTUyNTAsIDU1NTAwLCA1NTc1MCwgNTYwMDAsIDU2MjUwLCA1NjUwMCwgNTY3NTAsIDU3MDAwLCA1NzI1MCwgNTc1MDAsIDU3NzUwLCA1ODAwMCwgNTgyNTAsIDU4NTAwLCA1ODc1MCwgNTkwMDAsIDU5MjUwLCA1OTUwMCwgNTk3NTAsIDYwMDAwLCA2MDI1MCwgNjA1MDAsIDYwNzUwLCA2MTAwMCwgNjEyNTAsIDYxNTAwLCA2MTc1MCwgNjIwMDAsIDYyMjUwLCA2MjUwMCwgNjI3NTAsIDYzMDAwLCA2MzI1MCwgNjM1MDAsIDYzNzUwLCA2NDAwMCwgNjQyNTAsIDY0NTAwLCA2NDc1MCwgNjUwMDAsIDY1MjUwLCA2NTUwMCwgNjU3NTAsIDY2MDAwLCA2NjI1MCwgNjY1MDAsIDY2NzUwLCA2NzAwMCwgNjcyNTAsIDY3NTAwLCA2Nzc1MCwgNjgwMDAsIDY4MjUwLCA2ODUwMCwgNjg3NTAsIDY5MDAwLCA2OTI1MCwgNjk1MDAsIDY5NzUwLCA3MDAwMCwgNzAyNTAsIDcwNTAwLCA3MDc1MCwgNzEwMDAsIDcxMjUwLCA3MTUwMCwgNzE3NTAsIDcyMDAwLCA3MjI1MCwgNzI1MDAsIDcyNzUwLCA3MzAwMCwgNzMyNTAsIDczNTAwLCA3Mzc1MCwgNzQwMDAsIDc0MjUwLCA3NDUwMCwgNzQ3NTAsIDc1MDAwLCA3NTI1MCwgNzU1MDAsIDc1NzUwLCA3NjAwMCwgNzYyNTAsIDc2NTAwLCA3Njc1MCwgNzcwMDAsIDc3MjUwLCA3NzUwMCwgNzc3NTAsIDc4MDAwLCA3ODI1MCwgNzg1MDAsIDc4NzUwLCA3OTAwMCwgNzkyNTAsIDc5NTAwLCA3OTc1MCwgODAwMDAsIDgwMjUwLCA4MDUwMCwgODA3NTAsIDgxMDAwLCA4MTI1MCwgODE1MDAsIDgxNzUwLCA4MjAwMCwgODIyNTAsIDgyNTAwLCA4Mjc1MCwgODMwMDAsIDgzMjUwLCA4MzUwMCwgODM3NTAsIDg0MDAwLCA4NDI1MCwgODQ1MDAsIDg0NzUwLCA4NTAwMCwgODUyNTAsIDg1NTAwLCA4NTc1MCwgODYwMDAsIDg2MjUwLCA4NjUwMCwgODY3NTAsIDg3MDAwLCA4NzI1MCwgODc1MDAsIDg3NzUwLCA4ODAwMCwgODgyNTAsIDg4NTAwLCA4ODc1MCwgODkwMDAsIDg5MjUwLCA4OTUwMCwgODk3NTAsIDkwMDAwLCA5MDI1MCwgOTA1MDAsIDkwNzUwLCA5MTAwMCwgOTEyNTAsIDkxNTAwLCA5MTc1MCwgOTIwMDAsIDkyMjUwLCA5MjUwMCwgOTI3NTAsIDkzMDAwLCA5MzI1MCwgOTM1MDAsIDkzNzUwLCA5NDAwMCwgOTQyNTAsIDk0NTAwLCA5NDc1MCwgOTUwMDAsIDk1MjUwLCA5NTUwMCwgOTU3NTAsIDk2MDAwLCA5NjI1MCwgOTY1MDAsIDk2NzUwLCA5NzAwMCwgOTcyNTAsIDk3NTAwLCA5Nzc1MCwgOTgwMDAsIDk4MjUwLCA5ODUwMCwgOTg3NTAsIDk5MDAwLCA5OTI1MCwgOTk1MDAsIDk5NzUwLCAxMDAwMDAsIDEwMDI1MCwgMTAwNTAwLCAxMDA3NTAsIDEwMTAwMCwgMTAxMjUwLCAxMDE1MDAsIDEwMTc1MCwgMTAyMDAwLCAxMDIyNTAsIDEwMjUwMCwgMTAyNzUwLCAxMDMwMDAsIDEwMzI1MCwgMTAzNTAwLCAxMDM3NTAsIDEwNDAwMCwgMTA0MjUwLCAxMDQ1MDAsIDEwNDc1MCwgMTA1MDAwLCAxMDUyNTAsIDEwNTUwMCwgMTA1NzUwLCAxMDYwMDAsIDEwNjI1MCwgMTA2NTAwLCAxMDY3NTAsIDEwNzAwMCwgMTA3MjUwLCAxMDc1MDAsIDEwNzc1MCwgMTA4MDAwLCAxMDgyNTAsIDEwODUwMCwgMTA4NzUwLCAxMDkwMDAsIDEwOTI1MCwgMTA5NTAwLCAxMDk3NTAsIDExMDAwMCwgMTEwMjUwLCAxMTA1MDAsIDExMDc1MCwgMTExMDAwLCAxMTEyNTAsIDExMTUwMCwgMTExNzUwLCAxMTIwMDAsIDExMjI1MCwgMTEyNTAwLCAxMTI3NTAsIDExMzAwMCwgMTEzMjUwLCAxMTM1MDAsIDExMzc1MCwgMTE0MDAwLCAxMTQyNTAsIDExNDUwMCwgMTE0NzUwLCAxMTUwMDAsIDExNTI1MCwgMTE1NTAwLCAxMTU3NTAsIDExNjAwMCwgMTE2MjUwLCAxMTY1MDAsIDExNjc1MCwgMTE3MDAwLCAxMTcyNTAsIDExNzUwMCwgMTE3NzUwLCAxMTgwMDAsIDExODI1MCwgMTE4NTAwLCAxMTg3NTAsIDExOTAwMCwgMTE5MjUwLCAxMTk1MDAsIDExOTc1MCwgMTIwMDAwLCAxMjAyNTAsIDEyMDUwMCwgMTIwNzUwLCAxMjEwMDAsIDEyMTI1MCwgMTIxNTAwLCAxMjE3NTAsIDEyMjAwMCwgMTIyMjUwLCAxMjI1MDAsIDEyMjc1MCwgMTIzMDAwLCAxMjMyNTAsIDEyMzUwMCwgMTIzNzUwLCAxMjQwMDAsIDEyNDI1MCwgMTI0NTAwLCAxMjQ3NTAsIDEyNTAwMCwgMTI1MjUwLCAxMjU1MDAsIDEyNTc1MCwgMTI2MDAwLCAxMjYyNTAsIDEyNjUwMCwgMTI2NzUwLCAxMjcwMDAsIDEyNzI1MCwgMTI3NTAwLCAxMjc3NTAsIDEyODAwMCwgMTI4MjUwLCAxMjg1MDAsIDEyODc1MCwgMTI5MDAwLCAxMjkyNTAsIDEyOTUwMCwgMTI5NzUwLCAxMzAwMDAsIDEzMDI1MCwgMTMwNTAwLCAxMzA3NTAsIDEzMTAwMCwgMTMxMjUwLCAxMzE1MDAsIDEzMTc1MCwgMTMyMDAwLCAxMzIyNTAsIDEzMjUwMCwgMTMyNzUwLCAxMzMwMDAsIDEzMzI1MCwgMTMzNTAwLCAxMzM3NTAsIDEzNDAwMCwgMTM0MjUwLCAxMzQ1MDAsIDEzNDc1MCwgMTM1MDAwLCAxMzUyNTAsIDEzNTUwMCwgMTM1NzUwLCAxMzYwMDAsIDEzNjI1MCwgMTM2NTAwLCAxMzY3NTAsIDEzNzAwMCwgMTM3MjUwLCAxMzc1MDAsIDEzNzc1MCwgMTM4MDAwLCAxMzgyNTAsIDEzODUwMCwgMTM4NzUwLCAxMzkwMDAsIDEzOTI1MCwgMTM5NTAwLCAxMzk3NTAsIDE0MDAwMCwgMTQwMjUwLCAxNDA1MDAsIDE0MDc1MCwgMTQxMDAwLCAxNDEyNTAsIDE0MTUwMCwgMTQxNzUwLCAxNDIwMDAsIDE0MjI1MCwgMTQyNTAwLCAxNDI3NTAsIDE0MzAwMF0sICJnbG9iYWxfbnVtX2dwdXMiOiAzMn0=

System Hardware

CPU count	48
Logical CPU count	96
GPU count	8
GPU type	NVIDIA A100-SXM4-40GB

W&B CLI Version

0.16.3

Group

mamba-tp1-bs16-fusedinner_r7kwr8b2_11cu2i6r

Config parameters are your model's inputs. Learn more

▶
Config parameters:{} 258 keys
- account:
  null
- activation:
  "silu"
- activation_checkpointing:
  null
- adlr_autoresume:
  false
- adlr_autoresume_interval:
  1,000
- amp:
  null
- apply_query_key_layer_scaling:
  false
- ▶
  attention_config:[] 24 items
- attention_dropout:
  0
- attention_softmax_in_fp32:
  false
- autotuning:
  null
- autotuning_run:
  null
- base_shapes_file:
  null
- batch_size:
  16
- bf16:
  null
- bias_dropout_fusion:
  false
- bias_gelu_fusion:
  false
- char_level_ppl:
  false
- checkpoint:
  null
- checkpoint_activations:
  true
- checkpoint_factor:
  250
- checkpoint_in_cpu:
  false
- checkpoint_num_layers:
  1
- checkpoint_scale:
  "linear"
- checkpoint_validation_with_forward_pass:
  false
- clip_grad:
  1
- comment:
  null
- comms_logger:
  null
- communication_data_type:
  null
- compression_training:
  null
- ▶
  config_files:{} 1 key
  - mamba-160m.yml:
    "{ "pipe_parallel_size": 0, "model_parallel_size": 1, "num_layers": 24, "hidden_size": 768, "num_attention_heads": 12, "seq_length": 2048, "max_position_embeddings": 2048, "pos_emb": "rotary", "rotary_pct": 0.25, "no_weight_tying": true, "gpt_j_residual": true, "output_layer_parallelism": "column", "attention_config": [[["mamba"], 24]], # "scaled_upper_triang_masked_softmax_fusion": true, # "bias_gelu_fusion": true, "mamba_selective_scan_fusion": true, "mamba_causal_conv_fusion": true, "mamba_inner_func_fusion": true, "mamba_selective_fp32_params": true, "activation": "silu", "norm": "rmsnorm", "rms_norm_epsilon": 1.0e-5, "output_layer_init_method": "single_residual_scaled_normal", # "init_method": "small_init", # "output_layer_init_method": "wang_init", "optimizer": { "type": "Adam", "params": { "lr": 0.0006, "betas": [0.9, 0.95], "eps": 1.0e-8 } }, "min_lr": 0.00006, "zero_optimization": { "stage": 1, "allgather_partitions": true, "allgather_bucket_size": 500000000, "overlap_comm": true, "reduce_scatter": true, "reduce_bucket_size": 500000000, "contiguous_gradients": true, "cpu_offload": false }, "train_micro_batch_size_per_gpu": 16, "gradient_accumulation_steps": 2, "data_impl": "mmap", "num_workers": 1, "checkpoint_activations": true, "checkpoint_num_layers": 1, "partition_activations": true, "synchronize_each_layer": true, "gradient_clipping": 1.0, "weight_decay": 0.1, "hidden_dropout": 0, "attention_dropout": 0, "fp16": { "fp16": true, "enabled": true, "loss_scale": 0, "loss_scale_window": 1000, "initial_scale_power": 12, "hysteresis": 2, "min_loss_scale": 1 }, "train_iters": 143001, "lr_decay_iters": 143000, "distributed_backend": "nccl", "lr_decay_style": "cosine", "warmup": 0.01, "checkpoint_factor": 250, # "extra_save_iters": [0,1,2,4,8,16,32,64,128,256,512], "eval_interval": 143000, "eval_iters": 10, "log_interval": 10, "steps_per_print": 10, "wall_clock_breakdown": true, "tokenizer_type": "HFTokenizer", "vocab_file": "/weka/pile/20B_tokenizer.json", # "save": "/weka/hailey/mamba-ckpts/mamba-160m-pythia-test-conv-bias", # "load": "/weka/hailey/mamba-ckpts/mamba-160m-pythia-test-conv-bias", # "s3_path": "s3://s-eai-neox-west/hailey/mamba/test-ckpts/mamba-160m-pythia-test-conv-bias", # "keep_last_n_checkpoints": 2, "train_data_paths": ["/weka/pile/pile_20B_tokenizer_text_document"], "valid_data_paths": ["/weka/pile/pile_20B_tokenizer_text_document"], "test_data_paths": ["/weka/pile/pile_20B_tokenizer_text_document"], "launcher": "slurm", "deepspeed_slurm": true, # "account": "eleuther", "no_ssh_check": true, "use_wandb": true, "wandb_group": "mamba-tp1-bs16-fusedinner", "wandb_team": "eleutherai", "wandb_project": "mamba-neox-tp-memsavings", } "
- contiguous_checkpointing:
  false
- coord_check:
  false
- create_moe_param_group:
  true
- csv_monitor:
  null
- curriculum_learning:
  null
- curriculum_seqlen:
  0
- data_efficiency:
  null
- data_impl:
  "mmap"
- data_path:
  null
- data_types:
  null
- deepscale:
  false
- deepscale_config:
  null
- deepspeed:
  true
- deepspeed_activation_checkpointing:
  true
- deepspeed_extra_args:
  null
- ▶
  zero_optimization:{} 8 keys
- zero_reduce_bucket_size:
  500,000,000
- zero_reduce_scatter:
  true
- zero_stage:
  1

Summary metrics are your model's outputs. Learn more

No summary metrics saved for this run.

Check the summary metrics documentation for more information.