ptst1110-ee10

Ptst1110-ee10's workspace

Runs

task_configs.tmlu_AST_biology.doc_to_text

task_configs.tmlu_AST_chemistry.doc_to_text

Finished

ptst1110-ee10

7mo ago

3m 27s

[]

100000

cuda:1

1234

pretrained=benchang1110/Qwen2.5-Taiwan-7B-Instruct

torch.bfloat16

7615616512

main

80a408cdd0f1062e395ec19b2692519770172e2f

1234

AST_biology

miulab/tmlu

["A","B","C","D"]

answer

{{question.strip()}} A. {{choices[0]}} B. {{choices[1]}} C. {{choices[2]}} D. {{choices[3]}}{% if choices is defined and choices|length > 4 %} E. {{choices[4]}}{% endif %}{% if choices is defined and choices|length > 5 %} F. {{choices[5]}}{% endif %} Answer:

first_n

dev

0.1

[{"metric":"acc","aggregation":"mean","higher_is_better":true}]

multiple_choice

def process_docs(dataset: datasets.Dataset) -> datasets.Dataset: def _helper(doc): # modifies the contents of a single # document in our dataset. answer_list = ["A", "B", "C", "D"] choices = [doc["A"], doc["B"], doc["C"], doc["D"]] if doc.get("E", None): answer_list.append("E") choices.append(doc["E"]) if doc.get("F", None): answer_list.append("F") choices.append(doc["F"]) out_doc = { "questions": doc["question"], "choices": choices, "goal": answer_list.index(doc["answer"]), } return out_doc return dataset.map(_helper) # returns back a datasets.Dataset object

false

tmlu_stem_tasks

tmlu_AST_biology

AST biology

test

false

AST_chemistry

miulab/tmlu

["A","B","C","D","E"]

answer

first_n

dev

0.1

1-1

of 1