This experiment was done for an assignment in a graduate level course in RL. Code implementations are based on CleanRL project works