“I don’t wanna take someone’s energy away,” the comedian explained