The singer also surprised fans with an appearance from Gracie Abrams; the pair debuted their collaboration “Us” live