Patrick keeps it classy by wearing all sweats to brunch