Atlanta Contemporary
Atlanta Contemporary on Means is free and small enough to do in an hour, and the High is quiet on a weekday morning.
For food DeKalb Farmers Market is the one, cheap and good and nobody cares if you look like you have been up all night.