When do locals go to Pacific Beach during June Gloom?

I have lived in San Diego for a short time and I keep hearing about May Gray and June Gloom. People say the weather is mild all year, but I arrived in summer and the mornings were cloudy in a way I did not expect. I want to plan beach days properly and not waste them. Is it normal to go to Pacific Beach in the afternoon once the clouds have gone, or do locals still head out early?