A lot of people think polarizers are just for skies but that is not true. I think polarizers are at their best when removing glare from specular (shiny) things like water on rocks and in streams or lakes, or the glint of sunlight on leaves. The polarizer knocks out the glare and often intensifies the colors in the image.
Here is an example from a simple little image I shot last week of a waterfall at Vermejo Park Ranch in New Mexico.
First the image without a polarizer. . . .
The result is pretty harsh looking. Nine seconds later, I shot the same scene with a polarizer rotated to knock out all of the shiny reflections on the stone.
Glare is usually distracting and it is usually good to minimize it. The polarizer eliminates most of the glare on the stone and you can now see the reds.
Another side effect of a polarizer is that the polarizer blocks some light from getting back to the sensor. Depending on the polarizer, you can lose up to two stops of light with the polarizer attached. When you need to slow the shutter speed, the polarizer can help! But when you need lots of shutter speed and the polarizer does not help the image, by all means take it off.
In this case, it was a very bright morning and I wanted a slow shutter speed to abstract the flowing water. I used my lowest ISO (50), smallest aperture (f/32), and the polarizer to get the shutter speed down to 1.3 seconds, which gave the waterfall a soft, abstract look.