So far I have read several different explanations on this phrase, including yarb and reesetee's (below). Another theory is that "goat" is here a corruption of "goad". More theories at http://www.zyra.org.uk/getgoat.htm. This could be a thesis in the making.
I always assumed it referred to one's inner goat, the goat being symbolic of capriciousness, wilfulness, stubbornness, and, further, of Pan and Satan. I.e. something that gets your goat, gets that goatish side of you to show itself.