Sense (since?) of Humor

I'm not the funniest guy in the world, and I am definitely not aspiring to be a stand up comedian. However, I do think that I can be fairly quick witted and clever. At the very least cheeky and fun. An interaction a few weeks back changed my sense of humor ego a lot though. 

A couple friends of mine and I were talking and one of them mentioned a slight lack of desire to travel very far to race his bike. I followed up his comment, without thinking much before I spoke, that I was racing as much as possible, near and far, because "it, bike racing, was more fun then hookers." He looked very shocked! Calmly he said, "I wouldn't know." At that point I was back tracking on my words as much as possible.

It made me realize that maybe I'm not as funny as I thought and that not everyone gets my sense of humor.