Stop *all* requests to IP-only GET requests in apache?

Offhand, I'm wondering if it's possible to reject all GET requests that use
only my IP address and not one of my virtual hosts. Currently, these
requests just get routed to my first virtual host (www.granroth.org)... but
what I would like is for them to get a dropped TCP connection or a refused or
something. There is really no good reason for anybody but me to be accessing
my web server using the IP address.

Any thoughts?
-- 
Kurt Granroth            | http://www.granroth.org
KDE Developer/Evangelist | SuSE Labs Open Source Developer
granroth@kde.org         | granroth@suse.com
            KDE -- Conquer Your Desktop