Vista x64 Ultimate here as well. I would not touch XP 64bit for the very reason you mentioned... drivers. MS has not nurtured the OS and not very many drivers are actively written for it. On the other hand, in order for a driver to be WHQL certified for Vista, it has to be designed for both x86 and x64 versions of Vista. So, if something works with Vista, it will work regardless of 32 or 64bit.
Unless you decide to try to get a 10 year old HP All in one printer working with Vista 64bit, or an old program that relies on kernel mode drivers (most virus scanners, partition managers, system level defraggers like disk keeper, etc), you will be fine.