I bought an Antec Fusion 430 mini atx enclosure, 2 x 2GB RAM sticks, a motherboard with integrated Nvidia 9400 video processor, and an intel Celeron CPU. I had a spare hard drive, DVD drive, and power supply to put inside. I was worried about the noise, but it's actually quite quiet. I think this is mostly because of the way the antec case is designed.
I went with intel because the best mini atx motherboard I could find was nvidia 8600. It was just a few dollars more to go with intel. I wanted an Nvidia based video card because of Nvidia's excellent drivers and video acceleration.
I put Vista and XMBC on it, and I was very disappointed with the video performance. I had read that nvidia drivers will offload video rendering from the CPU. My CPU was spiked while playing anything encoded with x264. I dug around and learned that you must purchase a video player to take advantage of acceleration on windows. XBMC was not going to work.
This left me with two apparent choices - buy the nvidia software and use it's unsuitable-for-tv interface or return my intel celeron ($40) and exchange it for a dual core ($120).
I was chatting with an XBMC project manager about this, and he informed me that nvidia had just released drivers for linux that would accerelate video rendering the way I want, and that XMBC Linux has a branch that takes advantage of it. This is a bit more trouble than the windows route; but it's free and it's exactly what I want - hardware accelerated XBMC.
So, I made an Ubuntu 8.10 CD, and installed it onto my HTPC in a dual boot configuration. Next I downloaded and installed Nvidia latest linux drivers. This was probably the easiest part. You download a bin file and run it. It unpacks itself and sets up the drivers for you. I was a little worried about installing something that wasn't in a repository; but from what I've read, you can get around any problems Ubuntu upgrade's cause by re running the Nvidia installation file.
Ubuntu's desktop has fancy features that you can enable once the driver is installed and working properly. I found a package called Emerald that makes windows transparent like on Vista.
Next I had to check out the source code for a special branch of the xbmc project and build it. I think the hardest part about this was getting all the dependencies installed. There's a list of them on the XBMC site, put they don't match the versions/names in the ubuntu repository exactly. Actually building XBMC took a while because of my slow celeron. Installing was a breeze. And it worked perfectly once it was done.
Audio worked fine, except getting audio out of HDMI. I found a helpful guide that stepped me through the process of enabling this feature. Basically, it went through disabling pulse audio, installing the latest alsa drivers, and enabling ESound. Then I could set the output using alsa mixer.
I launched XBMC and loaded a 1080p x264 video. It played beautifully. Running "top" told me the CPU usuage was at 7% which is exactly right if the video card is doing all the work.
The negatives are that whenever Ubuntu decides to upgrade the linux kernel, ( not very often ) I'll have a bit of work to do afterword: re-install nvidia drivers, re-build xbmc. Also, the antec remote doesn't work with linux (UPDATE: I got it to work; but what a PIA. Post about it forthcoming.)
Here are some links if you're interested in doing the same thing:
HDMI audio and Ubuntu
Nvidia drivers for linux
XBMC instructions <- follow the build from source install and use the vdpau branch.