Do you want a traditional desktop computer or an all-in-one? An all-in-one is a modular desktop unit with an integrated display. They look terrific -- taking up less space and producing less cord and other clutter than traditional desktop setups. But an all-in-one computer can be pricier than a conventional desktop or sacrifice processing power for display quality. Also, you can't upgrade the monitor and can rarely upgrade components.
What extras will you need? You'll need a monitor, keyboard, mouse or trackpad, as well as cables for these unless they are wireless. Some computer systems ship with all or some of these items, but with others you are on your own. If speakers aren't provided, you'll have to supply those as well. Not all desktops come equipped with built-in optical drives, but you can connect an external optical drive or additional hard drive storage via the USB ports. The same goes for a memory card reader, too.
Will you be using your computer as a home theater PC (HTPC)? If so, you'll want a desktop computer with an HDMI output for sending HD content to your TV (or monitor). As a bare minimum, you can use an HDCP-compliant DVI output and a DVI-to-HDMI connector. An integrated optical drive is also a plus -- preferably a Blu-ray drive -- but you can also add an external drive post-purchase. Some desktop computers designed for HTPC use ship with a remote control as well.
Is gaming a concern? If you are serious about gaming, be sure to choose your computer with that in mind. Most cheaper desktop computers rely on integrated graphics. Some have discrete graphics, but use a low-end graphics card. Some systems are upgradable at purchase -- or later by the owner -- to use more advanced graphics solutions, but that adds to the ultimate bottom line. However, all but the lowest-powered systems can handle less intense games without a hitch. If there's a specific game you know you'll want to play, check its hardware requirements before you buy.
Do you want the flexibility to upgrade your system as your needs grow? All-in-ones and compact desktops typically offer no or very limited expandability. If you'd like to add more memory, additional hard drives, graphics cards, etc. later on, your best bet is a traditional full-sized tower.