Bruce Schneirer points to a technical report from Microsoft describing a clever prototype - called GhostBuster - developed for detecting arbitrary persistent and stealthy software, such as rootkits,Trojans, and software keyloggers. It's a really elegent idea, based on a simple observation: the rootkit must exist on disk to be persistent, but must lie to programs running within the infected OS in order to hide.
The GhostBuster program runs from CD drive from within the (possibly corrupted) OS, the checker program runs: stopping all other user programs, flushing the caches, and then doing a complete checksum of all files on the disk and a scan of any registry keys that could autostart the system, writing out the results to a file on the hard drive.Then the user is instructed to press the reset button, the CD boots its own OS, and the scan is repeated. Any differences indicate a rootkit or other stealth software, without the need for knowing what particular rootkits are or the proper checksums for the programs installed on disk.
In order to fool GhostBuster, the rootkit must
1) detect that such a checking program is running and either not lie to it or change the output as it's written to disk (in the limit this becomes the halting problem for the rootkit designer),
2) integrate into the BIOS rather than the OS (tricky, platform specific, and not always possible), or
3) give up on either being persistent or stealthy. Thus this doesn't eliminate rootkits entirely, but is a pretty mortal blow to persistent rootkits.
Bruce thinks this is a great idea, but points to a problem. GhostBuster is only a research prototype, and , Microsoft has no plans to turn it into a commercial tool. This is too good an idea to abandon. Microsoft, if you're listening, you should release this tool to the world. Make it public domain. Make it open source, even.