Douglas Thrift's Search Engine is an indexing search engine for use on small
websites such as personal or small business sites.  It is designed to be
very similar to Google for end users and its output is customizable.  For
indexing, it supports both the Robots Exclusion Protocol and the Robots META
Tag as specified at http://www.robotstxt.org/wc/exclusion.html.
