URLGatekeeper

a class to track robots.txt rules across multiple servers

The URLGatekeeper class is accessible via the fmtorrent.master.aggregator.feedfinder module.

Methods

f __init__(self) ...

f can_fetch(self, url) ...

f get(self, url) ...

See the source for more information.