URLGatekeeper

a class to track robots.txt rules across multiple servers


Methods

f __init__(self) ...

f can_fetch(self, url) ...

f get(self, url) ...

See the source for more information.