urllib.error — 由 urllib.request 引发的异常类
urllib.error
urllib.robotparser
源代码： Lib/urllib/robotparser.py
此模块提供单个类， RobotFileParser , which answers questions about whether or not a particular user agent can fetch a URL on the web site that published the robots.txt file. For more details on the structure of robots.txt files, see http://www.robotstxt.org/orig.html .
RobotFileParser
robots.txt
This class provides methods to read, parse and answer questions about the robots.txt file at url .
Sets the URL referring to a robots.txt 文件。
读取 robots.txt URL and feeds it to the parser.
剖析行自变量。
返回 True 若 useragent is allowed to fetch the url according to the rules contained in the parsed robots.txt 文件。
True
Returns the time the robots.txt file was last fetched. This is useful for long-running web spiders that need to check for new robots.txt files periodically.
Sets the time the robots.txt file was last fetched to the current time.
返回值为 Crawl-delay 参数从 robots.txt 为 useragent in question. If there is no such parameter or it doesn’t apply to the useragent specified or the robots.txt entry for this parameter has invalid syntax, return None .
Crawl-delay
None
Added in version 3.6.
Returns the contents of the Request-rate 参数从 robots.txt 作为 命名元组 RequestRate(requests, seconds) . If there is no such parameter or it doesn’t apply to the useragent specified or the robots.txt entry for this parameter has invalid syntax, return None .
Request-rate
RequestRate(requests, seconds)
Returns the contents of the Sitemap 参数从 robots.txt in the form of a list() . If there is no such parameter or the robots.txt entry for this parameter has invalid syntax, return None .
Sitemap
list()
Added in version 3.8.
The following example demonstrates basic use of the RobotFileParser 类：
>>> import urllib.robotparser >>> rp = urllib.robotparser.RobotFileParser() >>> rp.set_url("http://www.musi-cal.com/robots.txt") >>> rp.read() >>> rrate = rp.request_rate("*") >>> rrate.requests 3 >>> rrate.seconds 20 >>> rp.crawl_delay("*") 6 >>> rp.can_fetch("*", "http://www.musi-cal.com/cgi-bin/search?city=San+Francisco") False >>> rp.can_fetch("*", "http://www.musi-cal.com/") True
http — HTTP 模块
http
键入搜索术语或模块、类、函数名称。