Scrapy可以爬取第一页，但无法跟随链接。

Question

我不明白为什么Scrapy可以爬取第一个页面，但不能跟进链接爬取后续页面。这一定与规则有关。非常感谢。谢谢！

from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.selector import HtmlXPathSelector
from craigslist_sample.items import CraigslistItem

class MySpider(CrawlSpider):
    name = "craig"
    allowed_domains = ["sfbay.craigslist.org"]
    start_urls = ["http://sfbay.craigslist.org/acc/"]   

    rules = (Rule (SgmlLinkExtractor(allow=("index100\.html", ),restrict_xpaths=('//p[@id="nextpage"]',))
    , callback="parse_items", follow= True),
    )   

    def parse_items(self, response):
        hxs = HtmlXPathSelector(response)
        titles = hxs.select("//p")
        items = []
        for titles in titles:
            item = CraigslistItem()
            item ["title"] = titles.select("a/text()").extract()
            item ["link"] = titles.select("a/@href").extract()
            items.append(item)
        return(items)

spider = MySpider()

- Michael

---

craigs_sample.items里面有什么内容？你能分享一下代码片段，这样from craigslist_sample.items import CraigslistItem就可以工作了。 - Calvin Cheng

Answer 1

Craig使用index100，index200，index300...作为下一页的标识，最大可用index900。

rules = (Rule (SgmlLinkExtractor(allow=("index\d00\.html", ),restrict_xpaths=('//p[@id="nextpage"]',))
, callback="parse_items", follow= True),
)

对我来说可以工作。

- iMom0