你好,我有一个基本的爬虫,用来获取给定域上的所有链接。 我想确保它能保持状态,以便可以从上次离开的地方继续执行。 我遵循了给定 URL http://doc.scrapy.org/en/latest/topics/jobs.html 。 但是,当我第一次尝试时,它可以正常运行,并使用 Ctrl+C 结束,但是当我尝试恢复它时,爬行停止在第一个 URL 上。
以下是结束时的日志:
这是我的爬虫:
以下是结束时的日志:
2016-08-29 16:51:08 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 896,
'downloader/request_count': 4,
'downloader/request_method_count/GET': 4,
'downloader/response_bytes': 35320,
'downloader/response_count': 4,
'downloader/response_status_count/200': 4,
'dupefilter/filtered': 149,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2016, 8, 29, 16, 51, 8, 837853),
'log_count/DEBUG': 28,
'log_count/INFO': 7,
'offsite/domains': 22,
'offsite/filtered': 23,
'request_depth_max': 1,
'response_received_count': 4,
'scheduler/dequeued': 2,
'scheduler/dequeued/disk': 2,
'scheduler/enqueued': 2,
'scheduler/enqueued/disk': 2,
'start_time': datetime.datetime(2016, 8, 29, 16, 51, 7, 821974)}
2016-08-29 16:51:08 [scrapy] INFO: Spider closed (finished)
这是我的爬虫:
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor
from Something.items import SomethingItem
class maxSpider(CrawlSpider):
name = 'something'
allowed_domains = ['thecheckeredflag.com', 'inautonews.com']
start_urls = ['http://www.thecheckeredflag.com/', 'http://www.inautonews.com/']
rules = (Rule(LinkExtractor(allow=()), callback='parse_obj', follow=True),)
def parse_obj(self,response):
for link in LinkExtractor(allow=self.allowed_domains,deny =() ).extract_links(response):
item = SomethingItem()
item['url'] = link.url
yield item
#print item
Scrapy版本:Scrapy 1.1.2
Python版本:2.7
我是Scrapy的新手,如果需要发布更多信息,请告诉我。
scrapy crawl somespider -s JOBDIR=crawls/somespider-1
? - dblclik