有没有一种特殊机制可以强制Scrapy打印出所有的Python异常/堆栈跟踪。
我犯了一个简单的错误,将一个列表属性搞错了,导致AttributeError没有完整地显示在日志中。显示出来的是:
2015-11-15 22:13:50 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 264,
'downloader/request_count': 1,
'downloader/request_method_count/GET': 1,
'downloader/response_bytes': 40342,
'downloader/response_count': 1,
'downloader/response_status_count/200': 1,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2015, 11, 15, 22, 13, 50, 860480),
'log_count/CRITICAL': 1,
'log_count/DEBUG': 1,
'log_count/INFO': 1,
'response_received_count': 1,
'scheduler/dequeued': 1,
'scheduler/dequeued/memory': 1,
'scheduler/enqueued': 1,
'scheduler/enqueued/memory': 1,
'spider_exceptions/AttributeError': 1,
'start_time': datetime.datetime(2015, 11, 15, 22, 13, 49, 222371)}
因此,它显示了AttributeError计数为1,但没有告诉我在哪里和如何出错,我不得不在代码中手动放置ipdb.set_trace()来找出它在哪里出错。Scrapy本身继续执行其他线程而没有打印任何内容。
ipdb>
AttributeError: "'list' object has no attribute 'match'"
> /Users/username/Programming/regent/regentscraper/spiders/regent_spider.py(139)request_listing_detail_pages_from_listing_id_list()
138 volatile_props = ListingScanVolatilePropertiesItem()
--> 139 volatile_props['position_in_search'] = list_of_listing_ids.match(listing_id) + rank_of_first_item_in_page
140
scrapy settings
# -*- coding: utf-8 -*-
# Scrapy settings for regentscraper project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
# http://doc.scrapy.org/en/latest/topics/settings.html
# http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html
# http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html
import sys
import os
import django
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__name__), os.pardir)))
print sys.path
os.environ['DJANGO_SETTINGS_MODULE'] = 'regent.settings'
django.setup() #new for Django 1.8
BOT_NAME = 'regentscraper'
SPIDER_MODULES = ['regentscraper.spiders']
NEWSPIDER_MODULE = 'regentscraper.spiders'
ITEM_PIPELINES = {
'regentscraper.pipelines.ListingScanPipeline': 300,
}