Scrapy错误：exceptions.IOError：无法识别图像文件

Question

Scrapy错误：exceptions.IOError：无法识别图像文件

3

我经常遇到以下错误，但不知道图片文件名或响应URL以跟踪它：

2012-08-20 08:14:34+0000 [spider] Unhandled Error
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 545, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 362, in callback
    self._startRunCallbacks(result)
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 458, in _startRunCallbacks
    self._runCallbacks()
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 545, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
--- <exception caught here> ---
  File "/usr/lib/pymodules/python2.7/scrapy/contrib/pipeline/images.py", line 204, in media_downloaded
    checksum = self.image_downloaded(response, request, info)
  File "/usr/lib/pymodules/python2.7/scrapy/contrib/pipeline/images.py", line 252, in image_downloaded
    for key, image, buf in self.get_images(response, request, info):
  File "/usr/lib/pymodules/python2.7/scrapy/contrib/pipeline/images.py", line 261, in get_images
    orig_image = Image.open(StringIO(response.body))
  File "/usr/lib/python2.7/dist-packages/PIL/Image.py", line 1980, in open
    raise IOError("cannot identify image file")
exceptions.IOError: cannot identify image file

那么，我该如何解决这个问题？因为它会在特定数量的错误后停止我的爬虫，而这个数量已经在settings.py中定义了。

- Mahmoud M. Abdel-Fattah

这个错误是在什么时候发生的？你是否使用类似PIL这样的工具来处理图像？ - Rostyslav Dzinko

请发布完整的堆栈跟踪（错误）。 - Steven Almeroth

@MahmoudM.Abdel-Fattah，这种错误通常发生在PIL试图解析只下载了部分的图像时。 - Rostyslav Dzinko

@MahmoudM.Abdel-Fattah，显然你必须将那段代码放入try-except块中。 - Rostyslav Dzinko

这听起来像是一个Scrapy问题。我想尝试在本地重现它：你能发布你的爬虫代码，或者至少是start_urls吗？ - Steven Almeroth

显示剩余4条评论

1个回答

网页内容由stack overflow 提供, 点击上面的

可以查看英文原文，
原文链接

- Steven Almeroth · Accepted Answer

有问题的代码行在scrapy.contrib.pipelines.images.ImagesPipeline中使用了PIL的Image.open()：

def get_images(self, response, request, info):
    key = self.image_key(request.url)
    orig_image = Image.open(StringIO(response.body))

media_downloaded() 函数中的 try 块会捕获此错误，但会发出自己的错误提示：

except Exception:
    log.err(spider=info.spider)

您可以使用以下方法来黑掉这个文件：

try:
    key = self.image_key(request.url)
    checksum = self.image_downloaded(response, request, info)
except ImageException, ex:
    log.msg(str(ex), level=log.WARNING, spider=info.spider)
    raise
except IOError, ex:
    log.msg(str(ex), level=log.WARNING, spider=info.spider)
    raise ImageException
except Exception:
    log.err(spider=info.spider)
    raise ImageException

但更好的选择是创建自己的管道并在pipelines.py文件中覆盖image_downloaded()方法：

from scrapy import log
from scrapy.contrib.pipeline.images import ImagesPipeline

class BkamImagesPipeline(ImagesPipeline):

    def image_downloaded(self, response, request, info):
        try:
            super(BkamImagesPipeline, self).image_downloaded(response, request, info)
        except IOError, ex:
            log.msg(str(ex), level=log.WARNING, spider=info.spider)

请确保在您的设置文件中声明此管道：

ITEM_PIPELINES = [
    'bkam.pipelines.BkamImagesPipeline',
]