我对Scrapy和Python都很陌生。我正在尝试编写一个爬取文章标题、链接和类似于RSS源的文章描述的爬虫，以帮助我的论文。我已经编写了以下的爬虫代码，但当我运行并导出为.txt文件时，返回的结果是空白的。我认为需要添加Item Loader，但我不确定。
Items.py
from scrapy.item import Item, Field
class NorthAfricaItem(Item):
title = Field()
link = Field()
desc = Field()
pass
蜘蛛
from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
from northafricatutorial.items import NorthAfricaItem
class NorthAfricaItem(BaseSpider):
name = "northafrica"
allowed_domains = ["http://www.north-africa.com/"]
start_urls = [
"http://www.north-africa.com/naj_news/news_na/index.1.html",
]
def parse(self, response):
hxs = HtmlXPathSelector(response)
sites = hxs.select('//ul/li')
items = []
for site in sites:
item = NorthAfricaItem()
item['title'] = site.select('a/text()').extract()
item['link'] = site.select('a/@href').extract()
item['desc'] = site.select('text()').extract()
items.append(item)
return items
更新
感谢Talvalin的帮助，通过一些尝试我已经解决了问题。我使用了一份在网上找到的基础脚本，但是当我使用shell时，我找到了正确的标签以获取所需内容。最终结果如下：
from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
from northafrica.items import NorthAfricaItem
class NorthAfricaSpider(BaseSpider):
name = "northafrica"
allowed_domains = ["http://www.north-africa.com/"]
start_urls = [
"http://www.north-africa.com/naj_news/news_na/index.1.html",
]
def parse(self, response):
hxs = HtmlXPathSelector(response)
sites = hxs.select('//ul/li')
items = []
for site in sites:
item = NorthAfricaItem()
item['title'] = site.select('//div[@class="short_holder"] /h2/a/text()').extract()
item['link'] = site.select('//div[@class="short_holder"]/h2/a/@href').extract()
item['desc'] = site.select('//span[@class="summary"]/text()').extract()
items.append(item)
return items
如果有人发现我做错了什么，请告诉我......但它有效。