我正在爬取一个网站,这个网站使用JavaScript进行下一页的跳转。我使用Splash在第一页上执行我的JavaScript代码,我能够跳转到第二页,但是我无法跳转到第三页、第四页、第五页等等。只有一页后,爬取就停止了。
我正在爬取的链接为:http://59.180.234.21:8788/user/viewallrecord.aspx 我的代码如下:
我正在爬取的链接为:http://59.180.234.21:8788/user/viewallrecord.aspx 我的代码如下:
import scrapy
from scrapy_splash import SplashRequest
from time import sleep
class MSEDCLSpider(scrapy.Spider):
name = "msedcl_spider"
scope_path = 'body > table:nth-child(11) tr > td.content_area > table:nth-child(4) tr:not(:first-child)'
ref_no_path = "td:nth-child(1) ::text"
title_path = "td:nth-child(2) ::text"
end_date_path = "td:nth-child(5) ::text"
fee_path = "td:nth-child(6) ::text"
start_urls = ["http://59.180.234.21:8788/user/viewallrecord.aspx"]
lua_src = """function main(splash)
local url = splash.args.url
splash:go(url)
splash:wait(2.0)
splash:runjs("document.querySelectorAll('#lnkNext')[0].click()")
splash:wait(4.0)
return {
splash:html(),
}
end
"""
def start_requests(self):
for url in self.start_urls:
yield SplashRequest(
url,
self.parse,
endpoint='execute',
method='POST',
dont_filter=True,
args={
'wait': 1.0,
'lua_source': self.lua_src,
},
)
def parse(self, response):
print response.status
scopes = response.css('#page-info').extract()[0]
print(response.url)
print(scopes)
我对scrapy和splash都是新手,请温柔一点。谢谢。