我有两个脚本,scraper.py和db_control.py。在scraper.py中,我有以下内容:
...
def scrape(category, field, pages, search, use_proxy, proxy_file):
...
loop = asyncio.get_event_loop()
to_do = [ get_pages(url, params, conngen) for url in urls ]
wait_coro = asyncio.wait(to_do)
res, _ = loop.run_until_complete(wait_coro)
...
loop.close()
return [ x.result() for x in res ]
...
在db_control.py中:
from scraper import scrape
...
while new < 15:
data = scrape(category, field, pages, search, use_proxy, proxy_file)
...
...
理论上,爬虫应该在未知的时间内启动,直到获取足够的数据。但是当new
不立即> 15
时,就会出现此错误:
File "/usr/lib/python3.4/asyncio/base_events.py", line 293, in run_until_complete
self._check_closed()
File "/usr/lib/python3.4/asyncio/base_events.py", line 265, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
但是,如果我只运行一次scrape()函数,脚本可以正常运行。所以我猜想重新创建
loop = asyncio.get_event_loop()
时存在问题,我尝试过此方法,但是没有任何改变。我该如何解决这个问题呢?当然,这些只是我的代码片段,如果您认为问题可能出现在其他地方,完整的代码可以在这里找到。