我有一个使用Scrapy编写的网页爬虫,可以获取数据项。我希望能够异步地将这些数据项插入到数据库中。
例如,我有一个事务,使用SQLAlchemy Core将一些项目插入到我的数据库中:
def process_item(self, item, spider):
with self.connection.begin() as conn:
conn.execute(insert(table1).values(item['part1'])
conn.execute(insert(table2).values(item['part2'])
我知道可以使用alchimia
将SQLAlchemy Core与Twisted异步配合使用。下面是alchimia
的文档代码示例。
我不明白的是如何在alchimia框架中使用上述代码。我该如何设置process_item
来使用反应堆?
我能做类似这样的事情吗?
@inlineCallbacks
def process_item(self, item, spider):
with self.connection.begin() as conn:
yield conn.execute(insert(table1).values(item['part1'])
yield conn.execute(insert(table2).values(item['part2'])
如何编写反应器部分?
或者在Scrapy管道中进行非阻塞数据库插入是否有更简单的方法?
为了参考,这里是来自alchimia
文档的代码示例:
from alchimia import TWISTED_STRATEGY
from sqlalchemy import (
create_engine, MetaData, Table, Column, Integer, String
)
from sqlalchemy.schema import CreateTable
from twisted.internet.defer import inlineCallbacks
from twisted.internet.task import react
@inlineCallbacks
def main(reactor):
engine = create_engine(
"sqlite://", reactor=reactor, strategy=TWISTED_STRATEGY
)
metadata = MetaData()
users = Table("users", metadata,
Column("id", Integer(), primary_key=True),
Column("name", String()),
)
# Create the table
yield engine.execute(CreateTable(users))
# Insert some users
yield engine.execute(users.insert().values(name="Jeremy Goodwin"))
yield engine.execute(users.insert().values(name="Natalie Hurley"))
yield engine.execute(users.insert().values(name="Dan Rydell"))
yield engine.execute(users.insert().values(name="Casey McCall"))
yield engine.execute(users.insert().values(name="Dana Whitaker"))
result = yield engine.execute(users.select(users.c.name.startswith("D")))
d_users = yield result.fetchall()
# Print out the users
for user in d_users:
print "Username: %s" % user[users.c.name]
if __name__ == "__main__":
react(main, [])