我正在尝试运行一个包含以下内容的Flask
应用:
- 实时处理API请求
- 将每个请求上传到
SQLalchemy
数据库中 - 作为后台进程运行
任务1
和任务2
为此,我有以下代码:
import concurrent.futures
import queue
from concurrent.futures import ThreadPoolExecutor
from flask import Flask, current_app
app = Flask(__name__)
q = queue.Queue()
def build_cache():
# 1. Yielding API requests on the fly
track_and_features = spotify.query_tracks() # <- a generator
while True:
q.put(next(track_and_features))
def upload_cache(tracks_and_features):
# 2. Uploading each request to a `SQLalchemy` database
with app.app_context():
Upload_Tracks(filtered_dataset=track_and_features)
return "UPLOADING TRACKS TO DATABASE"
@app.route("/cache")
def cache():
# 3. Do `1` and `2` as a background process
with concurrent.futures.ThreadPoolExecutor() as executor:
future_to_track = {executor.submit(build_cache): "TRACKER DONE"}
while future_to_track:
# check for status of the futures which are currently working
done, not_done = concurrent.futures.wait(
future_to_track,
timeout=0.25,
return_when=concurrent.futures.FIRST_COMPLETED,
)
# if there is incoming work, start a new future
while not q.empty():
# fetch a track from the queue
track = q.get()
# Start the load operation and mark the future with its TRACK
future_to_track[executor.submit(upload_cache, track)] = track
# process any completed futures
for future in done:
track = future_to_track[future]
try:
data = future.result()
except Exception as exc:
print("%r generated an exception: %s" % (track, exc))
del future_to_track[future]
return "Cacheing playlist in the background..."
上述方法都可以使用,但不能作为后台进程。调用
cache()
时,应用程序会挂起,并且只有当该进程完成时才会恢复。我使用
gunicorn -c gconfig.py app:app -w 4 --threads 12
来运行它,那么我的问题出在哪里呢?# 1st background process
def build_cache():
# only ONE JOB
tracks_and_features = spotify.query_tracks() # <- not a generator
while True:
print(next(tracks_and_features))
# background cache
@app.route("/cache")
def cache():
executor.submit(build_cache)
return "Cacheing playlist in the background..."
然后进程在后台运行。
但是,如果我添加另一个作业:
def build_cache():
tracks_and_features = spotify.query_tracks()
while True:
# SQLalchemy db
Upload_Tracks(filtered_dataset=next(tracks_and_features))
背景图片再次无法正常工作。
简而言之:
只有在一次只运行一个任务时,背景图片才能正常工作(这也是使用队列的初衷限制)。
看起来问题是将后台进程绑定到SQLalchemy,不确定。完全迷失了。
gunicorn -c gconfig.py app:app -w 4 --threads 12
命令来运行应用程序。 - 8-Bit Borges