该问题最近出现，之前健康的容器在创建shutit会话时现在进入了睡眠循环。该问题仅在Cloud Run上出现，本地没有出现。
最小可重现代码：
Dockerfile
已尝试的解决方法：
最小可重现代码：
requirements.txt
Flask==2.0.1
gunicorn==20.1.0
shutit
Dockerfile
FROM python:3.9
# Allow statements and log messages to immediately appear in the Cloud Run logs
ENV PYTHONUNBUFFERED True
COPY requirements.txt ./
RUN pip install -r requirements.txt
# Copy local code to the container image.
ENV APP_HOME /myapp
WORKDIR $APP_HOME
COPY . ./
CMD exec gunicorn \
--bind :$PORT \
--worker-class "sync" \
--workers 1 \
--threads 1 \
--timeout 0 \
main:app
main.py
import os
import shutit
from flask import Flask, request
app = Flask(__name__)
# just to prove api works
@app.route('/ping', methods=['GET'])
def ping():
os.system('echo pong')
return 'OK'
# issue replication
@app.route('/healthcheck', methods=['GET'])
def healthcheck():
os.system("echo 'healthcheck'")
# hangs inside create_session
shell = shutit.create_session(echo=True, loglevel='debug')
# never shell.send reached
shell.send('echo Hello World', echo=True)
# never returned
return 'OK'
if __name__ == '__main__':
app.run(host='127.0.0.1', port=8080, debug=True)
cloudbuild.yaml
steps:
- id: "build_container"
name: "gcr.io/kaniko-project/executor:latest"
args:
- --destination=gcr.io/$PROJECT_ID/borked-service-debug:latest
- --cache=true
- --cache-ttl=99h
- id: "configure infrastructure"
name: "gcr.io/cloud-builders/gcloud"
entrypoint: "bash"
args:
- "-c"
- |
set -euxo pipefail
REGION="europe-west1"
CLOUD_RUN_SERVICE="borked-service-debug"
SA_NAME="$${CLOUD_RUN_SERVICE}@${PROJECT_ID}.iam.gserviceaccount.com"
gcloud beta run deploy $${CLOUD_RUN_SERVICE} \
--service-account "$${SA_NAME}" \
--image gcr.io/${PROJECT_ID}/$${CLOUD_RUN_SERVICE}:latest \
--allow-unauthenticated \
--platform managed \
--concurrency 1 \
--max-instances 10 \
--timeout 1000s \
--cpu 1 \
--memory=1Gi \
--region "$${REGION}"
循环访问Cloud Run日志：
Setting up prompt
In session: host_child, trying to send: export PS1_ORIGIN_ENV=$PS1 && PS1='OR''IGIN_ENV:rkkfQQ2y# ' && PROMPT_COMMAND='sleep .05||sleep 1'
================================================================================
Sending>>> export PS1_ORIGIN_ENV=$PS1 && PS1='OR''IGIN_ENV:rkkfQQ2y# ' && PROMPT_COMMAND='sleep .05||sleep 1'<<<, expecting>>>['\r\nORIGIN_ENV:rkkfQQ2y# ']<<<
Sending in pexpect session (68242035994000): export PS1_ORIGIN_ENV=$PS1 && PS1='OR''IGIN_ENV:rkkfQQ2y# ' && PROMPT_COMMAND='sleep .05||sleep 1'
Expecting: ['\r\nORIGIN_ENV:rkkfQQ2y# ']
export PS1_ORIGIN_ENV=$PS1 && PS1='OR''IGIN_ENV:rkkfQQ2y# ' && PROMPT_COMMAND='sleep .05||sleep 1'
root@localhost:/myapp# export PS1_ORIGIN_ENV=$PS1 && PS1='OR''IGIN_ENV:rkkfQQ2y# ' && PROMPT_COMMAND='sleep .05||sleep 1'
Stopped sleep .05
Stopped sleep 1
pexpect: buffer: b'' before: b'cm9vdEBsb2NhbGhvc3Q6L3B1YnN1YiMgIGV4cx' after: b'DQpPUklHSU5fRU5WOnJra2ZRUTJ5IyA='
Resetting default expect to: ORIGIN_ENV:rkkfQQ2y#
In session: host_child, trying to send: stty cols 65535
================================================================================
Sending>>> stty cols 65535<<<, expecting>>>ORIGIN_ENV:rkkfQQ2y# <<<
Sending in pexpect session (68242035994000): stty cols 65535
Expecting: ORIGIN_ENV:rkkfQQ2y#
ORIGIN_ENV:rkkfQQ2y# stty cols 65535
stty cols 65535
Stopped stty cols 65535
Stopped sleep .05
Stopped sleep 1
已尝试的解决方法：
- 不同地区：欧洲（1、2级）、亚洲和美国。
- 使用 Docker 构建，而非 kaniko。
- 为容器分配不同的 CPU 和内存大小。
- 最小容器数为 1-5（以确保 CPU 始终分配给容器）。
--no-cpu-throttling也没有改变情况。- 最大容器数为 1-30。
- 使用不同的 GCP 项目。
- 使用不同的 Docker 基础映像（3.5-3.9 + 不同的 shas，从一年前到最近的版本）。