连接Kubernetes Redis服务时出现连接被拒绝错误

4
我在我的集群上拥有一个单实例 Redis 部署/服务:

Redis.yaml

---

apiVersion: v1
kind: Service
metadata:
  name: myapp-redis
  labels:
    name: myapp-redis
spec:
  ports:
  - port: 6379
    targetPort: 6379
  selector:
    name: myapp-redis

---

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: myapp-redis
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

---

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: myapp-redis
  labels:
    name: myapp-redis
spec:
  selector:
    matchLabels:
      name: myapp-redis
  strategy:
    type: Recreate
  replicas: 1
  template:
    metadata:
      labels:
        name: myapp-redis
    spec:
      containers:
      - name: myapp-redis
        image: registry/myapp-redis:0.0.0-alpha.13
        imagePullPolicy: Always
        ports:
        - containerPort: 6379
        volumeMounts:
        - name: myapp-redis
          mountPath: /etc/redis/
      imagePullSecrets:
      - name: regsecret
      volumes:
      - name: myapp-redis
        persistentVolumeClaim:
          claimName: myapp-redis

---

Redis服务描述

我从kubectl describe svc myapp-redis -n mw-dev中得到了以下内容:

Name:              myapp-redis
Namespace:         mw-dev
Labels:            name=myapp-redis
Annotations:       kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"name":"myapp-redis"},"name":"myapp-redis","namespace":"mw-dev"},"sp...
Selector:          name=myapp-redis
Type:              ClusterIP
IP:                10.3.0.137
Port:              <unset>  6379/TCP
TargetPort:        6379/TCP
Endpoints:         10.2.2.173:6379
Session Affinity:  None
Events:            <none>

检查redis是否正在运行

确保数据库正在运行,我可以使用kubectl exec -it myapp-redis-[..] sh -n mw-dev在pod内打开shell,并使用redis-cli -a test ping来ping数据库。如果这样做,我会收到一个PONG,因此似乎密码(test)已解决并且数据库已启动。

连接Python应用程序到redis服务的问题

然而,如果我尝试将运行Python应用程序的pod连接到redis数据库,我会从Python应用程序中收到连接被拒绝的错误。

kubectl logs myapp-backend-596... -n mw-dev

[...]
  File "/usr/local/lib/python3.6/site-packages/aioredis/stream.py", line 19, in open_connection
    lambda: protocol, host, port, **kwds)
  File "uvloop/loop.pyx", line 1733, in create_connection
  File "uvloop/loop.pyx", line 1712, in uvloop.loop.Loop.create_connection
ConnectionRefusedError: [Errno 111] Connection refused

这是Python应用程序的配置:

Backend.yaml

---

apiVersion: v1
kind: Service
metadata:
  name: myapp-backend
  labels:
    name: myapp-backend
spec:
  ports:
  - port: 8000
    targetPort: 8000
  selector:
    name: myapp-backend

---

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: myapp-backend
  labels:
    name: myapp-backend
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        name: myapp-backend
    spec:
      containers:
      - name: myapp-backend
        image: registry/myapp-backend:0.0.0-alpha.13
        imagePullPolicy: Always
        ports:
        - containerPort: 8000
        env:
        - name: REDIS_HOST
          value: 'myapp-redis'
        - name: REDIS_PASSWORD
          value: 'test'
      imagePullSecrets:
      - name: regsecret

---

Python后端Pod描述

这是我从kubectl describe po myapp-backend-58... -n mw-dev中获取的:

Name:           myapp-backend-585d...
Namespace:      mw-dev
Node:           worker-2/ip...
Start Time:     Sat, 03 Feb 2018 13:08:01 +0100
Labels:         name=myapp-backend
                pod-template-hash=myhash
Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"mw-dev","name":"myapp-backend-58...","uid":"e13...
Status:         Running
IP:             10.2.2.180
Controlled By:  ReplicaSet/myapp-backend-58...
Containers:
  myapp-backend:
    Container ID:   docker://78cfc218d...
    Image:          registry/myapp-backend:0.0.0-alpha.13
    Image ID:       docker-pullable://registry/mw-dev/myapp-backend@sha256:785a...
    Port:           8000/TCP
    State:        registryg
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Sat, 03 Feb 2018 13:55:07 +0100
      Finished:     Sat, 03 Feb 2018 13:55:08 +0100
    Ready:          False
    Restart Count:  14
    Environment:
      REDIS_HOST:      myapp-redis
      REDIS_PASSWORD:  test
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-7... (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          False 
  PodScheduled   True 
Volumes:
  default-token-7cm7c:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-7...
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.alpha.kubernetes.io/notReady:NoExecute for 300s
                 node.alpha.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                 Age                  From                          Message
  ----     ------                 ----                 ----                          -------
  Normal   Scheduled              50m                  default-scheduler             Successfully assigned myapp-backend-58... to worker-2
  Normal   SuccessfulMountVolume  50m                  kubelet, worker-2  MountVolume.SetUp succeeded for volume "default-token-7..."
  Warning  BackOff                50m (x4 over 50m)    kubelet, worker-2  Back-off restarting failed container
  Normal   Pulling                50m (x4 over 50m)    kubelet, worker-2  pulling image "registry/mw-dev/myapp-backend:0.0.0-alpha.13"
  Normal   Pulled                 50m (x4 over 50m)    kubelet, worker-2  Successfully pulled image "registry/mw-dev/myapp-backend:0.0.0-alpha.13"
  Normal   Created                50m (x4 over 50m)    kubelet, worker-2  Created container
  Normal   Started                50m (x4 over 50m)    kubelet, worker-2  Started container
  Warning  FailedSync             52s (x229 over 50m)  kubelet, worker-2  Error syncing pod

运行中的Pods

kubectl get pods --all-namespaces:

NAMESPACE     NAME                                                          READY     STATUS    RESTARTS   AGE
kube-system   cert-manager-cert-manager-59fff59c7b-vdnd7                    2/2       Running   4          3d
kube-system   digitalocean-cloud-controller-manager-6d6b675bfd-nxqq2        1/1       Running   0          3d
kube-system   digitalocean-provisioner-d4c79dfb4-mhb5d                      1/1       Running   0          3d
kube-system   heapster-56bf7c7896-9rv4z                                     1/1       Running   0          3d
kube-system   kube-apiserver-wp7b4                                          1/1       Running   5          10d
kube-system   kube-controller-manager-586c9b745b-gkqk4                      1/1       Running   2          10d
kube-system   kube-controller-manager-586c9b745b-pdhw7                      1/1       Running   1          10d
kube-system   kube-dns-7d74988c8b-z9zs2                                     3/3       Running   0          10d
kube-system   kube-flannel-5wlk6                                            2/2       Running   0          10d
kube-system   kube-flannel-khsvq                                            2/2       Running   0          10d
kube-system   kube-flannel-skt2m                                            2/2       Running   4          10d
kube-system   kube-proxy-cwqv8                                              1/1       Running   2          10d
kube-system   kube-proxy-mg8jx                                              1/1       Running   0          10d
kube-system   kube-proxy-vmw8g                                              1/1       Running   0          10d
kube-system   kube-scheduler-7686847675-5kkhn                               1/1       Running   1          10d
kube-system   kube-scheduler-7686847675-lkm98                               1/1       Running   2          10d
kube-system   kubernetes-dashboard-7658f8d76-svtzh                          1/1       Running   0          3d
kube-system   loadbalancer-nginx-ingress-controller-8649c7986b-jndzz        1/1       Running   3          3d
kube-system   loadbalancer-nginx-ingress-default-backend-6fb9444c64-bpz4g   1/1       Running   0          3d
kube-system   pod-checkpointer-kfcpp                                        1/1       Running   0          10d
kube-system   pod-checkpointer-kfcpp-spc1aitu1i-master-1                    1/1       Running   0          10d
kube-system   tiller-deploy-fb8d7b69c-6xrpn                                 1/1       Running   2          3d
mw-dev        myapp-backend-6c4b56d9b7-2mfbs                                1/1       Running   0          21m
mw-dev        myapp-frontend-7478fd456b-5ztvq                               1/1       Running   0          1d
mw-dev        myapp-redis-67d45d97d7-7wxtj                                  1/1       Running   0          1d

确保 Python 应用程序接收到正确的环境变量值

Python 应用程序打印出它用于连接数据库的值。查看 Pod 日志,我可以看到这些值与 Backend.yaml 中给出的值相同(REDIS_HOST=myapp-redisREDIS_PASSWORD=test)。

在 Docker 中本地运行正常

如果我在我的笔记本电脑上使用 Docker 本地运行 Redis 容器和 Python 应用程序容器,则它们可以成功连接。

集群信息

该集群使用 nginx-ingress 控制器将服务暴露给互联网。我不确定是否重要,因为我需要将 Python Pod 内部连接到 Redis 服务。

该集群由 1 个 Master 节点、两个 Worker 节点和一个负载均衡器组成,所有节点均在 DigitalOcean 上运行。

现在怎么办?

此时我不知道如何进一步调试此问题。我已经在网上搜索了几个小时,但没有找到解决方案。任何建议都将不胜感激!


1
如果您使用服务ClusterIP从后端Pod对Redis数据库进行ping操作,是否有效? - Jose Armesto
@fiunchinho 看起来是这样的。一个 ping 返回:10.3.0.137:6379 (10.3.0.137): 56 data bytes - Rotareti
我的意思是redis-cli ping命令。你能把你的操作步骤粘贴到问题中吗? - Jose Armesto
如果我从另一个 Pod 运行以下命令:redis-cli -h 10.3.0.137 -p 6379 -a test ping,我会得到以下错误信息:无法连接到 Redis 10.3.0.137:6379:连接被拒绝 - Rotareti
运行 kubectl get pods --all-namespaces 命令来查看内部 Kubernetes Pod 是否正在运行,例如 kube-dns。 - Jose Armesto
我在“运行的Pod”下面添加了输出到问题中。 - Rotareti
1个回答

9

连接被拒绝错误是由redis配置引起的。

我不得不将redis主机从localhost更改为0.0.0.0,以允许外部连接。

redis.conf中,我更改了这一行:

bind 127.0.0.1

转换为:

bind 0.0.0.0

6
我有一个类似的问题,但在redis.conf中使用bind并不能帮助我,redis-cli仍然无法连接到k8s IP。还有其他建议吗? - Valentin
有完全相同的问题。有人解决了吗?即使在端口6379上telnet到pod ip也失败了。 - Samir Shaik

网页内容由stack overflow 提供, 点击上面的
可以查看英文原文,
原文链接