Apache负载均衡Tomcat WebSocket

Question

我目前正在开发一个websocket应用程序，部署在Tomcat服务器上。由于用户数量巨大，我希望将工作负载分配到多个Tomcat实例中。我决定使用Apache进行负载平衡。

现在我在实现Apache负载平衡和websockets请求的黏性会话时遇到了问题。这是我的Apache配置：

ProxyRequests off
SSLProxyEngine on
RewriteEngine On

<Proxy balancer://http-localhost/>
    BalancerMember  https://mcsgest1.desy.de:8443/Whiteboard/ route=jvm1 keepalive=On smax=1 connectiontimeout=10 retry=600 timeout=900 ttl=900
    BalancerMember  https://mcsgest1.desy.de:8444/Whiteboard/ route=jvm2 keepalive=On smax=1 connectiontimeout=10 retry=600 timeout=900 ttl=900
    ProxySet lbmethod=byrequests
    ProxySet stickysession=JSESSIONID|sid scolonpathdelim=On
</Proxy>

<Proxy balancer://ws-localhost/>
    BalancerMember  wss://mcsgest1.desy.de:8443/Whiteboard/ route=jvm1 keepalive=On smax=1 connectiontimeout=10 retry=600 timeout=900 ttl=900
    BalancerMember  wss://mcsgest1.desy.de:8444/Whiteboard/ route=jvm2 keepalive=On smax=1 connectiontimeout=10 retry=600 timeout=900 ttl=900
    ProxySet lbmethod=byrequests
    ProxySet stickysession=JSESSIONID|sid scolonpathdelim=On
</Proxy>

RewriteCond       %{HTTP:Upgrade} =websocket
RewriteRule       /jddd/(.*)      balancer://ws-localhost/$1 [P,L]
ProxyPassReverse  /jddd/          balancer://ws-localhost/
RewriteCond       %{HTTP:Upgrade} !=websocket
RewriteRule       /jddd/(.*)      balancer://http-localhost/$1 [P,L]
ProxyPassReverse  /jddd/          balancer://http-localhost/

第一个https请求被平衡到8443端口。升级后的wss请求也被转发到了8443端口。

第二个https请求包含第一个请求的sessionID： https://...&sid=C28C13EEEC525D203F8CA4E827605E0B.jvm1

根据Apache日志文件，我可以看到该sessionID被用于stickySession：

...为stickysession sid找到值C28C13EEEC525D203F8CA4E827605E0B.jvm1

...找到路由jvm1

...balancer：//http-localhost：worker（htttps://mcsgest1.desy.de:8443/Whiteboard/）被重写为htttps://mcsgest1.desy.de:8443/Whiteboard//?file=octocenter.xml&address=///&sid=C28C13EEEC525D203F8CA4E827605E0B.jvm1

第二个https请求仍然在8443端口，但升级到websocket协议后，ws-balancer不会评估sessionID并重写到8444：

...balancer://ws-localhost: worker (wss://mcsgest1.desy.de:8444/Whiteboard/)被重写为 wss://mcsgest1.desy.de:8444/Whiteboard//whiteboardendpoint

我需要在Apache配置中更改什么以启用wss协议的stickysession？我真的需要两个负载均衡器（http和ws）才能获得websocket平衡吗？

- Elke

Answer 1

您不需要为WebSockets单独使用负载均衡器，因为初始HTTP请求已经具有HTTP cookie并属于正确的实例。

您只需要检测连接升级并根据cookie的粘性部分手动路由请求即可。

确保为WebSockets加载代理模块 - mod_proxy_wstunnel。

例如：

SSLProxyEngine on
RewriteEngine On

<Proxy balancer://http-localhost/>
    BalancerMember  https://mcsgest1.desy.de:8443/Whiteboard/ route=jvm1 keepalive=On smax=1 connectiontimeout=10 retry=600 timeout=900 ttl=900
    BalancerMember  https://mcsgest1.desy.de:8444/Whiteboard/ route=jvm2 keepalive=On smax=1 connectiontimeout=10 retry=600 timeout=900 ttl=900
    ProxySet lbmethod=byrequests
    ProxySet stickysession=JSESSIONID|sid scolonpathdelim=On
</Proxy>

RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC]
RewriteCond %{HTTP:CONNECTION} Upgrade$ [NC]
RewriteCond %{HTTP_COOKIE} ^.*(JSESSIONID|sid)=([^=]*)\.jvm1 [NC]
RewriteRule .* wss://mcsgest1.desy.de:8443%{REQUEST_URI} [P,L]

RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC]
RewriteCond %{HTTP:CONNECTION} Upgrade$ [NC]
RewriteCond %{HTTP_COOKIE} ^.*(JSESSIONID|sid)=([^=]*)\.jvm2 [NC]
RewriteRule .* wss://mcsgest1.desy.de:8444%{REQUEST_URI} [P,L]

RewriteRule  /jddd/(.*)  balancer://http-localhost$1  [P,L]

ProxyPreserveHost On
ProxyRequests Off

ProxyPass /jddd/  balancer://http-localhost
ProxyPassReverse /jddd/ balancer://http-localhost

解释：

# if header upgrade = WebSocket
RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC]  
# and header connection contains Upgrade (header may be like this: connection=keep-alive, upgrade)
RewriteCond %{HTTP:CONNECTION} Upgrade$ [NC] 
# and header cookie contains JSESSIONID or sid, ending with sticky part - .jvm1 in that case
RewriteCond %{HTTP_COOKIE} ^.*(JSESSIONID|sid)=([^=]*)\.jvm1 [NC] 
#than we route request to application server via mod_proxy (P flag) and end rewrite rule check
RewriteRule .* wss://mcsgest1.desy.de:8443%{REQUEST_URI} [P] 

- user1516873

---

我的HTTP请求在/路径上，我的WebSocket请求在/push路径上。 这对于负载均衡的HTTP请求和WebSocket请求都适用吗？ - gbenroscience

你需要重复升级部分吗？我的意思是：RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC] RewriteCond %{HTTP:CONNECTION} Upgrade$ [NC] - gbenroscience

我按照您的指示进行设置，但是从Chrome控制台中，jvm#未添加到会话ID中。是否需要进行特殊配置？ - Maosheng Wang

@MaoshengWang，你应该在Tomcat配置中设置jvmRoute，例如 <Engine name="Catalina" defaultHost="localhost" jvmRoute="jvm1">。 - user1516873

我的后端，不是Tomcat，似乎需要找到另一种方法。 - Maosheng Wang