Kubernetes下SpringBoot应用零错误率滚动更新

为应用添加就绪探针

spec.template.spec.containers[0]:
添加探针

1
2
3
4
5
6
7
8
9
10
readinessProbe:
httpGet:
path: /health
port: 80
scheme: HTTP
initialDelaySeconds: 0
timeoutSeconds: 2
successThreshold: 1
failureThreshold: 2
periodSeconds: 2

redinessProbe是就绪探针, 如果没有此探针, Pod在启动成功后即为就绪状态. 此时应用可能正处于启动状态. 会导致访问到该Pod的请求失败.

Pod启动后k8s每2s访问一次/health
访问成功1次则表示Pod就绪
访问失败2次则表示Pod未就绪
每次访问超时时间为2s

SpringBoot 停机

  • SpringBoot需要优雅停机(等待30s后强制关闭应用)
  • 停机前先更新health接口, 然后等待kubernetes检测未就绪后才真正停止tomcat的链接器
1
2
3
4
5
NAME              DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
k8s-update-test 1 1 0 1 21h
k8s-update-test 1 2 1 1 21h
k8s-update-test 1 2 1 2 21h
k8s-update-test 1 1 1 1 21h

第3条会有一瞬间存在2个可用的, 此时其实旧的Pod已经正在关闭, 因此转发到该Pod的http请求会处理失败

更改后, 第3条处, 其实旧的Pod并没有真正关闭. 访问仍然可用. 等k8s检测到health不可用时会将流量切到新部署的应用. 此时旧的Pod还处于可用状态并会保持一小段时间(0-1s左右)
在此之后, 旧Pod的connection会暂停(而在0-1s之前流量已经不再转发至该Pod了). 已经建立的连接会继续处理(最长等待30s).
等旧Pod停机后, 此次更新结束.

  • java代码更改
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
@Configuration
public class GracefulShutdownConfiguration {
@Bean
public GracefulShutdown gracefulShutdown() {
return new GracefulShutdown();
}

@Bean
public EmbeddedServletContainerCustomizer tomcatCustomizer() {
return container -> {
if (container instanceof TomcatEmbeddedServletContainerFactory) {
((TomcatEmbeddedServletContainerFactory) container).addConnectorCustomizers(gracefulShutdown());
}
};
}

private static class GracefulShutdown implements TomcatConnectorCustomizer,
ApplicationListener<ContextClosedEvent> {

private static final Logger log = LoggerFactory.getLogger(GracefulShutdown.class);

private volatile Connector connector;

@Override
public void customize(Connector connector) {
this.connector = connector;
}

@Override
public void onApplicationEvent(ContextClosedEvent event) {

SystemStatus.HTTP_HEALTH.set(false);
log.info("Wait 5 seconds to give enough time for Kubernetes readiness probe to fail");
try {
TimeUnit.SECONDS.sleep(5);
} catch (InterruptedException e) {
e.printStackTrace();
}

this.connector.pause();
log.info("Shutting down ... (max wait 30s)");
Executor executor = this.connector.getProtocolHandler().getExecutor();
if (executor instanceof ThreadPoolExecutor) {
try {
ThreadPoolExecutor threadPoolExecutor = (ThreadPoolExecutor) executor;
threadPoolExecutor.shutdown();
if (!threadPoolExecutor.awaitTermination(30, TimeUnit.SECONDS)) {
log.warn("Tomcat thread pool did not shut down gracefully within "
+ "30 seconds. Proceeding with forceful shutdown");
}
} catch (InterruptedException ex) {
Thread.currentThread().interrupt();
}
}
}

}
}
1
2
3
4
5
6
7
8
@GetMapping("health")
public Result health(HttpServletResponse response) {
if (!SystemStatus.HTTP_HEALTH.get()) {
response.setStatus(HttpServletResponse.SC_SERVICE_UNAVAILABLE);
return Result.error(null);
}
return Result.success(null);
}
1
2
3
public class SystemStatus {
public static final AtomicBoolean HTTP_HEALTH = new AtomicBoolean(true);
}

https://github.com/spring-projects/spring-boot/issues/4657
https://github.com/RisingStack/kubernetes-graceful-shutdown-example

0%