Edu/Support

쿠버네티스 환경에서 Redis-Cluster Java App 구축 ②

Failover 테스트

🖧 Redis-Cluster 파드 다이어그램

┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ Redis-0 │ │ Redis-1 │ │ Redis-2 │ │ 🟢Master-1 │ │ 🟢Master-2 │ │ 🟢Master-3 │ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │ │ │ │ ┌────────────────│──────────────────┘ │ │ └──────────────────┐ └──────────────────┐ │ │ │ │ ▼ ▼ ▼ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ Redis-3 │ │ Redis-4 │ │ Redis-5 │ │ 🟡Replica-1 │ │ 🟡Replica-2 │ │ 🟡Replica-3 │ └──────────────┘ └──────────────┘ └──────────────┘

1. Failover(장애조치) 테스트 시작

SSH 터미널을 2개 열어서, Terminal-1에서는 App 파드 생성, 레디스 파드 다운 등을 실행하고, Ternimal-2에서 App 로그를 본다.

1) Java(Spring) App 파드 생성

서버 (Terminal-1 [~/kube/app/cluster/])에서 실행
$ kubectl apply -f app-deployment.yaml -> 파드 생성

💻 App 로그

서버 (Terminal-2 [~/kube/app/cluster])에서 실행
$ kubectl logs -f -l app=spring-redis-cluster-app
1초개 10개씩 입력([SUCCESS])되는 것을 확인할 수 있다. :: Spring Boot :: (v3.4.3) T11:04:25.996 : Starting KubeRedisClusterAsyncApplication v0.0.1-SNAPSHOT using Java 21.0.12 with PID 1 (/app/app.jar started by root in /app) T11:04:28.709 : Tomcat started on port 8080 (http) with context path '/' T11:04:28.732 :📍Started KubeRedisClusterAsyncApplication in 3.328 seconds (process running for 3.885) T11:04:28.736 : ----- 쿠버네티스 Redis-Cluster Async App(Spring Boot 3.4.3 / Lettuce, Java 21) ------ T11:04:29.624 :📍[SUCCESS] Key: keyA-00000010 -> Normal operation in progress... T11:04:30.624 : [SUCCESS] Key: keyA-00000020 -> Normal operation in progress... T11:04:31.624 : [SUCCESS] Key: keyA-00000030 -> Normal operation in progress... ... (반복) 중간 생략 ... T11:04:46.623 : [SUCCESS] Key: keyA-00000180 -> Normal operation in progress... T11:04:47.623 : [SUCCESS] Key: keyA-00000190 -> Normal operation in progress... T11:04:48.623 :📍[SUCCESS] Key: keyA-00000200 -> Normal operation in progress...

2) 레디스(Redis-0) 💥 파드 다운(delete)

Terminal-1) $ date; kubectl delete pod redis-cluster-0 -> 시각: 11:04:49

💻 Redis-0 로그

① 49초 레디스 다운 (+0초)
② 50초 레디스 재시작 (+1초)
③ 54초 마스터 -> 복제로 전환 (+5초) 1:M 11:04:49.302 *📍레디스 다운: User requested shutdown... 1:M 11:04:50.862 *📍레디스 재시작: Server initialized 1:S 11:04:54.587 *📍마스터 -> 복제로 전환: Connecting to MASTER 10.42.0.175:18504

Redis-0 전체 로그 ▼

1:signal-handler (1788660289) Received SIGTERM scheduling shutdown...
1:M 11:04:49.302 *📍User requested shutdown...
1:M 11:04:49.302 * 1 of 1 replicas are in sync when shutting down.
1:M 11:04:49.302 * Calling fsync() on the AOF file.
1:M 11:04:49.307 * Removing the pid file.
1:M 11:04:49.307 # Redis is now ready to exit, bye bye...
1:C 11:04:50.859 # WARNING: Changing databases number from 16 to 1 since we are in cluster mode
1:C 11:04:50.859 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 11:04:50.859 * Redis version=8.8.1, bits=64, commit=00000000, modified=1, pid=1, just started
1:C 11:04:50.859 * Configuration loaded
1:M 11:04:50.859 * monotonic clock: POSIX clock_gettime
                   Redis Open Source            
1:M 11:04:50.861 * Node configuration loaded, I'm aa8df00ebc2f7ac3001804dd01b75181249b16e1
1:M 11:04:50.862 *📍Server initialized
1:M 11:04:50.863 * Reading RDB base file on AOF loading...
1:M 11:04:50.863 * Loading RDB produced by version 8.8.1
1:M 11:04:50.863 * RDB age 1128089 seconds
1:M 11:04:50.863 * RDB memory usage when created 553.05 Mb
1:M 11:04:50.863 * RDB is base AOF
1:M 11:04:54.009 * Done loading RDB, keys loaded: 3240388, keys expired: 0.
1:M 11:04:54.009 * DB loaded from base file aof.2.base.rdb: 3.147 seconds
1:M 11:04:54.578 * DB loaded from incr file aof.2.incr.aof: 0.568 seconds
1:M 11:04:54.578 * DB loaded from append only file: 3.716 seconds
1:M 11:04:54.582 * Opening AOF incr file aof.2.incr.aof on server start
1:M 11:04:54.582 * Ready to accept connections tcp
1:M 11:04:54.587 * Configuration change detected. Reconfiguring myself as a replica of 11fae5f5710ff5dff65ea6c7865ca0a628458362 (redis-cluster-4)
1:S 11:04:54.587 * Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
1:S 11:04:54.587 *📍Connecting to MASTER 10.42.0.175:18504
1:S 11:04:54.587 * MASTER <-> REPLICA sync started
1:S 11:04:54.587 * Cluster state changed: ok
1:S 11:04:54.590 * Non blocking connect for SYNC fired the event.
1:S 11:04:54.597 * Master replied to PING, replication can continue...
1:S 11:04:54.598 * Trying a partial resynchronization (request fd41c9f456f555d2f81711f9d2367c254d4811e7:527152773).
1:S 11:04:54.598 * PSYNC is not possible, initialize RDB channel.
1:S 11:04:57.165 * Starting to receive RDB and replication stream in parallel.
1:S 11:04:57.186 * MASTER <-> REPLICA sync: receiving streamed RDB from master with EOF to disk
1:S 11:04:59.286 * Discarding previously cached master state.
1:S 11:04:59.286 * MASTER <-> REPLICA sync: Loading DB in memory
1:S 11:04:59.293 * MASTER <-> REPLICA sync: Flushing old data
1:S 11:04:59.525 * Loading RDB produced by version 8.8.1
1:S 11:04:59.525 * RDB age 2 seconds
1:S 11:04:59.525 * RDB memory usage when created 371.71 Mb
1:S 11:05:00.978 * Done loading RDB, keys loaded: 1943407, keys expired: 0.
1:S 11:05:00.978 * Removing the RDB file obtained from the master. This replica has persistence disabled
1:S 11:05:00.978 * MASTER <-> REPLICA sync: Finished with success
1:S 11:05:00.978 * MASTER <-> REPLICA sync: Starting AOF after a successful sync
1:S 11:05:00.978 * Creating AOF incr file temp-aof.incr on background rewrite
1:S 11:05:00.984 * Background append only file rewriting started by pid 13
1:S 11:05:00.984 * MASTER <-> REPLICA sync: Starting to stream replication buffer into the db (463 bytes).
1:S 11:05:00.985 * MASTER <-> REPLICA sync: Successfully streamed replication buffer into the db (463 bytes in total)
13:C 11:05:03.114 * BGSAVE done, 1943407 keys saved, 0 keys skipped, 244865779 bytes written.
13:C 11:05:03.124 * Successfully created the temporary AOF base file temp-rewriteaof-bg-13.aof
13:C 11:05:03.129 * Fork CoW for AOF rewrite: current 1 MB, peak 1 MB, average 1 MB
1:S 11:05:03.197 * Background AOF rewrite terminated with success
1:S 11:05:03.197 * Successfully renamed the temporary AOF base file temp-rewriteaof-bg-13.aof into aof.3.base.rdb
1:S 11:05:03.197 * Successfully renamed the temporary AOF incr file temp-aof.incr into aof.3.incr.aof
1:S 11:05:03.202 * Removing the history file aof.2.incr.aof in the background
1:S 11:05:03.202 * Removing the history file aof.2.base.rdb in the background
1:S 11:05:03.208 * Background AOF rewrite finished successfully

💻 Redis-4 로그

① 49초 마스터와 연결 끊김 (+0초)
② 54초 복제 -> 마스터로 전환(승격) (+5초)
1:S 11:04:49.335 *📍마스터와 연결 끊김: Connection with master lost. 1:M 11:04:54.557 *📍복제 -> 마스트로 승격: Discarding previously cached master state.

Redis-4 전체 로그 ▼

1:S 11:04:49.335 *📍Connection with master lost.
1:S 11:04:49.335 * Caching the disconnected master state.
1:S 11:04:49.336 * Reconnecting to MASTER 10.42.0.174:18504
1:S 11:04:49.336 * MASTER <-> REPLICA sync started
1:S 11:04:49.336 # Error condition on socket for SYNC: Connection refused
1:S 11:04:50.129 * Connecting to MASTER 10.42.0.174:18504
1:S 11:04:50.130 * MASTER <-> REPLICA sync started
1:S 11:04:53.562 * FAIL message received from 6f50eafcd2a642c90b1b3d69cfb3b619342ae9ea (redis-cluster-1) about aa8df00ebc2f7ac3001804dd01b75181249b16e1 (redis-cluster-0)
1:S 11:04:53.644 * Start of election delayed for 817 milliseconds (rank #0, offset 527152772).
1:S 11:04:54.548 * Starting a failover election for epoch 12.
1:S 11:04:54.557 * Failover election won: I'm the new master.
1:S 11:04:54.557 * configEpoch set to 12 after successful failover
1:M 11:04:54.557 *📍Discarding previously cached master state.
1:M 11:04:54.557 * Setting secondary replication ID to 696ad08778fd3b3ef0f9151ef8f964d0cfed880b, valid up to offset: 527152773. New replication ID is 50b86caf74cae2c28d5e66e362f9e0fc4c1230da
1:M 11:04:54.583 * Address updated for node aa8df00ebc2f7ac3001804dd01b75181249b16e1 (redis-cluster-0), now 10.42.0.208:18504
1:M 11:04:54.590 * A failover occurred in shard 68bd5bbafb1ea037e69667f5c9cd85d514ddf106; node aa8df00ebc2f7ac3001804dd01b75181249b16e1 () lost 0 slot(s) to node 11fae5f5710ff5dff65ea6c7865ca0a628458362 (redis-cluster-4) with a config epoch of 12
1:M 11:04:54.598 * Replica 10.42.0.208:18504 asks for synchronization
1:M 11:04:54.598 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for 'fd41c9f456f555d2f81711f9d2367c254d4811e7', my replication IDs are '50b86caf74cae2c28d5e66e362f9e0fc4c1230da' and '696ad08778fd3b3ef0f9151ef8f964d0cfed880b')
1:M 11:04:54.598 * Replica 10.42.0.208:18504 is capable of rdb channel synchronization, and partial sync isn't possible. Full sync will continue with dedicated rdb channel.
1:M 11:04:54.598 * Replica 10.42.0.208:18504 asks for synchronization
1:M 11:04:54.598 * Full resync requested by replica 10.42.0.208:18504 (rdb-channel)
1:M 11:04:54.598 * Delay next BGSAVE for diskless SYNC
1:M 11:04:54.649 * Clear FAIL state for node aa8df00ebc2f7ac3001804dd01b75181249b16e1 (redis-cluster-0):replica is reachable again.
1:M 11:04:57.165 * Starting BGSAVE for SYNC with target: replicas sockets (rdb-channel)
1:M 11:04:57.165 * Starting to deliver RDB and replication stream to replica: 10.42.0.208:18504
1:M 11:04:57.173 * Background RDB transfer started by pid 26 to replica socket
26:C 11:04:59.267 * BGSAVE done, 1943407 keys saved, 0 keys skipped, 244865912 bytes written.
26:C 11:04:59.273 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
1:M 11:04:59.288 * Background RDB transfer terminated with success
1:M 11:04:59.288 * Synchronization with replica 10.42.0.208:18504 succeeded
1:M 11:04:59.288 * Streamed RDB transfer with replica 10.42.0.208:18504 succeeded (socket). Waiting for REPLCONF ACK from replica to enable streaming
1:M 11:04:59.288 * RDB transfer completed, rdb only replica (10.42.0.208:18504) should be disconnected asap
1:M 11:04:59.288 * Connection with replica (rdbchannel) 10.42.0.208:18504 lost.

💻 App 로그

Terminal-2) $ kubectl logs -f -l app=spring-redis-sentinel-app
① 49초 레디스 연결 끊김 -> 재연결 시도 (+0초)
② 50초 Redis-1,2에는 데이터 입력됨 (+1초) ③ 53초~54초 Redis-0 4건 입력 실패 (+4초~+5초) 이후 Redis-4가 마스터로 전환(승격)되고 정상 입력됨
④ 59초 Redis-0 접속 성공 -> 입력 성공 (+10초) T11:04:49.396 INFO :📍Reconnecting, last destination was redis-cluster-0.redis-cluster-service.default.svc.cluster.local/10.42.0.174:18504 T11:04:50.623 INFO :📍[SUCCESS] Key: keyA-00000220 -> Normal operation in progress... T11:04:53.342 ERROR :📍[FAIL] Key: keyA-00000207 -> Error after retries: Redis exception T11:04:54.424 ERROR :📍[FAIL] Key: keyA-00000218 -> Error after retries: Redis exception T11:04:59.096 INFO :📍Reconnected to redis-cluster-0.redis-cluster-service.default.svc.cluster.local/<unresolved>:18504

App 전체 로그 ▼

T11:04:49.396  INFO [xecutorLoop-1-4] i.l.core.protocol.ConnectionWatchdog     :📍Reconnecting, last destination was redis-cluster-0.redis-cluster-service.default.svc.cluster.local/10.42.0.174:18504
T11:04:49.399  WARN [xecutorLoop-1-4] i.l.core.protocol.ConnectionWatchdog     : Cannot reconnect to [redis-cluster-0.redis-cluster-service.default.svc.cluster.local/<unresolved>:18504]: Connection refused: redis-cluster-0.redis-cluster-service.default.svc.cluster.local/10.42.0.174:18504
T11:04:50.623  INFO [         task-8] c.r.K.RedisOperationService              :📍[SUCCESS] Key: keyA-00000220 -> Normal operation in progress...
T11:04:51.284  WARN [ioEventLoop-4-3] i.l.c.c.t.DefaultClusterTopologyRefresh  : Unable to connect to [redis-cluster-0.redis-cluster-service.default.svc.cluster.local/<unresolved>:18504]: connection timed out after 1000 ms: redis-cluster-0.redis-cluster-service.default.svc.cluster.local/10.42.0.174:18504
T11:04:51.623  INFO [         task-5] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000230 -> Normal operation in progress...
T11:04:52.623  INFO [         task-1] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000240 -> Normal operation in progress...
T11:04:53.283  WARN [ioEventLoop-4-7] i.l.c.c.t.DefaultClusterTopologyRefresh  : Unable to connect to [redis-cluster-0.redis-cluster-service.default.svc.cluster.local/<unresolved>:18504]: connection timed out after 1000 ms: redis-cluster-0.redis-cluster-service.default.svc.cluster.local/10.42.0.174:18504
T11:04:53.342 ERROR [         task-7] c.r.K.RedisOperationService              :📍[FAIL] Key: keyA-00000207 -> Error after retries: Redis exception
T11:04:53.623  INFO [         task-1] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000250 -> Normal operation in progress...
T11:04:53.624 ERROR [         task-4] c.r.K.RedisOperationService              :  [FAIL] Key: keyA-00000210 -> Error after retries: Redis exception
T11:04:54.024 ERROR [         task-6] c.r.K.RedisOperationService              :  [FAIL] Key: keyA-00000214 -> Error after retries: Redis exception
T11:04:54.424 ERROR [         task-2] c.r.K.RedisOperationService              :📍 [FAIL] Key: keyA-00000218 -> Error after retries: Redis exception
T11:04:54.896  WARN [ioEventLoop-4-3] i.l.c.c.t.DefaultClusterTopologyRefresh  : Unable to connect to [redis-cluster-0.redis-cluster-service.default.svc.cluster.local/<unresolved>:18504]: connection timed out after 1000 ms: redis-cluster-0.redis-cluster-service.default.svc.cluster.local/10.42.0.174:18504
T11:04:54.993  INFO [xecutorLoop-1-5] i.l.core.protocol.ConnectionWatchdog     : Reconnecting, last destination was redis-cluster-0.redis-cluster-service.default.svc.cluster.local/<unresolved>:18504
T11:04:55.623  INFO [         task-4] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000270 -> Normal operation in progress...
T11:04:55.623  INFO [         task-1] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000260 -> Normal operation in progress...
T11:04:55.995  WARN [ioEventLoop-4-5] i.l.core.protocol.ConnectionWatchdog     : Cannot reconnect to [redis-cluster-0.redis-cluster-service.default.svc.cluster.local/<unresolved>:18504]: connection timed out after 1000 ms: redis-cluster-0.redis-cluster-service.default.svc.cluster.local/10.42.0.174:18504
T11:04:55.996  WARN [ioEventLoop-4-6] i.l.c.c.t.DefaultClusterTopologyRefresh  : Unable to connect to [redis-cluster-0.redis-cluster-service.default.svc.cluster.local/<unresolved>:18504]: connection timed out after 1000 ms: redis-cluster-0.redis-cluster-service.default.svc.cluster.local/10.42.0.174:18504
T11:04:56.623  INFO [         task-5] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000280 -> Normal operation in progress...
T11:04:57.196  WARN [ioEventLoop-4-2] i.l.c.c.t.DefaultClusterTopologyRefresh  : Unable to connect to [redis-cluster-0.redis-cluster-service.default.svc.cluster.local/<unresolved>:18504]: connection timed out after 1000 ms: redis-cluster-0.redis-cluster-service.default.svc.cluster.local/10.42.0.174:18504
T11:04:57.623  INFO [         task-3] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000290 -> Normal operation in progress...
T11:04:58.284  WARN [ioEventLoop-4-4] i.l.c.c.t.DefaultClusterTopologyRefresh  : Unable to connect to [redis-cluster-0.redis-cluster-service.default.svc.cluster.local/<unresolved>:18504]: connection timed out after 1000 ms: redis-cluster-0.redis-cluster-service.default.svc.cluster.local/10.42.0.174:18504
T11:04:58.623  INFO [         task-1] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000300 -> Normal operation in progress...
T11:04:59.096  INFO [ioEventLoop-4-7] i.l.core.protocol.ReconnectionHandler    :📍Reconnected to redis-cluster-0.redis-cluster-service.default.svc.cluster.local/<unresolved>:18504
T11:04:59.623  INFO [         task-7] c.r.K.RedisOperationService              :📍[SUCCESS] Key: keyA-00000310 -> Normal operation in progress...
T11:05:00.623  INFO [         task-5] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000320 -> Normal operation in progress...
T11:05:01.623  INFO [         task-3] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000330 -> Normal operation in progress...
T11:05:02.623  INFO [         task-1] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000340 -> Normal operation in progress...
T11:05:03.623  INFO [         task-7] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000350 -> Normal operation in progress...
T11:05:04.623  INFO [         task-5] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000360 -> Normal operation in progress...
T11:05:05.623  INFO [         task-3] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000370 -> Normal operation in progress...
T11:05:06.623  INFO [         task-1] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000380 -> Normal operation in progress...
T11:05:07.623  INFO [         task-7] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000390 -> Normal operation in progress...
T11:05:08.623  INFO [         task-5] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000400 -> Normal operation in progress...
T11:05:09.623  INFO [         task-3] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000410 -> Normal operation in progress...
T11:05:10.623  INFO [         task-1] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000420 -> Normal operation in progress...
T11:05:11.623  INFO [         task-7] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000430 -> Normal operation in progress...
T11:05:12.623  INFO [         task-5] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000440 -> Normal operation in progress...
T11:05:13.623  INFO [         task-3] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000450 -> Normal operation in progress...
T11:05:14.623  INFO [         task-1] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000460 -> Normal operation in progress...
T11:05:15.623  INFO [         task-7] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000470 -> Normal operation in progress...
T11:05:16.623  INFO [         task-5] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000480 -> Normal operation in progress...
T11:05:17.622  INFO [         task-3] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000490 -> Normal operation in progress...
T11:05:18.623  INFO [         task-1] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000500 -> Normal operation in progress...
T11:05:19.623  INFO [         task-7] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000510 -> Normal operation in progress...
T11:05:20.623  INFO [         task-5] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000520 -> Normal operation in progress...
T11:05:21.623  INFO [         task-3] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000530 -> Normal operation in progress...
T11:05:22.622  INFO [         task-1] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000540 -> Normal operation in progress...
T11:05:23.623  INFO [         task-7] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000550 -> Normal operation in progress...
T11:05:24.623  INFO [         task-5] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000560 -> Normal operation in progress...
T11:05:25.623  INFO [         task-3] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000570 -> Normal operation in progress...
T11:05:26.623  INFO [         task-1] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000580 -> Normal operation in progress...
T11:05:27.623  INFO [         task-7] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000590 -> Normal operation in progress...
T11:05:28.623  INFO [         task-5] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000600 -> Normal operation in progress...
T11:05:29.623  INFO [         task-3] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000610 -> Normal operation in progress...
T11:05:30.623  INFO [         task-1] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000620 -> Normal operation in progress...
T11:05:31.623  INFO [         task-7] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000630 -> Normal operation in progress...
T11:05:32.626  INFO [         task-5] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000640 -> Normal operation in progress...
T11:05:33.623  INFO [         task-3] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000650 -> Normal operation in progress...
T11:05:34.623  INFO [         task-1] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000660 -> Normal operation in progress...
T11:05:35.623  INFO [         task-7] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000670 -> Normal operation in progress...
T11:05:36.622  INFO [         task-5] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000680 -> Normal operation in progress...
T11:05:37.623  INFO [         task-3] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000690 -> Normal operation in progress...
T11:05:38.624  INFO [         task-1] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000700 -> Normal operation in progress...
T11:05:39.622  INFO [         task-7] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000710 -> Normal operation in progress...
T11:05:40.622  INFO [         task-5] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000720 -> Normal operation in progress...
T11:05:41.622  INFO [         task-3] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000730 -> Normal operation in progress...
T11:05:42.623  INFO [         task-1] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000740 -> Normal operation in progress...
T11:05:43.622  INFO [         task-7] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000750 -> Normal operation in progress...
T11:05:44.622  INFO [         task-5] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000760 -> Normal operation in progress...
T11:05:45.622  INFO [         task-3] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000770 -> Normal operation in progress...
T11:05:46.622  INFO [         task-1] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000780 -> Normal operation in progress...
T11:05:47.622  INFO [         task-7] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000790 -> Normal operation in progress...
T11:05:48.622  INFO [         task-5] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000800 -> Normal operation in progress...
T11:05:49.623  INFO [         task-3] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000810 -> Normal operation in progress...

🖧 Redis-Cluster 파드 다이어그램: Failover 상태 (Redis-4 Master-1)

┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ Redis-0 │ │ Redis-1 │ │ Redis-2 │ │ 🟡Replica-2 │ │ 🟢Master-2 │ │ 🟢Master-3 │ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ ▲ │ │ │ ┌────────────────│──────────────────┘ │ │ └──────────────────┐ └──────────────────┐ │ │ │ │ ▼ │ ▼ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ Redis-3 │ │ Redis-4 │ │ Redis-5 │ │ 🟡Replica-1 │ │ 🟢Master-1 │ │ 🟡Replica-3 │ └──────────────┘ └──────────────┘ └──────────────┘

2. Failback(원상복구) 테스트

1) 레디스(Redis-4) 파드 💥 다운(delete)

Terminal-1) $ date; kubectl delete pod redis-cluster-4 -> 시각: 11:05:49

💻 Redis-4 로그

① 49초 레디스 다운 (+0초)
② 52초 레디스 재시작 (+3초)
③ 55초 마스터 -> 복제로 전환 (+6초) 1:signal-handler (1788660349) Received SIGTERM scheduling shutdown... 1:M 11:05:49.852 *📍레디스 다운: User requested shutdown... 1:M 11:05:52.004 *📍레디스 재시작: Server initialized 1:S 11:05:55.865 *📍마스터 -> 복제로 전환: Before turning into a replica, ...

Redis-4 전체 로그 ▼

1:signal-handler (1788660349) Received SIGTERM scheduling shutdown...
1:M 11:05:49.852 *📍User requested shutdown...
1:M 11:05:49.852 * Waiting for replicas before shutting down.
1:M 11:05:50.054 * 1 of 1 replicas are in sync when shutting down.
1:M 11:05:50.054 * Calling fsync() on the AOF file.
1:M 11:05:50.060 * Removing the pid file.
1:M 11:05:50.060 # Redis is now ready to exit, bye bye...
1:C 11:05:52.001 # WARNING: Changing databases number from 16 to 1 since we are in cluster mode
1:C 11:05:52.001 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 11:05:52.001 * Redis version=8.8.1, bits=64, commit=00000000, modified=1, pid=1, just started
1:C 11:05:52.001 * Configuration loaded
1:M 11:05:52.002 * monotonic clock: POSIX clock_gettime
                   Redis Open Source
1:M 11:05:52.003 * Node configuration loaded, I'm 11fae5f5710ff5dff65ea6c7865ca0a628458362
1:M 11:05:52.004 *📍Server initialized
1:M 11:05:52.005 * Reading RDB base file on AOF loading...
1:M 11:05:52.005 * Loading RDB produced by version 8.8.1
1:M 11:05:52.005 * RDB age 1127569 seconds
1:M 11:05:52.005 * RDB memory usage when created 553.06 Mb
1:M 11:05:52.005 * RDB is base AOF
1:M 11:05:55.187 * Done loading RDB, keys loaded: 3240388, keys expired: 0.
1:M 11:05:55.187 * DB loaded from base file aof.3.base.rdb: 3.183 seconds
1:M 11:05:55.856 * DB loaded from incr file aof.3.incr.aof: 0.668 seconds
1:M 11:05:55.856 * DB loaded from append only file: 3.851 seconds
1:M 11:05:55.860 * Opening AOF incr file aof.3.incr.aof on server start
1:M 11:05:55.860 * Ready to accept connections tcp
1:M 11:05:55.865 * Configuration change detected. Reconfiguring myself as a replica of aa8df00ebc2f7ac3001804dd01b75181249b16e1 (redis-cluster-0)
1:S 11:05:55.865 *📍Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
1:S 11:05:55.865 * Connecting to MASTER 10.42.0.208:18504
1:S 11:05:55.865 * MASTER <-> REPLICA sync started
1:S 11:05:55.866 * Cluster state changed: ok
1:S 11:05:55.870 * Non blocking connect for SYNC fired the event.
1:S 11:05:55.874 * Master replied to PING, replication can continue...
1:S 11:05:55.874 * Trying a partial resynchronization (request c663cc92a44ee197afdeb6e737aa6dde6ddbcb29:527159456).
1:S 11:05:55.875 * PSYNC is not possible, initialize RDB channel.
1:S 11:05:58.085 * Starting to receive RDB and replication stream in parallel.
1:S 11:05:58.094 * MASTER <-> REPLICA sync: receiving streamed RDB from master with EOF to disk
1:S 11:06:00.140 * Discarding previously cached master state.
1:S 11:06:00.140 * MASTER <-> REPLICA sync: Loading DB in memory
1:S 11:06:00.145 * MASTER <-> REPLICA sync: Flushing old data
1:S 11:06:00.380 * Loading RDB produced by version 8.8.1
1:S 11:06:00.380 * RDB age 2 seconds
1:S 11:06:00.380 * RDB memory usage when created 332.27 Mb
1:S 11:06:02.021 * Done loading RDB, keys loaded: 1943407, keys expired: 0.
1:S 11:06:02.021 * Removing the RDB file obtained from the master. This replica has persistence disabled
1:S 11:06:02.021 * MASTER <-> REPLICA sync: Finished with success
1:S 11:06:02.021 * MASTER <-> REPLICA sync: Starting AOF after a successful sync
1:S 11:06:02.021 * Creating AOF incr file temp-aof.incr on background rewrite
1:S 11:06:02.033 * Background append only file rewriting started by pid 19
1:S 11:06:02.033 * MASTER <-> REPLICA sync: Starting to stream replication buffer into the db (463 bytes).
1:S 11:06:02.034 * MASTER <-> REPLICA sync: Successfully streamed replication buffer into the db (463 bytes in total)
19:C 11:06:04.126 * BGSAVE done, 1943407 keys saved, 0 keys skipped, 244865779 bytes written.
19:C 11:06:04.130 * Successfully created the temporary AOF base file temp-rewriteaof-bg-19.aof
19:C 11:06:04.139 * Fork CoW for AOF rewrite: current 152 MB, peak 152 MB, average 80 MB
1:S 11:06:04.245 * Background AOF rewrite terminated with success
1:S 11:06:04.246 * Successfully renamed the temporary AOF base file temp-rewriteaof-bg-19.aof into aof.4.base.rdb
1:S 11:06:04.246 * Successfully renamed the temporary AOF incr file temp-aof.incr into aof.4.incr.aof
1:S 11:06:04.248 * Removing the history file aof.3.incr.aof in the background
1:S 11:06:04.248 * Removing the history file aof.3.base.rdb in the background
1:S 11:06:04.251 * Background AOF rewrite finished successfully

💻 Redis-0 로그

① 50초 마스터와 연결 끊김 (+1초)
② 55초 복제 -> 마스터로 전환 (+5초)
1:S 11:05:50.092 *📍Connection with master lost. 1:M 11:05:55.784 *📍Discarding previously cached master state.

Redis-0 전체 로그 ▼

1:S 11:05:50.092 *📍Connection with master lost.
1:S 11:05:50.092 * Caching the disconnected master state.
1:S 11:05:50.092 * Reconnecting to MASTER 10.42.0.175:18504
1:S 11:05:50.092 * MASTER <-> REPLICA sync started
1:S 11:05:50.092 # Error condition on socket for SYNC: Connection refused
1:S 11:05:51.053 * Connecting to MASTER 10.42.0.175:18504
1:S 11:05:51.053 * MASTER <-> REPLICA sync started
1:S 11:05:54.777 * FAIL message received from 9eba06300dd9acd8f994aa9faa8765009c7de4e9 (redis-cluster-2) about 11fae5f5710ff5dff65ea6c7865ca0a628458362 (redis-cluster-4)
1:S 11:05:54.869 * Start of election delayed for 869 milliseconds (rank #0, offset 527159455).
1:S 11:05:55.773 * Starting a failover election for epoch 13.
1:S 11:05:55.784 * Failover election won: I'm the new master.
1:S 11:05:55.784 * configEpoch set to 13 after successful failover
1:M 11:05:55.784 *📍Discarding previously cached master state.
1:M 11:05:55.784 * Setting secondary replication ID to 50b86caf74cae2c28d5e66e362f9e0fc4c1230da, valid up to offset: 527159456. New replication ID is 9f1b65b33a8f3f098ce81e49a44dba0863e1ee65
1:M 11:05:55.862 * Address updated for node 11fae5f5710ff5dff65ea6c7865ca0a628458362 (redis-cluster-4), now 10.42.0.209:18504
1:M 11:05:55.870 * A failover occurred in shard 68bd5bbafb1ea037e69667f5c9cd85d514ddf106; node 11fae5f5710ff5dff65ea6c7865ca0a628458362 () lost 0 slot(s) to node aa8df00ebc2f7ac3001804dd01b75181249b16e1 (redis-cluster-0) with a config epoch of 13
1:M 11:05:55.874 * Clear FAIL state for node 11fae5f5710ff5dff65ea6c7865ca0a628458362 (redis-cluster-4):replica is reachable again.
1:M 11:05:55.874 * Replica 10.42.0.209:18504 asks for synchronization
1:M 11:05:55.874 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for 'c663cc92a44ee197afdeb6e737aa6dde6ddbcb29', my replication IDs are '9f1b65b33a8f3f098ce81e49a44dba0863e1ee65' and '50b86caf74cae2c28d5e66e362f9e0fc4c1230da')
1:M 11:05:55.874 * Replica 10.42.0.209:18504 is capable of rdb channel synchronization, and partial sync isn't possible. Full sync will continue with dedicated rdb channel.
1:M 11:05:55.875 * Replica 10.42.0.209:18504 asks for synchronization
1:M 11:05:55.875 * Full resync requested by replica 10.42.0.209:18504 (rdb-channel)
1:M 11:05:55.875 * Delay next BGSAVE for diskless SYNC
1:M 11:05:58.085 * Starting BGSAVE for SYNC with target: replicas sockets (rdb-channel)
1:M 11:05:58.085 * Starting to deliver RDB and replication stream to replica: 10.42.0.209:18504
1:M 11:05:58.089 * Background RDB transfer started by pid 20 to replica socket
20:C 11:06:00.135 * BGSAVE done, 1943407 keys saved, 0 keys skipped, 244865912 bytes written.
20:C 11:06:00.139 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
1:M 11:06:00.202 * Background RDB transfer terminated with success
1:M 11:06:00.202 * Synchronization with replica 10.42.0.209:18504 succeeded
1:M 11:06:00.202 * Streamed RDB transfer with replica 10.42.0.209:18504 succeeded (socket). Waiting for REPLCONF ACK from replica to enable streaming
1:M 11:06:00.202 * RDB transfer completed, rdb only replica (10.42.0.209:18504) should be disconnected asap
1:M 11:06:00.202 * Connection with replica (rdbchannel) 10.42.0.209:18504 lost.

💻 App 로그

① 50초 Redis-4 연결 끊김 (+1초)
② 54초~56초 5건 입력 실패 (+4초~+6초) 이후 Redis-0이 마스터로 전화(승격)되고 정상 입력됨
③ 59초 Redis-4 연결 성공 (+10초)
T11:05:50.193 INFO 📍Reconnecting, last destination was redis-cluster-4.redis-cluster-service.default.svc.cluster.local/10.42.0.175:18504 T11:05:54.723 ERROR 📍[FAIL] Key: keyA-00000821 -> Error after retries: Redis exception T11:05:56.223 ERROR 📍[FAIL] Key: keyA-00000836 -> Error after retries: Redis exception T11:05:59.896 INFO 📍Reconnected to redis-cluster-4.redis-cluster-service.default.svc.cluster.local/<unresolved>:18504

App 전체 로그 ▼

T11:05:50.193  INFO [xecutorLoop-1-1] i.l.core.protocol.ConnectionWatchdog     :📍Reconnecting, last destination was redis-cluster-4.redis-cluster-service.default.svc.cluster.local/10.42.0.175:18504
T11:05:50.194  WARN [ioEventLoop-4-8] i.l.core.protocol.ConnectionWatchdog     : Cannot reconnect to [redis-cluster-4.redis-cluster-service.default.svc.cluster.local/<unresolved>:18504]: Connection refused: redis-cluster-4.redis-cluster-service.default.svc.cluster.local/10.42.0.175:18504
T11:05:50.623  INFO [         task-1] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000820 -> Normal operation in progress...
T11:05:51.284  WARN [ioEventLoop-4-5] i.l.c.c.t.DefaultClusterTopologyRefresh  : Unable to connect to [redis-cluster-4.redis-cluster-service.default.svc.cluster.local/<unresolved>:18504]: connection timed out after 1000 ms: redis-cluster-4.redis-cluster-service.default.svc.cluster.local/10.42.0.175:18504
T11:05:51.623  INFO [         task-8] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000830 -> Normal operation in progress...
T11:05:52.622  INFO [         task-4] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000840 -> Normal operation in progress...
T11:05:53.283  WARN [ioEventLoop-4-1] i.l.c.c.t.DefaultClusterTopologyRefresh  : Unable to connect to [redis-cluster-4.redis-cluster-service.default.svc.cluster.local/<unresolved>:18504]: connection timed out after 1000 ms: redis-cluster-4.redis-cluster-service.default.svc.cluster.local/10.42.0.175:18504
T11:05:53.622  INFO [         task-8] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000850 -> Normal operation in progress...
T11:05:54.597  WARN [ioEventLoop-4-5] i.l.c.c.t.DefaultClusterTopologyRefresh  : Unable to connect to [redis-cluster-4.redis-cluster-service.default.svc.cluster.local/<unresolved>:18504]: connection timed out after 1000 ms: redis-cluster-4.redis-cluster-service.default.svc.cluster.local/10.42.0.175:18504
T11:05:54.723 ERROR [         task-2] c.r.K.RedisOperationService              :📍[FAIL] Key: keyA-00000821 -> Error after retries: Redis exception
T11:05:55.123 ERROR [         task-6] c.r.K.RedisOperationService              :  [FAIL] Key: keyA-00000825 -> Error after retries: Redis exception
T11:05:55.523 ERROR [         task-7] c.r.K.RedisOperationService              :  [FAIL] Key: keyA-00000829 -> Error after retries: Redis exception
T11:05:55.697  WARN [ioEventLoop-4-8] i.l.c.c.t.DefaultClusterTopologyRefresh  : Unable to connect to [redis-cluster-4.redis-cluster-service.default.svc.cluster.local/<unresolved>:18504]: connection timed out after 1000 ms: redis-cluster-4.redis-cluster-service.default.svc.cluster.local/10.42.0.175:18504
T11:05:55.793  INFO [xecutorLoop-1-5] i.l.core.protocol.ConnectionWatchdog     : Reconnecting, last destination was redis-cluster-4.redis-cluster-service.default.svc.cluster.local/<unresolved>:18504
T11:05:55.823 ERROR [         task-3] c.r.K.RedisOperationService              :  [FAIL] Key: keyA-00000832 -> Error after retries: Redis exception
T11:05:56.223 ERROR [         task-5] c.r.K.RedisOperationService              :📍[FAIL] Key: keyA-00000836 -> Error after retries: Redis exception
T11:05:56.225  INFO [         task-5] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000870 -> Normal operation in progress...
T11:05:56.795  WARN [ioEventLoop-4-6] i.l.core.protocol.ConnectionWatchdog     : Cannot reconnect to [redis-cluster-4.redis-cluster-service.default.svc.cluster.local/<unresolved>:18504]: connection timed out after 1000 ms: redis-cluster-4.redis-cluster-service.default.svc.cluster.local/10.42.0.175:18504
T11:05:56.796  WARN [ioEventLoop-4-3] i.l.c.c.t.DefaultClusterTopologyRefresh  : Unable to connect to [redis-cluster-4.redis-cluster-service.default.svc.cluster.local/<unresolved>:18504]: connection timed out after 1000 ms: redis-cluster-4.redis-cluster-service.default.svc.cluster.local/10.42.0.175:18504
T11:05:56.832  INFO [         task-3] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000880 -> Normal operation in progress...
T11:05:57.124  INFO [         task-6] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000860 -> Normal operation in progress...
T11:05:57.623  INFO [         task-4] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000890 -> Normal operation in progress...
T11:05:57.995  WARN [ioEventLoop-4-6] i.l.c.c.t.DefaultClusterTopologyRefresh  : Unable to connect to [redis-cluster-4.redis-cluster-service.default.svc.cluster.local/<unresolved>:18504]: connection timed out after 1000 ms: redis-cluster-4.redis-cluster-service.default.svc.cluster.local/10.42.0.175:18504
T11:05:58.623  INFO [         task-8] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000900 -> Normal operation in progress...
T11:05:59.284  WARN [ioEventLoop-4-8] i.l.c.c.t.DefaultClusterTopologyRefresh  : Unable to connect to [redis-cluster-4.redis-cluster-service.default.svc.cluster.local/<unresolved>:18504]: connection timed out after 1000 ms: redis-cluster-4.redis-cluster-service.default.svc.cluster.local/10.42.0.175:18504
T11:05:59.622  INFO [         task-3] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000910 -> Normal operation in progress...
T11:05:59.896  INFO [ioEventLoop-4-7] i.l.core.protocol.ReconnectionHandler    :📍Reconnected to redis-cluster-4.redis-cluster-service.default.svc.cluster.local/<unresolved>:18504
T11:06:00.623  INFO [         task-4] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000920 -> Normal operation in progress...
T11:06:01.622  INFO [         task-1] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000930 -> Normal operation in progress...
T11:06:02.622  INFO [         task-8] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000940 -> Normal operation in progress...
T11:06:03.623  INFO [         task-3] c.r.K.RedisOperationService              :  [SUCCESS] Key: keyA-00000950 -> Normal operation in progress...

🖧 Redis-Cluster 파드 다이어그램: 원상복구 상태 (Redis-0 Master-1)

🧑‍💻 테스트 검토 및 핵심 요약

이번 장애 조치(Failover/Failback) 테스트를 통해 쿠버네티스 환경의 Redis-Cluster와 Java(Spring Boot / Lettuce) 애플리케이션 간의 실제 장애 대응 동작 및 클러스터 토폴로지 재구성 과정을 검증했습니다.

⏱️ 장애 조치 타임라인 (Timeline)

구분	시각 (상대 시간)	Redis Cluster 노드 동작	Java (Spring) App 동작
장애 발생	11:04:49 (+0초)	Master(Redis-0) 파드 종료 (delete)	Redis-0 연결 끊김 및 백그라운드 재접속 시도. 📍정상 노드(Redis-1, 2)를 향하는 데이터는 Async 처리에 의해 블로킹 없이 계속 입력 성공 (11:04:50~)
재시도 한계 도달	11:04:53~54 (+4~5초)	Replica(Redis-4)가 장애 감지 후 승격 투표(Election) 진행 중	Redis-0 슬롯으로 향하던 일부 요청(4건)이 백그라운드 5회(5초) 재시도 임계치를 넘겨 최종 실패(FAIL) 처리됨
Failover 완료	11:04:54 (+5초)	투표 완료 후 Replica(Redis-4)를 신규 Master로 승격	클러스터 토폴로지 갱신 시도 (ClusterTopologyRefresh)
완전 정상화	11:04:55 (+6초)	신규 Master(Redis-4)가 해당 슬롯(0~5460) 쓰기/읽기 시작	변경된 토폴로지가 적용되어 모든 슬롯에 대한 데이터가 다시 100% 정상(SUCCESS) 쓰기 재개됨
구 마스터 재기동	11:04:59 (+10초)	재시작된 구 마스터(Redis-0)가 신규 마스터(Redis-4)의 Replica 노드로 재구성됨	Redis-0 노드 연결 재확립(Reconnected) 완료

💡 실무 적용을 위한 핵심 시사점 (Key Takeaways) ▼

1. Redis-0 마스터가 다운되어도 살아있는 Redis-1,2에는 정상적으로 입력됨
2. Cluster 노드 간 장애 감지 및 승격 시간 (약 4~5초):
Master 노드가 다운되면 나머지 노드들이 cluster-node-timeout 동안 상태를 확인(FAIL 판정)한 후, Replica 노드가 자동 승격하기까지 약 4~5초의 다운타임이 발생합니다.
3. Spring / Lettuce 클라이언트의 Dynamic Topology Refresh(토폴로지 자동 갱신)의 중요성:
Redis Cluster 환경에서는 Master 노드가 변경되면 해당 노드가 담당하던 해시 슬롯(Hash Slot)의 소유권이 넘어갑니다. Spring App(Lettuce)의 클러스터 토폴로지 자동 갱신 옵션이 동작하여 변경된 Master IP와 슬롯 정보를 실시간으로 감지했기 때문에 별도의 서비스 재시작 없이 연동이 유지되었습니다.
4. Client-Side Retry(5초)의 효과와 한계:
첫 번째 테스트(Redis-0 다운)에서는 App의 Retry(5초) 범위 내인 5초 시점에 Failover가 완료되어 성공률 100%(유실 0개)를 기록했습니다. 반면 두 번째 테스트(Redis-4 다운)에서는 토폴로지 재조율 시간 차이로 인해 1개의 실패 요청(keyA-00000607)이 발생했습니다. 따라서 실무에서는 App Retry뿐만 아니라 비동기 큐 처리 및 재시도 메커니즘을 함께 설계해야 완벽한 무중단을 보장할 수 있습니다.
5. Cluster의 자동 노드 역할 재구성(Re-configuration):
복구된 구 Master(Redis-0) 노드가 다시 켜지면, 클러스터 버스를 통해 변경된 노드 상태를 수신하고 자동으로 신규 Master(Redis-4)의 Replica 노드로 전환됩니다. 따라서 관리자의 수동 개입 없이도 클러스터 전체의 HA(고가용성) 타깃 수량이 복구됩니다.
6. 사용자 경험(UX) 및 타임아웃 고려사항:
클라이언트 관점에서는 장애 발생 시 약 4~5초간의 응답 지연(Latency)이 발생합니다. 웹/앱 프론트엔드 API 호출 시 사용자 화면이 멈추지 않도록 Front-end Timeout 및 Circuit Breaker(Resilience4j 등)를 적절히 조합하는 것을 권장합니다.

🛡️ Front-end Timeout 및 Circuit Breaker 설명 ▼

🛡️ 레디스 장애 시 시스템 마비를 막는 안전장치

센티널의 장애 조치(Failover) 동안 발생하는 4~5초간의 응답 지연(Latency)은 백엔드 서버와 사용자 화면으로 장애가 전파되는 연쇄 장애(Cascading Failure)를 유발할 수 있습니다. 이를 방지하기 위해 실무에서는 다음 두 가지 안전장치를 반드시 결합하여 구축합니다.

1. Front-end Timeout (프론트엔드 타임아웃)

사용자 브라우저나 모바일 앱이 백엔드 API를 호출한 후 "응답이 오지 않을 때 무한정 대기하지 않고 요청을 포기하는 최대 시간"입니다.

문제 상황 (Timeout 미설정 시): 레디스 장애 동안 4초간 응답이 멈추면, 답답한 사용자가 '새로고침'이나 '버튼'을 연타합니다. 이로 인해 백엔드 Tomcat/Spring 쓰레드(Thread)가 고갈되어 웹 서버 전체가 다운됩니다.
해결 방안 (2~3초 Timeout 적용): 2초 내 응답이 없으면 즉시 요청을 중단하고 "일시적인 네트워크 지연입니다. 잠시 후 다시 시도해 주세요"라는 안내를 띄워 백엔드 폭주를 차단합니다.

2. Circuit Breaker (서킷 브레이커 - Resilience4j)

전기 배전반의 '차단기(두꺼비집)'처럼 백엔드(Spring) 애플리케이션 내부에 설치하여, 레디스 장애 감지 시 차단기를 열어(Open) 레디스로의 접속 시도를 즉시 중단시키는 라이브러리입니다.

[정상 상태] 🟢 CLOSED ──(에러/지연 급증)──► 🔴 OPEN [차단 상태: 즉시 Fallback 실행] ▲ │ │ │ (설정한 대기시간 후) └────────────── (성공 검증) ─────────── 🟡 HALF-OPEN [소수 요청 테스트]

🟢 CLOSED (정상): 모든 캐시/데이터 요청을 레디스로 정상 전달합니다.
🔴 OPEN (차단): 레디스 지연/에러율이 지정치를 넘으면 차단기를 열어 레디스 호출을 아예 시도조차 하지 않고, 미리 준비된 대체 로직(Fallback: 예: Main RDB 직접 조회)을 즉시 실행합니다.
🟡 HALF-OPEN (검증): 레디스가 복구되었는지 확인하기 위해 소수의 요청만 살짝 보내보고, 정상 동작이 확인되면 다시 🟢 CLOSED 상태로 복구합니다.

🔄 장애 발생 시 두 기술의 협업 프로세스

0초 (레디스 Master 다운): Sentinel이 Failover를 시작하며 응답 지연 발생
1~2초 (Circuit Breaker 작동): Spring App의 Resilience4j가 에러/지연을 감지하고 차단기를 🔴 OPEN으로 전환 ➔ 레디스 대신 RDB를 조회하거나 기본값 반환(Fallback)
2초 (Front-end Timeout 작동): 프론트엔드가 요청을 안전하게 끊어내어 무한 로딩 및 사용자 연타로 인한 WAS 폭주 방지
4~5초 (Failover 완료 & 자동 복구): Sentinel이 신규 Master 승격을 완료하면, Resilience4j가 이를 감지하여 🟢 CLOSED로 돌려놓고 캐시 서비스 자동 정상화

<< K8s Redis-Cluster App ①	K8s Redis-Sentinel 준비 ① >>

Email 답글이 올라오면 이메일로 알려드리겠습니다.