Edu/Support

쿠버네티스 환경에서 Redis-Cluster 구축 실전 가이드 ③

Failover(장애조치) 테스트

🖧 1. Redis-Cluster 파드 다이어그램 (초기 상태)

┌────────────────┐ ┌────────────────┐ ┌───────────────┐ │🟢Redis-0(231) │ │🟢Redis-1(233) │ │🟢Redis-2(235) │ │ Master-1 │ │ Master-2 │ │ Master-3 │ └──────┬─────────┘ └──────┬─────────┘ └──────┬────────┘ │ │ │ │ ┌──────────────────│────────────────────┘ │ │ └────────────────────┐ └──────────────────┐ │ │ │ │ ▼ ▼ ▼ ┌────────────────┐ ┌────────────────┐ ┌───────────────┐ │🟡Redis-3(237) │ │🟡Redis-4(239) │ │🟡Redis-5(241) │ │ Replica-1 │ │ Replica-2 │ │ Replica-3 │ └────────────────┘ └────────────────┘ └───────────────┘

1. Failover(장애조치) 테스트

Redis-0 파드를 다운시켜서 Redis-4가 Replica에서 Master로 전환(승격)되는지 확인
redis.conf replica-priority 값은 Redis-Cluster에서 사용되지 않는다. Redis-Sentinel에서 사용된다. 대신 가장 최신 데이터를 받은 복제가 마스터가 된다. 여기서는 마스터별로 복제가 하나이므로 Redis-0의 복제인 Redis-4가 마스터가 된다.
💥 파드 다운 후 재시작하면 IP가 변경된다.

Redis-0 💥 다운(delete)

$ kubectl delete pod redis-cluster-0 pod "redis-cluster-0" deleted

💻 Redis-0 LOG: Master -> Replica

① 24초 kill 시그널을 받고 레디스 다운 (+0초)
② 26초 레디스 자동 재시작 (+2초)
③ 35초 Master -> Replica(Slave)로 전환 (+11초) 1:M 14:42:24.187 #📍kill 시그널을 받고 다운: Redis is now ready to exit, bye bye... 1:C 14:42:26.706 *📍2초 후 다시 시작: oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 1:S 14:42:35.038 *📍Master -> Replica(Slave)로 전환: Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.

Redis-0 전체 로그 ▼

$ kubectl exec -it redis-cluster-0 -- tail -f /data/redis.log
1:signal-handler (1786167744) Received SIGTERM scheduling shutdown... 
1:M 14:42:24.182 * User requested shutdown...
1:M 14:42:24.187 #📍kill 시그널을 받고 다운: Redis is now ready to exit, bye bye...
1:C 14:42:26.706 *📍2초 후 다시 시작: oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 
1:M 14:42:26.708 * Node configuration loaded, I'm 66390566e7f48211e8bed3a6a0e49155cbe8b54b
1:M 14:42:26.709 * Server initialized
1:M 14:42:26.709 * Reading RDB base file on AOF loading...
1:M 14:42:26.709 * Loading RDB produced by version 8.8.1
1:M 14:42:26.709 * RDB age 3615 seconds
1:M 14:42:26.709 * RDB memory usage when created 2.16 Mb
1:M 14:42:26.710 * RDB is base AOF
1:M 14:42:26.710 * Done loading RDB, keys loaded: 0, keys expired: 0.
1:M 14:42:26.710 * DB loaded from base file aof.1.base.rdb: 0.001 seconds
1:M 14:42:35.027 * DB loaded from incr file aof.1.incr.aof: 8.317 seconds
1:M 14:42:35.027 *➡8.3초 소요: DB loaded from append only file: 8.318 seconds	
1:M 14:42:35.031 * Opening AOF incr file aof.1.incr.aof on server start
1:M 14:42:35.031 * Ready to accept connections tcp
1:M 14:42:35.038 * Configuration change detected. Reconfiguring myself as a replica of ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4)
1:S 14:42:35.038 *📍Master -> Replica(Slave)로 전환: Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
1:S 14:42:35.038 *➡새 마스터와 연결: Connecting to MASTER 10.42.0.239:18504 
1:S 14:42:35.038 *➡새 마스터로 부터 데이터를 받기 시작함: MASTER <-> REPLICA sync started	
1:S 14:42:35.039 * Cluster state changed: ok
1:S 14:42:35.042 * Non blocking connect for SYNC fired the event.
1:S 14:42:35.046 * Master replied to PING, replication can continue...
1:S 14:42:35.046 * Trying a partial resynchronization (request b95c76e4720380a0cdced96e236842d144aeb5b8:918944394).
1:S 14:42:35.047 * PSYNC is not possible, initialize RDB channel.
1:S 14:42:38.638 * Starting to receive RDB and replication stream in parallel.
1:S 14:42:38.676 * MASTER <-> REPLICA sync: receiving streamed RDB from master with EOF to disk
1:S 14:42:42.179 * Discarding previously cached master state.
1:S 14:42:42.179 * MASTER <-> REPLICA sync: Loading DB in memory
1:S 14:42:42.188 * MASTER <-> REPLICA sync: Flushing old data
1:S 14:42:43.300 * Loading RDB produced by version 8.8.1
1:S 14:42:43.300 * RDB age 5 seconds
1:S 14:42:43.300 * RDB memory usage when created 650.33 Mb
1:S 14:42:45.600 * Done loading RDB, keys loaded: 3053053, keys expired: 0.
1:S 14:42:45.600 * Removing the RDB file obtained from the master. This replica has persistence disabled
1:S 14:42:45.600 *➡새 마스터로 부터 데이터 받고 로딩 완료: MASTER <-> REPLICA sync: Finished with success 
1:S 14:42:45.600 *➡AOF rewrite 시작: MASTER <-> REPLICA sync: Starting AOF after a successful sync  
1:S 14:42:45.600 * Creating AOF incr file temp-aof.incr on background rewrite
1:S 14:42:45.613 * Background append only file rewriting started by pid 19
1:S 14:42:45.613 * MASTER <-> REPLICA sync: Starting to stream replication buffer into the db (0 bytes).
1:S 14:42:45.613 * MASTER <-> REPLICA sync: Successfully streamed replication buffer into the db (0 bytes in total)
19:C 14:42:48.921 * BGSAVE done, 3053053 keys saved, 0 keys skipped, 384717496 bytes written.
19:C 14:42:48.927 * Successfully created the temporary AOF base file temp-rewriteaof-bg-19.aof
19:C 14:42:48.933 * Fork CoW for AOF rewrite: current 2 MB, peak 2 MB, average 1 MB
1:S 14:42:49.027 * Background AOF rewrite terminated with success
1:S 14:42:49.027 * Successfully renamed the temporary AOF base file temp-rewriteaof-bg-19.aof into aof.2.base.rdb
1:S 14:42:49.027 * Successfully renamed the temporary AOF incr file temp-aof.incr into aof.2.incr.aof
1:S 14:42:49.031 * Removing the history file aof.1.incr.aof in the background
1:S 14:42:49.031 * Removing the history file aof.1.base.rdb in the background
1:S 14:42:49.036 *➡AOF rewrite 완료: Background AOF rewrite finished successfully

💻 Redis-1 LOG: Master

① 28초 Redis-0 다운 확정(Fail/odown) (+4초) -> cluster-node-timeout 3000ms
② 29초 Redis-4 새 마스터 승격 (+5초)
$ kubectl exec -it redis-cluster-1 -- tail -f /data/redis.log 1:M 14:42:28.507 *📍Redis-0 다운 확정: Marking node 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0) as failing (quorum reached). 1:M 14:42:29.608 *📍Redis-4 새 마스터로 승격: Failover auth granted to ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4) for epoch 7 1:M 14:42:35.033 *➡Redis-0 파드 재시작 ➔ 새 IP(10.42.0.242) 감지: Address updated for node 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0), now 10.42.0.242:18504 1:M 14:42:35.078 *➡Redis-0 복제 노드로 재연결되며 FAIL 상태 해제: Clear FAIL state for node 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0):replica is reachable again.

💻 Redis-4 LOG: Replica -> Master

① 24초 Redis-0 다운 감지 (+0초)
② 29초 마스터로 승격 (+5초)
$ kubectl exec -it redis-cluster-4 -- tail -f /data/redis.log 1:S 14:42:24.235 *📍Redis-0 다운 감지: Connection with master lost. 1:M 14:42:29.613 *📍마스터로 승격: Discarding previously cached master state.

Redis-4 전체 로그 ▼

$ kubectl exec -it redis-cluster-4 -- tail -f /data/redis.log
1:S 14:42:24.235 *📍Redis-0 다운 감지: Connection with master lost.	 
1:S 14:42:24.235 * Caching the disconnected master state.
1:S 14:42:24.235 *➡연결 시도 Reconnecting to MASTER 10.42.0.231:18504 
1:S 14:42:24.235 * MASTER <-> REPLICA sync started
1:S 14:42:24.235 #➡연결 실패 Error condition on socket for SYNC: Connection refused	
1:S 14:42:24.579 * Connecting to MASTER 10.42.0.231:18504
1:S 14:42:24.579 * MASTER <-> REPLICA sync started
1:S 14:42:28.507 * FAIL message received from 3b690009fb9a90ff6b43765363c4c0b81b63d103 (redis-cluster-1) about 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0)
1:S 14:42:28.598 * Start of election delayed for 972 milliseconds (rank #0, offset 918944393).
1:S 14:42:29.602 * Starting a failover election for epoch 7.
1:S 14:42:29.613 * Failover election won: I'm the new master.
1:S 14:42:29.613 * configEpoch set to 7 after successful failover					
1:M 14:42:29.613 *📍마스터로 승격: Discarding previously cached master state.	
1:M 14:42:29.613 * Setting secondary replication ID to c76d2514db200f6d13ba070d8ef349efe33bc2cf, valid up to offset: 918944394. New replication ID is caab310287582352de716fc92f8542ef435b125d
1:M 14:42:35.033 * Address updated for node 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0), now 10.42.0.242:18504
1:M 14:42:35.042 * A failover occurred in shard 14464984890143edab6e29a8503e3c93981c8809; node 66390566e7f48211e8bed3a6a0e49155cbe8b54b () lost 0 slot(s) to node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4) with a config epoch of 7
1:M 14:42:35.047 *➡옛 마스터(현재는 복제)가 데이터 동기화 요청: Replica 10.42.0.242:18504 asks for synchronization 
1:M 14:42:35.047 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for 'b95c76e4720380a0cdced96e236842d144aeb5b8', my replication IDs are 'caab310287582352de716fc92f8542ef435b125d' and 'c76d2514db200f6d13ba070d8ef349efe33bc2cf')
1:M 14:42:35.047 * Replica 10.42.0.242:18504 is capable of rdb channel synchronization, and partial sync isn't possible. Full sync will continue with dedicated rdb channel.
1:M 14:42:35.047 * Replica 10.42.0.242:18504 asks for synchronization
1:M 14:42:35.047 * Full resync requested by replica 10.42.0.242:18504 (rdb-channel)
1:M 14:42:35.047 * Delay next BGSAVE for diskless SYNC
1:M 14:42:35.124 * Clear FAIL state for node 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0):replica is reachable again.
1:M 14:42:38.638 *➡보낼 데이터 만들기 시작: Starting BGSAVE for SYNC with target: replicas sockets (rdb-channel)
1:M 14:42:38.638 * Starting to deliver RDB and replication stream to replica: 10.42.0.242:18504
1:M 14:42:38.652 * Background RDB transfer started by pid 20 to replica socket
20:C 14:42:42.174 * BGSAVE done, 3053053 keys saved, 0 keys skipped, 384717629 bytes written.
20:C 14:42:42.183 * Fork CoW for RDB: current 2 MB, peak 2 MB, average 1 MB
1:M 14:42:42.267 * Background RDB transfer terminated with success
1:M 14:42:42.267 *➡새 복제에게 데이터 보내기(동기화) 완료: Synchronization with replica 10.42.0.242:18504 succeeded 
1:M 14:42:42.267 * Streamed RDB transfer with replica 10.42.0.242:18504 succeeded (socket). Waiting for REPLCONF ACK from replica to enable streaming
1:M 14:42:42.267 * RDB transfer completed, rdb only replica (10.42.0.242:18504) should be disconnected asap
1:M 14:42:42.267 * Connection with replica (rdbchannel) 10.42.0.242:18504 lost.

🧮 Failover(장애조치) 후 상황

Redis-0 IP가 '10.42.0.231'에서 '10.42.0.242'로 변경됐다. [redis@Rocky9 kube]$ k get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE redis-cluster-0 1/1 Running 0 3m3s 10.42.0.242 rocky9 IP:231->242 redis-cluster-1 1/1 Running 0 63m 10.42.0.233 rocky9 redis-cluster-2 1/1 Running 0 63m 10.42.0.235 rocky9 redis-cluster-3 1/1 Running 0 63m 10.42.0.237 rocky9 redis-cluster-4 1/1 Running 0 63m 10.42.0.239 rocky9 redis-cluster-5 1/1 Running 0 63m 10.42.0.241 rocky9

🖧 Redis-Cluster 파드 다이어그램 (Failover 후)

┌────────────────┐ ┌────────────────┐ ┌───────────────┐ │🟡Redis-0(242) │ │🟢Redis-1(233) │ │🟢Redis-2(235) │ │ Replica-2 │ │ Master-2 │ │ Master-3 │ └──────┬─────────┘ └──────┬─────────┘ └──────┬────────┘ ▲ │ │ │ ┌──────────────────│────────────────────┘ │ │ └────────────────────┐ └──────────────────┐ │ │ │ │ ▼ │ ▼ ┌────────────────┐ ┌────────────────┐ ┌───────────────┐ │🟡Redis-3(237) │ │🟢Redis-4(239) │ │🟡Redis-5(241) │ │ Replica-1 │ │ Master-1 │ │ Replica-3 │ └────────────────┘ └────────────────┘ └───────────────┘

2. Failback(원상복구) 테스트

현재 마스터인 Redis-4를 다운시켜서 Redis-0가 Replica에서 Master로 전환(승격)되는지 확인
📚 'Failback(원상복구) 테스트'는 Redis-0 파드가 다시 마스터가 되도록 한다는 의미에서 사용했다.
(Failover 테스트와 구분하기 위해서 사용했다)

💻 Redis-4 💥다운(delete)

$ kubectl delete pod redis-cluster-4 pod "redis-cluster-4" deleted

💻 Redis-0 LOG: Replica -> Master

① 52초 Redis-4 연결 단절 (+0초)
② 56초 Redis-4 다운 확정(Fail/odown) (+4초)
③ 57초 Replica -> Master로 전환(승격) (+5초)
1:S 15:19:52.289 *📍Redis-4 연결 단절: Connection with master lost. 1:S 15:19:56.549 *📍Redis-4 다운 확정: Marking node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4) as failing (quorum reached). 1:M 15:19:57.276 *📍Replica -> Master로 전환(승격): Discarding previously cached master state.

Redis-0 전체 로그 ▼

1:S 15:19:52.289 *📍Redis-4 연결 단절: Connection with master lost.	
1:S 15:19:52.289 * Caching the disconnected master state.
1:S 15:19:52.289 *➡Redis-4에 연결 시도: Reconnecting to MASTER 10.42.0.239:18504	
1:S 15:19:52.289 * MASTER <-> REPLICA sync started
1:S 15:19:52.289 # Error condition on socket for SYNC: Connection refused
1:S 15:19:52.644 * Connecting to MASTER 10.42.0.239:18504
1:S 15:19:52.644 * MASTER <-> REPLICA sync started
1:S 15:19:56.549 *📍Redis-4 다운 확정(odown): Marking node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4) as failing (quorum reached).
1:S 15:19:56.560 * Start of election delayed for 639 milliseconds (rank #0, offset 918947529).
1:S 15:19:57.267 * Starting a failover election for epoch 8.
1:S 15:19:57.276 * Failover election won: I'm the new master.
1:S 15:19:57.276 * configEpoch set to 8 after successful failover                              
1:M 15:19:57.276 *📍Replica(Slave) -> Master로 전환(승격): Discarding previously cached master state.
1:M 15:19:57.276 * Setting secondary replication ID to caab310287582352de716fc92f8542ef435b125d, valid up to offset: 918947530. New replication ID is 2e46928d6c500b1c71c1038b815e61b9c059cc82
1:M 15:20:02.615 * Address updated for node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4), now 10.42.0.243:18504
1:M 15:20:02.632 * A failover occurred in shard 14464984890143edab6e29a8503e3c93981c8809; node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 () lost 0 slot(s) to node 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0) with a config epoch of 8
1:M 15:20:02.637 * Replica 10.42.0.243:18504 asks for synchronization
1:M 15:20:02.637 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for 'f0e297826ed9c693bb4b5abccd5c415d8319ac10', my replication IDs are '2e46928d6c500b1c71c1038b815e61b9c059cc82' and 'caab310287582352de716fc92f8542ef435b125d')
1:M 15:20:02.637 * Replica 10.42.0.243:18504 is capable of rdb channel synchronization, and partial sync isn't possible. Full sync will continue with dedicated rdb channel.
1:M 15:20:02.638 * Replica 10.42.0.243:18504 asks for synchronization
1:M 15:20:02.638 * Full resync requested by replica 10.42.0.243:18504 (rdb-channel)
1:M 15:20:02.638 * Delay next BGSAVE for diskless SYNC
1:M 15:20:02.689 *➡Clear FAIL state for node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4):replica is reachable again.
1:M 15:20:05.701 * Starting BGSAVE for SYNC with target: replicas sockets (rdb-channel)
1:M 15:20:05.701 * Starting to deliver RDB and replication stream to replica: 10.42.0.243:18504
1:M 15:20:05.708 * Background RDB transfer started by pid 28 to replica socket
28:C 15:20:08.931 * BGSAVE done, 3053053 keys saved, 0 keys skipped, 384717629 bytes written.
28:C 15:20:08.938 * Fork CoW for RDB: current 2 MB, peak 2 MB, average 1 MB
1:M 15:20:09.019 * Background RDB transfer terminated with success
1:M 15:20:09.019 * Synchronization with replica 10.42.0.243:18504 succeeded
1:M 15:20:09.019 * Streamed RDB transfer with replica 10.42.0.243:18504 succeeded (socket). Waiting for REPLCONF ACK from replica to enable streaming
1:M 15:20:09.019 * RDB transfer completed, rdb only replica (10.42.0.243:18504) should be disconnected asap
1:M 15:20:09.020 * Connection with replica (rdbchannel) 10.42.0.243:18504 lost.

💻 Redis-1 LOG: Master

① 56초 Redis-4 다운 확정 (+4초)
② 57초 Redis-0 새 마스터로 승격 (+5초)
1:M 15:19:56.549 *📍Redis-4 다운 확정: FAIL message received from 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0) about ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4) 1:M 15:19:57.271 *📍Redis-0 새 마스터로 승격: Failover auth granted to 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0) for epoch 8 1:M 15:20:02.615 *➡Redis-4 파드 재시작 -> 새 IP(10.42.0.243) 감지: Address updated for node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4), now 10.42.0.243:18504 1:M 15:20:02.692 *➡Redis-4 복제 노드로 재연결되며 FAIL 상태 해제: Clear FAIL state for node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4):replica is reachable again.

💻 Redis-4 LOG: Master -> Replica

① 19분 52초 kill 시그널을 받고 다운 (+0초)
② 19분 54초 파드(Redis) 재시작 (+2초)
③ 20분 02초 Master -> Replica(Slave)로 전환 (+10초) 1:M 15:19:52.237 #📍kill 시그널을 받고 다운: Redis is now ready to exit, bye bye... 1:M 15:19:54.305 *📍파드(Redis) 재시작: Server initialized 1:S 15:20:02.629 *📍Master -> Replica(Slave)로 전환: Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.

Redis-4 전체 로그 ▼

1:signal-handler (1786169992) Received SIGTERM scheduling shutdown...
1:M 15:19:52.232 * User requested shutdown...
1:M 15:19:52.237 #📍kill 시그널을 받고 다운: Redis is now ready to exit, bye bye...
1:M 15:19:54.305 *📍파드(Redis) 재시작: Server initialized
1:M 15:19:54.306 * Reading RDB base file on AOF loading...
1:M 15:19:54.306 * Loading RDB produced by version 8.8.1
1:M 15:19:54.306 * RDB age 5681 seconds
1:M 15:19:54.306 * RDB memory usage when created 2.39 Mb
1:M 15:19:54.306 * RDB is base AOF
1:M 15:19:54.306 * Done loading RDB, keys loaded: 0, keys expired: 0.
1:M 15:19:54.306 * DB loaded from base file aof.2.base.rdb: 0.001 seconds
1:M 15:20:02.608 * DB loaded from incr file aof.2.incr.aof: 8.302 seconds
1:M 15:20:02.609 *➡Data Load 8.3초 소요: DB loaded from append only file: 8.303 seconds 
1:M 15:20:02.613 * Opening AOF incr file aof.2.incr.aof on server start
1:M 15:20:02.613 * Ready to accept connections tcp
1:M 15:20:02.629 * Configuration change detected. Reconfiguring myself as a replica of 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0)
1:S 15:20:02.629 *📍Master -> Replica(Slave)로 전환: Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
1:S 15:20:02.629 *➡새 마스터에 접속: Connecting to MASTER 10.42.0.242:18504	
1:S 15:20:02.629 * MASTER <-> REPLICA sync started
1:S 15:20:02.629 * Cluster state changed: ok
1:S 15:20:02.632 * Non blocking connect for SYNC fired the event.
1:S 15:20:02.637 * Master replied to PING, replication can continue...
1:S 15:20:02.637 * Trying a partial resynchronization (request f0e297826ed9c693bb4b5abccd5c415d8319ac10:918947530).
1:S 15:20:02.637 * PSYNC is not possible, initialize RDB channel.
1:S 15:20:05.701 * Starting to receive RDB and replication stream in parallel.
1:S 15:20:05.716 * MASTER <-> REPLICA sync: receiving streamed RDB from master with EOF to disk
1:S 15:20:08.938 * Discarding previously cached master state.
1:S 15:20:08.938 * MASTER <-> REPLICA sync: Loading DB in memory
1:S 15:20:08.947 * MASTER <-> REPLICA sync: Flushing old data
1:S 15:20:10.042 * Loading RDB produced by version 8.8.1
1:S 15:20:10.042 * RDB age 5 seconds
1:S 15:20:10.042 * RDB memory usage when created 522.06 Mb
1:S 15:20:12.351 * Done loading RDB, keys loaded: 3053053, keys expired: 0.
1:S 15:20:12.351 * Removing the RDB file obtained from the master. This replica has persistence disabled
1:S 15:20:12.351 *➡동기화 완료: MASTER <-> REPLICA sync: Finished with success  
1:S 15:20:12.351 * MASTER <-> REPLICA sync: Starting AOF after a successful sync
1:S 15:20:12.351 * Creating AOF incr file temp-aof.incr on background rewrite
1:S 15:20:12.370 * Background append only file rewriting started by pid 19
1:S 15:20:12.370 * MASTER <-> REPLICA sync: Starting to stream replication buffer into the db (0 bytes).
1:S 15:20:12.370 * MASTER <-> REPLICA sync: Successfully streamed replication buffer into the db (0 bytes in total)
19:C 15:20:15.668 * BGSAVE done, 3053053 keys saved, 0 keys skipped, 384717496 bytes written.
19:C 15:20:15.673 * Successfully created the temporary AOF base file temp-rewriteaof-bg-19.aof
19:C 15:20:15.682 * Fork CoW for AOF rewrite: current 2 MB, peak 2 MB, average 1 MB
1:S 15:20:15.790 * Background AOF rewrite terminated with success
1:S 15:20:15.790 * Successfully renamed the temporary AOF base file temp-rewriteaof-bg-19.aof into aof.3.base.rdb
1:S 15:20:15.790 * Successfully renamed the temporary AOF incr file temp-aof.incr into aof.3.incr.aof
1:S 15:20:15.795 * Removing the history file aof.2.incr.aof in the background
1:S 15:20:15.795 * Removing the history file aof.2.base.rdb in the background
1:S 15:20:15.798 *➡AOF Rewrite 완료: Background AOF rewrite finished successfully

🧮 Failback(원상복구) 후 상황

Redis-4 IP가 '10.42.0.239'에서 '10.42.0.243'로 변경됐다. [kube]$ k get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE redis-cluster-0 1/1 Running 0 47m 10.42.0.242 rocky9 📍IP:231->242 redis-cluster-1 1/1 Running 0 107m 10.42.0.233 rocky9 redis-cluster-2 1/1 Running 0 107m 10.42.0.235 rocky9 redis-cluster-3 1/1 Running 0 107m 10.42.0.237 rocky9 redis-cluster-4 1/1 Running 0 9m34s 10.42.0.243 rocky9 📍IP:239->243 redis-cluster-5 1/1 Running 0 106m 10.42.0.241 rocky9

🧮 Cluster info 확인

[kube]$ ./cluster_info.sh cluster_state:ok cluster_slots_assigned:16384 cluster_slots_ok:16384 ➡📍모든 슬롯 OK cluster_slots_pfail:0 cluster_slots_fail:0 cluster_known_nodes:6 cluster_size:3

🧮 Cluster nodes 확인

[kube]$ ./cluster_nodes.sh 66390566e7f48211e8bed3a6a0e49155cbe8b54b 10.42.0.242:18504@28504,redis-cluster-0.redis-cluster-service.default.svc.cluster.local master - 0 1786170813036 8 connected 0-5460 3b690009fb9a90ff6b43765363c4c0b81b63d103 10.42.0.233:18504@28504,redis-cluster-1.redis-cluster-service.default.svc.cluster.local myself,master - 0 0 2 connected 5461-10922 02069ae6b4f98c31d5f4e70f243bc9a006b68443 10.42.0.235:18504@28504,redis-cluster-2.redis-cluster-service.default.svc.cluster.local master - 0 1786170813539 3 connected 10923-16383 3722046802e006615ac1d976e62e4c94c58af586 10.42.0.237:18504@28504,redis-cluster-3.redis-cluster-service.default.svc.cluster.local slave 02069ae6b4f98c31d5f4e70f243bc9a006b68443 0 1786170813237 3 connected ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 10.42.0.243:18504@28504,redis-cluster-4.redis-cluster-service.default.svc.cluster.local slave 66390566e7f48211e8bed3a6a0e49155cbe8b54b 0 1786170814041 8 connected 541fa7cc95095c4b0390bb8ae9d787aa1201c9e2 10.42.0.241:18504@28504,redis-cluster-5.redis-cluster-service.default.svc.cluster.local slave 3b690009fb9a90ff6b43765363c4c0b81b63d103 0 1786170813539 2 connected

🖧 Redis-Cluster 파드 다이어그램 (Failback 후)

┌────────────────┐ ┌────────────────┐ ┌───────────────┐ │🟢Redis-0(242) │ │🟢Redis-1(233) │ │🟢Redis-2(235) │ │ Master-1 │ │ Master-2 │ │ Master-3 │ └──────┬─────────┘ └──────┬─────────┘ └──────┬────────┘ │ │ │ │ ┌──────────────────│────────────────────┘ │ │ └────────────────────┐ └──────────────────┐ │ │ │ │ ▼ ▼ ▼ ┌────────────────┐ ┌────────────────┐ ┌───────────────┐ │🟡Redis-3(237) │ │🟡Redis-4(243) │ │🟡Redis-5(241) │ │ Replica-1 │ │ Replica-2 │ │ Replica-3 │ └────────────────┘ └────────────────┘ └───────────────┘

🧑‍💻 결론

Redis-Cluster Failover(장애 조치), Failback(원상 복구) 테스트 성공
파드(pod)를 다운, 재시작하면 IP가 변경되지만 아무 문제없이 처리된다.

Failover(장애조치) 테스트: Redis-0 SLEEP(멈춤)

SLEEP 테스트는 레디스가 어떤 이유로든 응답하지 못하는 상황을 만들어서, 이 경우도 제대로 Failover(장애 조치)가 되는지 확인하는 테스트이다.
💥 'Sleep 10'은 레디스를 10초 동안 멈추는 것이므로 IP는 변경되지 않는다.

1. Failover(장애조치) 테스트

• Failover(장애조치) 테스트: Redis-0 Sleep 10 -> Redis-4가 Replica에서 Master로 전환(승격)되는지 확인

Redis-0 SLEEP 10

$ kubectl exec -it redis-cluster-0 -- redis-cli -p 18504 -a redisgate DEBUG sleep 10

💻 Redis-0 LOG: Master -> Replica

① 17초 SLEEP 시작 (+0초)
② 27초 SLEEP 종료 (+10초)
③ 27초 Master -> Replica로 전환 (+10초)
1:M 17:08:27.199 *📍SLEEP 종료: Connection with replica 10.42.0.243:18504 lost. 1:S 17:08:27.200 *📍Master -> Replica로 전환: Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.

Redis-0 전체 로그 ▼

1:M 17:08:27.199 *📍SLEEP 종료: Connection with replica 10.42.0.243:18504 lost.
1:M 17:08:27.200 * Configuration change detected. Reconfiguring myself as a replica of ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4)
1:S 17:08:27.200 *📍 Master -> Replica로 전환: Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
1:S 17:08:27.200 * Connecting to MASTER 10.42.0.243:18504
1:S 17:08:27.200 * MASTER <-> REPLICA sync started
1:S 17:08:27.205 * Non blocking connect for SYNC fired the event.
1:S 17:08:27.210 * Master replied to PING, replication can continue...
1:S 17:08:27.211 * Trying a partial resynchronization (request 2e46928d6c500b1c71c1038b815e61b9c059cc82:918956630).
1:S 17:08:27.211 * Successful partial resynchronization with master.
1:S 17:08:27.211 * Master replication ID changed to 8d3577deeb6b8113718c642303195a34c4b78d09
1:S 17:08:27.211 *➡부분동기화 완료: MASTER <-> REPLICA sync: Master accepted a Partial Resynchronization.

💻 Redis-1 LOG: Master

① 21초 Redis-0 다운 확정(Fail/odown) (+4초) -> cluster-node-timeout 3000ms
② 22초 Redis-4 Replica -> Master로 전환 (+5초)
③ 27초 Redis-0 재시작 (+10초)
1:M 17:08:21.074 *📍Redis-0 다운 확정: FAIL message received from ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4) about 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0) 1:M 17:08:22.138 *📍Redis-4 Replica -> Master로 전환: Failover auth granted to ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4) for epoch 9 1:M 17:08:27.205 *📍Redis-0 재시작: Clear FAIL state for node 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0):master without slots is reachable again.

💻 Redis-4 LOG: Replica -> Master

① 21초 Redis-0 다운 확정(Fail/odown) (+4초)
② 22초 Replica -> Master로 전환 (+5초)
1:S 17:08:21.073 *📍Redis-0 다운 확정: Marking node 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0) as failing (quorum reached). 1:S 17:08:22.144 *📍Replica(Slave) -> Master로 전환: Failover election won: I'm the new master.

Redis-4 로그 전체 ▼

1:S 17:08:21.073 *📍Redis-0 다운 확정: Marking node 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0) as failing (quorum reached).
1:S 17:08:21.131 * Start of election delayed for 944 milliseconds (rank #0, offset 918956629).
1:S 17:08:22.134 * Starting a failover election for epoch 9.
1:S 17:08:22.144 *📍Replica(Slave) -> Master로 전환: Failover election won: I'm the new master.
1:S 17:08:22.144 * configEpoch set to 9 after successful failover                               
1:M 17:08:22.144 * Connection with master lost.
1:M 17:08:22.144 * Caching the disconnected master state.
1:M 17:08:22.144 * Discarding previously cached master state.
1:M 17:08:22.144 * Setting secondary replication ID to 2e46928d6c500b1c71c1038b815e61b9c059cc82, valid up to offset: 918956630. New replication ID is 8d3577deeb6b8113718c642303195a34c4b78d09
1:M 17:08:27.205 * Clear FAIL state for node 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0):master without slots is reachable again.
1:M 17:08:27.205 * A failover occurred in shard 14464984890143edab6e29a8503e3c93981c8809; node 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0) lost 0 slot(s) to node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4) with a config epoch of 9
1:M 17:08:27.211 * Replica 10.42.0.242:18504 asks for synchronization
1:M 17:08:27.211 * Partial resynchronization request from 10.42.0.242:18504 accepted. Sending 0 bytes of backlog starting from offset 918956630.

🧮 Failover(장애조치) 후 상황

IP는 변경되지 않습니다. 파드 자체는 계속 살아있습니다. 'AGE' 항목 참조. [kube]$ k get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE redis-cluster-0 1/1 Running 0 146m 10.42.0.242 rocky9 redis-cluster-1 1/1 Running 0 3h26m 10.42.0.233 rocky9 redis-cluster-2 1/1 Running 0 3h26m 10.42.0.235 rocky9 redis-cluster-3 1/1 Running 0 3h26m 10.42.0.237 rocky9 redis-cluster-4 1/1 Running 0 108m 10.42.0.243 rocky9 redis-cluster-5 1/1 Running 0 3h26m 10.42.0.241 rocky9

🧮 Cluster info 조회

cluster_state:ok cluster_slots_assigned:16384 cluster_slots_ok:16384 cluster_slots_pfail:0 cluster_slots_fail:0 cluster_known_nodes:6 cluster_size:3

🧮 Cluster nodes 조회

66390566e7f48211e8bed3a6a0e49155cbe8b54b 10.42.0.242:18504@28504,redis-cluster-0.redis-cluster-service.default.svc.cluster.local 🟡slave ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 0 1786176794190 9 connected 3b690009fb9a90ff6b43765363c4c0b81b63d103 10.42.0.233:18504@28504,redis-cluster-1.redis-cluster-service.default.svc.cluster.local myself,🟢master - 0 0 2 connected 5461-10922 02069ae6b4f98c31d5f4e70f243bc9a006b68443 10.42.0.235:18504@28504,redis-cluster-2.redis-cluster-service.default.svc.cluster.local 🟢master - 0 1786176794089 3 connected 10923-16383 3722046802e006615ac1d976e62e4c94c58af586 10.42.0.237:18504@28504,redis-cluster-3.redis-cluster-service.default.svc.cluster.local 🟡slave 02069ae6b4f98c31d5f4e70f243bc9a006b68443 0 1786176793184 3 connected ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 10.42.0.243:18504@28504,redis-cluster-4.redis-cluster-service.default.svc.cluster.local 🟢master - 0 1786176794000 9 connected 0-5460 541fa7cc95095c4b0390bb8ae9d787aa1201c9e2 10.42.0.241:18504@28504,redis-cluster-5.redis-cluster-service.default.svc.cluster.local 🟡slave 3b690009fb9a90ff6b43765363c4c0b81b63d103 0 1786176793586 2 connected

🖧 Redis-Cluster 파드 다이어그램 (Failover 후)

┌────────────────┐ ┌────────────────┐ ┌───────────────┐ │🟡Redis-0(242) │ │🟢Redis-1(233) │ │🟢Redis-2(235) │ │ Replica-2 │ │ Master-2 │ │ Master-3 │ └────────────────┘ └──────┬─────────┘ └──────┬────────┘ ▲ │ │ │ ┌──────────────────│────────────────────┘ │ │ └────────────────────┐ └──────────────────┐ │ │ │ │ ▼ │ ▼ ┌────────────────┐ ┌────┴───────────┐ ┌───────────────┐ │🟡Redis-3(237) │ │🟢Redis-4(243) │ │🟡Redis-5(241) │ │ Replica-1 │ │ Master-1 │ │ Replica-3 │ └────────────────┘ └────────────────┘ └───────────────┘

2. Failback(원상복구) 테스트

• Failback(원상복구) 테스트: Redis-4 Sleep 10 -> Redis-0가 Replica에서 Master로 전환(승격)되는지 확인

💻 Redis-4 SLEEP 10

$ kubectl exec -it redis-cluster-4 -- redis-cli -p 18504 -a redisgate DEBUG sleep 10

💻 Redis-0 LOG: Replica -> Master

① 26초 SLEEP 명령 실행 (+0초)
② 30초 Redis-4 다운 확정(Fail/odown) (+4초)
③ 31초 Redis-0 Replica -> Master로 전환 (+5초)
④ 36초 Redis-4 재시작 (+10초)
1:S 20:19:30.481 *📍Redis-4 다운 확정: FAIL message received from 3b690009fb9a90ff6b43765363c4c0b81b63d103 (redis-cluster-1) about ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4) 1:S 20:19:31.588 *📍Replica(Slave) -> Master 전환: Failover election won: I'm the new master. 1:M 20:19:36.349 *📍Redis-4 재시작: Clear FAIL state for node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4):master without slots is reachable again.

Redis-0 전체 로그 ▼

1:S 20:19:30.481 *📍Redis-4 다운 확정: FAIL message received from 3b690009fb9a90ff6b43765363c4c0b81b63d103 (redis-cluster-1) about ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4)
1:S 20:19:30.568 * Start of election delayed for 971 milliseconds (rank #0, offset 918972673).
1:S 20:19:31.573 * Starting a failover election for epoch 10.
1:S 20:19:31.588 *📍Replica(Slave) -> Master 전환: Failover election won: I'm the new master.
1:S 20:19:31.588 * configEpoch set to 10 after successful failover
1:M 20:19:31.588 * Connection with master lost.
1:M 20:19:31.588 * Caching the disconnected master state.
1:M 20:19:31.588 * Discarding previously cached master state.
1:M 20:19:31.588 * Setting secondary replication ID to 8d3577deeb6b8113718c642303195a34c4b78d09, valid up to offset: 918972674. New replication ID is 806c6073fffb5fe2ac6d318f6aceb9d2aa090c18
1:M 20:19:36.349 *📍Redis-4 재시작: Clear FAIL state for node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4):master without slots is reachable again.
1:M 20:19:36.349 * A failover occurred in shard 14464984890143edab6e29a8503e3c93981c8809; node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4) lost 0 slot(s) to node 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0) with a config epoch of 10
1:M 20:19:36.354 * Replica 10.42.0.243:18504 asks for synchronization
1:M 20:19:36.354 * Partial resynchronization request from 10.42.0.243:18504 accepted. Sending 0 bytes of backlog starting from offset 918972674.

💻 Redis-1 LOG: Master

① 30초 Redis-4 다운 확정(Fail/odown) (+4초)
② 31초 Redis-0 Replica -> Master로 전환(승격) (+5초)
③ 36초 Redis-4 재시작 (+10초)
1:M 20:19:30.481 *📍Redis-4 다운(FAIL) 확정: Marking node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4) as failing (quorum reached). 1:M 20:19:31.577 *📍Replica -> Master로 전환(승격): Failover auth granted to 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0) for epoch 10 1:M 20:19:36.349 *📍Clear FAIL state for node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4):master without slots is reachable again.

💻 Redis-2 LOG: Redis-1 로그와 동일하여 생략함

💻 Redis-4 LOG: Master -> Replica

① 26초 SLEEP 명령 실행 (+0초)
① 36초 SLEEP 명령 종료 -> 복제(Redis-0)와 연결 끊김 (+10초)
② 36초 Master -> Replica로 전환 (+10초)
1:M 20:19:36.342 *📍복제(Redis-0)와 연결 끊김: Connection with replica 10.42.0.242:18504 lost. 1:S 20:19:36.343 *📍Master -> Replica(Slave) 전환: Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.

Redis-4 로그 전체 ▼

1:M 20:19:36.342 *📍복제(Redis-0)와 연결 끊김: Connection with replica 10.42.0.242:18504 lost.
1:M 20:19:36.342 # Failover auth denied to 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0): its master is up
1:M 20:19:36.343 * Configuration change detected. Reconfiguring myself as a replica of 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0)
1:S 20:19:36.343 *📍Master -> Replica(Slave) 전환: Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
1:S 20:19:36.343 * Connecting to MASTER 10.42.0.242:18504
1:S 20:19:36.343 * MASTER <-> REPLICA sync started
1:S 20:19:36.349 * Non blocking connect for SYNC fired the event.
1:S 20:19:36.354 * Master replied to PING, replication can continue...
1:S 20:19:36.354 * Trying a partial resynchronization (request 8d3577deeb6b8113718c642303195a34c4b78d09:918972674).
1:S 20:19:36.354 * Successful partial resynchronization with master.
1:S 20:19:36.354 * Master replication ID changed to 806c6073fffb5fe2ac6d318f6aceb9d2aa090c18
1:S 20:19:36.354 * MASTER <-> REPLICA sync: Master accepted a Partial Resynchronization.

🧮 Failback(원상복구) 후 상황

[kube]$ k get pods -o wide -> IP 변경없음, 다시 시작하지 않음 NAME READY STATUS RESTARTS AGE IP NODE redis-cluster-0 1/1 Running 0 5h37m 10.42.0.242 rocky9 redis-cluster-1 1/1 Running 0 6h38m 10.42.0.233 rocky9 redis-cluster-2 1/1 Running 0 6h37m 10.42.0.235 rocky9 redis-cluster-3 1/1 Running 0 6h37m 10.42.0.237 rocky9 redis-cluster-4 1/1 Running 0 5h 10.42.0.243 rocky9 redis-cluster-5 1/1 Running 0 6h37m 10.42.0.241 rocky9

🧮 Cluster info 조회

cluster_state:ok cluster_slots_assigned:16384 cluster_slots_ok:16384 cluster_slots_pfail:0 cluster_slots_fail:0 cluster_known_nodes:6 cluster_size:3

🧮 Cluster nodes 조회

66390566e7f48211e8bed3a6a0e49155cbe8b54b 10.42.0.242:18504@28504,redis-cluster-0.redis-cluster-service.default.svc.cluster.local 🟢master - 0 1786188079000 10 connected 0-5460 3b690009fb9a90ff6b43765363c4c0b81b63d103 10.42.0.233:18504@28504,redis-cluster-1.redis-cluster-service.default.svc.cluster.local myself,🟢master - 0 0 2 connected 5461-10922 02069ae6b4f98c31d5f4e70f243bc9a006b68443 10.42.0.235:18504@28504,redis-cluster-2.redis-cluster-service.default.svc.cluster.local 🟢master - 0 1786188078548 3 connected 10923-16383 3722046802e006615ac1d976e62e4c94c58af586 10.42.0.237:18504@28504,redis-cluster-3.redis-cluster-service.default.svc.cluster.local 🟡slave 02069ae6b4f98c31d5f4e70f243bc9a006b68443 0 1786188079552 3 connected ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 10.42.0.243:18504@28504,redis-cluster-4.redis-cluster-service.default.svc.cluster.local 🟡slave 66390566e7f48211e8bed3a6a0e49155cbe8b54b 0 1786188079552 10 connected 541fa7cc95095c4b0390bb8ae9d787aa1201c9e2 10.42.0.241:18504@28504,redis-cluster-5.redis-cluster-service.default.svc.cluster.local 🟡slave 3b690009fb9a90ff6b43765363c4c0b81b63d103 0 1786188079000 2 connected

🖧 Redis-Cluster 파드 다이어그램 (Failback 후)

🧑‍💻 결론

SLEEP 명령을 사용한 Redis-Cluster Failover(장애 조치), Failback(원상 복구) 테스트 성공
SLEEP 명령을 사용하면 레디스가 다운되지도 않고, IP가 변경되지도 않지만, 3초 이상 응답이 없으면 장애 조치가 실행된다.

➡ 여기까지는 Redis-Cluster 단독으로 테스트했습니다.
Java(Spring) App와 연결해서 작동하는 것까지 보려면 여기를 클릭하세요.

<< K8s Redis-Cluster 구축 ②	K8s Redis-Cluster App ① >>

Email 답글이 올라오면 이메일로 알려드리겠습니다.