Edu/Support

쿠버네티스 환경에서 Redis-Sentinel 구축 실전 가이드 ③

Failover(장애조치) 테스트

🖧 Redis-Sentinel 파드 다이어그램 (초기 상태)

┌───────────────┐ ┌───────────────┐ ┌───────────────┐ │Sentinel-0(98) │ │Sentinel-1(100)│ │Sentinel-2(102)│ └───────┬───────┘ └───────┬───────┘ └───────┬───────┘ └───────────────────┼───────────────────┘ ▼ ┌───────────────┐ │ Redis-0(92) │ │ 🟢Master │ └──────┬────────┘ ┌──────────┴───────────┐ ▼ ▼ ┌───────────────┐ ┌───────────────┐ │ Redis-1(94) │ │ Redis-2(96) │ │ 🟡Replica-1 │ │ 🟡Replica-2 │ └───────────────┘ └───────────────┘

1. Failover(장애조치) 테스트

Redis-0를 다운시켜서 Redis-1이 Replica에서 Master로 전환(승격)되는지 확인
복제 Redis-1,2 중 Redis-1이 마스터가 되는 이유: redis.conf replica-priority 값: Redis-0 100, Redis-1 110, Redis-2 120 -> 값이 작은 Redis-1이 마스터로 전환된다.
💥 파드 다운 후 재시작하면 IP가 변경된다.

Redis-0 💥 다운(delete)

$ kubectl delete pod redis-0 pod "redis-0" deleted

💻 Redis-0 LOG: Master -> Replica

① 03초 kill 시그널을 받고 레디스 다운 (+0초)
② 05초 레디스 자동 재시작 (+2초)
③ 17초 센티널로부터 명령를 받아 Master -> Replica(Slave)로 전환 (+14초) 1:M 11:03:03.043 #📍kill 시그널을 받고 다운: Redis is now ready to exit, bye bye... 1:C 11:03:05.808 *📍2초 후 다시 시작: oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 1:S 11:03:17.398 *📍센티널로부터 신호를 받아 Master -> Replica(Slave)로 전환: Before turning into a replica,...

Redis-0 전체 로그 ▼

1:signal-handler (1786413782) Received SIGTERM scheduling shutdown... 
1:M 11:03:03.037 * User requested shutdown...
1:M 11:03:03.043 #📍kill 시그널을 받고 다운: Redis is now ready to exit, bye bye...				
1:C 11:03:05.808 *📍2초 후 다시 시작: oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 11:03:05.808 * Redis version=8.8.2, bits=64, commit=00000000, modified=1, pid=1, just started
1:C 11:03:05.808 * Configuration loaded				
1:M 11:03:05.808 *➡일단 Master로 시작: monotonic clock: POSIX clock_gettime
1:M 11:03:05.810 * Server initialized
1:M 11:03:05.811 * Reading RDB base file on AOF loading...
1:M 11:03:05.811 * DB loaded from base file aof.1.base.rdb: 0.001 seconds
1:M 11:03:10.458 * DB loaded from incr file aof.1.incr.aof: 4.648 seconds
1:M 11:03:10.458 *➡4.6초 소요: DB loaded from append only file: 4.648 seconds 
1:M 11:03:10.462 * Opening AOF incr file aof.1.incr.aof on server start
1:M 11:03:10.462 * Ready to accept connections tcp				
1:S 11:03:17.398 *📍센티널로부터 신호를 받아 Master -> Replica(Slave)로 전환: Before turning into a replica, ... 
1:S 11:03:17.398 * Connecting to MASTER redis-1.redis-service.default.svc.cluster.local:18504
1:S 11:03:17.399 * MASTER <-> REPLICA sync started
1:S 11:03:17.399 *➡센티널로 부터 'REPLICAOF redis-1' 명령 받음: REPLICAOF redis-1.redis-service.default.svc.cluster.local:18504 enabled (user request from 'id=6 name=sentinel-...')
1:S 11:03:17.403 *➡redis.conf 기록: CONFIG REWRITE executed with success.
1:S 11:03:17.404 * Master replied to PING, replication can continue...
1:S 11:03:27.929 * Done loading RDB, keys loaded: 2955803, keys expired: 0.
1:S 11:03:27.930 * MASTER <-> REPLICA sync: Finished with success
1:S 11:03:27.930 * MASTER <-> REPLICA sync: Starting AOF after a successful sync
1:S 11:03:31.263 * Background AOF rewrite finished successfully

💻 Redis-1 LOG: Replica -> Master

① 03초 Redis-0 다운 감지 (+0초)
② 06초 Sentinel-1로 부터 명령(replicaof noone)을 받아 복제 -> 마스터로 전환 (+3초)
1:S 11:03:03.090 *📍Redis-0 다운 감지: Connection with master lost. 1:M 11:03:06.504 *📍센티널로부터 명령를 받아 Replica(Slave) -> Master로 전환: Discarding previously cached master state.

Redis-1 전체 로그 ▼

1:S 11:03:03.090 *📍Redis-0 다운 감지: Connection with master lost. 
1:S 11:03:03.090 * Caching the disconnected master state.
1:S 11:03:03.091 *➡재연결 시도: Reconnecting to MASTER redis-0.redis-service.default.svc.cluster.local:18504  
1:S 11:03:03.091 * MASTER <-> REPLICA sync started
1:S 11:03:03.091 #➡연결 실패: Error condition on socket for SYNC: Connection refused 
1:S 11:03:03.905 * Connecting to MASTER redis-0.redis-service.default.svc.cluster.local:18504
1:S 11:03:03.905 * MASTER <-> REPLICA sync started					 
1:M 11:03:06.504 *📍센티널로부터 명령를 받아 Replica(Slave) -> Master로 전환: Discarding previously cached master state.
1:M 11:03:06.504 * MASTER MODE enabled (user request from 'id=8 addr=10.42.0.100:49936 laddr=10.42.0.94:18504 fd=16 name=sentinel-bde0e191-cmd...')
1:M 11:03:06.508 *➡redis.conf 기록('REPLICAOF' 삭제): CONFIG REWRITE executed with success.
1:M 11:03:07.355 *➡redis-2로 부터 동기화 요청 받음: Replica redis-2.redis-service.default.svc.cluster.local:18504 asks for synchronization
1:M 11:03:07.356 * Partial resynchronization request from redis-2.redis-service.default.svc.cluster.local:18504 accepted. Sending 596 bytes of backlog starting from offset 432382052.
1:M 11:03:17.404 * Replica redis-0.redis-service.default.svc.cluster.local:18504 asks for synchronization
1:M 11:03:17.404 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for '27e7d7f2495d430017d646aef6776a55870fac45', my replication IDs are 'd735e0345181173ecc516d24ffb58e11e54cc224' and '80d34a3b7754b6bc41d8e95c4645a44540ec73b2')  
1:M 11:03:17.404 * Replica redis-0.redis-service.default.svc.cluster.local:18504 is capable of rdb channel synchronization, and partial sync isn't possible. Full sync will continue with dedicated rdb channel.
1:M 11:03:17.404 *➡redis-0로 부터 동기화 요청 받음: Replica redis-0.redis-service.default.svc.cluster.local:18504 asks for synchronization
1:M 11:03:17.405 * Full resync requested by replica redis-0.redis-service.default.svc.cluster.local:18504 (rdb-channel)
1:M 11:03:17.405 * Delay next BGSAVE for diskless SYNC
1:M 11:03:20.981 * Starting BGSAVE for SYNC with target: replicas sockets (rdb-channel)
1:M 11:03:20.981 * Starting to deliver RDB and replication stream to replica: redis-0.redis-service.default.svc.cluster.local:18504
1:M 11:03:20.994 * Background RDB transfer started by pid 20 to replica socket
20:C 11:03:24.223 * BGSAVE done, 2955803 keys saved, 0 keys skipped, 354696590 bytes written.
20:C 11:03:24.231 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
1:M 11:03:24.309 * Background RDB transfer terminated with success
1:M 11:03:24.309 * Synchronization with replica redis-0.redis-service.default.svc.cluster.local:18504 succeeded
1:M 11:03:24.309 * Streamed RDB transfer with replica redis-0.redis-service.default.svc.cluster.local:18504 succeeded (socket). Waiting for REPLCONF ACK from replica to enable streaming
1:M 11:03:24.309 * RDB transfer completed, rdb only replica (redis-0.redis-service.default.svc.cluster.local:18504) should be disconnected asap
1:M 11:03:24.309 * Connection with replica (rdbchannel) redis-0.redis-service.default.svc.cluster.local:18504 lost.

💻 Redis-2 LOG

① 03초 Redis-0 다운 감지 (+0초)
② 07초 새 마스터(Redis-1)에 연결 (+4초) 1:S 11:03:03.090 *📍Redis-0 다운 감지: Connection with master lost. 1:S 11:03:07.349 *📍새 마스터에 연결: Connecting to MASTER redis-1.redis-service.default.svc.cluster.local:18504

Redis-2 전체 로그 ▼

1:S 11:03:03.090 *📍Redis-0 다운 감지: Connection with master lost. 
1:S 11:03:03.090 * Caching the disconnected master state.
1:S 11:03:03.091 *➡재연결 시도: Reconnecting to MASTER redis-0.redis-service.default.svc.cluster.local:18504  
1:S 11:03:03.091 * MASTER <-> REPLICA sync started
1:S 11:03:03.091 #➡연결 실패: Error condition on socket for SYNC: Connection refused 
1:S 11:03:03.965 * Connecting to MASTER redis-0.redis-service.default.svc.cluster.local:18504
1:S 11:03:03.965 * MASTER <-> REPLICA sync started
1:S 11:03:07.349 *📍새 마스터에 연결: Connecting to MASTER redis-1.redis-service.default.svc.cluster.local:18504  
1:S 11:03:07.350 * MASTER <-> REPLICA sync started
1:S 11:03:07.350 *➡센티널로 부터 'REPLICAOF redis-1' 명령 받음: REPLICAOF redis-1.redis-service.default.svc.cluster.local:18504 enabled (user request from 'id=8 addr=10.42.0.100:33902 laddr=10.42.0.96:18504 fd=16 name=sentinel-bde0e191-cmd ...')  
1:S 11:03:07.355 *➡redis.conf 기록: CONFIG REWRITE executed with success.
1:S 11:03:07.355 * Non blocking connect for SYNC fired the event.
1:S 11:03:07.355 * Master replied to PING, replication can continue...
1:S 11:03:07.355 * Trying a partial resynchronization (request 80d34a3b7754b6bc41d8e95c4645a44540ec73b2:432382052).
1:S 11:03:07.356 * Successful partial resynchronization with master.
1:S 11:03:07.356 * Master replication ID changed to d735e0345181173ecc516d24ffb58e11e54cc224
1:S 11:03:07.356 * MASTER <-> REPLICA sync: Master accepted a Partial Resynchronization.

💻 Sentinel-0 LOG

① 06.1초 redis-0 다운 감지(+sdown) (+3초) -> sentinel.conf down-after-milliseconds mymaster 3000 ms
② 06.2초 redis-0 다운 확정(+odown) (+3초) -> 다른 센티널들과 통신해서 과반수가 동의하면 확정
1:X 11:03:06.131 #📍redis-0 다운 감지: +sdown master mymaster redis-0.redis-service.default.svc.cluster.local 18504 1:X 11:03:06.242 #📍redis-0 다운 확정: +odown master mymaster redis-0.redis-service.default.svc.cluster.local 18504 #quorum 2/2

Sentinel-0 전체 로그 ▼

1:X 11:03:06.131 #📍redis-0 다운 감지: +sdown master mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:06.231 * Sentinel new configuration saved on disk
1:X 11:03:06.231 # +new-epoch 1
1:X 11:03:06.242 * Sentinel new configuration saved on disk
1:X 11:03:06.242 # +vote-for-leader bde0e1917a4b878a3d48742d44f932639a1de44a 1
1:X 11:03:06.242 #📍redis-0 다운 확정: +odown master mymaster redis-0.redis-service.default.svc.cluster.local 18504 #quorum 2/2
1:X 11:03:06.242 * Next failover delay: I will not start a failover before Tue Aug 11 11:05:06 2026
1:X 11:03:07.350 # +config-update-from sentinel bde0e1917a4b878a3d48742d44f932639a1de44a sentinel-1.sentinel-service 28504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:07.350 #➡redis-1를 마스터로 전환 확정: +switch-master mymaster redis-0.redis-service.default.svc.cluster.local 18504 redis-1.redis-service.default.svc.cluster.local 18504
1:X 11:03:07.351 * +slave slave redis-2.redis-service.default.svc.cluster.local:18504 redis-2.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504  
1:X 11:03:07.351 * +slave slave redis-0.redis-service.default.svc.cluster.local:18504 redis-0.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 11:03:07.356 *➡변경 내용을 sentinel.conf에 기록: Sentinel new configuration saved on disk

💻 Sentinel-1 LOG

① 06.1초 redis-0 다운 감지(+sdown) (+3초) -> sentinel.conf down-after-milliseconds mymaster 3000 ms
② 06.2초 redis-0 다운 확정(+odown) (+3초) -> 다른 센티널들과 통신해서 과반수가 동의하면 확정
③ 06.3초 센티널끼리 투표해서 'Sentinel-1'을 Leader로 선출(+elected-leader)
④ 06.4초 redis-1을 마스터로 변경(+failover-state-send-slaveof-noone)
1:X 11:03:06.138 #📍redis-0 다운 감지: +sdown master mymaster redis-0.redis-service.default.svc.cluster.local 18504 1:X 11:03:06.222 #📍redis-0 다운 확정: +odown master mymaster redis-0.redis-service.default.svc.cluster.local 18504 #quorum 3/2 1:X 11:03:06.327 #📍Leader로 선출: +elected-leader master mymaster redis-0.redis-service.default.svc.cluster.local 18504 1:X 11:03:06.403 *📍redis-1을 마스터로 변경: +failover-state-send-slaveof-noone slave redis-1.redis-service.default.svc.cluster.local:18504 redis-1.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504

Sentinel-1 전체 로그 ▼

1:X 11:03:06.138 #📍redis-0 다운 감지: +sdown master mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:06.222 #📍redis-0 다운 확정: +odown master mymaster redis-0.redis-service.default.svc.cluster.local 18504 #quorum 3/2
1:X 11:03:06.222 # +new-epoch 1
1:X 11:03:06.222 # +try-failover master mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:06.226 * Sentinel new configuration saved on disk
1:X 11:03:06.226 # +vote-for-leader bde0e1917a4b878a3d48742d44f932639a1de44a 1
1:X 11:03:06.243 * 984e3350ee6299ced794cd4fb6820e0d202f0a6e voted for bde0e1917a4b878a3d48742d44f932639a1de44a 1
1:X 11:03:06.243 * b2919c14e1a99a3ca317cf7f91ba75f1d40b50d1 voted for bde0e1917a4b878a3d48742d44f932639a1de44a 1
1:X 11:03:06.327 #📍Leader로 선출: +elected-leader master mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:06.327 # +failover-state-select-slave master mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:06.403 # +selected-slave slave redis-1.redis-service.default.svc.cluster.local:18504 redis-1.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:06.403 *📍redis-1을 마스터로 변경: +failover-state-send-slaveof-noone slave redis-1.redis-service.default.svc.cluster.local:18504 redis-1.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:06.503 * +failover-state-wait-promotion slave redis-1.redis-service.default.svc.cluster.local:18504 redis-1.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:07.295 * Sentinel new configuration saved on disk
1:X 11:03:07.296 # +promoted-slave slave redis-1.redis-service.default.svc.cluster.local:18504 redis-1.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:07.296 # +failover-state-reconf-slaves master mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:07.349 *➡redis-2에게 redis-1이 마스터로 변경되었다고 알림: +slave-reconf-sent slave redis-2.redis-service.default.svc.cluster.local:18504 redis-2.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504  
1:X 11:03:08.322 * +slave-reconf-inprog slave redis-2.redis-service.default.svc.cluster.local:18504 redis-2.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:08.322 * +slave-reconf-done slave redis-2.redis-service.default.svc.cluster.local:18504 redis-2.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:08.381 # -odown master mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:08.381 # +failover-end master mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:08.381 # +switch-master mymaster redis-0.redis-service.default.svc.cluster.local 18504 redis-1.redis-service.default.svc.cluster.local 18504
1:X 11:03:08.381 * +slave slave redis-2.redis-service.default.svc.cluster.local:18504 redis-2.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 11:03:08.382 * +slave slave redis-0.redis-service.default.svc.cluster.local:18504 redis-0.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 11:03:08.394 * Sentinel new configuration saved on disk

💻 Sentinel-2 LOG

① 06.1초 redis-0 다운 감지(+sdown) (+3초) -> sentinel.conf down-after-milliseconds mymaster 3000 ms
② 07.2초 redis-0 다운 확정(+odown) (+4초) -> 다른 센티널들과 통신해서 과반수가 동의하면 확정
1:X 11:03:06.114 #📍+sdown master mymaster redis-0.redis-service.default.svc.cluster.local 18504 1:X 11:03:07.239 #📍+odown master mymaster redis-0.redis-service.default.svc.cluster.local 18504 #quorum 3/2

Sentinel-2 전체 로그 ▼

1:X 11:03:06.114 #📍+sdown master mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:06.232 * Sentinel new configuration saved on disk
1:X 11:03:06.232 # +new-epoch 1
1:X 11:03:06.243 * Sentinel new configuration saved on disk
1:X 11:03:06.243 # +vote-for-leader bde0e1917a4b878a3d48742d44f932639a1de44a 1
1:X 11:03:07.239 #📍+odown master mymaster redis-0.redis-service.default.svc.cluster.local 18504 #quorum 3/2
1:X 11:03:07.239 * Next failover delay: I will not start a failover before Tue Aug 11 11:05:06 2026
1:X 11:03:07.350 # +config-update-from sentinel bde0e1917a4b878a3d48742d44f932639a1de44a sentinel-1.sentinel-service 28504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:07.350 #➡redis-1을 마스터로 전환 확정+switch-master mymaster redis-0.redis-service.default.svc.cluster.local 18504 redis-1.redis-service.default.svc.cluster.local 18504
1:X 11:03:07.351 * +slave slave redis-2.redis-service.default.svc.cluster.local:18504 redis-2.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 11:03:07.351 * +slave slave redis-0.redis-service.default.svc.cluster.local:18504 redis-0.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 11:03:07.356 * Sentinel new configuration saved on disk
1:X 11:03:17.398 *➡redis-0에게 redis-1이 마스터로 변경되었다고 알림: +convert-to-slave slave redis-0.redis-service.default.svc.cluster.local:18504 redis-0.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504

🧮 Failover(장애조치) 후 상황

redis-0 IP가 '10.42.0.92'에서 '10.42.0.103'로 변경됐다. [redis@Rocky9 kube]$ k get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE redis-0 1/1 Running 0 7s 10.42.0.103 rocky9 IP:92->103 redis-1 1/1 Running 0 23m 10.42.0.94 rocky9 redis-2 1/1 Running 0 23m 10.42.0.96 rocky9 sentinel-0 1/1 Running 0 23m 10.42.0.98 rocky9 sentinel-1 1/1 Running 0 22m 10.42.0.100 rocky9 sentinel-2 1/1 Running 0 22m 10.42.0.102 rocky9

🧮 redis.conf 확인

Redis LOG: 'CONFIG REWRITE'에서 변경했다.
Redis-0(103): replicaof redis-1.redis-service.default.svc.cluster.local 18504 Redis-1(94): replicaof 없음 -> Redis-1이 마스터이기 때문에 Redis-2(96): replicaof redis-1.redis-service.default.svc.cluster.local 18504

🧮 sentinel.conf 확인

Sentinel LOG: 'Sentinel new configuration saved on disk'에서 변경했다.
redis-0,2가 복제라고 기록, sentinel-0,1,2 내용 동일. # Generated by CONFIG REWRITE sentinel monitor mymaster redis-1.redis-service.default.svc.cluster.local 18504 2 sentinel known-replica mymaster redis-0.redis-service.default.svc.cluster.local 18504 sentinel known-replica mymaster redis-2.redis-service.default.svc.cluster.local 18504

🖧 Redis-Sentinel 파드 다이어그램 (Failover 후)

┌───────────────┐ ┌───────────────┐ ┌───────────────┐ │Sentinel-0(98) │ │Sentinel-1(100)│ │Sentinel-2(102)│ └───────┬───────┘ └───────┬───────┘ └───────┬───────┘ └─────┬─────────────┴───────────────────┘ │ │ ┌───────────────┐ │ │ Redis-0(103) │ │ │ 🟡Replica-1 │ │ └───────────────┘ │ ▲ │ ┌──────────┴───────────┐ ▼ │ ▼ ┌──────┴────────┐ ┌───────────────┐ │ Redis-1(94) │ │ Redis-2(96) │ │ 🟢Master │ │ 🟡Replica-2 │ └───────────────┘ └───────────────┘

2. Failback(원상복구) 테스트

현재 마스터인 Redis-1를 다운시켜서 Redis-0가 Replica에서 Master로 전환(승격)되는지 확인
📚 'Failback(원상복구) 테스트'는 Redis-0 파드가 다시 마스터가 되도록 한다는 의미에서 사용했다.
(Failover 테스트와 구분하기 위해서 사용했다)

💻 Redis-1 💥다운(delete)

$ kubectl delete pod redis-1 pod "redis-1" deleted

💻 Redis-0 LOG: Replica -> Master

① 29분 55초 Redis-1 연결 단절 (+0초)
② 29분 58초 센티널로부터 명령를 받아 Replica(Slave) -> Master로 전환(승격) (+3초) 1:S 13:29:55.437 *📍Redis-1 연결 단절: Connection with master lost. 1:M 13:29:58.714 *📍센티널로부터 신호를 받아 Replica(Slave) -> Master로 전환: Discarding previously cached master state.

Redis-0 전체 로그 ▼

1:S 13:29:55.437 *📍Redis-1 연결 단절: Connection with master lost.
1:S 13:29:55.437 * Caching the disconnected master state.
1:S 13:29:55.437 *➡재연결 시도: Reconnecting to MASTER redis-1.redis-service.default.svc.cluster.local:18504
1:S 13:29:55.438 * MASTER <-> REPLICA sync started
1:S 13:29:55.438 #➡연결 실패: Error condition on socket for SYNC: Connection refused
1:S 13:29:55.765 * Connecting to MASTER redis-1.redis-service.default.svc.cluster.local:18504
1:S 13:29:55.766 # Unable to connect to MASTER: Operation now in progress
1:S 13:29:56.771 * Connecting to MASTER redis-1.redis-service.default.svc.cluster.local:18504
1:S 13:29:56.772 # Unable to connect to MASTER: Operation now in progress
1:S 13:29:57.776 * Connecting to MASTER redis-1.redis-service.default.svc.cluster.local:18504
1:S 13:29:57.777 # Unable to connect to MASTER: Operation now in progress
1:M 13:29:58.714 *📍센티널로부터 신호를 받아 Replica(Slave) -> Master로 전환: Discarding previously cached master state.
1:M 13:29:58.714 * Setting secondary replication ID to d735e0345181173ecc516d24ffb58e11e54cc224, valid up to offset: 434858832. New replication ID is b9c98fef85ebf2b43197de7c0505082ef78ccef1
1:M 13:29:58.714 * MASTER MODE enabled (user request from 'id=15 addr=10.42.0.98:55870 laddr=10.42.0.103:18504 fd=18 name=sentinel-984e3350-cmd ...')
1:M 13:29:58.720 * CONFIG REWRITE executed with success.
1:M 13:29:59.655 * Replica redis-2.redis-service.default.svc.cluster.local:18504 asks for synchronization
1:M 13:29:59.655 * Partial resynchronization request from redis-2.redis-service.default.svc.cluster.local:18504 accepted. Sending 596 bytes of backlog starting from offset 434858832.
1:M 13:30:11.770 * Replica redis-1.redis-service.default.svc.cluster.local:18504 asks for synchronization
1:M 13:30:11.770 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for '1d3bd6057b7649148d5200d8a78f296834efc05e', my replication IDs are 'b9c98fef85ebf2b43197de7c0505082ef78ccef1' and 'd735e0345181173ecc516d24ffb58e11e54cc224')
1:M 13:30:11.770 * Replica redis-1.redis-service.default.svc.cluster.local:18504 is capable of rdb channel synchronization, and partial sync isn't possible. Full sync will continue with dedicated rdb channel.
1:M 13:30:11.770 * Replica redis-1.redis-service.default.svc.cluster.local:18504 asks for synchronization
1:M 13:30:11.771 * Full resync requested by replica redis-1.redis-service.default.svc.cluster.local:18504 (rdb-channel)
1:M 13:30:11.771 * Delay next BGSAVE for diskless SYNC
1:M 13:30:14.852 * Starting BGSAVE for SYNC with target: replicas sockets (rdb-channel)
1:M 13:30:14.852 * Starting to deliver RDB and replication stream to replica: redis-1.redis-service.default.svc.cluster.local:18504
1:M 13:30:14.860 * Background RDB transfer started by pid 14 to replica socket
14:C 13:30:17.948 * BGSAVE done, 2955803 keys saved, 0 keys skipped, 354696590 bytes written.
14:C 13:30:17.956 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
1:M 13:30:17.973 * Background RDB transfer terminated with success
1:M 13:30:17.973 * Synchronization with replica redis-1.redis-service.default.svc.cluster.local:18504 succeeded
1:M 13:30:17.973 * Streamed RDB transfer with replica redis-1.redis-service.default.svc.cluster.local:18504 succeeded (socket). Waiting for REPLCONF ACK from replica to enable streaming
1:M 13:30:17.973 * RDB transfer completed, rdb only replica (redis-1.redis-service.default.svc.cluster.local:18504) should be disconnected asap
1:M 13:30:17.973 * Connection with replica (rdbchannel) redis-1.redis-service.default.svc.cluster.local:18504 lost.

💻 Redis-1 LOG: Master -> Replica

① 29분 55초 kill 시그널을 받고 다운 (+0초)
② 29분 57초 파드 재시작 (+2초)
③ 30분 11초 센티널로부터 명령를 받아 Master -> Replica(Slave)로 전환 (+16초) 1:M 13:29:55.389 #📍kill 시그널을 받고 다운: Redis is now ready to exit, bye bye... 1:M 13:29:57.787 *📍2초 후 다시 시작: Server initialized 1:S 13:30:11.754 *📍센티널로부터 신호를 받아 Master -> Replica(Slave)로 전환: Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.

Redis-1 전체 로그 ▼

1:signal-handler (1786422594)➡kill 시그널: Received SIGTERM scheduling shutdown...
1:M 13:29:54.480 * User requested shutdown...
1:M 13:29:55.389 #📍kill 시그널을 받고 다운: Redis is now ready to exit, bye bye...
1:C 13:29:57.785 *oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 13:29:57.785 * Redis version=8.8.2, bits=64, commit=00000000, modified=1, pid=1, just started
1:C 13:29:57.785 * Configuration loaded
1:M 13:29:57.785 *➡일단 Master로 시작: monotonic clock: POSIX clock_gettime
1:M 13:29:57.787 *📍2초 후 다시 시작: Server initialized
1:M 13:29:57.787 * Reading RDB base file on AOF loading...
1:M 13:29:57.788 * Loading RDB produced by version 8.8.2
1:M 13:29:57.788 * RDB age 10198 seconds
1:M 13:29:57.788 * RDB memory usage when created 0.70 Mb
1:M 13:29:57.788 * RDB is base AOF
1:M 13:29:57.788 * Done loading RDB, keys loaded: 0, keys expired: 0.
1:M 13:29:57.788 * DB loaded from base file aof.2.base.rdb: 0.001 seconds
1:M 13:30:02.391 * DB loaded from incr file aof.2.incr.aof: 4.604 seconds
1:M 13:30:02.391 * DB loaded from append only file: 4.604 seconds
1:M 13:30:02.397 * Opening AOF incr file aof.2.incr.aof on server start
1:M 13:30:02.397 * Ready to accept connections tcp
1:S 13:30:11.754 *📍센티널로부터 신호를 받아 Master -> Replica(Slave)로 전환: Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
1:S 13:30:11.754 * Connecting to MASTER redis-0.redis-service.default.svc.cluster.local:18504
1:S 13:30:11.755 * MASTER <-> REPLICA sync started
1:S 13:30:11.755 * REPLICAOF redis-0.redis-service.default.svc.cluster.local:18504 enabled (user request from 'id=6 addr=10.42.0.100:51752 laddr=10.42.0.104:18504 fd=13 name=sentinel-bde0e191-cmd ...')
1:S 13:30:11.769 * CONFIG REWRITE executed with success.
1:S 13:30:11.769 * Non blocking connect for SYNC fired the event.
1:S 13:30:11.770 * Master replied to PING, replication can continue...
1:S 13:30:11.770 * Trying a partial resynchronization (request 1d3bd6057b7649148d5200d8a78f296834efc05e:434858832).
1:S 13:30:11.770 * PSYNC is not possible, initialize RDB channel.
1:S 13:30:14.852 * Starting to receive RDB and replication stream in parallel.
1:S 13:30:14.871 * MASTER <-> REPLICA sync: receiving streamed RDB from master with EOF to disk
1:S 13:30:17.954 * Discarding previously cached master state.
1:S 13:30:17.954 * MASTER <-> REPLICA sync: Loading DB in memory
1:S 13:30:17.960 * MASTER <-> REPLICA sync: Flushing old data
1:S 13:30:18.983 * Loading RDB produced by version 8.8.2
1:S 13:30:18.983 * RDB age 4 seconds
1:S 13:30:18.983 * RDB memory usage when created 476.64 Mb
1:S 13:30:21.623 * Done loading RDB, keys loaded: 2955803, keys expired: 0.
1:S 13:30:21.623 * Removing the RDB file obtained from the master. This replica has persistence disabled
1:S 13:30:21.623 * MASTER <-> REPLICA sync: Finished with success
1:S 13:30:21.623 * MASTER <-> REPLICA sync: Starting AOF after a successful sync
1:S 13:30:21.623 * Creating AOF incr file temp-aof.incr on background rewrite
1:S 13:30:21.634 * Background append only file rewriting started by pid 13
1:S 13:30:21.634 * MASTER <-> REPLICA sync: Starting to stream replication buffer into the db (1933 bytes).
1:S 13:30:21.635 * MASTER <-> REPLICA sync: Successfully streamed replication buffer into the db (1933 bytes in total)
13:C 13:30:24.866 * BGSAVE done, 2955803 keys saved, 0 keys skipped, 354696457 bytes written.
13:C 13:30:24.872 * Successfully created the temporary AOF base file temp-rewriteaof-bg-13.aof
13:C 13:30:24.881 * Fork CoW for AOF rewrite: current 1 MB, peak 1 MB, average 1 MB
1:S 13:30:24.950 * Background AOF rewrite terminated with success
1:S 13:30:24.950 * Successfully renamed the temporary AOF base file temp-rewriteaof-bg-13.aof into aof.3.base.rdb
1:S 13:30:24.950 * Successfully renamed the temporary AOF incr file temp-aof.incr into aof.3.incr.aof
1:S 13:30:24.953 * Removing the history file aof.2.incr.aof in the background
1:S 13:30:24.953 * Removing the history file aof.2.base.rdb in the background
1:S 13:30:24.960 * Background AOF rewrite finished successfully

💻 Redis-2 LOG

① 29분 55초 Redis-1과 연결 끊김 (+0초)
② 29분 59초 센티널로부터 Redis-0가 새 마스터가 되었다는 신호를 받고 연결 (+4초) 1:S 13:29:55.437 *📍Redis-1과 연결 끊김: Connection with master lost. 1:S 13:29:59.650 *📍센티널로부터 Redis-0가 새 마스터가 되었다는 신호를 받고 연결: Connecting to MASTER redis-0.redis-service.default.svc.cluster.local:18504

Redis-2 전체 로그 ▼

1:S 13:29:55.437 *📍Redis-1과 연결 끊김: Connection with master lost.
1:S 13:29:55.437 * Caching the disconnected master state.
1:S 13:29:55.437 *➡Redis-1에 연결 시도: Reconnecting to MASTER redis-1.redis-service.default.svc.cluster.local:18504
1:S 13:29:55.437 * MASTER <-> REPLICA sync started
1:S 13:29:55.438 #➡Redis-1에 연결 실패: Error condition on socket for SYNC: Connection refused
1:S 13:29:56.341 * Connecting to MASTER redis-1.redis-service.default.svc.cluster.local:18504
1:S 13:29:56.342 # Unable to connect to MASTER: Operation now in progress
1:S 13:29:57.347 * Connecting to MASTER redis-1.redis-service.default.svc.cluster.local:18504
1:S 13:29:57.347 # Unable to connect to MASTER: Operation now in progress
1:S 13:29:58.352 * Connecting to MASTER redis-1.redis-service.default.svc.cluster.local:18504
1:S 13:29:58.353 # Unable to connect to MASTER: Operation now in progress
1:S 13:29:59.357 * Connecting to MASTER redis-1.redis-service.default.svc.cluster.local:18504
1:S 13:29:59.357 # Unable to connect to MASTER: Success
1:S 13:29:59.650 *📍센티널로부터 Redis-0가 새 마스터가 되었다는 신호를 받고 연결: Connecting to MASTER redis-0.redis-service.default.svc.cluster.local:18504
1:S 13:29:59.650 * MASTER <-> REPLICA sync started
1:S 13:29:59.650 * REPLICAOF redis-0.redis-service.default.svc.cluster.local:18504 enabled (user request from 'id=16 addr=10.42.0.98:54564 laddr=10.42.0.96:18504 fd=17 name=sentinel-984e3350-cmd ...')
1:S 13:29:59.654 * CONFIG REWRITE executed with success.
1:S 13:29:59.654 * Non blocking connect for SYNC fired the event.
1:S 13:29:59.655 * Master replied to PING, replication can continue...
1:S 13:29:59.655 * Trying a partial resynchronization (request d735e0345181173ecc516d24ffb58e11e54cc224:434858832).
1:S 13:29:59.655 * Successful partial resynchronization with master.
1:S 13:29:59.655 * Master replication ID changed to b9c98fef85ebf2b43197de7c0505082ef78ccef1
1:S 13:29:59.655 * MASTER <-> REPLICA sync: Master accepted a Partial Resynchronization.

💻 Sentinel-0 LOG

① 58.4초 redis-1 다운 감지(+sdown) (+3초) -> sentinel.conf down-after-milliseconds mymaster 3000 ms
② 58.5초 redis-1 다운 확정(+odown) (+3초) -> 다른 센티널들과 통신해서 과반수가 동의하면 확정
③ 58.5초 센티널끼리 투표해서 'Sentinel-0'을 Leader로 선출(+elected-leader)
④ 58.6초 redis-0을 마스터로 변경(+failover-state-send-slaveof-noone)
1:X 13:29:58.457 #📍Redis-1 다운 감지: +sdown master mymaster redis-1.redis-service.default.svc.cluster.local 18504 1:X 13:29:58.534 #📍Redis-1 다운 확정: +odown master mymaster redis-1.redis-service.default.svc.cluster.local 18504 #quorum 2/2 1:X 13:29:58.593 #📍리더로 선출: +elected-leader master mymaster redis-1.redis-service.default.svc.cluster.local 18504 1:X 13:29:58.652 *📍redis-0를 마스터로 변경: +failover-state-send-slaveof-noone slave redis-0.redis-service.default.svc.cluster.local:18504 redis-0.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504

Sentinel-0 전체 로그 ▼

1:X 13:29:56.546 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:58.302 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:58.457 #📍Redis-1 다운 감지: +sdown master mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:58.521 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:58.534 #📍Redis-1 다운 확정: +odown master mymaster redis-1.redis-service.default.svc.cluster.local 18504 #quorum 2/2
1:X 13:29:58.534 # +new-epoch 2
1:X 13:29:58.534 # +try-failover master mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:58.538 * Sentinel new configuration saved on disk
1:X 13:29:58.538 # +vote-for-leader 984e3350ee6299ced794cd4fb6820e0d202f0a6e 2
1:X 13:29:58.548 * bde0e1917a4b878a3d48742d44f932639a1de44a voted for 984e3350ee6299ced794cd4fb6820e0d202f0a6e 2
1:X 13:29:58.549 * b2919c14e1a99a3ca317cf7f91ba75f1d40b50d1 voted for 984e3350ee6299ced794cd4fb6820e0d202f0a6e 2
1:X 13:29:58.593 #📍리더로 선출: +elected-leader master mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:58.593 # +failover-state-select-slave master mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:58.652 # +selected-slave slave redis-0.redis-service.default.svc.cluster.local:18504 redis-0.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:58.652 *📍redis-0를 마스터로 변경: +failover-state-send-slaveof-noone slave redis-0.redis-service.default.svc.cluster.local:18504 redis-0.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:58.714 * +failover-state-wait-promotion slave redis-0.redis-service.default.svc.cluster.local:18504 redis-0.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:59.354 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:59.587 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:59.596 * Sentinel new configuration saved on disk
1:X 13:29:59.596 # +promoted-slave slave redis-0.redis-service.default.svc.cluster.local:18504 redis-0.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:59.596 # +failover-state-reconf-slaves master mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:59.649 * +slave-reconf-sent slave redis-2.redis-service.default.svc.cluster.local:18504 redis-2.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:30:00.611 * +slave-reconf-inprog slave redis-2.redis-service.default.svc.cluster.local:18504 redis-2.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:30:00.611 * +slave-reconf-done slave redis-2.redis-service.default.svc.cluster.local:18504 redis-2.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:30:00.687 #➡Redis-1 재시작 확인: -odown master mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:30:00.687 # +failover-end master mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:30:00.687 #➡Redis-1에게 Redis-0가 마스터라고 알림: +switch-master mymaster redis-1.redis-service.default.svc.cluster.local 18504 redis-0.redis-service.default.svc.cluster.local 18504
1:X 13:30:00.688 * +slave slave redis-2.redis-service.default.svc.cluster.local:18504 redis-2.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 13:30:00.689 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:30:00.689 * +slave slave redis-1.redis-service.default.svc.cluster.local:18504 redis-1.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 13:30:00.693 * Sentinel new configuration saved on disk
1:X 13:30:00.747 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'

💻 Sentinel-1 LOG

① 58.5초 redis-1 다운 감지(+sdown) (+3초) -> sentinel.conf down-after-milliseconds mymaster 3000 ms
② 58.6초 redis-1 다운 확정(+odown) (+3초) -> 다른 센티널들과 통신해서 과반수가 동의하면 확정
1:X 13:29:58.519 #📍+sdown master mymaster redis-1.redis-service.default.svc.cluster.local 18504 1:X 13:29:58.620 #📍+odown master mymaster redis-1.redis-service.default.svc.cluster.local 18504 #quorum 3/2

Sentinel-1 전체 로그 ▼

1:X 13:29:56.497 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:58.302 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:58.459 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:58.519 #📍+sdown master mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:58.539 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:58.543 * Sentinel new configuration saved on disk
1:X 13:29:58.543 # +new-epoch 2
1:X 13:29:58.548 * Sentinel new configuration saved on disk
1:X 13:29:58.548 # +vote-for-leader 984e3350ee6299ced794cd4fb6820e0d202f0a6e 2
1:X 13:29:58.620 #📍+odown master mymaster redis-1.redis-service.default.svc.cluster.local 18504 #quorum 3/2
1:X 13:29:58.620 * Next failover delay: I will not start a failover before Tue Aug 11 13:31:59 2026
1:X 13:29:59.354 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:59.592 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:59.650 # +config-update-from sentinel 984e3350ee6299ced794cd4fb6820e0d202f0a6e sentinel-0.sentinel-service 28504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:59.650 #➡redis-0를 마스터로 전환 확정: +switch-master mymaster redis-1.redis-service.default.svc.cluster.local 18504 redis-0.redis-service.default.svc.cluster.local 18504
1:X 13:29:59.651 * +slave slave redis-2.redis-service.default.svc.cluster.local:18504 redis-2.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 13:29:59.652 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:59.652 * +slave slave redis-1.redis-service.default.svc.cluster.local:18504 redis-1.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 13:29:59.657 *➡변경 내용을 sentinel.conf에 기록: Sentinel new configuration saved on disk
1:X 13:29:59.664 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:30:00.612 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:30:00.689 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:30:11.754 * +convert-to-slave slave redis-1.redis-service.default.svc.cluster.local:18504 redis-1.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504

💻 Sentinel-2 LOG

① 58.3초 redis-1 다운 감지(+sdown) (+3초) -> sentinel.conf down-after-milliseconds mymaster 3000 ms
② 59.4초 redis-1 다운 확정(+odown) (+3초) -> 다른 센티널들과 통신해서 과반수가 동의하면 확정
1:X 13:29:58.301 #📍+sdown master mymaster redis-1.redis-service.default.svc.cluster.local 18504 1:X 13:29:59.454 #📍+odown master mymaster redis-1.redis-service.default.svc.cluster.local 18504 #quorum 3/2

Sentinel-2 전체 로그 ▼

1:X 13:29:56.552 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:58.301 #📍+sdown master mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:58.459 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:58.521 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:58.539 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:58.543 * Sentinel new configuration saved on disk
1:X 13:29:58.543 # +new-epoch 2
1:X 13:29:58.549 * Sentinel new configuration saved on disk
1:X 13:29:58.549 # +vote-for-leader 984e3350ee6299ced794cd4fb6820e0d202f0a6e 2
1:X 13:29:59.454 #📍+odown master mymaster redis-1.redis-service.default.svc.cluster.local 18504 #quorum 3/2
1:X 13:29:59.454 * Next failover delay: I will not start a failover before Tue Aug 11 13:31:58 2026
1:X 13:29:59.587 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:59.592 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:59.650 # +config-update-from sentinel 984e3350ee6299ced794cd4fb6820e0d202f0a6e sentinel-0.sentinel-service 28504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:59.650 #➡redis-0를 마스터로 전환 확정: +switch-master mymaster redis-1.redis-service.default.svc.cluster.local 18504 redis-0.redis-service.default.svc.cluster.local 18504
1:X 13:29:59.651 * +slave slave redis-2.redis-service.default.svc.cluster.local:18504 redis-2.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504  
1:X 13:29:59.652 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:59.652 * +slave slave redis-1.redis-service.default.svc.cluster.local:18504 redis-1.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 13:29:59.657 *➡변경 내용을 sentinel.conf에 기록: Sentinel new configuration saved on disk
1:X 13:29:59.694 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:30:00.612 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:30:00.708 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'

🧮 Failback(원상복구) 후 상황

$ kubectl get pods -o wide redis-0 1/1 Running 0 147m 10.42.0.103 rocky9 📍IP:92->103 redis-1 1/1 Running 0 39s 10.42.0.104 rocky9 📍IP:94->104 redis-2 1/1 Running 0 170m 10.42.0.96 rocky9 sentinel-0 1/1 Running 0 170m 10.42.0.98 rocky9 sentinel-1 1/1 Running 0 170m 10.42.0.100 rocky9 sentinel-2 1/1 Running 0 170m 10.42.0.102 rocky9

🧮 redis.conf 확인

Redis-0: replicaof 없음 -> Master Redis-1: replicaof redis-0.redis-service.default.svc.cluster.local 18504 Redis-2: replicaof redis-0.redis-service.default.svc.cluster.local 18504

🧮 sentinel.conf 확인

Redis-0 마스터, Redis-1,2 복제 sentinel monitor mymaster redis-0.redis-service.default.svc.cluster.local 18504 2 sentinel known-replica mymaster redis-1.redis-service.default.svc.cluster.local 18504 sentinel known-replica mymaster redis-2.redis-service.default.svc.cluster.local 18504

🖧 Redis-Sentinel 파드 다이어그램 (원상복구)

┌───────────────┐ ┌───────────────┐ ┌───────────────┐ │Sentinel-0(98) │ │Sentinel-1(100)│ │Sentinel-2(102)│ └───────┬───────┘ └───────┬───────┘ └───────┬───────┘ └───────────────────┼───────────────────┘ ▼ ┌───────────────┐ │ Redis-0(103) │ │ 🟢Master │ └──────┬────────┘ ┌──────────┴───────────┐ ▼ ▼ ┌───────────────┐ ┌───────────────┐ │ Redis-1(104) │ │ Redis-2(96) │ │ 🟡Replica-1 │ │ 🟡Replica-2 │ └───────────────┘ └───────────────┘

🧑‍💻 결론

Redis-Sentinel Failover(장애 조치), Failback(원상 복구) 테스트 성공
파드(pod)를 다운, 재시작하면 IP가 변경되지만 아무 문제없이 처리된다.
장애 조치(복구) 시간은 4초 정도 걸린다.

➡ 여기까지는 Redis-Sentinel 단독으로 테스트했습니다.
Java(Spring) App와 연결해서 작동하는 것까지 보려면 여기를 클릭하세요.

<< K8s Redis-Sentinel 구축 ②	K8s Redis-Sentinel App ① >>

Email 답글이 올라오면 이메일로 알려드리겠습니다.