Edu/Support

쿠버네티스 환경에서 Redis-Standalone 구축 실전 가이드 ③

Failover(장애조치) 테스트

🖧 1. Redis-Standalone 파드 다이어그램 (초기 상태)

┌──────────────┐ │ Redis(220) │ │ 🟢Master │ └──────────────┘

1. Failover(장애조치) 테스트

Redis-0 파드를 다운시킨다. 복제는 없다.
💥 파드 다운 후 재시작하면 IP가 변경된다.

Redis-0 💥 다운(delete)

💻 Terminal-1

$ kubectl delete pod redis-0 -> 시각: 22:11:15
  pod "redis-0" deleted

💻 Redis-0 LOG

① 15초 kill 시그널을 받고 레디스 다운 (+0초)
② 18초 레디스 자동 재시작 (+3초)
③ 22초 데이터 로딩 완료(약 5초 소요) (+7초)
④ 22초 레디스 준비 완료 (+7초)

📄 redis.log

1:M 22:11:15.321 *📍kill 시그널을 받고 레디스 다운: User requested shutdown...
1:C 22:11:18.053 *📍레디스 자동 재시작: oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:M 22:11:22.782 *📍데이터 로딩 완료: DB loaded from append only file: 4.728 seconds
1:M 22:11:22.788 *📍레디스 준비 완료: Ready to accept connections tcp

Redis-0 전체 로그 ▼

📄 redis.log

$ kubectl exec -it redis-0 -- tail -f /data/redis.log
1:signal-handler (1788786675) Received SIGTERM scheduling shutdown...
1:M 22:11:15.321 *📍User requested shutdown...
1:M 22:11:15.321 * Calling fsync() on the AOF file.
1:M 22:11:15.327 * Removing the pid file.
1:M 22:11:15.327 # Redis is now ready to exit, bye bye...

1:C 22:11:18.053 *📍oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 22:11:18.053 * Redis version=8.8.2, bits=64, commit=00000000, modified=1, pid=1, just started
1:C 22:11:18.053 * Configuration loaded
1:M 22:11:18.053 * monotonic clock: POSIX clock_gettime
1:M 22:11:18.054 * Server initialized
1:M 22:11:18.055 * Reading RDB base file on AOF loading...
1:M 22:11:18.055 * Loading RDB produced by version 8.8.2
1:M 22:11:18.055 * RDB age 22168 seconds
1:M 22:11:18.055 * RDB memory usage when created 0.68 Mb
1:M 22:11:18.055 * RDB is base AOF
1:M 22:11:18.055 * Done loading RDB, keys loaded: 0, keys expired: 0.
1:M 22:11:18.055 * DB loaded from base file aof.1.base.rdb: 0.001 seconds
1:M 22:11:22.782 * DB loaded from incr file aof.1.incr.aof: 4.727 seconds
1:M 22:11:22.782 *📍DB loaded from append only file: 4.728 seconds
1:M 22:11:22.788 * Opening AOF incr file aof.1.incr.aof on server start
1:M 22:11:22.788 *📍Ready to accept connections tcp

💻 App 로그

① 15초 연결 끊김 -> 재접속 실패 (+0초)
② 19초 5회 재시도 후 실패 에러 발생 (+4초)
③ 24초 재접속 -> 실패 (+9초)
④ 32초 재접속 -> 성공 (+17초)
⑤ 32초 데이터 입력 성공 (+17초)

💻 Terminal-2

$ kubectl logs -f -l app=redis-standalone-app
T22:11:15.442  WARN :📍Cannot reconnect to redis-0.redis-service.default.svc.cluster.local: Name does not resolve 
  java.net.UnknownHostException: redis-0.redis-service.default.svc.cluster.local: Name does not resolve
T22:11:19.409 ERROR :📍[FAIL] Key: keyA-00005649 -> Error after retries: Redis exception
T22:11:24.433  WARN :📍Cannot reconnect to redis-0.redis-service.default.svc.cluster.local
  java.net.UnknownHostException: redis-0.redis-service.default.svc.cluster.local
T22:11:32.636 INFO :📍Reconnected to redis-0.redis-service.default.svc.cluster.local/<unresolved>:18504
T22:11:32.711 INFO :📍[SUCCESS] Key: keyA-00005690 -> Normal operation in progress...

App 전체 로그 ▼

📄 App 전체 로그

[redis@Rocky9 standalone]$ kubectl logs -f -l app=redis-standalone-app
T22:11:11.503  INFO c.r.K.RedisOperationService          :  [SUCCESS] Key: keyA-00005610 -> Normal operation in progress...
T22:11:12.503  INFO c.r.K.RedisOperationService          :  [SUCCESS] Key: keyA-00005620 -> Normal operation in progress...
T22:11:13.503  INFO c.r.K.RedisOperationService          :  [SUCCESS] Key: keyA-00005630 -> Normal operation in progress...
T22:11:14.503  INFO c.r.K.RedisOperationService          :  [SUCCESS] Key: keyA-00005640 -> Normal operation in progress...
T22:11:15.435  INFO i.l.core.protocol.ConnectionWatchdog : Reconnecting, last destination was redis-0.redis-service.default.svc.cluster.local/10.42.0.220:18504
T22:11:15.442  WARN i.l.core.protocol.ConnectionWatchdog :📍Cannot reconnect to [redis-0.redis-service.default.svc.cluster.local/<unresolved>:18504]: redis-0.redis-service.default.svc.cluster.local: Name does not resolve

📍java.net.UnknownHostException: redis-0.redis-service.default.svc.cluster.local: Name does not resolve

T22:11:19.409 ERROR c.r.K.RedisOperationService          :📍[FAIL] Key: keyA-00005649 -> Error after retries: Redis exception
T22:11:19.504 ERROR c.r.K.RedisOperationService          :  [FAIL] Key: keyA-00005650 -> Error after retries: Redis exception
T22:11:19.604 ERROR c.r.K.RedisOperationService          :  [FAIL] Key: keyA-00005651 -> Error after retries: Redis exception
T22:11:19.704 ERROR c.r.K.RedisOperationService          :  [FAIL] Key: keyA-00005652 -> Error after retries: Redis exception
T22:11:19.804 ERROR c.r.K.RedisOperationService          :  [FAIL] Key: keyA-00005653 -> Error after retries: Redis exception
T22:11:19.904 ERROR c.r.K.RedisOperationService          :  [FAIL] Key: keyA-00005654 -> Error after retries: Redis exception
T22:11:20.004 ERROR c.r.K.RedisOperationService          :  [FAIL] Key: keyA-00005655 -> Error after retries: Redis exception
T22:11:20.104 ERROR c.r.K.RedisOperationService          :  [FAIL] Key: keyA-00005656 -> Error after retries: Redis exception
T22:11:23.411 ERROR c.r.K.RedisOperationService          :  [FAIL] Key: keyA-00005657 -> Error after retries: Redis exception
T22:11:23.506 ERROR c.r.K.RedisOperationService          :  [FAIL] Key: keyA-00005658 -> Error after retries: Redis exception
T22:11:23.606 ERROR c.r.K.RedisOperationService          :  [FAIL] Key: keyA-00005659 -> Error after retries: Redis exception
T22:11:23.706 ERROR c.r.K.RedisOperationService          :  [FAIL] Key: keyA-00005660 -> Error after retries: Redis exception
T22:11:23.805 ERROR c.r.K.RedisOperationService          :  [FAIL] Key: keyA-00005661 -> Error after retries: Redis exception
T22:11:23.905 ERROR c.r.K.RedisOperationService          :  [FAIL] Key: keyA-00005662 -> Error after retries: Redis exception
T22:11:24.005 ERROR c.r.K.RedisOperationService          :  [FAIL] Key: keyA-00005663 -> Error after retries: Redis exception
T22:11:24.105 ERROR c.r.K.RedisOperationService          :  [FAIL] Key: keyA-00005664 -> Error after retries: Redis exception
T22:11:24.432  INFO i.l.core.protocol.ConnectionWatchdog : Reconnecting, last destination was redis-0.redis-service.default.svc.cluster.local/<unresolved>:18504
T22:11:24.433  WARN i.l.core.protocol.ConnectionWatchdog :📍Cannot reconnect to [redis-0.redis-service.default.svc.cluster.local/<unresolved>:18504]: redis-0.redis-service.default.svc.cluster.local

📍java.net.UnknownHostException: redis-0.redis-service.default.svc.cluster.local

T22:11:27.413 ERROR c.r.K.RedisOperationService           :  [FAIL] Key: keyA-00005665 -> Error after retries: Redis exception
T22:11:27.507 ERROR c.r.K.RedisOperationService           :  [FAIL] Key: keyA-00005666 -> Error after retries: Redis exception
T22:11:27.607 ERROR c.r.K.RedisOperationService           :  [FAIL] Key: keyA-00005667 -> Error after retries: Redis exception
T22:11:27.707 ERROR c.r.K.RedisOperationService           :  [FAIL] Key: keyA-00005668 -> Error after retries: Redis exception
T22:11:27.807 ERROR c.r.K.RedisOperationService           :  [FAIL] Key: keyA-00005669 -> Error after retries: Redis exception
T22:11:27.907 ERROR c.r.K.RedisOperationService           :  [FAIL] Key: keyA-00005670 -> Error after retries: Redis exception
T22:11:28.007 ERROR c.r.K.RedisOperationService           :  [FAIL] Key: keyA-00005671 -> Error after retries: Redis exception
T22:11:28.107 ERROR c.r.K.RedisOperationService           :  [FAIL] Key: keyA-00005672 -> Error after retries: Redis exception
T22:11:31.414 ERROR c.r.K.RedisOperationService           :  [FAIL] Key: keyA-00005673 -> Error after retries: Redis exception
T22:11:31.509 ERROR c.r.K.RedisOperationService           :  [FAIL] Key: keyA-00005674 -> Error after retries: Redis exception
T22:11:31.609 ERROR c.r.K.RedisOperationService           :  [FAIL] Key: keyA-00005675 -> Error after retries: Redis exception
T22:11:31.709 ERROR c.r.K.RedisOperationService           :  [FAIL] Key: keyA-00005676 -> Error after retries: Redis exception
T22:11:31.808 ERROR c.r.K.RedisOperationService           :  [FAIL] Key: keyA-00005677 -> Error after retries: Redis exception
T22:11:31.908 ERROR c.r.K.RedisOperationService           :  [FAIL] Key: keyA-00005678 -> Error after retries: Redis exception
T22:11:32.008 ERROR c.r.K.RedisOperationService           :  [FAIL] Key: keyA-00005679 -> Error after retries: Redis exception
T22:11:32.108 ERROR c.r.K.RedisOperationService           :  [FAIL] Key: keyA-00005680 -> Error after retries: Redis exception
T22:11:32.632  INFO i.l.core.protocol.ConnectionWatchdog  : Reconnecting, last destination was redis-0.redis-service.default.svc.cluster.local/<unresolved>:18504
T22:11:32.636  INFO i.l.core.protocol.ReconnectionHandler :📍Reconnected to redis-0.redis-service.default.svc.cluster.local/<unresolved>:18504
T22:11:32.711  INFO c.r.K.RedisOperationService           :📍[SUCCESS] Key: keyA-00005690 -> Normal operation in progress...
T22:11:32.733  INFO c.r.K.RedisOperationService           :  [SUCCESS] Key: keyA-00005700 -> Normal operation in progress...
T22:11:32.737  INFO c.r.K.RedisOperationService           :  [SUCCESS] Key: keyA-00005710 -> Normal operation in progress...
T22:11:32.741  INFO c.r.K.RedisOperationService           :  [SUCCESS] Key: keyA-00005720 -> Normal operation in progress...
T22:11:32.745  INFO c.r.K.RedisOperationService           :  [SUCCESS] Key: keyA-00005730 -> Normal operation in progress...

🧮 Failover(장애조치) 후 상황

Redis-0 IP가 '10.42.0.220'에서 '10.42.0.222'로 변경됐다.

💻 Terminal-1

$ k get pods -o wide
NAME    READY STATUS  RESTARTS  AGE  IP            NODE  
redis-0 1/1   Running 0         1h   10.42.0.222   rocky9

🖧 Redis-Cluster 파드 다이어그램 (Failover 후)

┌──────────────┐ │ Redis(222) │ │ 🟢Master │ └──────────────┘

🧑‍💻 결론

Redis-Standalone Failover(장애 조치) 테스트 성공
파드(pod)를 다운, 재시작하면 IP가 변경된다. 약 17초 후 App가 재접속 성공해서 데이터를 입력한다.
약 30초(여유 있게) 정도 다운되어도 큰 문제가 없는 서비스라면 Redis-Standalone을 사용해 볼 수도 있다.

➡ 여기까지는 Redis-Cluster 단독으로 테스트했습니다.
Java(Spring) App와 연결해서 작동하는 것까지 보려면 여기를 클릭하세요.

<< K8s Redis-Standalone ②	K8s Redis Scale-Out(확장) >>

Email 답글이 올라오면 이메일로 알려드리겠습니다.