Redis-Cluster
标签:Redis

Cluster

1. 数据分布概论

数据分区:

将一个大的数据按照一定的分区规则讲数据划分为不同的子集来。

顺序分布于哈希分布:

对比

分布方式 特点 典型产品
哈希分布 数据分散度高
键值分布与业务无关(经过hash计算了)
无法顺序访问
支持批量操作
一致性哈希Memcache
Redis Cluster
其他缓存产品
顺序分布 数据分散度易倾斜
键值与业务相关
可顺序性访问
支持批量操作
BigTable
HBase

2. 节点hash取余分区

上面左边每次增加单个节点,再进行取余重新分区,数据迁移将达到80%,如果采用后面的节点翻倍的方式,数据的迁移约50%。

因为迁移后,第一次是没办法从缓存中获取到的,要从数据库中回写,大量的回写将会导致数据库的压力。但方式简单,如果缓存的数据不大的话,可以使用。

3. 一致性哈希

一致性哈希将数据看成一个环,每个节点负责一部分区域,当插入一个key的时候,hash后落在某个区间内,再顺时针去找离他最近的那个节点存储,如上图。

上图的四个数据hash后都落在n2上,这个时候插入了n5。

这样添加一个节点之后,不会影响节点的数据,只会影响n2的数据。但是它没办法将原来n2上的数据迁移到自己上,只能从数据库里面去取。

一致性哈希采用的还是客户端分片的方式(哈希+顺时针查找),它只影响临近节点,但是还是有数据迁移发生,它能保证最小的数据迁移,但插入单个的节点,没办法做到负载均衡,其他节点的压力还是很大,所以一般做翻倍扩容(4->8)。

4. 虚拟槽分区

5. 基本架构

分布式情况下,每个节点之间是互相通信的,每个节点都负责读写。

cluster-enabled:yes 表示以集群方式启动。

meet操作:

meet操作确保任意两个节点之间都可以通信。

指派槽:

当客户端访问某个key的时候,先对key计算hash值,再对16383取余,看是否在某个槽内,如果不在的话,将返回一个槽告诉客户端在这个槽上,因为它们彼此之间知道对方信息。

6. 安装配置

Step 1:编写配置文件

port 7000
daemonize yes
dir "/usr/local/redis/data"
logfile "7000.log"
dbfilename "dump-7000.rdb"
cluster-enabled yes
cluster-config-file nodes-7000.conf
cluster-require-full-coverage no

Step 2: 复制多份

[root@localhost cluster]# sed 's/7000/7001/g' redis-7000.conf > redis-7001.conf  
[root@localhost cluster]# sed 's/7000/7002/g' redis-7000.conf > redis-7002.conf  
[root@localhost cluster]# sed 's/7000/7003/g' redis-7000.conf > redis-7003.conf  
[root@localhost cluster]# sed 's/7000/7004/g' redis-7000.conf > redis-7004.conf  
[root@localhost cluster]# sed 's/7000/7005/g' redis-7000.conf > redis-7005.conf  

Step 3:启动 新建redis-cluster-start.sh并执行

#!/bin/bash
/usr/local/redis/bin/redis-server /usr/local/redis/cluster/redis-7000.conf
/usr/local/redis/bin/redis-server /usr/local/redis/cluster/redis-7001.conf
/usr/local/redis/bin/redis-server /usr/local/redis/cluster/redis-7002.conf
/usr/local/redis/bin/redis-server /usr/local/redis/cluster/redis-7003.conf
/usr/local/redis/bin/redis-server /usr/local/redis/cluster/redis-7004.conf
/usr/local/redis/bin/redis-server /usr/local/redis/cluster/redis-7005.conf

Step 4: 查看进程信息:

[root@localhost myshell]# ps -ef | grep redis
root      3114  2785  0 14:10 pts/0    00:00:00 vim redis-cluster.sh
root      3399     1  0 14:21 ?        00:00:00 /usr/local/redis/bin/redis-server *:7000 [cluster]
root      3401     1  1 14:21 ?        00:00:00 /usr/local/redis/bin/redis-server *:7001 [cluster]
root      3406     1  0 14:21 ?        00:00:00 /usr/local/redis/bin/redis-server *:7002 [cluster]
root      3408     1  0 14:21 ?        00:00:00 /usr/local/redis/bin/redis-server *:7003 [cluster]
root      3416     1  0 14:21 ?        00:00:00 /usr/local/redis/bin/redis-server *:7004 [cluster]
root      3421     1  0 14:21 ?        00:00:00 /usr/local/redis/bin/redis-server *:7005 [cluster]
root      3439  2785  0 14:21 pts/0    00:00:00 grep --color=auto redis

可以看到每个进程后面都跟了cluster

查看集群配置:

[root@localhost redis]# ./bin/redis-cli -p 7000 cluster nodes
48667b9a473814e258a3ce192d8613eb54d1c271 :7000@17000 myself,master - 0 0 0 connected

查看信息:

[root@localhost redis]# ./bin/redis-cli -p 7000 cluster info
cluster_state:fail
cluster_slots_assigned:0
cluster_slots_ok:0
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:1
cluster_size:0
cluster_current_epoch:0
cluster_my_epoch:0
cluster_stats_messages_sent:0
cluster_stats_messages_received:0

可见当前的集群状态是fail的,槽也没有分配的。

Step 5:执行meet操作

#! /bin/bash
/usr/local/redis/bin/redis-cli -p 7000 cluster meet 192.168.91.136 7001
/usr/local/redis/bin/redis-cli -p 7000 cluster meet 192.168.91.136 7002
/usr/local/redis/bin/redis-cli -p 7000 cluster meet 192.168.91.136 7003
/usr/local/redis/bin/redis-cli -p 7000 cluster meet 192.168.91.136 7004
/usr/local/redis/bin/redis-cli -p 7000 cluster meet 192.168.91.136 7005

查看meet结果:

[root@localhost data]# cat nodes-7000.conf 
3d531d0d33cfafa60ab57da410c018e6c116c669 192.168.91.136:7004@17004 master - 0 1529303776359 2 connected
15d8b40c13df426c9d33dec29282cd8c8b91a2ad 192.168.91.136:7001@17001 master - 0 1529303779377 4 connected
95c24138d598b9719b25cfb6391856074e1bd00d 192.168.91.136:7003@17003 master - 0 1529303776000 0 connected
60f22252b656e69641bfae600d2984356ba196d9 192.168.91.136:7002@17002 master - 0 1529303777363 3 connected
48667b9a473814e258a3ce192d8613eb54d1c271 192.168.91.136:7000@17000 myself,master - 0 1529303777000 1 connected
125d42f70cab711739ef1ba784d9f31d8ae8eea7 192.168.91.136:7005@17005 master - 0 1529303778370 5 connected
vars currentEpoch 5 lastVoteEpoch 0

可见7000和其他meet成功,其他的类似,注意只需要一个节点执行meet操作,其他的都可以互联了

比如我现在对7001进行info操作:

[root@localhost redis]# ./bin/redis-cli -p 7001 cluster info
cluster_state:fail
cluster_slots_assigned:0
cluster_slots_ok:0
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:0
cluster_current_epoch:5
cluster_my_epoch:4
cluster_stats_messages_ping_sent:220
cluster_stats_messages_pong_sent:227
cluster_stats_messages_meet_sent:2
cluster_stats_messages_sent:449
cluster_stats_messages_ping_received:223
cluster_stats_messages_pong_received:222
cluster_stats_messages_meet_received:4
cluster_stats_messages_received:449

可以看见上面已经发行了6个节点。

Step 6:给master添加槽

三个主节点平均分16384个槽,则

7000:0~5641

7001:5642~10922,

7002:10923~16383

编写脚本redis-cluster-addslots.sh分配槽:

start=$1
end=$2
port=$3
for slot in `seq ${start} ${end}`
do
	echo "slot:${slot}"
	/usr/local/redis/bin/redis-cli -p ${port} cluster addslots ${slot}
done

执行:

sh redis-cluster-addslots.sh 0 5641 7000
sh redis-cluster-addslots.sh 5642 10922 7001
sh redis-cluster-addslots.sh 10923 16383 7002

分配完成后查看信息:

[root@localhost redis]# ./bin/redis-cli -p 7000 cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:5
cluster_my_epoch:1
cluster_stats_messages_ping_sent:958
cluster_stats_messages_pong_sent:1014
cluster_stats_messages_fail_sent:20
cluster_stats_messages_sent:1992
cluster_stats_messages_ping_received:1014
cluster_stats_messages_pong_received:958
cluster_stats_messages_fail_received:18
cluster_stats_messages_received:1990

看见上面的16384个槽已经被分配了

查看nodes信息:

[root@localhost data]# cat nodes-7000.conf 
60f22252b656e69641bfae600d2984356ba196d9 192.168.91.136:7002@17002 master - 0 1529583971653 3 connected 10923-16383
48667b9a473814e258a3ce192d8613eb54d1c271 192.168.91.136:7000@17000 myself,master - 0 1529583967000 1 connected 0-5641
3d531d0d33cfafa60ab57da410c018e6c116c669 192.168.91.136:7004@17004 master - 0 1529583969644 2 connected
125d42f70cab711739ef1ba784d9f31d8ae8eea7 192.168.91.136:7005@17005 master - 0 1529583968000 5 connected
95c24138d598b9719b25cfb6391856074e1bd00d 192.168.91.136:7003@17003 master - 0 1529583970648 0 connected
15d8b40c13df426c9d33dec29282cd8c8b91a2ad 192.168.91.136:7001@17001 master - 0 1529583969000 4 connected 5642-10922
vars currentEpoch 5 lastVoteEpoch 0

看见上面已经分配了槽

Step 6:cluster主从分配

查看node信息:

./bin/redis-cli -p 7000 cluster nodes
60f22252b656e69641bfae600d2984356ba196d9 192.168.91.136:7002@17002 master - 0 1529584417296 3 connected 10923-16383
48667b9a473814e258a3ce192d8613eb54d1c271 192.168.91.136:7000@17000 myself,master - 0 1529584416000 1 connected 0-5641
3d531d0d33cfafa60ab57da410c018e6c116c669 192.168.91.136:7004@17004 master - 0 1529584416288 2 connected
125d42f70cab711739ef1ba784d9f31d8ae8eea7 192.168.91.136:7005@17005 master - 0 1529584418000 5 connected
95c24138d598b9719b25cfb6391856074e1bd00d 192.168.91.136:7003@17003 master - 0 1529584419309 0 connected
15d8b40c13df426c9d33dec29282cd8c8b91a2ad 192.168.91.136:7001@17001 master - 0 1529584419000 4 connected 5642-10922

执行分配7003->7000、7004->7001、7005->7002

 ./bin/redis-cli -p 7003 cluster replicate 48667b9a473814e258a3ce192d8613eb54d1c271
./bin/redis-cli -p 7004 cluster replicate 15d8b40c13df426c9d33dec29282cd8c8b91a2ad
 ./bin/redis-cli -p 7005 cluster replicate 60f22252b656e69641bfae600d2984356ba196d9

检查分配关系:

[root@localhost redis]# ./bin/redis-cli -p 7000 cluster nodes
60f22252b656e69641bfae600d2984356ba196d9 192.168.91.136:7002@17002 master - 0 1529584690004 3 connected 10923-16383
48667b9a473814e258a3ce192d8613eb54d1c271 192.168.91.136:7000@17000 myself,master - 0 1529584690000 1 connected 0-5641
3d531d0d33cfafa60ab57da410c018e6c116c669 192.168.91.136:7004@17004 slave 15d8b40c13df426c9d33dec29282cd8c8b91a2ad 0 1529584691014 4 connected
125d42f70cab711739ef1ba784d9f31d8ae8eea7 192.168.91.136:7005@17005 slave 60f22252b656e69641bfae600d2984356ba196d9 0 1529584688000 5 connected
95c24138d598b9719b25cfb6391856074e1bd00d 192.168.91.136:7003@17003 slave 48667b9a473814e258a3ce192d8613eb54d1c271 0 1529584692021 4 connected
15d8b40c13df426c9d33dec29282cd8c8b91a2ad 192.168.91.136:7001@17001 master - 0 1529584690000 4 connected 5642-10922

或者以槽位单位查看:

[root@localhost redis]# ./bin/redis-cli -p 7000 cluster slots
1) 1) (integer) 10923
   2) (integer) 16383
   3) 1) "192.168.91.136"
      2) (integer) 7002
      3) "60f22252b656e69641bfae600d2984356ba196d9"
   4) 1) "192.168.91.136"
      2) (integer) 7005
      3) "125d42f70cab711739ef1ba784d9f31d8ae8eea7"
2) 1) (integer) 0
   2) (integer) 5641
   3) 1) "192.168.91.136"
      2) (integer) 7000
      3) "48667b9a473814e258a3ce192d8613eb54d1c271"
   4) 1) "192.168.91.136"
      2) (integer) 7003
      3) "95c24138d598b9719b25cfb6391856074e1bd00d"
3) 1) (integer) 5642
   2) (integer) 10922
   3) 1) "192.168.91.136"
      2) (integer) 7001
      3) "15d8b40c13df426c9d33dec29282cd8c8b91a2ad"
   4) 1) "192.168.91.136"
      2) (integer) 7004
      3) "3d531d0d33cfafa60ab57da410c018e6c116c669"

7. 利用ruby安装

7.1 安装ruby环境

Step 1:下载

cd /opt/soft
wget https://cache.ruby-lang.org/pub/ruby/2.3/ruby-2.3.1.tar.gz

Step 2:安装ruby

tar -xvf ruby-2.3.1.tar.gz
cd ruby-2.3.1/
./configure -prefix=/usr/local/ruby
make
make install
cd /usr/local/ruby
cp bin/ruby /usr/local/bin

Step 3:安装rubygem redis

wget http://rubygems.org/downloads/redis-3.3.0.gem
sudo gem install -l redis-3.3.0.gem
sudo gem list -- check redis gem

Step 4:安装redis-trib.rb

cp ${REDIS_HOME}/src/redis-trib.rb /usr/local/bin

7.2 使用redis-trib.rb 配置

使用 ./src/redis-trib.rb create --replicas 1 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005

[root@localhost redis]# ./src/redis-trib.rb create --replicas 1 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
127.0.0.1:7000
127.0.0.1:7001
127.0.0.1:7002
Adding replica 127.0.0.1:7004 to 127.0.0.1:7000
Adding replica 127.0.0.1:7005 to 127.0.0.1:7001
Adding replica 127.0.0.1:7003 to 127.0.0.1:7002
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: 98cdd9928767f879cdc39453c54f5249beecc35e 127.0.0.1:7000
   slots:0-5460 (5461 slots) master
M: dfbb938a4e1a3b031d6a5fdf4955ec9ae23c9ff0 127.0.0.1:7001
   slots:5461-10922 (5462 slots) master
M: 4472f9374bb20b40f5b65d781bf33410bc7d5386 127.0.0.1:7002
   slots:10923-16383 (5461 slots) master
S: ba2c1179961f12edc4ea22a46674bb0e4fda1abb 127.0.0.1:7003
   replicates dfbb938a4e1a3b031d6a5fdf4955ec9ae23c9ff0
S: 45c2e0c0505e1347f9d4c4dca64a361823edf901 127.0.0.1:7004
   replicates 4472f9374bb20b40f5b65d781bf33410bc7d5386
S: b45227eb8412073c644c724e6b9fedf207e4aefa 127.0.0.1:7005
   replicates 98cdd9928767f879cdc39453c54f5249beecc35e
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join.....
>>> Performing Cluster Check (using node 127.0.0.1:7000)
M: 98cdd9928767f879cdc39453c54f5249beecc35e 127.0.0.1:7000
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
S: b45227eb8412073c644c724e6b9fedf207e4aefa 127.0.0.1:7005
   slots: (0 slots) slave
   replicates 98cdd9928767f879cdc39453c54f5249beecc35e
S: ba2c1179961f12edc4ea22a46674bb0e4fda1abb 127.0.0.1:7003
   slots: (0 slots) slave
   replicates dfbb938a4e1a3b031d6a5fdf4955ec9ae23c9ff0
S: 45c2e0c0505e1347f9d4c4dca64a361823edf901 127.0.0.1:7004
   slots: (0 slots) slave
   replicates 4472f9374bb20b40f5b65d781bf33410bc7d5386
M: 4472f9374bb20b40f5b65d781bf33410bc7d5386 127.0.0.1:7002
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
M: dfbb938a4e1a3b031d6a5fdf4955ec9ae23c9ff0 127.0.0.1:7001
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

  • 11 min read

CONTRIBUTORS


  • 11 min read