Redis/Memcache代理服务Twemproxy简介



简介 twemproxy，也叫nutcracker，是twtter开源的Redis和Memcache代理服务器。 功能 Fast. Lightweight. Maintains persistent server connections. Keeps connection count on the backend caching servers low. Enables pipelining of requests and respon

简介

twemproxy，也叫nutcracker，是twtter开源的Redis和Memcache代理服务器。

功能

Fast.
Lightweight.
Maintains persistent server connections.
Keeps connection count on the backend caching servers low.
Enables pipelining of requests and responses.
Supports proxying to multiple servers.
Supports multiple server pools simultaneously.
Shard data automatically across multiple servers.
Implements the complete memcached ascii and redis protocol.
Easy configuration of server pools through a YAML file.
Supports multiple hashing modes including consistent hashing and distribution.
Can be configured to disable nodes on failures.
Observability through stats exposed on stats monitoring port.
Works with Linux, *BSD, OS X and Solaris (SmartOS)

安装

从官网下载安装包编译并安装：distribution tarball:

[root@~/software/nutcracker-0.3.0]# ./configure --prefix=/usr/local/nutcracker-0.3.0
[root@~/software/nutcracker-0.3.0]# make
[root@~/software/nutcracker-0.3.0]# make install
[root@~/software/nutcracker-0.3.0]# cp -R conf /usr/local/nutcracker-0.3.0
[root@~/software/nutcracker-0.3.0]# ln -s /usr/local/nutcracker-0.3.0 /usr/local/nutcracker
[root@~/software/nutcracker-0.3.0]# ln -s /usr/local/nutcracker/sbin/nutcracker /usr/local/bin/nutcracker

配置

twemproxy支持通过YAML语法配置，其支持的指令集如下：

listen: The listening address and port (name:port or ip:port) for this server pool.
hash: The name of the hash function. Possible values are:
one_at_a_time
md5
crc16
crc32 (crc32 implementation compatible with libmemcached)
crc32a (correct crc32 implementation as per the spec)
fnv1_
fnv1a_
fnv1_32
fnv1a_32
hsieh
murmur
jenkins
hash_tag: A two character string that specifies the part of the key used for hashing. Eg "{}" or "$$".Hash tag enable mapping different keys to the same server as long as the part of the key within the tag is the same.
distribution: The key distribution mode. Possible values are:
ketama
modula
random
timeout: The timeout value in msec that we wait for to establish a connection to the server or receive a response from a server. By default, we wait indefinitely.
backlog: The TCP backlog argument. Defaults to 512.
preconnect: A boolean value that controls if nutcracker should preconnect to all the servers in this pool on process start. Defaults to false.
redis: A boolean value that controls if a server pool speaks redis or memcached protocol. Defaults to false.
server_connections: The maximum number of connections that can be opened to each server. By default, we open at most 1 server connection.
auto_eject_hosts: A boolean value that controls if server should be ejected temporarily when it fails consecutively server_failure_limit times. See liveness recommendations for information. Defaults to false.
server_retry_timeout: The timeout value in msec to wait for before retrying on a temporarily ejected server, when auto_eject_host is set to true. Defaults to 30000 msec.
server_failure_limit: The number of conseutive failures on a server that would leads to it being temporarily ejected when auto_eject_host is set to true. Defaults to 2.
servers: A list of server address, port and weight (name:port:weight or ip:port:weight) for this server pool.

以本文为例，配置一个Redis代理和一个Memcache代理服务如下：

[root@~/software/nutcracker-0.3.0]# cat /usr/local/nutcracker/conf/nutcracker.yml
alpha:
 listen: 127.0.0.1:22121
 hash: fnv1a_
 hash_tag: "{}"
 distribution: ketama
 auto_eject_hosts: false
 timeout: 400
 redis: true
 servers:
 - 127.0.0.1:6379:1

beta:
 listen: 127.0.0.1:22122
 hash: fnv1a_
 distribution: ketama
 timeout: 400
 backlog: 1024
 preconnect: true
 auto_eject_hosts: true
 server_retry_timeout: 2000
 server_failure_limit: 3
 servers:
 - 127.0.0.1:11211:1
 - 127.0.0.1:11212:1

其中，Redis实例 127.0.0.1:6379 和 Memcache实例 127.0.0.1:11211 127.0.0.1:11212 请预先启动好。

/etc/init.d/redis_6379 start
/usr/local/memcached-1.4.20/bin/memcached -d -m  -l 127.0.0.1 -p 11211 -u root
/usr/local/memcached-1.4.20/bin/memcached -d -m  -l 127.0.0.1 -p 11212 -u root

测试

https://github.com/billfeller/billfeller.github.io/blob/master/code/twenproxyTest.php

启动服务：

[root@/usr/local/nutcracker/conf]# nutcracker -c nutcracker.yml
[Sun Dec 14 20:59:04 2014] nc.c:187 nutcracker-0.3.0 built for Linux 2.6.32-431.23.3.el6.x86_ x86_ started on pid 14359
[Sun Dec 14 20:59:04 2014] nc.c:192 run, rabbit run / dig that hole, forget the sun / and when at last the work is done / don't sit down / it's time to dig another one

执行测试脚本：

[root@~/wade/git/billfeller.github.io/code]# /usr/local/php/bin/php twenproxyTest.php 
bool(true)
string(1) "1"
int(1)
bool(false)
bool(true)
int(2)
bool(true)
int(3)
bool(true)
bool(false)