Redis

REmote DIctionary Service is a key-value database.

Connect to database

In [1]:

import redis

In [2]:

r = redis.Redis()

In [3]:

r.ping()

Out[3]:

True

Clear database

In [4]:

r.flushdb()

Out[4]:

True

Simple data types

Set and get a single value

In [5]:

r.set('a', 'adenosine')

Out[5]:

True

In [6]:

r.get('a')

Out[6]:

b'adenosine'

Set and get multiple values

In [7]:

r.mset(c='cytosine', t='thymidine', g='guanosine')

Out[7]:

True

In [8]:

r.mget(list('tcga'))

Out[8]:

[b'thymidine', b'cytosine', b'guanosine', b'adenosine']

Deletion

In [9]:

r.delete('a')

Out[9]:

1

In [10]:

r.keys()

Out[10]:

[b't', b'c', b'g']

In [11]:

r.delete('c', 't', 'g')

Out[11]:

3

In [12]:

r.keys()

Out[12]:

[]

Transactions

Transactions are achieved by creating and executing pipeline. This is useful not just for atomicity, but also to reduce communication costs.

In [13]:

pipe = r.pipeline()
pipe.set('a', 0).incr('a').incr('a').incr('a').execute()

Out[13]:

[True, 1, 2, 3]

In [14]:

r.get('a')

Out[14]:

b'3'

Expiring values

In [15]:

import time

In [16]:

r.setex('foo', 'bar', 3)
print(r.get('foo'))
time.sleep(3)
print(r.get('foo'))

b'bar'
None

Alternative

In [17]:

r.set('foo', 'bar')
r.expire('foo', 3)
print(r.get('foo'))
time.sleep(3)
print(r.get('foo'))

b'bar'
None

Complex data types

In [18]:

r.hmset('nuc', dict(a='adenosine', c='cytosine', t='thymidine', g='guanosine'))

Out[18]:

True

In [19]:

r.hget('nuc', 'a')

Out[19]:

b'adenosine'

In [20]:

r.hmget('nuc', list('ctg'))

Out[20]:

[b'cytosine', b'thymidine', b'guanosine']

In [21]:

r.hkeys('nuc')

Out[21]:

[b't', b'a', b'g', b'c']

In [22]:

r.hvals('nuc')

Out[22]:

[b'thymidine', b'adenosine', b'guanosine', b'cytosine']

In [23]:

r.rpush('xs', 1, 2, 3)

Out[23]:

3

In [24]:

r.lpush('xs', 4, 5, 6)

Out[24]:

6

In [25]:

r.llen('xs')

Out[25]:

6

In [26]:

r.lrange('xs', 0, r.llen('xs'))

Out[26]:

[b'6', b'5', b'4', b'1', b'2', b'3']

In [27]:

r.lrange('xs', 0, -1)

Out[27]:

[b'6', b'5', b'4', b'1', b'2', b'3']

Using list as a queue

In [28]:

r.lpush('q', 1, 2, 3)

Out[28]:

3

In [29]:

while r.llen('q'):
    print(r.rpop('q'))

b'1'
b'2'
b'3'

Using list as stack

In [30]:

r.lpush('q', 1, 2, 3)

Out[30]:

3

In [31]:

while r.llen('q'):
    print(r.lpop('q'))

b'3'
b'2'
b'1'

Transferring values across lists

In [32]:

r.lpush('l1', 1,2,3)

Out[32]:

3

In [33]:

while r.llen('l1'):
    r.rpoplpush('l1', 'l2')
r.llen('l1'), r.llen('l2')

Out[33]:

(0, 3)

In [34]:

for key in r.scan_iter('l2'):
    print(key)

b'l2'

In [35]:

r.lpush('l1', 1,2,3)

Out[35]:

3

Sets

In [36]:

r.sadd('s1', 1,2,3)

Out[36]:

3

In [37]:

r.sadd('s1', 2,3,4)

Out[37]:

1

In [38]:

r.smembers('s1')

Out[38]:

{b'1', b'2', b'3', b'4'}

In [39]:

r.sadd('s2', 4,5,6)

Out[39]:

3

In [40]:

r.sdiff(['s1', 's2'])

Out[40]:

{b'1', b'2', b'3'}

In [41]:

r.sinter(['s1', 's2'])

Out[41]:

{b'4'}

In [42]:

r.sunion(['s1', 's2'])

Out[42]:

{b'1', b'2', b'3', b'4', b'5', b'6'}

Sorted sets

This is equivalent to a priority queue.

In [43]:

r.zadd('jobs', 'job1', 3, 'job2', 7, 'job3', 1, 'job4', 2, 'job5', 6)

Out[43]:

5

In [44]:

r.zincrby('jobs', 'job5', 2)

Out[44]:

8.0

In [45]:

r.zrange('jobs', 0, -1, withscores=True)

Out[45]:

[(b'job3', 1.0),
 (b'job4', 2.0),
 (b'job1', 3.0),
 (b'job2', 7.0),
 (b'job5', 8.0)]

In [46]:

r.zrevrange('jobs', 0, -1, withscores=True)

Out[46]:

[(b'job5', 8.0),
 (b'job2', 7.0),
 (b'job1', 3.0),
 (b'job4', 2.0),
 (b'job3', 1.0)]

Union and intersection store

In [47]:

s1 = 'time flies like an arrow'
s2 = 'fruit flies like a banana'

In [48]:

from collections import Counter

In [49]:

c1 = Counter(s1.split())

In [50]:

c2 = Counter(s2.split())

In [51]:

r.zadd('c1', **c1)

Out[51]:

5

In [52]:

r.zadd('c2', **c2)

Out[52]:

5

In [53]:

r.zrange('c1', 0, -1, withscores=True)

Out[53]:

[(b'an', 1.0),
 (b'arrow', 1.0),
 (b'flies', 1.0),
 (b'like', 1.0),
 (b'time', 1.0)]

In [54]:

r.zrange('c2', 0, -1, withscores=True)

Out[54]:

[(b'a', 1.0),
 (b'banana', 1.0),
 (b'flies', 1.0),
 (b'fruit', 1.0),
 (b'like', 1.0)]

In [55]:

r.zunionstore('c3', ['c1', 'c2'])

Out[55]:

8

In [56]:

r.zrange('c3', 0, -1, withscores=True)

Out[56]:

[(b'a', 1.0),
 (b'an', 1.0),
 (b'arrow', 1.0),
 (b'banana', 1.0),
 (b'fruit', 1.0),
 (b'time', 1.0),
 (b'flies', 2.0),
 (b'like', 2.0)]

In [57]:

r.zinterstore('c4', ['c1', 'c2'])

Out[57]:

2

In [58]:

r.zrange('c4', 0, -1, withscores=True)

Out[58]:

[(b'flies', 2.0), (b'like', 2.0)]

Publisher/Subscriber

In [59]:

p = r.pubsub()

In [60]:

p.subscribe('python', 'perl', 'sql')

In [61]:

p.get_message()

Out[61]:

{'channel': b'sql', 'data': 1, 'pattern': None, 'type': 'subscribe'}

In [62]:

p.get_message()

Out[62]:

{'channel': b'python', 'data': 2, 'pattern': None, 'type': 'subscribe'}

In [63]:

p.get_message()

Out[63]:

{'channel': b'perl', 'data': 3, 'pattern': None, 'type': 'subscribe'}

In [64]:

p.channels

Out[64]:

{b'perl': None, b'python': None, b'sql': None}

In [65]:

p2 = r.pubsub()

In [66]:

p2.psubscribe('p*')

In [67]:

p2.patterns

Out[67]:

{b'p*': None}

Interpretation of message

From redis-puy

Every message read from a PubSub instance will be a dictionary with the following keys.

  • type: One of the following: ‘subscribe’, ‘unsubscribe’, ‘psubscribe’, ‘punsubscribe’, ‘message’, ‘pmessage’
  • channel: The channel [un]subscribed to or the channel a message was published to
  • pattern: The pattern that matched a published message’s channel. Will be None in all cases except for ‘pmessage’ types.
  • data: The message data. With [un]subscribe messages, this value will be the number of channels and patterns the connection is currently subscribed to. With [p]message messages, this value will be the actual published message.
In [68]:

r.publish('python', 'use blank spaces')
r.publish('python', 'no semi-colons')
r.publish('perl', 'use spaceship operator')
r.publish('sql', 'select this')
r.publish('haskell', 'functional is cool')

Out[68]:

0

In [69]:

p.get_message()

Out[69]:

{'channel': b'python',
 'data': b'use blank spaces',
 'pattern': None,
 'type': 'message'}

In [70]:

p.get_message()

Out[70]:

{'channel': b'python',
 'data': b'no semi-colons',
 'pattern': None,
 'type': 'message'}

In [71]:

p.get_message()

Out[71]:

{'channel': b'perl',
 'data': b'use spaceship operator',
 'pattern': None,
 'type': 'message'}

In [72]:

p.get_message()

Out[72]:

{'channel': b'sql', 'data': b'select this', 'pattern': None, 'type': 'message'}

In [73]:

p.get_message()

In [74]:

p.unsubscribe('r')

In [75]:

p.channels

Out[75]:

{b'perl': None, b'python': None, b'sql': None}

In [76]:

r.publish('python', 'use blank spaces 2')
r.publish('python', 'no semi-colons 2')
r.publish('perl', 'use spaceship operator 2')
r.publish('sql', 'select this 2')
r.publish('haskell', 'functional is cool 2')

Out[76]:

0

In [77]:

for i in range(10):
    m = p.get_message()
    if m:
        print('%-10s%s' % (m['channel'], m['data']))

b'r'      3
b'python' b'use blank spaces 2'
b'python' b'no semi-colons 2'
b'perl'   b'use spaceship operator 2'
b'sql'    b'select this 2'

In [78]:

for i in range(10):
    m = p2.get_message()
    if m:
        print('%-10s%s' % (m['channel'], m['data']))

b'p*'     1
b'python' b'use blank spaces'
b'python' b'no semi-colons'
b'perl'   b'use spaceship operator'
b'python' b'use blank spaces 2'
b'python' b'no semi-colons 2'
b'perl'   b'use spaceship operator 2'

Multiple databases

In [79]:

r2 = redis.Redis(db=1)
r2.flushdb()

Out[79]:

True

In [80]:

for c in ['c1', 'c2', 'c3', 'c4']:
    r.move(c, 1)

In [81]:

for key in r2.scan_iter('c?'):
    print(r2.zrange(key, 0, -1, withscores=True))

[(b'a', 1.0), (b'banana', 1.0), (b'flies', 1.0), (b'fruit', 1.0), (b'like', 1.0)]
[(b'an', 1.0), (b'arrow', 1.0), (b'flies', 1.0), (b'like', 1.0), (b'time', 1.0)]
[(b'a', 1.0), (b'an', 1.0), (b'arrow', 1.0), (b'banana', 1.0), (b'fruit', 1.0), (b'time', 1.0), (b'flies', 2.0), (b'like', 2.0)]
[(b'flies', 2.0), (b'like', 2.0)]

Clean up

There is no need to close the connections when we use the Redis() object. This is taken care of automatically

python def execute_command(self, *args, **options):     "Execute a command and return a parsed response"     pool = self.connection_pool     command_name = args[0]     connection = pool.get_connection(command_name, **options)     try:          connection.send_command(*args)         return self.parse_response(connection, command_name, **options)     except (ConnectionError, TimeoutError) as e:         connection.disconnect()         if not connection.retry_on_timeout and isinstance(e, TimeoutError):             raise         connection.send_command(*args)         return self.parse_response(connection, command_name, **options)     finally:         pool.release(connection)

Benchmark redis

In [82]:

%%bash

redis-benchmark -q -n 10000 -c 50

PING_INLINE: 52083.33 requests per second
PING_BULK: 47846.89 requests per second
SET: 36496.35 requests per second
GET: 34013.61 requests per second
INCR: 35587.19 requests per second
LPUSH: 38022.81 requests per second
RPUSH: 32051.28 requests per second
LPOP: 33670.04 requests per second
RPOP: 31847.13 requests per second
SADD: 32894.74 requests per second
SPOP: 55865.92 requests per second
LPUSH (needed to benchmark LRANGE): 55555.55 requests per second
LRANGE_100 (first 100 elements): 13831.26 requests per second
LRANGE_300 (first 300 elements): 6635.70 requests per second
LRANGE_500 (first 450 elements): 4703.67 requests per second
LRANGE_600 (first 600 elements): 3703.70 requests per second
MSET (10 keys): 41493.78 requests per second