Help us understand the problem. What is going on with this article?

RedisにIDのリスト(100万件)を格納するときgzip圧縮するべきか

More than 3 years have passed since last update.

前提

$ ruby -v
ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-darwin15]
$ grep redis Gemfile.lock
    redis (3.3.1)
    redis-actionpack (5.0.1)
      redis-rack (>= 1, < 3)
      redis-store (>= 1.1.0, < 1.4.0)
    redis-activesupport (5.0.1)
      redis-store (~> 1.2.0)
    redis-rack (2.0.0)
      redis-store (~> 1.2.0)
    redis-rails (5.0.1)
      redis-actionpack (~> 5.0.0)
      redis-activesupport (~> 5.0.0)
      redis-store (~> 1.2.0)
    redis-store (1.2.0)
      redis (>= 2.2)
  redis (~> 3.0)
  redis-rails

$ bundle exec rails c

[1] pry(main)>

テストデータ

[1] pry(main)> ids = (1..1_000_000).to_a

Marshal vs JSON

[2] pry(main)> json = ""
=> ""
[3] pry(main)> marshaled = ""
=> ""

[4] pry(main)> Benchmark.realtime { marshaled = Marshal.dump(ids) }
=> 0.06611821899423376
[5] pry(main)> marshaled.size
=> 4934095

[6] pry(main)> Benchmark.realtime { json = ids.to_json }
=> 0.7926974160072859
[7] pry(main)> json.size
=> 6888897

raw vs gzip

  • redis は docker で立ち上げ、ポートフォワーディングしてホストから繋いでいます
[8] pry(main)> host = "192.168.99.100"
=> "192.168.99.100"
[9] pry(main)> port = 6379
=> 6379
[10] pry(main)> db = 0
=> 0
[11] pry(main)> redis = Redis.new(url: "redis://#{host}:#{port}/#{db}")
=> #<Redis client v3.3.1 for redis://192.168.99.100:6379/0>

[12] pry(main)> res = ""
[13] pry(main)> Benchmark.realtime { redis.set('cachekey', marshaled) }
=> 0.08158121499582194
[14] pry(main)> Benchmark.realtime { res = redis.get('cachekey') }
=> 0.34710412900312804
[15] pry(main)> res.size
=> 4905295

[15] pry(main)> gzip = ""
=> ""
[15] pry(main)> gzip2 = ""
=> ""
[15] pry(main)> res2 = ""
=> ""
[16] pry(main)> Benchmark.realtime { gzip = Zlib::Deflate.deflate(marshaled); redis.set('cachekey_gzip', gzip) }
=> 0.6087514339888003
[17] pry(main)> Benchmark.realtime { gzip2 = redis.get('cachekey_gzip'); res2 = Zlib::Inflate.inflate(gzip2) }
=> 0.06563029202516191
[18] pry(main)> gzip.size
=> 1505444
[19] pry(main)> gzip2.size
=> 1448046
[20] pry(main)> res2.size
=> 4934095

結論(になっていない)

  • サンプル数は少ないが数回叩いても大きな差異はなかった
  • to_json より marshal のほうがよさげ
  • 圧縮すると redis の get は速くなるが圧縮に時間はかかるので gzip圧縮しないほうが全体としては高速という結果
  • ただ圧縮するとredisのメモリは節約できる
  • 速度重視だとして、実際にはネットワークI/O がもっとかかるのでこの結果だけでは圧縮するべきかどうかは何ともいえない
Why not register and get more from Qiita?
  1. We will deliver articles that match you
    By following users and tags, you can catch up information on technical fields that you are interested in as a whole
  2. you can read useful information later efficiently
    By "stocking" the articles you like, you can search right away
Comments
No comments
Sign up for free and join this conversation.
If you already have a Qiita account
Why do not you register as a user and use Qiita more conveniently?
You need to log in to use this function. Qiita can be used more conveniently after logging in.
You seem to be reading articles frequently this month. Qiita can be used more conveniently after logging in.
  1. We will deliver articles that match you
    By following users and tags, you can catch up information on technical fields that you are interested in as a whole
  2. you can read useful information later efficiently
    By "stocking" the articles you like, you can search right away
ユーザーは見つかりませんでした