Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bringing this up to Crystal 1.9.2 compatibility: #10

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,4 @@
/shard.lock

/tmp/*
bin
3 changes: 3 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,7 +1,10 @@
CRYSTAL_BIN ?= $(shell which crystal)

test:
docker-compose up -d
sleep 2
$(CRYSTAL_BIN) spec
docker-compose down

benchmark:
$$(mkdir tmp -p)
Expand Down
41 changes: 34 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,15 @@ dependencies:
version: ~> 0.1.0
```

Temporarily, if you wan't the newest version from akitaonrails' fork:

```yaml
dependencies:
kiwi:
github: akitaonrails/kiwi
branch: master
```

## Usage

All the stores have the same simple interface defined by
Expand All @@ -52,6 +61,11 @@ store.clear
# Or your can use Hash-like methods:
store["key"] = "new value"
store["key"] # => "new "value"

# fetch with a block:
store.fetch("key") do
"value"
end
```

### FileStore
Expand Down Expand Up @@ -96,17 +110,28 @@ require "kiwi/memcached_store"
store = Kiwi::MemcachedStore.new(Memcached::Client.new)
```

### Expires

Almost all stores, but the FileStore, can receive a "expires_in" argument to set a default expiring time span:

```crystal
store1 = Kiwi::MemcachedStore.new(Memcached::Client.new, expires_in: 5.minutes)
store2 = Kiwi::RedisStore.new(Redis::PooledClient.new, expires_in: 1.hour)
```

This is a cache library, so cached data is supposed to eventually expire without manual intervention.

## Benchmark

The following table shows **operations per second** for every particular store on my machine.

| | set | get | get(empty) | delete |
| ------------------:| -------:| -------:| ----------:| --------:|
| **MemoryStore** | 3056000 | 4166000 | 4074000 | 10473000 |
| **LevelDBStore** | 120000 | 193000 | 253000 | 37000 |
| **RedisStore** | 41000 | 42000 | 42000 | 21000 |
| **MemcachedStore** | 38000 | 41000 | 40000 | 21000 |
| **FileStore** | 27000 | 66000 | 73000 | 8000 |
| | set | get | get(empty) | delete |
| ------------------ | ------- | ------- | ---------- | ------- |
| **MemoryStore** | 2096000 | 3023000 | 3171000 | 3453000 |
| **LevelDBStore** | 690000 | 518000 | 627000 | 360000 |
| **RedisStore** | 24000 | 30000 | 25000 | 13000 |
| **MemcachedStore** | 11000 | 10000 | 11000 | 5000 |
| **FileStore** | 80000 | 118000 | 117000 | 90000 |

Data information:
* Key size: 5-100 bytes.
Expand All @@ -132,6 +157,7 @@ make benchmark

Run specs for all stores:
```
# you must have docker-compose installed
make test
```

Expand All @@ -145,3 +171,4 @@ crystal spec ./spec/kiwi/file_store_spec.cr

- [greyblake](https://github.com/greyblake) Sergey Potapov - creator, maintainer.
- [mauricioabreu](https://github.com/mauricioabreu) Mauricio de Abreu Antunes - thanks for MemcachedStore.
- [akitaonrails](https://akitando.com) Fabio Akita - adding type checks and expiration feature
21 changes: 10 additions & 11 deletions benchmark.cr
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,10 @@ require "./src/kiwi/memcached_store"

N = 100_000

def benchmark(name)
start_time = Time.now
def benchmark(name, &)
start_time = Time.utc
yield
writing_time = Time.now - start_time
writing_time = Time.utc - start_time
speed = (N.to_f / writing_time.to_f)
rounded_speed = ((speed / 1000).round * 1000).to_i
puts " #{name}: #{rounded_speed} ops/sec"
Expand All @@ -33,7 +33,6 @@ end
def measure(stores)
puts "Genrating data..."
data = gen_data
empty_keys = (0..N).map { |i| "empty-key-#{i}" }

result = Hash(String, Hash(String, Int32)).new

Expand All @@ -51,19 +50,19 @@ def measure(stores)
end

store_metrics["get"] = benchmark("get") do
data.each do |key, val|
data.each do |key, _|
store.get(key)
end
end

store_metrics["get_empty"] = benchmark("get (empty)") do
data.each do |key, val|
data.each do |key, _|
store.get(key)
end
end

store_metrics["delete"] = benchmark("delete") do
data.each do |key, val|
data.each do |key, _|
store.delete(key)
end
end
Expand All @@ -76,11 +75,11 @@ def measure(stores)
end

def print_table(result)
store_col_size = result.keys.map(&.size).max
set_col_size = result.values.map { |metrics| metrics["set"].to_s.size }.max
get_col_size = result.values.map { |metrics| metrics["get"].to_s.size }.max
store_col_size = result.keys.max_of(&.size)
set_col_size = result.values.max_of(&.["set"].to_s.size)
get_col_size = result.values.max_of(&.["get"].to_s.size)
get_empty_col_size = "get(empty)".size
delete_col_size = result.values.map { |metrics| metrics["delete"].to_s.size }.max
delete_col_size = result.values.max_of(&.["delete"].to_s.size)

# header
puts "| " + " " * store_col_size +
Expand Down
16 changes: 16 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
version: '3'
services:
memcached:
image: memcached
ports:
- "11211:11211"

redis:
image: redis
ports:
- "6379:6379"

leveldb:
image: ekristen/leveldb
ports:
- "2012:2012"
7 changes: 6 additions & 1 deletion shard.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ version: 0.1.0

authors:
- Sergey Potapov <[email protected]>
- Fabio Akita <[email protected]>

license: MIT

Expand All @@ -12,7 +13,11 @@ development_dependencies:
branch: master
redis:
github: stefanwille/crystal-redis
version: ~> 1.8.0
version: ~> 2.9.1
leveldb:
github: crystal-community/leveldb
branch: master
timecop:
github: crystal-community/timecop.cr
ameba:
github: crystal-ameba/ameba
1 change: 1 addition & 0 deletions spec/kiwi/memcached_store_spec.cr
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,5 @@ require "../../src/kiwi/memcached_store"

describe Kiwi::MemcachedStore do
behaves_like_store Kiwi::MemcachedStore.new(Memcached::Client.new)
behaves_like_expiring_store(Kiwi::MemcachedStore.new(Memcached::Client.new, 1.second))
end
1 change: 1 addition & 0 deletions spec/kiwi/memory_store_spec.cr
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,5 @@ require "../../src/kiwi/memory_store"

describe Kiwi::MemoryStore do
behaves_like_store Kiwi::MemoryStore.new
behaves_like_expiring_store Kiwi::MemoryStore.new(1.seconds)
end
1 change: 1 addition & 0 deletions spec/kiwi/redis_store_spec.cr
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,5 @@ require "../../src/kiwi/redis_store"

describe Kiwi::RedisStore do
behaves_like_store Kiwi::RedisStore.new(Redis.new(database: 13))
behaves_like_expiring_store Kiwi::RedisStore.new(Redis.new(database: 13), 1.second)
end
51 changes: 40 additions & 11 deletions spec/spec_helper.cr
Original file line number Diff line number Diff line change
@@ -1,35 +1,64 @@
require "spec"
require "timecop"
require "../src/kiwi"

macro behaves_like_store(store_definition)
it "behaves like a store" do
store = {{store_definition}}

# set
store.set("key1", "value1").should(eq("value1"), "#set fails")
store.set("key2", "value2").should(eq("value2"), "#set fails")
store.set("key3", "value3").should(eq("value3"), "#set fails")
store.set("key1", "value1").should eq("value1")
store.set("key2", "value2").should eq("value2")
store.set("key3", "value3").should eq("value3")

# get
store.get("key1").should(eq("value1"), "#get fails")
store.get("key2").should(eq("value2"), "#get fails")
store.get("key3").should(eq("value3"), "#get fails")
store.get("none").should(eq(nil), "#get fails")
store.get("key1").should eq("value1")
store.get("key2").should eq("value2")
store.get("key3").should eq("value3")
store.get("none").should be_nil

# delete
store.delete("key1").should eq "value1"
store.get("key1").should eq nil
store.get("key1").should be_nil
store.get("key2").should eq "value2"
store.get("key3").should eq "value3"

# []= and [] aliases
store["key1"] = "abc"
store["key1"].should eq "abc"

# fetch
value = store.fetch("key9") do
"value9"
end
store["key9"].should eq "value9"
value.should eq "value9"

# clear
store.clear.should eq store
store.get("key1").should eq nil
store.get("key2").should eq nil
store.get("key3").should eq nil
store.get("key1").should be_nil
store.get("key2").should be_nil
store.get("key3").should be_nil
end
end

macro behaves_like_expiring_store(store_definition)
it "behaves like a store with expires capabilities" do
store = {{store_definition}}

store.set("key1", "value1")
store.get("key1").should eq("value1")
sleep 1.5 # sleep because Timecop won't affect real memcached/redis
Timecop.travel(Time.utc + 5.seconds) do
store.get("key1").should be_nil
end

store.fetch("key2") do
"value2"
end
sleep 1.5
Timecop.travel(Time.utc + 10.seconds) do
store["key2"].should be_nil
end
end
end
18 changes: 8 additions & 10 deletions src/kiwi/file_store.cr
Original file line number Diff line number Diff line change
Expand Up @@ -9,38 +9,36 @@ module Kiwi
create_dir
end

def get(key)
def get(key : String) : String?
file = file_for_key(key)
File.exists?(file) ? File.read(file) : nil
end

def set(key, val)
def set(key : String, val : String) : String
create_dir unless dir_created?

file = file_for_key(key)
File.write(file, val)
val
end

def delete(key)
def delete(key : String) : String?
create_dir unless dir_created?

file = file_for_key(key)
if File.exists?(file)
value = File.read(file)
return nil unless File.exists?(file)

File.read(file).tap do |_|
File.delete(file)
value
else
nil
end
end

def clear
def clear : Store
remove_dir
self
end

private def file_for_key(key)
private def file_for_key(key : String)
hex = Digest::SHA1.hexdigest(key)
File.join(@dir, hex)
end
Expand Down
27 changes: 20 additions & 7 deletions src/kiwi/leveldb_store.cr
Original file line number Diff line number Diff line change
Expand Up @@ -6,24 +6,37 @@ module Kiwi
def initialize(@leveldb : ::LevelDB::DB)
end

def set(key, val)
def set(key : String, val : String) : String
@leveldb.put(key, val)
val
end

def get(key)
def get(key : String) : String?
@leveldb.get(key)
end

def delete(key)
val = get(key)
@leveldb.delete(key)
val
def delete(key : String) : String?
get(key).tap do |val|
@leveldb.delete(key) if val
end
end

def clear
def clear : Store
@leveldb.clear
self
end

# exclusive LevelDB API
def create_snapshot : LevelDB::Snapshot
@leveldb.create_snapshot
end

def set_snapshot(snapshot : LevelDB::Snapshot)
@leveldb.set_snapshot(snapshot)
end

def unset_snapshot
@leveldb.unset_snapshot
end
end
end
Loading