Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: Read replica support #476

Merged
merged 27 commits into from
Jun 12, 2023
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
2ce89df
added variables for tracking time since last write
a-alhusaini May 5, 2023
dcaa7d2
added variables to store reader and writer db connections
a-alhusaini May 7, 2023
b4010f4
switched from time.utc to time.monotonic for tracking time since last…
a-alhusaini May 7, 2023
fcec217
Added methods to switch between connections.
a-alhusaini May 7, 2023
d6658ea
added instance variant of switch_to_writer_adapter for use with callb…
a-alhusaini May 7, 2023
b22e62b
Fixed typo in def switch_to_writer_adapter
a-alhusaini May 7, 2023
9bc3743
added logic to automatically switch to reader adapter
a-alhusaini May 7, 2023
438f04c
functions in querying now dynamically change adapter based on need
a-alhusaini May 7, 2023
d2fc1d0
ensure all methods in query builder that need primary database switch…
a-alhusaini May 7, 2023
c2eb32d
added better error message for invalid connections
a-alhusaini May 10, 2023
2bd49f1
groundwork for replica testing
a-alhusaini May 10, 2023
21878b2
fixed error where mysql tests don't run
a-alhusaini May 10, 2023
866e00a
Granite::Base.adapter class method now actually fetches current adapt…
a-alhusaini May 10, 2023
720005d
fixed typo. Invalid adapter_type for pg_with_replica
a-alhusaini May 10, 2023
f6cf749
fixed True to true. Added table name to ReplicatedChat.
a-alhusaini May 10, 2023
4e97c87
update specs
kalinon May 11, 2023
4f1ef67
Merge pull request #1 from kalinon/track_last_query
a-alhusaini May 11, 2023
431ed8e
spec updates
kalinon May 11, 2023
e531675
Merge pull request #2 from kalinon/track_last_query
a-alhusaini May 11, 2023
4976d11
Update .gitignore
crimson-knight May 15, 2023
6ab17b0
moved connection management logic to seperate module. Fixed bug where…
a-alhusaini May 20, 2023
74ff74c
fixed error where reader connection switch ignored specified wait per…
a-alhusaini May 21, 2023
8ecc010
moved default value for connection switch wait period to granite::co…
a-alhusaini May 21, 2023
76133f3
moved connection macro to connection management module
a-alhusaini May 21, 2023
e11cf7c
cleaned up code for fetching first connection
a-alhusaini May 21, 2023
058b414
finalized syntax for adding new connections to granite::connections
a-alhusaini May 21, 2023
122be6c
optimization: when reader & writer database are the same do not dupli…
a-alhusaini May 30, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .env
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
PG_DATABASE_URL=postgres://granite:password@localhost:5432/granite_db
MYSQL_DATABASE_URL=mysql://granite:password@localhost:3306/granite_db
SQLITE_DATABASE_URL=sqlite3:./granite.db
SQLITE_REPLICA_URL=sqlite3:./granite_replica.db
CURRENT_ADAPTER=pg
PG_VERSION=15.2
MYSQL_VERSION=5.7
Expand Down
14 changes: 10 additions & 4 deletions spec/adapter/adapters_spec.cr
Original file line number Diff line number Diff line change
Expand Up @@ -13,20 +13,26 @@ end
describe Granite::Connections do
describe "registration" do
it "should allow connections to be be saved and looked up" do
Granite::Connections.registered_connections.size.should eq 3
Granite::Connections.registered_connections.size.should eq 4

if connection = Granite::Connections["mysql"]
connection.url.should eq ENV["MYSQL_DATABASE_URL"]
connection[:writer].url.should eq ENV["MYSQL_DATABASE_URL"]
else
connection.should_not be_falsey
end
if connection = Granite::Connections["pg"]
connection.url.should eq ENV["PG_DATABASE_URL"]
connection[:writer].url.should eq ENV["PG_DATABASE_URL"]
else
connection.should_not be_falsey
end
if connection = Granite::Connections["sqlite"]
connection.url.should eq ENV["SQLITE_DATABASE_URL"]
connection[:writer].url.should eq ENV["SQLITE_DATABASE_URL"]
else
connection.should_not be_falsey
end
if connection = Granite::Connections["sqlite_replica"]
connection[:writer].url.should eq ENV["SQLITE_DATABASE_URL"]
connection[:reader].url.should eq ENV["SQLITE_REPLICA_URL"]
else
connection.should_not be_falsey
end
Expand Down
17 changes: 17 additions & 0 deletions spec/granite/connection_switching_spec.cr
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
require "spec"

describe "Granite::Base track time since last write" do
it "should ensure that time since last write to model is saved" do
# This is a good way to test last write time
# It is unlikely that the test suite takes more than an hour to run
(Chat.time_since_last_write < 1.hours).should be_true

# Update the time since hte last write and make sure it updated to the time within the currnet second
Chat.update_last_write_time

# Time since last write should be less than a second since we just updated it (see line above)
unless (Chat.time_since_last_write < 1.second)
crimson-knight marked this conversation as resolved.
Show resolved Hide resolved
raise "ERROR!!!!!"
end
end
end
4 changes: 4 additions & 0 deletions spec/spec_helper.cr
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,10 @@ Granite::Connections << Granite::Adapter::Mysql.new(name: "mysql", url: ENV["MYS
Granite::Connections << Granite::Adapter::Pg.new(name: "pg", url: ENV["PG_DATABASE_URL"])
Granite::Connections << Granite::Adapter::Sqlite.new(name: "sqlite", url: ENV["SQLITE_DATABASE_URL"])

# Connections with replicas
# TODO: Experiment to find a better API.
Granite::Connections.<<(name: "sqlite_replica", writer: ENV["SQLITE_DATABASE_URL"], reader: ENV["SQLITE_REPLICA_URL"], adapter_type: Granite::Adapter::Sqlite)

require "spec"
require "../src/granite"
require "../src/adapter/**"
Expand Down
38 changes: 38 additions & 0 deletions src/granite/base.cr
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,39 @@ abstract class Granite::Base
extend Integrators
extend Select

@@last_write_time = Time.monotonic

def self.last_write_time
@@last_write_time
end

# This is done this way because callbacks don't work on class mthods
def self.update_last_write_time
@@last_write_time = Time.monotonic
end

def update_last_write_time
self.class.update_last_write_time
end

def self.time_since_last_write
Time.monotonic - @@last_write_time
end

def time_since_last_write
self.class.time_since_last_write
end

def self.switch_to_reader_adapter
if time_since_last_write > 2.seconds
@@adapter = @@reader_adapter
end
end

def self.switch_to_writer_adapter
@@adapter = @@writer_adapter
end

macro inherited
protected class_getter select_container : Container = Container.new(table_name: table_name, fields: fields)

Expand Down Expand Up @@ -70,5 +103,10 @@ abstract class Granite::Base

disable_granite_docs? def initialize
end

crimson-knight marked this conversation as resolved.
Show resolved Hide resolved
after_save :update_last_write_time
after_update :update_last_write_time
after_create :update_last_write_time
after_destroy :update_last_write_time
end
end
17 changes: 12 additions & 5 deletions src/granite/connections.cr
Original file line number Diff line number Diff line change
@@ -1,16 +1,23 @@
module Granite
class Connections
class_getter registered_connections = [] of Granite::Adapter::Base
class_getter registered_connections = [] of {writer: Granite::Adapter::Base, reader: Granite::Adapter::Base}
crimson-knight marked this conversation as resolved.
Show resolved Hide resolved

# Registers the given *adapter*. Raises if an adapter with the same name has already been registered.
def self.<<(adapter : Granite::Adapter::Base) : Nil
raise "Adapter with name '#{adapter.name}' has already been registered." if @@registered_connections.any? { |conn| conn.name == adapter.name }
@@registered_connections << adapter
raise "Adapter with name '#{adapter.name}' has already been registered." if @@registered_connections.any? { |conn| conn[:writer].name == adapter.name }
@@registered_connections << {writer: adapter, reader: adapter}
end

# TODO: Find cleaner type restriction method
def self.<<(*, name : String, reader : String, writer : String, adapter_type : Granite::Adapter::Base.class) : Nil
reader_adapter = adapter_type.new(name: name, url: reader)
writer_adapter = adapter_type.new(name: name, url: writer)
@@registered_connections << {writer: writer_adapter, reader: reader_adapter}
end

# Returns a registered connection with the given *name*, otherwise `nil`.
def self.[](name : String) : Granite::Adapter::Base?
registered_connections.find { |conn| conn.name == name }
def self.[](name : String) : {writer: Granite::Adapter::Base, reader: Granite::Adapter::Base}?
registered_connections.find { |conn| conn[:writer].name == name }
end
end
end
7 changes: 5 additions & 2 deletions src/granite/table.cr
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ end
module Granite::Tables
module ClassMethods
def adapter : Granite::Adapter::Base
Granite::Connections.registered_connections.first? || raise "No connections have been registered."
Granite::Connections.registered_connections.first?.not_nil![:writer] || raise "No connections have been registered."
a-alhusaini marked this conversation as resolved.
Show resolved Hide resolved
end

def primary_name
Expand Down Expand Up @@ -57,6 +57,9 @@ module Granite::Tables

# specify the database connection you will be using for this model.
macro connection(name)
class_getter adapter : Granite::Adapter::Base = Granite::Connections[{{(name.is_a?(StringLiteral) ? name : name.id.stringify)}}] || raise "No registered connection with the name '{{name.id}}'"
class_getter writer_adapter : Granite::Adapter::Base = Granite::Connections[{{(name.is_a?(StringLiteral) ? name : name.id.stringify)}}].not_nil![:writer]
class_getter reader_adapter : Granite::Adapter::Base = Granite::Connections[{{(name.is_a?(StringLiteral) ? name : name.id.stringify)}}].not_nil![:reader]
class_getter adapter : Granite::Adapter::Base = @@writer_adapter

end
end