Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide a streaming result set #64

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions relate/src/main/scala/com/lucidchart/relate/SqlQuery.scala
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,22 @@ trait Sql {
*/
def results()(implicit connection: Connection): ResultSet = normalStatement.results()

/**
* Provides a java.sql.ResultSet that streams records from the database.
* This allows for interacting with large data sets with less risk of OutOfMemoryErrors.
* Many JDBC connectors will not allow for additional queries to the connection until the
* returned ResultSet has been closed.
* @param fetchSize the number of rows to fetch at a time, defaults to 100. If the JDBC Driver
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where does this default come from? It seems like the fetchSize is required here.

* is MySQL, the fetchSize will always default to Int.MinValue, as MySQL's JDBC implementation
* ignores all other fetchSize values and only streams if fetchSize is Int.MinValue
* @param connection the db connection to use when executing the query
* @return java.sql.ResultSet that streams data from the database
*/
def streamingResults(fetchSize: Int)(implicit connection: Connection): ResultSet = {
val prepared = streamedStatement(fetchSize)
prepared.results()
}

/**
* Execute a statement
* @param connection the db connection to use when executing the query
Expand Down
17 changes: 17 additions & 0 deletions relate/src/test/scala/RelateITSpec.scala
Original file line number Diff line number Diff line change
Expand Up @@ -465,6 +465,23 @@ class RelateITSpec extends Specification with Db {
}
}

"results" should {
val ids = Array(1L, 2L, 3L)
"work from ResultSet" in withConnection { implicit connection =>
val resultSet = sql"SELECT * FROM pokedex WHERE id in ($ids)".results()
val pokemonNames = SqlResult(resultSet).asList(pokedexParser).map(_.name)

(pokemonNames must contain("Squirtle")) and (pokemonNames must contain("Wartortle")) and (pokemonNames must contain("Blastoise"))
}

"work from streaming ResultSet" in withConnection { implicit connection =>
val resultSet = sql"SELECT * FROM pokedex WHERE id in ($ids)".streamingResults(1)
val pokemonNames = SqlResult(resultSet).asList(pokedexParser).map(_.name)

(pokemonNames must contain("Squirtle")) and (pokemonNames must contain("Wartortle")) and (pokemonNames must contain("Blastoise"))
}
}

"update" should {
"update matched rows and not update unmatched rows" in withConnection { implicit connection =>
val correct = List(
Expand Down