Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement faster insertion #17

Draft
wants to merge 3 commits into
base: master
Choose a base branch
from
Draft

Implement faster insertion #17

wants to merge 3 commits into from

Conversation

ire4ever1190
Copy link
Owner

@ire4ever1190 ire4ever1190 commented May 22, 2023

Implements faster insertion of items by reusing the prepared statement instead of recreating it for each item. From benchmarks it looks to be roughly 4x faster. I'm also using arrays instead seqs for storing DbValue which gives a minor performance boost in the benchmark (6ms -> 5ms)

Benchmark code

import benchy
import src/ponairi {.all.}
import strformat
import std/random

type
  Person = object
    name: string
    age: int
    alive: bool

  Item = object
    id {.primary.}: int
    name: string
    something: int

# I am uncreative with names
const names = ["Jake", "James", "John", "Jack"]

proc oldInsert*[T: SomeTable](db: DbConn, items: openArray[T]) =
  ## Inserts the list of items into the database.
  ## This gets ran in a transaction so if an error happens then none
  ## of the items are saved to the database
  db.transaction:
    for item in items:
      db.insert item

proc oldUpsert[T: SomeTable](db: DbConn, items: openArray[T], exclude: static[openArray[string]] = []) =
  db.transaction:
    for item in items:
      db.upsertImpl(item, exclude)

proc main() =
  var
    people: seq[Person]
    items: seq[Item]
  for i in 0..<10000:
    people &= Person(name: sample(names), age: rand(1..100), alive: rand(bool))
    items &= Item(id: i, name: sample(names), something: rand(1..1000))

  let db = newConn(":memory:")

  template recreate() =
    db.drop(Person)
    db.create(Person)
  recreate()

  timeIt "Old insertion":
    db.oldInsert(people)

  recreate()

  timeIt "New insertion":
    db.insert(people)

  recreate()
  db.create(Item)
  db.insert(items)

  timeIt "Old upsert":
    db.oldUpsert(items)

  timeIt "New upsert":
    db.upsert(items)


  close db
main()

nim r -d:release bench.nim

   min time    avg time  std dv   runs name
  22.913 ms   24.901 ms  ±1.870   x200 Old insertion
   4.646 ms    5.367 ms  ±0.724   x914 New insertion
  42.152 ms   45.104 ms  ±2.409   x111 Old upsert
   5.740 ms    5.843 ms  ±0.110   x849 New upsert

Todo

  • Implement for normal inserting
  • Implement for upsert

Use array instead of seq for storing parameters which means we only use required memory

This gives a small performance boost

Will also look into applying this to `queryWithWhere` so that delete and exists reuse the same array code
Remove unused functions

Don't run tests with {.all.} import so that the state of the tests matches how people normally would use ponairi
@ire4ever1190
Copy link
Owner Author

Will merge when lowdb pushes a new release (So that I am not depending on devel)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant