Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Export document to jsonFile one by one #131

Open
wants to merge 6 commits into
base: v2
Choose a base branch
from
Open

Export document to jsonFile one by one #131

Changes from 3 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
1b905e6
write document one by one
letterbeezps Jun 20, 2023
517fd8c
add err check
letterbeezps Jun 20, 2023
2bb9d36
add close
letterbeezps Jun 20, 2023
fe14b77
export json : catch errors inside the function
letterbeezps Jun 21, 2023
6cd0197
fix conflict
letterbeezps Mar 14, 2024
fa7578e
capture export internal error
letterbeezps Mar 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 27 additions & 9 deletions json.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@ package clover
import (
"bufio"
"encoding/json"
"io/ioutil"
"os"

d "github.com/ostafen/clover/v2/document"
Expand All @@ -19,23 +18,42 @@ func (db *DB) ExportCollection(collectionName string, exportPath string) error {
if !exists {
return ErrCollectionNotExist
}

result, err := db.FindAll(query.NewQuery(collectionName))
q := query.NewQuery(collectionName)
collectionCount, err := db.Count(q)
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would just call this variable count, for simplicity

if err != nil {
return err
}

docs := make([]map[string]interface{}, 0)
for _, doc := range result {
docs = append(docs, doc.AsMap())
f, err := os.Create(exportPath)
if err != nil {
return err
}
defer f.Close()
if _, err = f.WriteString("["); err != nil {
return err
}

jsonString, err := json.Marshal(docs)
n := 0
err = db.ForEach(q, func(doc *d.Document) bool {
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Count() and the ForEach() statements run in separate transactions. Thus, if new documents are inserted between them, the ForEach() statement will iterate a number of documents n > collectionCount, which would lead to some problem with the statement:

if n != collectionCount {
	jsonString += ","
}

To prevent this issue, we can ensure the number of documents iterated by the ForEach() is exactly collectionCount using Limit():

err = db.ForEach(q.Limit(collectionCount), func(doc *d.Document) bool {

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You remind me that is can also cause problems when documents are deleted between them.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a suggestion for ForEach() function, let consumer() return error instead of bool. If there is any failure inside this function, we can get the error info directly

func (db *DB) ForEach(q *query.Query, consumer func(_ *d.Document) error) error {
	q, err := normalizeCriteria(q)
	if err != nil {
		return err
	}

	return db.IterateDocs(q, func(doc *d.Document) error {
		if err := consumer(doc); err != nil {
			return err
		}
		return nil
	})
}

n++
jsonByte, err := json.Marshal(doc.AsMap())
if err != nil {
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be a good idea to recover this error in the outer scope.
If there is any failure inside this function, then the export process would not report an error, but the result would be clearly not correct.

return false
}
jsonString := string(jsonByte)
if n != collectionCount {
jsonString += ","
}
_, err = f.WriteString(jsonString)
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess same thing for this error

return err == nil
})
if err != nil {
return err
}

return ioutil.WriteFile(exportPath, jsonString, os.ModePerm)
if _, err = f.WriteString("]"); err != nil {
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When possible, like in this case, I prefer using the := operator

return err
}
return nil
}

// ImportCollection imports a collection from a JSON file.
Expand Down