[`Improve`]: scan operations for `IReadOnlyStoreView` and fix of `Find` with all 0xff `keyPrefix` #3688

nan01ab · 2025-01-21T17:46:11Z

Description

Add scan operations for IReadOnlyStoreView
ScanPrefix and ScanRange are clearer than Find, and FindRange

A better way to fix #3681

Type of change

Optimization (the change is only an optimization)
Style (the change is only a code style for better maintenance or standard purpose)
[] Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

Checklist:

My code follows the style guidelines of this project
I have performed a self-review of my code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes
Any dependent changes have been merged and published in downstream modules

cschuchardt88

What is the purpose of the cache? What problems is classes trying to solved or has solved? I am to believe that the cache is suppose to be a faster lookup than from disk. If this is the case, I will fix all cache classes. These classes are really slow, compared to using leveldb, rocksdb or any other key/value store.

nan01ab · 2025-01-22T00:14:11Z

What is the purpose of the cache? What problems is classes trying to solved or has solved? I am to believe that the cache is suppose to be a faster lookup than from disk. If this is the case, I will fix all cache classes. These classes are really slow, compared to using leveldb, rocksdb or any other key/value store.

The performance is not the main purpose.
Other purposes:

Separate the readonly API as an interface. The Seek was omitted.
Nullable in Storage classes #3670, an issue about all 0xff keyPrefix discussed.
And [Fix]: keyPrefix in DataCache.Find can be all 0xff. #3681, I tried to fix it, but it changes the API behaviour, so there may be some incompatibilities with this fix.
If all 0xff prefix, you can use ScanPrefix instead now.
[Fix]: IStore.Seek should start from the last key if keyPrefix is null or empty #3680, I tried to fix an issue about Find(emptyKeyPrefix, SeekDirection.Backward), but it changes the API behavior too.
So more info has added to tell how to start scanning from the end with ScanPrefix and ScanRange.
Find, FindRange are scan operations. Names like ScanXXX are clearer. ScanXXX as extensions makes the interface simpler.

nan01ab · 2025-01-22T00:32:09Z

What is the purpose of the cache? What problems is classes trying to solved or has solved? I am to believe that the cache is suppose to be a faster lookup than from disk. If this is the case, I will fix all cache classes. These classes are really slow, compared to using leveldb, rocksdb or any other key/value store.

Which classes are readlly slow ?

src/Neo/Extensions/Persistence/ReadOnlyViewExtensions.cs

src/Neo/Persistence/ReadOnlyStoreView.cs

src/Neo/Persistence/DataCache.cs

src/Neo/Extensions/Persistence/ReadOnlyViewExtensions.cs

Co-authored-by: Christopher Schuchardt <[email protected]>

src/Neo/Extensions/Persistence/ReadOnlyViewExtensions.cs

Co-authored-by: Shargon <[email protected]>

cschuchardt88 · 2025-01-31T02:09:57Z

@nan01ab We still want #3688 (comment) updated.

@shargon hold until updated.

nan01ab · 2025-01-31T02:16:16Z

@nan01ab We still want #3688 (comment) updated.

@shargon hold until updated.

You can submit your suggestion.
But I suggest not to change the exception type, which is not necessary

cschuchardt88 · 2025-01-31T02:28:38Z

@nan01ab We still want #3688 (comment) updated.
@shargon hold until updated.

You can submit your suggestion. But I suggest not to change the exception type, which is not necessary

How is not necessary. The text is misleading. The exception is misleading. Tell why this is such a big deal to you?

What the text is telling people is, that right now you can't use all 0xff bytes for an array. So if I recall the method again in 5 minutes; it may work. English is my 1st and only language. So trust me. It makes neo look stupid and unprofessional.

Co-authored-by: Christopher Schuchardt <[email protected]>

nan01ab · 2025-01-31T03:50:33Z

@nan01ab We still want #3688 (comment) updated.
@shargon hold until updated.

You can submit your suggestion. But I suggest not to change the exception type, which is not necessary

How is not necessary. The text is misleading. The exception is misleading. Tell why this is such a big deal to you?

What the text is telling people is, that right now you can't use all 0xff bytes for an array. So if I recall the method again in 5 minutes; it may work. English is my 1st and only language. So trust me. It makes neo look stupid and unprofessional.

Great suggestion.
Some unit test issues after committing the suggestion.
I will fix later

Changes requests have been resolved.

cschuchardt88 · 2025-01-31T09:04:48Z

tests/Neo.UnitTests/Extensions/UT_ReadOnlyViewExtension.cs

@@ -112,7 +112,7 @@ public void TestFindEmptyPrefix()
            action.Should().Throw<ArgumentException>();

            action = () => dataCache.Find([0xff], SeekDirection.Backward).ToArray();
-            action.Should().Throw<ArgumentException>();
+            action.Should().Throw<NotSupportedException>();


Using [0xff] should work. The way leveldb and rocksdb all work is, it seeks to the key 0xff and moves backwards from there.

so results would be in this order:

1. 0xff 2. 0xfe 3. 0xfd 4. 0xfc 5. 0xfb 6. 0xfa 7. ...etc...

All you have to do is use ByteArrayComparer class. I dont understand what the problem is or why no one can change this. its only 2 lines of code.

cschuchardt88

Please stop supporting inconsistent code, that is not inline with leveldb or rocksdb functionality.

@Jim8y @shargon

This class is so old and outdated. So many problems that need to be addressed.

src/Neo/Extensions/Persistence/ReadOnlyViewExtensions.cs

cschuchardt88 · 2025-01-31T09:09:35Z

src/Neo/Extensions/Persistence/ReadOnlyViewExtensions.cs

+                : ByteArrayComparer.Reverse;
+            foreach (var (key, value) in view.Seek(inclusiveStartKey, direction))
+            {
+                if (comparer.Compare(key.ToArray(), exclusiveEndKey) < 0)


You want the the key. If the prefix is a key?

Suggested change

if (comparer.Compare(key.ToArray(), exclusiveEndKey) < 0)

if (comparer.Compare(key.ToArray(), exclusiveEndKey) <= 0)

You want the the key. If the prefix is a key?

exclusiveEndKey

because of exclusive.

Why go to all the trouble to edit or change this. If your not going to fix the issues with the class?

Why go to all the trouble to edit or change this. If your not going to fix the issues with the class?

Because this is ScanRange/FindRange, and range is [StartKey, EndKey)

You want the the key. If the prefix is a key?

start key and end key are not prefix.

Where are you getting this information? What is not to understand? IStore doesn't work this way. Find has bugs in DataCache.

https://github.com/neo-project/neo/blob/master/src/Neo/Persistence/DataCache.cs#L264

Like I said, DataCache has bugs.

LevelDBStore

neo/src/Plugins/LevelDBStore/IO/Data/LevelDB/Helper.cs

Lines 21 to 43 in 84a9fd5

public static IEnumerable<(byte[], byte[])> Seek(this DB db, ReadOptions options, byte[]? keyOrPrefix, SeekDirection direction)

{

keyOrPrefix ??= [];

using var it = db.CreateIterator(options);

if (direction == SeekDirection.Forward)

{

for (it.Seek(keyOrPrefix); it.Valid(); it.Next())

yield return new(it.Key()!, it.Value()!);

}

else

{

// SeekForPrev

it.Seek(keyOrPrefix);

if (!it.Valid())

it.SeekToLast();

else if (it.Key().AsSpan().SequenceCompareTo(keyOrPrefix) > 0)

it.Prev();

for (; it.Valid(); it.Prev())

yield return new(it.Key()!, it.Value()!);

}

}

RocksDbStore

neo/src/Plugins/RocksDBStore/Plugins/Storage/Store.cs

Lines 41 to 52 in 84a9fd5

public IEnumerable<(byte[] Key, byte[] Value)> Seek(byte[]? keyOrPrefix, SeekDirection direction = SeekDirection.Forward)

{

keyOrPrefix ??= [];

using var it = _db.NewIterator();

if (direction == SeekDirection.Forward)

for (it.Seek(keyOrPrefix); it.Valid(); it.Next())

yield return (it.Key(), it.Value());

else

for (it.SeekForPrev(keyOrPrefix); it.Valid(); it.Prev())

yield return (it.Key(), it.Value());

}

MemoryStore

neo/src/Neo/Persistence/MemoryStore.cs

Lines 51 to 63 in 84a9fd5

public IEnumerable<(byte[] Key, byte[] Value)> Seek(byte[]? keyOrPrefix, SeekDirection direction = SeekDirection.Forward)

{

keyOrPrefix ??= [];

if (direction == SeekDirection.Backward && keyOrPrefix.Length == 0) yield break;

var comparer = direction == SeekDirection.Forward ? ByteArrayComparer.Default : ByteArrayComparer.Reverse;

IEnumerable<KeyValuePair<byte[], byte[]>> records = _innerData;

if (keyOrPrefix.Length > 0)

records = records.Where(p => comparer.Compare(p.Key, keyOrPrefix) >= 0);

records = records.OrderBy(p => p.Key, comparer);

foreach (var pair in records)

yield return (pair.Key[..], pair.Value[..]);

}

Like I said, DataCache has bugs.

LevelDBStore

neo/src/Plugins/LevelDBStore/IO/Data/LevelDB/Helper.cs

Lines 21 to 43 in 84a9fd5

public static IEnumerable<(byte[], byte[])> Seek(this DB db, ReadOptions options, byte[]? keyOrPrefix, SeekDirection direction)

{

keyOrPrefix ??= [];

using var it = db.CreateIterator(options);

if (direction == SeekDirection.Forward)

{

for (it.Seek(keyOrPrefix); it.Valid(); it.Next())

yield return new(it.Key()!, it.Value()!);

}

else

{

// SeekForPrev

it.Seek(keyOrPrefix);

if (!it.Valid())

it.SeekToLast();

else if (it.Key().AsSpan().SequenceCompareTo(keyOrPrefix) > 0)

it.Prev();

for (; it.Valid(); it.Prev())

yield return new(it.Key()!, it.Value()!);

}

}

RocksDbStore

neo/src/Plugins/RocksDBStore/Plugins/Storage/Store.cs

Lines 41 to 52 in 84a9fd5

public IEnumerable<(byte[] Key, byte[] Value)> Seek(byte[]? keyOrPrefix, SeekDirection direction = SeekDirection.Forward)

{

keyOrPrefix ??= [];

using var it = _db.NewIterator();

if (direction == SeekDirection.Forward)

for (it.Seek(keyOrPrefix); it.Valid(); it.Next())

yield return (it.Key(), it.Value());

else

for (it.SeekForPrev(keyOrPrefix); it.Valid(); it.Prev())

yield return (it.Key(), it.Value());

}

MemoryStore

neo/src/Neo/Persistence/MemoryStore.cs

Lines 51 to 63 in 84a9fd5

public IEnumerable<(byte[] Key, byte[] Value)> Seek(byte[]? keyOrPrefix, SeekDirection direction = SeekDirection.Forward)

{

keyOrPrefix ??= [];

if (direction == SeekDirection.Backward && keyOrPrefix.Length == 0) yield break;

var comparer = direction == SeekDirection.Forward ? ByteArrayComparer.Default : ByteArrayComparer.Reverse;

IEnumerable<KeyValuePair<byte[], byte[]>> records = _innerData;

if (keyOrPrefix.Length > 0)

records = records.Where(p => comparer.Compare(p.Key, keyOrPrefix) >= 0);

records = records.OrderBy(p => p.Key, comparer);

foreach (var pair in records)

yield return (pair.Key[..], pair.Value[..]);

}

If DataCache has bugs, you can submit a PR to fix it.

This is not **Find**.

ScanRange / FindRange return items within range [StartKey(inclusive), EndKey(exclusive) )

prefix is misleading word.

it doesnt matter. In the stores the key are in order. If you convert all keys to an integer they start from zero through Infinity (or biggest key in the store). so if last key is [0xff, 0xff] and i use prefix 0xff the seeker goto 0xff key.

Example with prefix 0xff for keys ranging from [0x01] to [0xff, 0xff]

Records returned in order:

Key: [0xff] Key: [0xfe] Key: [0xfd] Key: [0xfc] Key: [0xfb] Key: [0xfa] .... etc ... Key: [0xdf] Key: [0xde] Key: [0xdd] Key: [0xdc] Key: [0xdb] Key: [0xda] .... etc ... Key: [0x01]

You can read this:

neo/src/Neo/Persistence/DataCache.cs

Lines 251 to 258 in 84a9fd5

/// <summary>

/// Finds the entries that between [start, end).

/// </summary>

/// <param name="start">The start key (inclusive).</param>

/// <param name="end">The end key (exclusive).</param>

/// <param name="direction">The search direction.</param>

/// <returns>The entries found with the desired range.</returns>

public IEnumerable<(StorageKey Key, StorageItem Value)> FindRange(byte[] start, byte[] end, SeekDirection direction = SeekDirection.Forward)

cschuchardt88 · 2025-01-31T09:15:39Z

src/Neo/Extensions/Persistence/ReadOnlyViewExtensions.cs

+        /// </exception>
+        internal static byte[] GetSeekPrefix(this byte[]? keyPrefix, ushort maxSizeWhenAll0xff = 4096 /* make it long enough */)
+        {
+            if (keyPrefix == null) // Backwards seek for null prefix is not supported for now.


This is inconsistent with MemoryStore, LevelDbStore and RocksDbStore. Why?

I tried to fix it. #3680

Fixed in #3701, but that PR isn't going to get merge. So, if you want you can create new PR and use my work.

Fixed in #3701, but that PR isn't going to get merge. So, if you want you can create new PR and use my work.

Yes, I tried to fix it.
But a requirement I received is to maintain consistent interface/api behavior.

who required this? Well don't listen them. As long as two developer approve a PR. It gets merged (with no issues and change requests).

cschuchardt88 · 2025-01-31T09:16:35Z

src/Neo/Extensions/Persistence/ReadOnlyViewExtensions.cs

+                if (maxSizeWhenAll0xff > 0)
+                    seekPrefix = ((byte)0xff).Repeat(maxSizeWhenAll0xff);
+                else
+                    throw new NotSupportedException("Array filled with max value (0xFF)", new ArgumentException(nameof(keyPrefix)));


Your already using comparer!!! What's the problem?

@shargon

Your already using comparer!!! What's the problem?

@shargon

If compatibility of interface behavior is not required, this exception can be removed.

Your already using comparer!!! What's the problem?

@shargon

Becuase keyPrefix is [0xff...] and direaction is backword.
The seekPrefix must greater than the last key in the store.

It all stores and caches must function the same way.

It all stores and caches must function the same way.

Which way?

The way all stores work is, they look for a key [0xff, 0xff, 0xff, 0xff, 0xff]'; for example, than returns that key if found, than everything from that point to the 1st key/record.

I know the key isn't all 0xff in this example. But if that key exists with all 0xff it would start there and go backwards.

Core Example:

neo/src/Plugins/ApplicationLogs/Store/LogStorageStore.cs

Lines 187 to 208 in 805678d

public IEnumerable<ContractLogState> FindContractState(UInt160 scriptHash, uint page, uint pageSize)

{

var prefix = new KeyBuilder(Prefix_Id, Prefix_Contract)

.Add(scriptHash)

.ToArray();

var prefixKey = new KeyBuilder(Prefix_Id, Prefix_Contract)

.Add(scriptHash)

.AddBigEndian(ulong.MaxValue) // Get newest to oldest (timestamp)

.ToArray();

uint index = 1;

foreach (var (key, value) in _snapshot.Seek(prefixKey, SeekDirection.Backward)) // Get newest to oldest

{

if (key.AsSpan().StartsWith(prefix))

{

if (index >= page && index < (pageSize + page))

yield return value.AsSerializable<ContractLogState>();

index++;

}

else

yield break;

}

}

Note: It took me some time to figure this out how backwards works. Think of it as an integer and you're doing currentRecordKey <= Int.MaxValue.

The way all stores work is, they look for a key [0xff, 0xff, 0xff, 0xff, 0xff]'; for example, than returns that key if found, than everything from that point to the 1st key/record.

I know the key isn't all 0xff in this example. But if that key exists with all 0xff it would start there and go backwards.

Core Example:

neo/src/Plugins/ApplicationLogs/Store/LogStorageStore.cs

Lines 187 to 208 in 805678d

public IEnumerable<ContractLogState> FindContractState(UInt160 scriptHash, uint page, uint pageSize)

{

var prefix = new KeyBuilder(Prefix_Id, Prefix_Contract)

.Add(scriptHash)

.ToArray();

var prefixKey = new KeyBuilder(Prefix_Id, Prefix_Contract)

.Add(scriptHash)

.AddBigEndian(ulong.MaxValue) // Get newest to oldest (timestamp)

.ToArray();

uint index = 1;

foreach (var (key, value) in _snapshot.Seek(prefixKey, SeekDirection.Backward)) // Get newest to oldest

{

if (key.AsSpan().StartsWith(prefix))

{

if (index >= page && index < (pageSize + page))

yield return value.AsSerializable<ContractLogState>();

index++;

}

else

yield break;

}

}

Note: It took me some time to figure this out how backwards works. Think of it as an integer and you're doing currentRecordKey < Int.MaxValue.

SeekDireaction.Bacworkd makes the logic more complicated.
You can read this function(and previous version) at first.

cschuchardt88

For now I'll approve. Because I'll be fixing these issue in future PR.

cschuchardt88 · 2025-01-31T13:19:03Z

src/Neo/Extensions/Persistence/ReadOnlyViewExtensions.cs

+            this IReadOnlyStoreView view,
+            byte[]? keyPrefix,
+            byte[]? seekPrefix,
+            SeekDirection direction = SeekDirection.Forward)


Just use comparer. Just forget about these direction things for a min. dotnet uses comparer why can't we? There are good reasons to use comparer instead of enums.

cschuchardt88 · 2025-01-31T13:20:28Z

src/Neo/Extensions/Persistence/ReadOnlyViewExtensions.cs

+                if (keyPrefix == null || key.ToArray().AsSpan().StartsWith(keyPrefix))
+                    yield return new(key, value);
+                else if (direction == SeekDirection.Forward || (seekPrefix == null || !key.ToArray().SequenceEqual(seekPrefix)))
+                    yield break;


Wouldn't have to do this if you use comparer. Just look at LevelDbStore and MemoryStore.

Wouldn't have to do this if you use comparer. Just look at LevelDbStore and MemoryStore.

These lines aren't new.
I moved these from DataCache.cs to here.
https://github.com/neo-project/neo/blob/master/src/Neo/Persistence/DataCache.cs#L242

Copying/Moving bugs doesn't make it ok.

Copying/Moving bugs doesn't make it ok.

This PR didn't change this.
You can submit a new PR to fix it.

cschuchardt88 · 2025-01-31T13:24:39Z

src/Neo/Extensions/Persistence/ReadOnlyViewExtensions.cs

+                : ByteArrayComparer.Reverse;
+            foreach (var (key, value) in view.Seek(inclusiveStartKey, direction))
+            {
+                if (comparer.Compare(key.ToArray(), exclusiveEndKey) < 0)


Where are you getting this information? What is not to understand? IStore doesn't work this way. Find has bugs in DataCache.

cschuchardt88 · 2025-01-31T13:28:01Z

if @shargon and @Jim8y is good with this PR can @superboyiii test?

shargon · 2025-02-01T18:29:46Z

if @shargon and @Jim8y is good with this PR can @superboyiii test?

I'm not against this pr, just need more time to review it well

nan01ab · 2025-02-02T07:16:29Z

if @shargon and @Jim8y is good with this PR can @superboyiii test?

I'm not against this pr, just need more time to review it well

Some problems already exist, which are not introduced by this PR.
There are two solutions to fix these issues:

Merge this PR, then fix these issues(on the changes of this PR).
Close this PR, then fix these issues, then resubmit this PR.

Jim8y · 2025-02-02T07:20:22Z

@cschuchardt88 for issues that are not related i suggest u to open another pr to fix, as to some english or minor issues maybe you can also fix in another pr, that will make ur ideas being applied faster.

cschuchardt88 · 2025-02-02T08:20:42Z

@cschuchardt88 for issues that are not related i suggest u to open another pr to fix, as to some english or minor issues maybe you can also fix in another pr, that will make ur ideas being applied faster.

I don't see the point of moving functions around. If you are to do that, at least fix the bugs in them. Put a little effort in, same with design. This isn't your own project, where its only used by you. Others have to support your copy and paste efforts. So some respect to code implementation would be nice. Why am I always the one to clean up everyones shit show?

nan01ab · 2025-02-02T09:14:41Z

@cschuchardt88 for issues that are not related i suggest u to open another pr to fix, as to some english or minor issues maybe you can also fix in another pr, that will make ur ideas being applied faster.

I don't see the point of moving functions around. If you are to do that, at least fix the bugs in them. Put a little effort in, same with design. This isn't your own project, where its only used by you. Others have to support your copy and paste efforts. So some respect to code implementation would be nice. Why am I always the one to clean up everyones shit show?

I have explained the purpose of this pr in the first reply.
A PR should be review-able(don’t change too many things).
The changes introduced by this pr are fully compatible with the previous interface behavior. So this pr does not introduce changes that may be incompatible.
Whether/how to fix these issues still needs to be discussed.

cschuchardt88 · 2025-02-02T09:35:19Z

I have explained the purpose of this pr in the first reply.

A PR should be review-able(don’t change too many things).

The changes introduced by this pr are fully compatible with the previous interface behavior. So this pr does not introduce changes that may be incompatible.

Whether/how to fix these issues still needs to be discussed.

Ill answer in order:

You said Add scan operations for IReadOnlyStoreView ScanPrefix and ScanRange are clearer than Find, and FindRange. So your changing the name of Find and FindRange. But in fact that's NOT what happened.
See [Add] Framework Design Guidelines #3708 if you want to put your two cent in.
That shouldn't be an excuse to not fix bugs. Bugs are High priority! Interface shouldn't change unless bugs and design have been worked out 1st.
We don't need this PR than. Until we discuss the change and problem you are trying to solved. Everything small adds up to something BIG. If it's shit people will throw it out (current state of neo).

improve: scan operations for ReadOnlyStoreView

3b7cd98

nan01ab mentioned this pull request Jan 21, 2025

[Fix]: keyPrefix in DataCache.Find can be all 0xff. #3681

Closed

14 tasks

nan01ab changed the title ~~[Improve]: scan operations for IReadOnlyStoreView~~ [Improve]: scan operations for IReadOnlyStoreView and fix of Find with all 0xff keyPrefix Jan 21, 2025

cschuchardt88 reviewed Jan 21, 2025

View reviewed changes

improve: scan operations for ReadOnlyStoreView

6ccb8ca

Improve: add more comments

5628012

nan01ab added the Waiting for Review label Jan 22, 2025

shargon reviewed Jan 22, 2025

View reviewed changes

src/Neo/Extensions/Persistence/ReadOnlyViewExtensions.cs Outdated Show resolved Hide resolved

src/Neo/Extensions/Persistence/ReadOnlyViewExtensions.cs Outdated Show resolved Hide resolved

shargon added 2 commits January 22, 2025 00:32

Update src/Neo/Extensions/Persistence/ReadOnlyViewExtensions.cs

a81b8f6

Merge branch 'master' into feat.scan-readonly-storeview

16c9ef2

cschuchardt88 previously requested changes Jan 22, 2025

View reviewed changes

src/Neo/Extensions/Persistence/ReadOnlyViewExtensions.cs Outdated Show resolved Hide resolved

nan01ab added 3 commits January 22, 2025 22:02

Improve: use ArrayExtensions.Repeat instead

86a8d75

Improve: use ArrayExtensions.Repeat instead

77fb843

Fix code format and add more comments

7477f87

cschuchardt88 reviewed Jan 22, 2025

View reviewed changes

Merge branch 'master' into feat.scan-readonly-storeview

1bb8715

Jim8y previously approved these changes Jan 23, 2025

View reviewed changes

Jim8y mentioned this pull request Jan 23, 2025

[Jan.23] Core Dev Regular Meeting Proposal Discussion #3690

Closed

Merge branch 'master' into feat.scan-readonly-storeview

0c7767a

shargon previously approved these changes Jan 23, 2025

View reviewed changes

Update src/Neo/Persistence/ReadOnlyStoreView.cs

3dde742

Co-authored-by: Christopher Schuchardt <[email protected]>

nan01ab dismissed stale reviews from shargon and Jim8y via 3dde742 January 23, 2025 11:16

nan01ab and others added 2 commits January 23, 2025 19:16

Update src/Neo/Extensions/Persistence/ReadOnlyViewExtensions.cs

746b2ad

Co-authored-by: Christopher Schuchardt <[email protected]>

Update src/Neo/Extensions/Persistence/ReadOnlyViewExtensions.cs

c06a38d

Co-authored-by: Christopher Schuchardt <[email protected]>

shargon reviewed Jan 24, 2025

View reviewed changes

src/Neo/Extensions/Persistence/ReadOnlyViewExtensions.cs Outdated Show resolved Hide resolved

Update src/Neo/Extensions/Persistence/ReadOnlyViewExtensions.cs

1eb7e1e

Co-authored-by: Shargon <[email protected]>

shargon previously approved these changes Jan 24, 2025

View reviewed changes

Merge branch 'master' into feat.scan-readonly-storeview

976dd32

Update src/Neo/Extensions/Persistence/ReadOnlyViewExtensions.cs

5beabbc

Co-authored-by: Christopher Schuchardt <[email protected]>

nan01ab dismissed stale reviews from shargon and Jim8y via 5beabbc January 31, 2025 03:31

cschuchardt88 self-requested a review January 31, 2025 05:04

Fix: unit test issue

4d7f58a

cschuchardt88 reviewed Jan 31, 2025

View reviewed changes

cschuchardt88 requested changes Jan 31, 2025

View reviewed changes

cschuchardt88 previously approved these changes Jan 31, 2025

View reviewed changes

cschuchardt88 self-assigned this Jan 31, 2025

cschuchardt88 requested review from shargon and Jim8y January 31, 2025 10:40

cschuchardt88 reviewed Jan 31, 2025

View reviewed changes

Jim8y and others added 3 commits February 1, 2025 09:52

Merge branch 'master' into feat.scan-readonly-storeview

46c28a8

fix: merge conflict

a417172

Merge branch 'master' into feat.scan-readonly-storeview

f959ca4

nan01ab dismissed cschuchardt88’s stale review via f959ca4 February 1, 2025 09:10

nan01ab closed this Feb 2, 2025

	if (comparer.Compare(key.ToArray(), exclusiveEndKey) < 0)
	if (comparer.Compare(key.ToArray(), exclusiveEndKey) <= 0)

	public static IEnumerable<(byte[], byte[])> Seek(this DB db, ReadOptions options, byte[]? keyOrPrefix, SeekDirection direction)
	{
	keyOrPrefix ??= [];

	using var it = db.CreateIterator(options);
	if (direction == SeekDirection.Forward)
	{
	for (it.Seek(keyOrPrefix); it.Valid(); it.Next())
	yield return new(it.Key()!, it.Value()!);
	}
	else
	{
	// SeekForPrev
	it.Seek(keyOrPrefix);
	if (!it.Valid())
	it.SeekToLast();
	else if (it.Key().AsSpan().SequenceCompareTo(keyOrPrefix) > 0)
	it.Prev();

	for (; it.Valid(); it.Prev())
	yield return new(it.Key()!, it.Value()!);
	}
	}

	public IEnumerable<(byte[] Key, byte[] Value)> Seek(byte[]? keyOrPrefix, SeekDirection direction = SeekDirection.Forward)
	{
	keyOrPrefix ??= [];

	using var it = _db.NewIterator();
	if (direction == SeekDirection.Forward)
	for (it.Seek(keyOrPrefix); it.Valid(); it.Next())
	yield return (it.Key(), it.Value());
	else
	for (it.SeekForPrev(keyOrPrefix); it.Valid(); it.Prev())
	yield return (it.Key(), it.Value());
	}

	public IEnumerable<(byte[] Key, byte[] Value)> Seek(byte[]? keyOrPrefix, SeekDirection direction = SeekDirection.Forward)
	{
	keyOrPrefix ??= [];
	if (direction == SeekDirection.Backward && keyOrPrefix.Length == 0) yield break;

	var comparer = direction == SeekDirection.Forward ? ByteArrayComparer.Default : ByteArrayComparer.Reverse;
	IEnumerable<KeyValuePair<byte[], byte[]>> records = _innerData;
	if (keyOrPrefix.Length > 0)
	records = records.Where(p => comparer.Compare(p.Key, keyOrPrefix) >= 0);
	records = records.OrderBy(p => p.Key, comparer);
	foreach (var pair in records)
	yield return (pair.Key[..], pair.Value[..]);
	}

	/// <summary>
	/// Finds the entries that between [start, end).
	/// </summary>
	/// <param name="start">The start key (inclusive).</param>
	/// <param name="end">The end key (exclusive).</param>
	/// <param name="direction">The search direction.</param>
	/// <returns>The entries found with the desired range.</returns>
	public IEnumerable<(StorageKey Key, StorageItem Value)> FindRange(byte[] start, byte[] end, SeekDirection direction = SeekDirection.Forward)

	public IEnumerable<ContractLogState> FindContractState(UInt160 scriptHash, uint page, uint pageSize)
	{
	var prefix = new KeyBuilder(Prefix_Id, Prefix_Contract)
	.Add(scriptHash)
	.ToArray();
	var prefixKey = new KeyBuilder(Prefix_Id, Prefix_Contract)
	.Add(scriptHash)
	.AddBigEndian(ulong.MaxValue) // Get newest to oldest (timestamp)
	.ToArray();
	uint index = 1;
	foreach (var (key, value) in _snapshot.Seek(prefixKey, SeekDirection.Backward)) // Get newest to oldest
	{
	if (key.AsSpan().StartsWith(prefix))
	{
	if (index >= page && index < (pageSize + page))
	yield return value.AsSerializable<ContractLogState>();
	index++;
	}
	else
	yield break;
	}
	}

[Improve]: scan operations for IReadOnlyStoreView and fix of Find with all 0xff keyPrefix #3688

[Improve]: scan operations for IReadOnlyStoreView and fix of Find with all 0xff keyPrefix #3688

Conversation

nan01ab commented Jan 21, 2025

Description

Type of change

Checklist:

cschuchardt88 left a comment • edited Loading

Choose a reason for hiding this comment

nan01ab commented Jan 22, 2025 • edited Loading

nan01ab commented Jan 22, 2025

cschuchardt88 commented Jan 31, 2025

nan01ab commented Jan 31, 2025 • edited Loading

cschuchardt88 commented Jan 31, 2025

nan01ab commented Jan 31, 2025 • edited Loading

cschuchardt88 Jan 31, 2025 • edited Loading

Choose a reason for hiding this comment

cschuchardt88 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nan01ab Jan 31, 2025 • edited Loading

Choose a reason for hiding this comment

cschuchardt88 Jan 31, 2025 • edited Loading

Choose a reason for hiding this comment

nan01ab Jan 31, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LevelDBStore

RocksDbStore

MemoryStore

Choose a reason for hiding this comment

LevelDBStore

RocksDbStore

MemoryStore

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nan01ab Jan 31, 2025 • edited Loading

Choose a reason for hiding this comment

nan01ab Jan 31, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cschuchardt88 Jan 31, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cschuchardt88 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nan01ab Feb 1, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cschuchardt88 commented Jan 31, 2025

shargon commented Feb 1, 2025

nan01ab commented Feb 2, 2025

Jim8y commented Feb 2, 2025

cschuchardt88 commented Feb 2, 2025 • edited Loading

nan01ab commented Feb 2, 2025

cschuchardt88 commented Feb 2, 2025 • edited Loading

Ill answer in order:

[`Improve`]: scan operations for `IReadOnlyStoreView` and fix of `Find` with all 0xff `keyPrefix` #3688

[`Improve`]: scan operations for `IReadOnlyStoreView` and fix of `Find` with all 0xff `keyPrefix` #3688

cschuchardt88 left a comment •

edited

Loading

nan01ab commented Jan 22, 2025 •

edited

Loading

nan01ab commented Jan 31, 2025 •

edited

Loading

nan01ab commented Jan 31, 2025 •

edited

Loading

cschuchardt88 Jan 31, 2025 •

edited

Loading

nan01ab Jan 31, 2025 •

edited

Loading

cschuchardt88 Jan 31, 2025 •

edited

Loading

nan01ab Jan 31, 2025 •

edited

Loading

nan01ab Jan 31, 2025 •

edited

Loading

nan01ab Jan 31, 2025 •

edited

Loading

cschuchardt88 Jan 31, 2025 •

edited

Loading

nan01ab Feb 1, 2025 •

edited

Loading

cschuchardt88 commented Feb 2, 2025 •

edited

Loading

cschuchardt88 commented Feb 2, 2025 •

edited

Loading