diff --git a/HISTORY.md b/HISTORY.md index 058f326c618..feffb34b4cb 100644 --- a/HISTORY.md +++ b/HISTORY.md @@ -1,6 +1,43 @@ # Rocksdb Change Log > NOTE: Entries for next release do not go here. Follow instructions in `unreleased_history/README.txt` +## 9.2.0 (05/01/2024) +### New Features +* Added two options `deadline` and `max_size_bytes` for CacheDumper to exit early +* Added a new API `GetEntityFromBatchAndDB` to `WriteBatchWithIndex` that can be used for wide-column point lookups with read-your-own-writes consistency. Similarly to `GetFromBatchAndDB`, the API can combine data from the write batch with data from the underlying database if needed. See the API comments for more details. +* [Experimental] Introduce two new cross-column-family iterators - CoalescingIterator and AttributeGroupIterator. The CoalescingIterator enables users to iterate over multiple column families and access their values and columns. During this iteration, if the same key exists in more than one column family, the keys in the later column family will overshadow the previous ones. The AttributeGroupIterator allows users to gather wide columns per Column Family and create attribute groups while iterating over keys across all CFs. +* Added a new API `MultiGetEntityFromBatchAndDB` to `WriteBatchWithIndex` that can be used for batched wide-column point lookups with read-your-own-writes consistency. Similarly to `MultiGetFromBatchAndDB`, the API can combine data from the write batch with data from the underlying database if needed. See the API comments for more details. +* *Adds a `SstFileReader::NewTableIterator` API to support programmatically read a SST file as a raw table file. +* Add an option to `WaitForCompactOptions` - `wait_for_purge` to make `WaitForCompact()` API wait for background purge to complete + +### Public API Changes +* DeleteRange() will return NotSupported() if row_cache is configured since they don't work together in some cases. +* Deprecated `CompactionOptions::compression` since `CompactionOptions`'s API for configuring compression was incomplete, unsafe, and likely unnecessary +* Using `OptionChangeMigration()` to migrate from non-FIFO to FIFO compaction +with `Options::compaction_options_fifo.max_table_files_size` > 0 can cause +the whole DB to be dropped right after migration if the migrated data is larger than +`max_table_files_size` + +### Behavior Changes +* Enabling `BlockBasedTableOptions::block_align` is now incompatible (i.e., APIs will return `Status::InvalidArgument`) with more ways of enabling compression: `CompactionOptions::compression`, `ColumnFamilyOptions::compression_per_level`, and `ColumnFamilyOptions::bottommost_compression`. +* Changed the default value of `CompactionOptions::compression` to `kDisableCompressionOption`, which means the compression type is determined by the `ColumnFamilyOptions`. +* `BlockBasedTableOptions::optimize_filters_for_memory` is now set to true by default. When `partition_filters=false`, this could lead to somewhat increased average RSS memory usage by the block cache, but this "extra" usage is within the allowed memory budget and should make memory usage more consistent (by minimizing internal fragmentation for more kinds of blocks). +* Dump all keys for cache dumper impl if `SetDumpFilter()` is not called +* `CompactRange()` with `CompactRangeOptions::change_level = true` and `CompactRangeOptions::target_level = 0` that ends up moving more than 1 file from non-L0 to L0 will return `Status::Aborted()`. +* On distributed file systems that support file system level checksum verification and reconstruction reads, RocksDB will now retry a file read if the initial read fails RocksDB block level or record level checksum verification. This applies to MANIFEST file reads when the DB is opened, and to SST file reads at all times. + +### Bug Fixes +* Fix a bug causing `VerifyFileChecksums()` to return false-positive corruption under `BlockBasedTableOptions::block_align=true` +* Provide consistent view of the database across the column families for `NewIterators()` API. +* Fixed feature interaction bug for `DeleteRange()` together with `ColumnFamilyOptions::memtable_insert_with_hint_prefix_extractor`. The impact of this bug would likely be corruption or crashing. +* Fixed hang in `DisableManualCompactions()` where compactions waiting to be scheduled due to conflicts would not be canceled promptly +* Fixed a regression when `ColumnFamilyOptions::max_successive_merges > 0` where the CPU overhead for deciding whether to merge could have increased unless the user had set the option `ColumnFamilyOptions::strict_max_successive_merges` +* Fixed a bug in `MultiGet()` and `MultiGetEntity()` together with blob files (`ColumnFamilyOptions::enable_blob_files == true`). An error looking up one of the keys could cause the results to be wrong for other keys for which the statuses were `Status::OK`. +* Fixed a bug where wrong padded bytes are used to generate file checksum and `DataVerificationInfo::checksum` upon file creation +* Correctly implemented the move semantics of `PinnableWideColumns`. +* Fixed a bug when the recycle_log_file_num in DBOptions is changed from 0 to non-zero when a DB is reopened. On a subsequent reopen, if a log file created when recycle_log_file_num==0 was reused previously, is alive and is empty, we could end up inserting stale WAL records into the memtable. +* *Fix a bug where obsolete files' deletion during DB::Open are not rate limited with `SstFilemManager`'s slow deletion feature even if it's configured. + ## 9.1.0 (03/22/2024) ### New Features * Added an option, `GetMergeOperandsOptions::continue_cb`, to give users the ability to end `GetMergeOperands()`'s lookup process before all merge operands were found. diff --git a/include/rocksdb/version.h b/include/rocksdb/version.h index 1842690b26e..24506703fbc 100644 --- a/include/rocksdb/version.h +++ b/include/rocksdb/version.h @@ -12,7 +12,7 @@ // NOTE: in 'main' development branch, this should be the *next* // minor or major version number planned for release. #define ROCKSDB_MAJOR 9 -#define ROCKSDB_MINOR 2 +#define ROCKSDB_MINOR 3 #define ROCKSDB_PATCH 0 // Do not use these. We made the mistake of declaring macros starting with diff --git a/unreleased_history/behavior_changes/block_align_compression_incompat.md b/unreleased_history/behavior_changes/block_align_compression_incompat.md deleted file mode 100644 index 7068a42679a..00000000000 --- a/unreleased_history/behavior_changes/block_align_compression_incompat.md +++ /dev/null @@ -1 +0,0 @@ -* Enabling `BlockBasedTableOptions::block_align` is now incompatible (i.e., APIs will return `Status::InvalidArgument`) with more ways of enabling compression: `CompactionOptions::compression`, `ColumnFamilyOptions::compression_per_level`, and `ColumnFamilyOptions::bottommost_compression`. diff --git a/unreleased_history/behavior_changes/default_compaction_options_compression.md b/unreleased_history/behavior_changes/default_compaction_options_compression.md deleted file mode 100644 index 2e4e87ff181..00000000000 --- a/unreleased_history/behavior_changes/default_compaction_options_compression.md +++ /dev/null @@ -1 +0,0 @@ -* Changed the default value of `CompactionOptions::compression` to `kDisableCompressionOption`, which means the compression type is determined by the `ColumnFamilyOptions`. diff --git a/unreleased_history/behavior_changes/default_optimize_filters_for_memory.md b/unreleased_history/behavior_changes/default_optimize_filters_for_memory.md deleted file mode 100644 index 0cd00120ae1..00000000000 --- a/unreleased_history/behavior_changes/default_optimize_filters_for_memory.md +++ /dev/null @@ -1 +0,0 @@ -`BlockBasedTableOptions::optimize_filters_for_memory` is now set to true by default. When `partition_filters=false`, this could lead to somewhat increased average RSS memory usage by the block cache, but this "extra" usage is within the allowed memory budget and should make memory usage more consistent (by minimizing internal fragmentation for more kinds of blocks). diff --git a/unreleased_history/behavior_changes/dump_all_keys.md b/unreleased_history/behavior_changes/dump_all_keys.md deleted file mode 100644 index 913cc044235..00000000000 --- a/unreleased_history/behavior_changes/dump_all_keys.md +++ /dev/null @@ -1 +0,0 @@ -Dump all keys for cache dumper impl if `SetDumpFilter()` is not called diff --git a/unreleased_history/behavior_changes/refit_to_l0.md b/unreleased_history/behavior_changes/refit_to_l0.md deleted file mode 100644 index 1c9813cace8..00000000000 --- a/unreleased_history/behavior_changes/refit_to_l0.md +++ /dev/null @@ -1 +0,0 @@ -`CompactRange()` with `CompactRangeOptions::change_level = true` and `CompactRangeOptions::target_level = 0` that ends up moving more than 1 file from non-L0 to L0 will return `Status::Aborted()`. diff --git a/unreleased_history/behavior_changes/retry_on_corruption.md b/unreleased_history/behavior_changes/retry_on_corruption.md deleted file mode 100644 index fe5a385b187..00000000000 --- a/unreleased_history/behavior_changes/retry_on_corruption.md +++ /dev/null @@ -1 +0,0 @@ -On distributed file systems that support file system level checksum verification and reconstruction reads, RocksDB will now retry a file read if the initial read fails RocksDB block level or record level checksum verification. This applies to MANIFEST file reads when the DB is opened, and to SST file reads at all times. diff --git a/unreleased_history/bug_fixes/block_align_checksum_mismatch.md b/unreleased_history/bug_fixes/block_align_checksum_mismatch.md deleted file mode 100644 index b784e32c5c8..00000000000 --- a/unreleased_history/bug_fixes/block_align_checksum_mismatch.md +++ /dev/null @@ -1 +0,0 @@ -Fix a bug causing `VerifyFileChecksums()` to return false-positive corruption under `BlockBasedTableOptions::block_align=true` diff --git a/unreleased_history/bug_fixes/consistent_view_for_new_iterators_api.md b/unreleased_history/bug_fixes/consistent_view_for_new_iterators_api.md deleted file mode 100644 index f537e474aad..00000000000 --- a/unreleased_history/bug_fixes/consistent_view_for_new_iterators_api.md +++ /dev/null @@ -1 +0,0 @@ -Provide consistent view of the database across the column families for `NewIterators()` API. diff --git a/unreleased_history/bug_fixes/del_range_and_memtable_insert_with_hint_prefix_extractor.md b/unreleased_history/bug_fixes/del_range_and_memtable_insert_with_hint_prefix_extractor.md deleted file mode 100644 index 7283fb95ac9..00000000000 --- a/unreleased_history/bug_fixes/del_range_and_memtable_insert_with_hint_prefix_extractor.md +++ /dev/null @@ -1 +0,0 @@ -* Fixed feature interaction bug for `DeleteRange()` together with `ColumnFamilyOptions::memtable_insert_with_hint_prefix_extractor`. The impact of this bug would likely be corruption or crashing. diff --git a/unreleased_history/bug_fixes/disable_manual_compaction_hang.md b/unreleased_history/bug_fixes/disable_manual_compaction_hang.md deleted file mode 100644 index c140874a1f4..00000000000 --- a/unreleased_history/bug_fixes/disable_manual_compaction_hang.md +++ /dev/null @@ -1 +0,0 @@ -* Fixed hang in `DisableManualCompactions()` where compactions waiting to be scheduled due to conflicts would not be canceled promptly diff --git a/unreleased_history/bug_fixes/max_successive_merges_regression.md b/unreleased_history/bug_fixes/max_successive_merges_regression.md deleted file mode 100644 index d00028014bb..00000000000 --- a/unreleased_history/bug_fixes/max_successive_merges_regression.md +++ /dev/null @@ -1 +0,0 @@ -* Fixed a regression when `ColumnFamilyOptions::max_successive_merges > 0` where the CPU overhead for deciding whether to merge could have increased unless the user had set the option `ColumnFamilyOptions::strict_max_successive_merges` diff --git a/unreleased_history/bug_fixes/multiget_partial_error_blob_dereference.md b/unreleased_history/bug_fixes/multiget_partial_error_blob_dereference.md deleted file mode 100644 index dfe5e7386b8..00000000000 --- a/unreleased_history/bug_fixes/multiget_partial_error_blob_dereference.md +++ /dev/null @@ -1 +0,0 @@ -* Fixed a bug in `MultiGet()` and `MultiGetEntity()` together with blob files (`ColumnFamilyOptions::enable_blob_files == true`). An error looking up one of the keys could cause the results to be wrong for other keys for which the statuses were `Status::OK`. diff --git a/unreleased_history/bug_fixes/pad_bug.md b/unreleased_history/bug_fixes/pad_bug.md deleted file mode 100644 index ef896585100..00000000000 --- a/unreleased_history/bug_fixes/pad_bug.md +++ /dev/null @@ -1 +0,0 @@ -Fixed a bug where wrong padded bytes are used to generate file checksum and `DataVerificationInfo::checksum` upon file creation diff --git a/unreleased_history/bug_fixes/pinnable_wide_columns_move.md b/unreleased_history/bug_fixes/pinnable_wide_columns_move.md deleted file mode 100644 index 8dceb277d3b..00000000000 --- a/unreleased_history/bug_fixes/pinnable_wide_columns_move.md +++ /dev/null @@ -1 +0,0 @@ -Correctly implemented the move semantics of `PinnableWideColumns`. diff --git a/unreleased_history/bug_fixes/recycle_logs_toggle_bug.md b/unreleased_history/bug_fixes/recycle_logs_toggle_bug.md deleted file mode 100644 index 2cfd8c46660..00000000000 --- a/unreleased_history/bug_fixes/recycle_logs_toggle_bug.md +++ /dev/null @@ -1 +0,0 @@ -Fixed a bug when the recycle_log_file_num in DBOptions is changed from 0 to non-zero when a DB is reopened. On a subsequent reopen, if a log file created when recycle_log_file_num==0 was reused previously, is alive and is empty, we could end up inserting stale WAL records into the memtable. diff --git a/unreleased_history/bug_fixes/slow_deletion_on_open.md b/unreleased_history/bug_fixes/slow_deletion_on_open.md deleted file mode 100644 index 338f1e6c4ff..00000000000 --- a/unreleased_history/bug_fixes/slow_deletion_on_open.md +++ /dev/null @@ -1 +0,0 @@ -*Fix a bug where obsolete files' deletion during DB::Open are not rate limited with `SstFilemManager`'s slow deletion feature even if it's configured. \ No newline at end of file diff --git a/unreleased_history/new_features/cache_dumper.md b/unreleased_history/new_features/cache_dumper.md deleted file mode 100644 index 9df7267feae..00000000000 --- a/unreleased_history/new_features/cache_dumper.md +++ /dev/null @@ -1 +0,0 @@ -Added two options `deadline` and `max_size_bytes` for CacheDumper to exit early diff --git a/unreleased_history/new_features/get_entity_from_batch_and_db.md b/unreleased_history/new_features/get_entity_from_batch_and_db.md deleted file mode 100644 index 34c583512ac..00000000000 --- a/unreleased_history/new_features/get_entity_from_batch_and_db.md +++ /dev/null @@ -1 +0,0 @@ -Added a new API `GetEntityFromBatchAndDB` to `WriteBatchWithIndex` that can be used for wide-column point lookups with read-your-own-writes consistency. Similarly to `GetFromBatchAndDB`, the API can combine data from the write batch with data from the underlying database if needed. See the API comments for more details. diff --git a/unreleased_history/new_features/multi_cf_iterators.md b/unreleased_history/new_features/multi_cf_iterators.md deleted file mode 100644 index 1305d1501d3..00000000000 --- a/unreleased_history/new_features/multi_cf_iterators.md +++ /dev/null @@ -1 +0,0 @@ -[Experimental] Introduce two new cross-column-family iterators - CoalescingIterator and AttributeGroupIterator. The CoalescingIterator enables users to iterate over multiple column families and access their values and columns. During this iteration, if the same key exists in more than one column family, the keys in the later column family will overshadow the previous ones. The AttributeGroupIterator allows users to gather wide columns per Column Family and create attribute groups while iterating over keys across all CFs. diff --git a/unreleased_history/new_features/multi_get_entity_from_batch_and_db.md b/unreleased_history/new_features/multi_get_entity_from_batch_and_db.md deleted file mode 100644 index 95efd40e3be..00000000000 --- a/unreleased_history/new_features/multi_get_entity_from_batch_and_db.md +++ /dev/null @@ -1 +0,0 @@ -Added a new API `MultiGetEntityFromBatchAndDB` to `WriteBatchWithIndex` that can be used for batched wide-column point lookups with read-your-own-writes consistency. Similarly to `MultiGetFromBatchAndDB`, the API can combine data from the write batch with data from the underlying database if needed. See the API comments for more details. diff --git a/unreleased_history/new_features/sst_file_reader_raw_table_iterator.md b/unreleased_history/new_features/sst_file_reader_raw_table_iterator.md deleted file mode 100644 index d3f26615981..00000000000 --- a/unreleased_history/new_features/sst_file_reader_raw_table_iterator.md +++ /dev/null @@ -1 +0,0 @@ -*Adds a `SstFileReader::NewTableIterator` API to support programmatically read a SST file as a raw table file. \ No newline at end of file diff --git a/unreleased_history/new_features/wait_for_purge_option.md b/unreleased_history/new_features/wait_for_purge_option.md deleted file mode 100644 index 518e022373c..00000000000 --- a/unreleased_history/new_features/wait_for_purge_option.md +++ /dev/null @@ -1,2 +0,0 @@ -Add an option to `WaitForCompactOptions` - `wait_for_purge` to make `WaitForCompact()` API wait for background purge to complete - diff --git a/unreleased_history/public_api_changes/delete_range_rowcache.md b/unreleased_history/public_api_changes/delete_range_rowcache.md deleted file mode 100644 index a2d66d3728c..00000000000 --- a/unreleased_history/public_api_changes/delete_range_rowcache.md +++ /dev/null @@ -1 +0,0 @@ -* DeleteRange() will return NotSupported() if row_cache is configured since they don't work together in some cases. \ No newline at end of file diff --git a/unreleased_history/public_api_changes/deprecate_compaction_options_compression.md b/unreleased_history/public_api_changes/deprecate_compaction_options_compression.md deleted file mode 100644 index 65dea565171..00000000000 --- a/unreleased_history/public_api_changes/deprecate_compaction_options_compression.md +++ /dev/null @@ -1 +0,0 @@ -* Deprecated `CompactionOptions::compression` since `CompactionOptions`'s API for configuring compression was incomplete, unsafe, and likely unnecessary diff --git a/unreleased_history/public_api_changes/migrate_fifo.md b/unreleased_history/public_api_changes/migrate_fifo.md deleted file mode 100644 index 93f0b44678c..00000000000 --- a/unreleased_history/public_api_changes/migrate_fifo.md +++ /dev/null @@ -1,4 +0,0 @@ -Using `OptionChangeMigration()` to migrate from non-FIFO to FIFO compaction -with `Options::compaction_options_fifo.max_table_files_size` > 0 can cause -the whole DB to be dropped right after migration if the migrated data is larger than -`max_table_files_size`