Release v0.4.2

What's Changed

We recommend all users currently using version v0.4 to upgrade to this version, as there is a serious bug in the query filter pushdown for memtable.

Highlights

Main changes:

• Row protocol support for OpenTSDB,
• The implementation of HistogramFold plan for Prometheus histogram type.
• There are also important bug fixes, such as the correction of range behavior in MemoryKvBackend & RaftEngineBackend, and the query predicates use the wrong schema to create physical expressions. #2642
• Additionally, we've made several updates to our CI/CD processes for better efficiency and communication.

Features

• Get the internal value size of ValueRef by @QuenKar. #2613
• Row protocol support for opentsdb by @Lilit0x. #2623
• Implement HistogramFold plan for Prometheus histogram type by @waynexia. #2626
• RepeatedTask adds execute-first-wait-later behavior by @paomian. #2625
• Ports InMemoryRowGroup from parquet crate by @evenyag. #2633
• Implement ObjectStoreManager for custom_storage by @NiwakaDev. #2621
• Hide internal error and unknown error message from end user by @sunng87. #2544
• Initial OTLP trace support by @yuanbohan. #2627
• Lookup manifest file size by @QuenKar. #2590
• Create_or_open always set writable by @QuenKar. #2641

Bug Fixes

• Correct the range behavior in MemoryKvBackend & RaftEngineBackend by @WenyXu. #2615
• Pass datanode config file in distributed mode sqlness by @waynexia. #2631
• Predicate shall use real schema to create physical exprs by @v0y4g3r. #2642

Others

• The 'publish-github-release' and 'release-cn-artifacts' have to wait for all the artifacts are built by @zyy17. #2622
• Remove the old version python by @zyy17. #2624
• Add slack notification for nightly ci failure by @sunng87. #2617
• Add size labeler by @waynexia. #2628
• Upgrade 0.4.2 by @daviderli614. #2644

Full Changelog: v0.4.1...v0.4.2

Release v0.5.0-nightly-20231023

What's Changed

• feat: get internal value size of ValueRef by @QuenKar in #2613
• ci: the 'publish-github-release' and 'release-cn-artifacts' have to wait for all the artifacts are built by @zyy17 in #2622
• ci: remove the old version python by @zyy17 in #2624
• ci: add slack notification for nightly ci failure by @sunng87 in #2617
• fix: correct the range behavior in MemoryKvBackend & RaftEngineBackend by @WenyXu in #2615
• feat: row protocol support for opentsdb by @Lilit0x in #2623
• ci: add size labeler by @waynexia in #2628
• feat: implement HistogramFold plan for prometheus histogram type by @waynexia in #2626
• feat: RepeatedTask adds execute-first-wait-later behavior. by @paomian in #2625
• fix: pass datanode config file in distributed mode sqlness by @waynexia in #2631

Full Changelog: v0.4.1...v0.5.0-nightly-20231023

Release v0.4.1

We recommend all users currently using version v0.4 to upgrade to v0.4.2 or above, as there is a serious bug in the query filter pushdown for memtable.

What's Changed

Highlights

• The steps of building an Android binary and the release of Windows artifacts
• Allow nest range expr in Range Query, it's really a useful feature #2557
• The implementation of greptime cli export.
• There have also been significant fixes including the correction to the readme document link, the removal of unused options, and the addition of 'build-android-artifacts'.
• Other changes include refinements to the releasing process of dev-builder images and the separation of the job of building these images.

Features

• Add the steps of building android binary by @zyy17 in #2567
• Add new action 'release-cn-artifacts' by @zyy17 in #2554
• Add more options for releasing dev-builder images by @zyy17 in #2573
• Release Windows artifacts by @MichaelScofield in #2574
• Change config name from kv_store to metadata_store by @waynexia in #2585
• Implement greptime cli export by @waynexia in #2535
• Add walconfig dir back by @Shoothzj in #2606
• Allow nest range expr in Range Query by @Taylor-lagrange in #2557

Fixes

• Fix to readme document link by @daviderli614 in #2566
• Remove unused options and add 'build-android-artifacts' by @zyy17 in #2572
• Cache capacity unit in sample config by @killme2008 in #2575
• Fix skopeo running errors by @zyy17 in #2581
• Ensure data_home directory created by @killme2008 in #2588
• Convert to ReadableSize & Durations by @masonyc in #2594
• Make nyc-taxi bench work again by @evenyag in #2599
• Fix auth in standalone mode by @shuiyisong in #2591
• Align frontend cmd name to rpc_* by @masonyc in #2609

Others

• Separate the job of building dev-builder images by @zyy17 in #2569
• Refine the process of releasing dev-builder images by @zyy17 in #2580
• Run windows tests every night instead of every commit by @sunng87 in #2577
• Downgrade builder version: ubuntu 22.04 -> ubuntu 20.04 for compatibility with older version glibc(>=2.31) by @zyy17 in #2592
• Pin linux runner to ubuntu-20.04 by @waynexia in #2586
• Update 0.4.0 tsbs result by @waynexia in #2597
• Refine release-cn-artifacts action by @zyy17 in #2600
• Correct image name of dev build by @zyy17 in #2603
• Deregister cli attach command by @waynexia in #2589
• Add export data to migrate tool by @shuiyisong in #2610
• Release new version '0.4.0' -> '0.4.1' by @zyy17 in #2611
• Stop the procedure manager if a new leader is elected by @WenyXu in #2576
• Make ReadableSize more readable by @QuenKar in #2614
• Change telemetry report URL to resolve connectivity issues by @paomian in #2608
• Change test_remove_outdated_meta_task sleep time to 40ms by @WenyXu in #2620

New Contributors

• @Shoothzj made their first contribution in #2606

Full Changelog: v0.4.0...v0.4.1

Release v0.5.0-nightly-20231016

What's Changed

• ci: add more options for releasing dev-builder images by @zyy17 in #2573
• fix: cache capacity unit in sample config by @killme2008 in #2575
• ci: refine the process of releasing dev-builder images by @zyy17 in #2580
• ci: run windows tests every night instead of every commit by @sunng87 in #2577
• ci: release Windows artifacts by @MichaelScofield in #2574
• ci: fix skopeo running errors by @zyy17 in #2581
• feat!: change config name from kv_store to metadata_store by @waynexia in #2585
• fix: ensure data_home directory created by @killme2008 in #2588
• chore(cli): deregister cli attach command by @waynexia in #2589
• ci: downgrade builder version: ubuntu 22.04 -> ubuntu 20.04 for compatible with older version glibc(>=2.31) by @zyy17 in #2592
• build(ci): pin linux runner to ubuntu-20.04 by @waynexia in #2586
• docs(benchmark): update 0.4.0 tsbs result by @waynexia in #2597
• fix!: convert to ReadableSize & Durations by @masonyc in #2594
• fix: make nyc-taxi bench work again by @evenyag in #2599
• fix: auth in standalone mode by @shuiyisong in #2591
• ci: refine release-cn-artifacts action by @zyy17 in #2600

Full Changelog: v0.5.0-nightly-20231011...v0.5.0-nightly-20231016

Release v0.5.0-nightly-20231013

What's Changed

• ci: add more options for releasing dev-builder images by @zyy17 in #2573
• fix: cache capacity unit in sample config by @killme2008 in #2575
• ci: refine the process of releasing dev-builder images by @zyy17 in #2580
• ci: run windows tests every night instead of every commit by @sunng87 in #2577
• ci: release Windows artifacts by @MichaelScofield in #2574
• ci: fix skopeo running errors by @zyy17 in #2581
• feat!: change config name from kv_store to metadata_store by @waynexia in #2585
• fix: ensure data_home directory created by @killme2008 in #2588
• chore(cli): deregister cli attach command by @waynexia in #2589
• ci: downgrade builder version: ubuntu 22.04 -> ubuntu 20.04 for compatible with older version glibc(>=2.31) by @zyy17 in #2592
• build(ci): pin linux runner to ubuntu-20.04 by @waynexia in #2586
• docs(benchmark): update 0.4.0 tsbs result by @waynexia in #2597
• fix!: convert to ReadableSize & Durations by @masonyc in #2594
• fix: make nyc-taxi bench work again by @evenyag in #2599
• fix: auth in standalone mode by @shuiyisong in #2591
• ci: refine release-cn-artifacts action by @zyy17 in #2600

Full Changelog: v0.5.0-nightly-20231011...v0.5.0-nightly-20231013

Release v0.5.0-nightly-20231011

What's Changed

• chore: add the steps of building android binary by @zyy17 in #2567
• fix: fix to readme document link by @daviderli614 in #2566
• ci: seperate the job of building dev-builder images by @zyy17 in #2569
• refactor: add new action 'release-cn-artifacts' by @zyy17 in #2554
• fix: remove unused options and add 'build-android-artifacts' by @zyy17 in #2572

Full Changelog: v0.4.0...v0.5.0-nightly-20231011

Release v0.4.0

We recommend all users currently using version v0.4 to upgrade to v0.4.2 or above, as there is a serious bug in the query filter pushdown for memtable.

What's Changed

Highlights

• Query and Storage Performance
  • Introduced the completely redesigned engine, Mito2, specifically tailored for time-series data, boasting a performance boost of 5x~20x compared to v0.3.
  • Inspired by Cassandra, we've incorporated the TWCS (Time Window Compaction Strategy) compaction method.
  • Support new object storage: Google Cloud Storage.
• Stability and Reliability
  • Expanded our 'sqlness' test scenarios.
  • Based on our improved Chaos Testing cases, v0.4 boasts enhanced distributed robustness.
  • Leveraged the Procedure framework to support DDL operations like Create, Alter, and Drop Table, harmonizing both standalone and distributed implementations.
  • Reconstructed the entire distributed architecture, optimized the heartbeat protocol, making the distributed version run more effective and stable.
• User Experience Improvements
  • Launched an entirely new Dashboard.
  • Enhanced range select grammar, now supporting nested use with regular functions.
  • Refined the DESC TABLE user experience.
  • Support external tables in the Apache ORC format.
  • Support for the gRPC protocol in both row and col format.
• Integration and Others
  • Support GreptimeDB metrics output in Vector.
  • Facilitated the initiation of GreptimeDB clusters using KubeBlocks.
  • Integrated with EMQX.
  • Builds for CentOS, Windows, and Android.
  • Released official client SDKs for JS, Erlang, and C++.

Feature

• feat: support gcs storage by @NiwakaDev in #1781
• feat: support where in show by @NiwakaDev in #1829
• feat: implement drop table procedure by @WenyXu in #1872
• feat!: rename WITH parameter ENDPOINT_URL to ENDPOINT by @WenyXu in #1904
• feat: add deregister_schema to CatalogManager by @NiwakaDev in #1911
• feat: switch to using create table procedure by @WenyXu in #1861
• feat: dedup rows while flushing memtables by @evenyag in #1916
• feat(mito2): Define basic structs for MitoEngine by @evenyag in #1928
• feat: add logical plan based prepare statement for postgresql by @sunng87 in #1813
• feat(config-endpoint): add initial implementation by @etolbakov in #1896
• feat: manual compact api by @v0y4g3r in #1912
• feat: meta procedure options by @WenyXu in #1937
• feat: switch to using drop table procedure by @WenyXu in #1901
• feat: support desc [table] <table_name> by @waynexia in #1944
• feat: implement truncate table parser by @DevilExileSu in #1932
• feat:add check port before start sqlness test by @Gump9 in #1895
• feat: implement alter table procedure by @WenyXu in #1878
• feat: simplify the usage of channel_manage by @fengjiachun in #1949
• feat: invalidate table cache after altering by @WenyXu in #1951
• feat: ask leader by @fengjiachun in #1957
• feat(mito): Implement WorkerGroup to handle requests by @evenyag in #1950
• feat: support append entries from multiple regions at a time by @v0y4g3r in #1959
• feat: update dashboard to v0.3.0 by @ZonaHex in #1968
• feat: expose metasrv datanode_client_options by @WenyXu in #1965
• feat: tool to migrate table metadata values by @MichaelScofield in #1971
• feat: impl time type by @killme2008 in #1961
• feat: support where in show databases by @NiwakaDev in #1962
• feat: initial implementation for OpenTelemetry otlp/http by @sunng87 in #1974
• feat: add timers for promql query by @sunng87 in #1994
• feat(mito): Define Version and metadata builders for mito2 by @evenyag in #1989
• feat: status_code in response header by @fengjiachun in #1982
• feat: truncate table execute by @DevilExileSu in #2002
• feat: Add more tags for OTLP metrics protocol by @sunng87 in #2003
• feat: benchmark table metadata managers by @MichaelScofield in #2008
• feat: add orc stream by @gobraves in #1981
• feat(mito): create region in mito2 engine by @evenyag in #1999
• feat: (upload binary s3)add nightly build tag by @daviderli614 in #2011
• feat: region lease improve by @fengjiachun in #2004
• feat: trace_id in query context by @shuiyisong in #2014
• feat(mito): Port parquet writer and reader to mito2 by @evenyag in #2018
• feat: set and retrieve trace id in log macro by @waynexia in #2016
• feat: accept influxdb request without timestamp even if table doesn't exist by @NiwakaDev in #2041
• feat: impl interval type by @QuenKar in #1952
• feat: rewrite the release pipeline to make it clean by @zyy17 in #2038
• feat(mito): Implement open for RegionManifestManager by @evenyag in #2036
• feat: prepare for implementing considering partition key in the distributed planner by @waynexia in #2000
• feat(mito): Implement open and close for mito2 regions by @evenyag in #2052
• feat: make the gRPC channel between Frontend and Datanode configurable by @sunray-ley in #2044
• feat: add Catalog and Schema Manager by @WenyXu in #2037
• feat: necessary Hash derive for types by @discord9 in #2075
• feat: impl pubsub in metasrv by @fengys1996 in #2045
• feat: improve /label and /labels APIs in prometheus server by @waynexia in #2087
• feat(mito): preparation to implementing write by @evenyag in #2085
• feat: define region group and sequence by @waynexia in #2100
• feat(mito): Implement mito2 Wal by @evenyag in #2103
• feat: truncate region by @DevilExileSu in #2097
• feat: conversion between interval and gRPC by @QuenKar in #2064
• feat: add schema and catalog key migration tool by @WenyXu in #2048
• feat(information_schema): implement table_factory method by @zhongzc in #2108
• feat: return metasrv leader addr by @WenyXu in #2110
• feat: truncate table in standalone mode by @DevilExileSu in #2090
• feat: use server inferenced types on statement describe by @sunng87 in #2032
• feat: Add write method to memtable trait by @evenyag in #2123
• feat: Support RangeSelect LogicalPlan rewrite by @Taylor-lagrange in #2058
• feat: print build output if it's failed in sqlness by @MichaelScofield in #2152
• feat: impl duration type by @QuenKar in #2117
• feat: compatible with postgres interval type by @QuenKar in #2146
• feat!: enable telemetry by default by @killme2008 in #2137
• feat(mito): Write wal and memtable by @evenyag in #2135
• feat(mito): checkpoint for mito2 by @waynexia in #2142
• feat!: switch to new catalog/schema key by @WenyXu in #2140
• feat: add table route manager and upgrade tool by @WenyXu in https://github.com/GreptimeTeam/greptim...

Release v0.4.0-nightly-20231002

What's Changed

• chore: add integration test for issue2437 by @NiwakaDev in #2481
• fix: print root cause error message to user facing interface by @MichaelScofield in #2486
• refactor(mito): remove #[allow(dead_code)] by @evenyag in #2479
• refactor: datetime time unit by @QuenKar in #2469
• fix: dn doesn't have chance to send a heartbeat to the new leader by @WenyXu in #2471
• feat: re-support query engine execute dml by @zhongzc in #2484
• fix: convert datetime to chrono datetime by @zhongzc in #2497
• feat: distribute truncate table in region server by @DevilExileSu in #2414
• feat: stack trace style debug print for error by @waynexia in #2489
• feat(mito): Add cache manager by @evenyag in #2488
• fix: check for table scan before expanding by @waynexia in #2491
• fix(mito): fix region drop task runs multiple times but never clean the dir by @evenyag in #2504
• refactor: simplify warn! and error! macros by @waynexia in #2503
• fix: re-create heartbeat stream ASAP by @WenyXu in #2499
• chore: tune return msg by @shuiyisong in #2506
• fix: allow .(dot) literal in table name by @MichaelScofield in #2483
• feat: push down filters to region engine by @zhongzc in #2513
• feat: defensively specify limit parameter for file stream by @zhongzc in #2517
• fix: drop table 0 rows affected by @fengjiachun in #2515
• feat: pushdown aggr, limit and sort plan by @waynexia in #2495
• docs: add event banner by @Wenjie0329 in #2518
• feat: support greatest function by @NiwakaDev in #2490
• fix: missing file engine with default options by @zhongzc in #2519

New Contributors

• @Wenjie0329 made their first contribution in #2518

Full Changelog: v0.4.0-nightly-20230925...v0.4.0-nightly-20231002

Release v0.4.0-nightly-20230925

What's Changed

• test: revert ignored tests by @killme2008 in #2455
• refactor: remove source and location in snafu display by @shuiyisong in #2428
• refactor: unify table metadata cache invalidator by @WenyXu in #2449
• fix: add slash after generated region_dir by @waynexia in #2463
• ci: expand upload retry timeout by @zyy17 in #2461
• fix: test on windows by @fengjiachun in #2462
• chore: bump opendal to 0.40 by @WenyXu in #2465
• refactor: remove SqlStatementExecutor by @waynexia in #2464
• feat: limit grpc message size by @TheWaWaR in #2459
• chore: remove unused region_stats method form table by @fengjiachun in #2458
• feat: support for show full tables by @Lilit0x in #2410
• feat: update proto and remove create_if_not_exists by @evenyag in #2467
• fix(mito): compaction scheduler schedules more tasks than expected by @evenyag in #2466
• refactor: not allowed int64 type as time index by @QuenKar in #2460
• chore: typo by @fengjiachun in #2470
• feat: implement storage for OTLP histogram by @sunng87 in #2282
• fix(mito): reset is_sorted to true after the merger finishing one series by @evenyag in #2474
• fix(mito): normalize region dir in RegionOpener by @evenyag in #2475
• chore: refactor options by @fengjiachun in #2476
• chore: fix typo by @zyy17 in #2477

New Contributors

• @TheWaWaR made their first contribution in #2459

Full Changelog: v0.4.0-nightly-20230920...v0.4.0-nightly-20230925

Release v0.4.0-nightly-20230920

What's Changed

• feat: prometheus row inserter by @fengjiachun in #2263
• fix: keep region failover state not changed upon failure by @MichaelScofield in #2261
• feat(mito): Skeleton for scanning a region by @evenyag in #2230
• feat: impl region engine for mito by @waynexia in #2269
• fix: fix typoes by @WenyXu in #2268
• feat(mito2): WAL replay by @v0y4g3r in #2264
• feat: avoid confusion in desc table by @fengjiachun in #2272
• fix: open region does not register catalog/schema by @WenyXu in #2271
• feat: create distributed Mito2 table by @MichaelScofield in #2246
• fix: use RegionId region number instead by @WenyXu in #2273
• refactor(table): remove Table impl for system by @zhongzc in #2270
• feat(mito): Flush framework for mito2 by @evenyag in #2262
• feat(mito2): add file purger and cooperate with scheduler to purge sst files by @Nateiru in #2251
• ci: upload the latest artifacts to 'latest/' directory of S3 bucket in scheduled and formal release by @zyy17 in #2276
• fix: deregister table after keeper closes table by @WenyXu in #2278
• fix: skip procedure if target route is not found by @WenyXu in #2277
• feat: schema level opts by @shuiyisong in #2283
• fix: json compatibility to null by @shuiyisong in #2287
• feat: update dashboard to v0.3.2 by @ZonaHex in #2295
• fix: qualify inputs on handling join in promql by @waynexia in #2297
• feat: sqlness test for interval type by @QuenKar in #2265
• feat: update dashboard to v0.3.3 by @ZonaHex in #2339
• fix: panic in try_into_vector() by @evenyag in #2351
• refactor: collecting memory usage during scan by @shuiyisong in #2353
• feat: migrate region server and mito2 engine by @waynexia in #2364
• feat: convert sql number to values with target type by @waynexia in #2370
• feat: use the latest command line options for sqlness runner by @waynexia in #2371
• fix: gRPC max mesage size limitation by @killme2008 in #2375
• chore: change error messages by @killme2008 in #2379
• fix: update sqlness result in distributed mode by @waynexia in #2381
• feat: engine name in heartbeat by @fengjiachun in #2377
• feat: atomic metadata by @fengjiachun in #2366
• fix: make test-integration able to compile by @MichaelScofield in #2384
• refactor: migrate OpenDal to 0.39 by @WenyXu in #2383
• fix: querying temporary table by @waynexia in #2387
• fix: alter table procedure forgets to update next column id by @evenyag in #2385
• feat(mito): Add writable flag to region by @evenyag in #2349
• fix(mito): incorrect field index in ProjectionMapper by @evenyag in #2388
• fix: update sqlness result for order_by by @waynexia in #2389
• chore: decrease mutable write buffer limit by @evenyag in #2390
• feat!: add engine name to DatanodeTableValue by @fengjiachun in #2395
• fix: alter table procedure panics while renaming table by @evenyag in #2397
• feat: set readonly first when deregister region by @fengjiachun in #2391
• feat: remove deprecated metadata keys by @fengjiachun in #2398
• refactor: remove table ident by @MichaelScofield in #2368
• chore: add #[serde(default)] to new added engine field by @shuiyisong in #2402
• fix: enable ignored promql unit tests by @waynexia in #2403
• refactor: catalog managers by @killme2008 in #2405
• feat: region storage path by @fengjiachun in #2404
• fix: resolve more integration tests by @MichaelScofield in #2406
• fix: validate partition columns by @MichaelScofield in #2393
• feat: reserve internal column by @Nateiru in #2396
• feat: implement truncate region for mito2 by @DevilExileSu in #2335
• feat: reopen corresponding regions on starting datanode by @waynexia in #2399
• fix: generate region path with given prefix by @waynexia in #2409
• feat: move table operations from frontend to operator crate by @killme2008 in #2411
• feat: clap wrapper around sqlness by @Lilit0x in #2400
• feat: heartbeat request with header by @fengjiachun in #2412
• feat!: unify naming with options by @fengjiachun in #2416
• feat: sync regions between RegionServer and RegionAliveKeeper by @WenyXu in #2417
• fix: missing datanode id on keep lease by @fengjiachun in #2415
• refactor: rename common-function-macro subcrate by @waynexia in #2418
• feat: scope spawned task with trace id by @waynexia in #2419
• feat(mito): Implements compaction scheduler by @evenyag in #2413
• fix: update several sqlness results by @waynexia in #2422
• docs: fix cargo doc errors and warnings by @evenyag in #2421
• feat: all distributed time together by @fengjiachun in #2423
• feat: migrate file engine from table to reigon by @zhongzc in #2365
• feat: start services after first heartbeat response processed by @WenyXu in #2424
• test: more integration test cases for external table by @zhongzc in #2429
• feat: type alias by @killme2008 in #2331
• fix: render comment in SHOW CREATE TABLE by @waynexia in #2427
• test: fix some integration tests by @zhongzc in #2432
• fix: fix incorrect matches by @WenyXu in #2430
• refactor: rename coordination to require_lease_before_startup by @WenyXu in #2431
• feat: impl duration datatype and vectors by @QuenKar in #2180
• feat: make scripts table work again by @killme2008 in #2420
• fix: check version before alter region by @WenyXu in #2433
• fix: compile error after updating protos by @killme2008 in #2435
• feat: type conversion between Values by @QuenKar in #2394
• fix: type cast bugs found by sqlness by @waynexia in #2438
• refactor: RegionRequestHandler -> RegionQueryHandler by @zhongzc in #2439
• fix: parse i64 to different kinds of timestamp by @QuenKar in #2440
• fix: fix open region missing path by @WenyXu in #2441
• feat(mito): make use of options in RegionCreate/OpenRequest by @evenyag in #2436
• refactor: remove the old mito engine by @waynexia in #2443
• fix: fix alter table verification by @WenyXu in #2437
• fix: remark region as inactive on ...