Skip to content

Commit

Permalink
Merge branch 'main' into xxh/fix-es-username
Browse files Browse the repository at this point in the history
  • Loading branch information
xxhZs authored Feb 4, 2025
2 parents c7e2f4c + 3ee72eb commit 0f5c4f6
Show file tree
Hide file tree
Showing 110 changed files with 2,050 additions and 1,018 deletions.
2 changes: 1 addition & 1 deletion .typos.toml
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ extend-exclude = [
"src/sqlparser/tests/testdata/",
"src/frontend/planner_test/tests/testdata",
"src/tests/sqlsmith/tests/freeze",
"src/license/src/manager.rs",
"src/license/**/*.rs", # JWT license key
"Cargo.lock",
"**/Cargo.toml",
"**/go.mod",
Expand Down
14 changes: 7 additions & 7 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

55 changes: 31 additions & 24 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,25 +9,18 @@

<div align="center">

### 🌊 Reimagine real-time data engineering.
### 🌊 Ride the Wave of Real-Time Data.

</div>

<p align="center">&nbsp;&nbsp;&nbsp;📚&nbsp;
<a
href="https://docs.risingwave.com/"
target="_blank"
><b>Documentation</b></a>&nbsp;&nbsp;&nbsp;🚀&nbsp;
<a
href="https://risingwave.com/slack"
target="_blank"
>
<b>Slack Community</b>
</a>
<p align="center">
<a href="https://docs.risingwave.com/">Docs</a> | <a href="https://docs.risingwave.com/get-started/rw-benchmarks-stream-processing">Benchmarks</a> | <a href="https://docs.risingwave.com/demos/overview">Demos</a>
</p>

<p align="center">

<div align="center">
<a
href="https://risingwave.com/slack"
href="https://go.risingwave.com/slack"
target="_blank"
>
<img alt="Slack" src="https://badgen.net/badge/Slack/Join%20RisingWave/0abd59?icon=slack" />
Expand All @@ -46,11 +39,11 @@
</a>
</div>
RisingWave is a Postgres-compatible SQL database engineered to provide the <i><b>simplest</b></i> and <i><b>most cost-efficient</b></i> approach for <b>processing</b>, <b>analyzing</b>, and <b>managing</b> real-time event streaming data.
RisingWave is the world's most advanced event stream processing platform engineered to provide the <i><b>simplest</b></i> and <i><b>most cost-efficient</b></i> approach for <b>processing</b>, <b>analyzing</b>, and <b>managing</b> real-time event streaming data. It provides both a Postgres-compatible [SQL interface](https://docs.risingwave.com/sql/overview) and a DataFrame-style [Python interface](https://docs.risingwave.com/python-sdk/intro).

RisingWave can <b>ingest</b> millions of events per second, continuously <b>join and analyze</b> live data streams with historical tables, <b>serve</b> ad-hoc queries in real-time, and <b>deliver</b> fresh, consistent results wherever needed.
RisingWave can <b>ingest</b> millions of events per second, continuously <b>join and analyze</b> live data streams with historical tables, <b>serve</b> ad-hoc queries at low latency, and <b>deliver</b> fresh, consistent results wherever needed.

![RisingWave](./docs/dev/src/images/architecture_20240908.png)
![RisingWave](./docs/dev/src/images/architecture_20250127.png)

## Try it out in 60 seconds

Expand All @@ -61,14 +54,28 @@ curl -L https://risingwave.com/sh | sh

To learn about other installation options, such as using a Docker image, see [Quick Start](https://docs.risingwave.com/docs/current/get-started/).

## When is RisingWave the perfect fit?
RisingWave is the ideal solution for:
## What Is RisingWave Optimized For?
RisingWave simplifies the development of real-time data pipelines and applications. It is purpose-built to:

* **Ingestion**: Ingest millions of events per second from both streaming and batch data sources.
* **Stream processing**: Perform real-time incremental data processing to join and analyze live data streams with historical tables.
* **Serving**: Persist data and serve ad-hoc queries with single-digit millisecond latency.
* **Delivery**: Deliver fresh, consistent results to data lakes (e.g., Apache Iceberg) or any destination.


## Why Is RisingWave Special?
RisingWave stands apart from traditional stream processing systems due to its:

### PostgreSQL Compatibility
* **Seamless integration**: Works with a wide range of tools in the PostgreSQL ecosystem.
* **Expressive SQL**: Supports structured, semi-structured, and unstructured data using a rich, familiar SQL dialect.
* **No manual state tuning**: Eliminates the need for complex state management configurations.

### Decoupled Compute and Storage
* **Optimized for complex queries**: Ensures high performance, especially for complex operations like joins and time windowing.
* **Fast failure recovery**: Recovers from system crashes within seconds.
* **Dynamic scaling**: Adjusts resources instantly to handle workload spikes.

* Managing real-time data sources like Kafka streams, database CDC, and more.
* Executing complex, on-the-fly queries, including joins, aggregations, and time windowing.
* Interactively and concurrently exploring consistent, up-to-the-moment results.
* Seamlessly delivering results to downstream systems.
* Processing both streaming and batch data with a unified codebase.


## In what use cases does RisingWave excel?
Expand Down
20 changes: 10 additions & 10 deletions ci/workflows/main-cron.yml
Original file line number Diff line number Diff line change
Expand Up @@ -144,7 +144,7 @@ steps:
files: "*-junit.xml"
format: "junit"
- ./ci/plugins/upload-failure-logs
timeout_in_minutes: 13
timeout_in_minutes: 18
retry: *auto-retry

- label: "end-to-end test (parallel, in-memory) (release)"
Expand All @@ -160,7 +160,7 @@ steps:
plugins:
- docker-compose#v5.5.0: *docker-compose
- ./ci/plugins/upload-failure-logs
timeout_in_minutes: 15
timeout_in_minutes: 20
retry: *auto-retry

- group: "end-to-end connector test (release)"
Expand Down Expand Up @@ -342,7 +342,7 @@ steps:

- label: "end-to-end test (madsim)"
key: "e2e-test-deterministic"
command: "TEST_NUM=32 timeout 130m ci/scripts/deterministic-e2e-test.sh"
command: "TEST_NUM=32 timeout 135m ci/scripts/deterministic-e2e-test.sh"
if: |
!(build.pull_request.labels includes "ci/main-cron/run-selected") && build.env("CI_STEPS") == null
|| build.pull_request.labels includes "ci/run-e2e-test-deterministic-simulation"
Expand All @@ -357,12 +357,12 @@ steps:
environment:
- GITHUB_TOKEN
- ./ci/plugins/upload-failure-logs
timeout_in_minutes: 135
timeout_in_minutes: 140
retry: *auto-retry

- label: "end-to-end test (madsim, random vnode count)"
key: "e2e-test-deterministic-random-vnode-count"
command: "TEST_NUM=32 RW_SIM_RANDOM_VNODE_COUNT=true timeout 130m ci/scripts/deterministic-e2e-test.sh"
command: "TEST_NUM=32 RW_SIM_RANDOM_VNODE_COUNT=true timeout 135m ci/scripts/deterministic-e2e-test.sh"
if: |
!(build.pull_request.labels includes "ci/main-cron/run-selected") && build.env("CI_STEPS") == null
|| build.pull_request.labels includes "ci/run-e2e-test-deterministic-simulation"
Expand All @@ -377,12 +377,12 @@ steps:
environment:
- GITHUB_TOKEN
- ./ci/plugins/upload-failure-logs
timeout_in_minutes: 135
timeout_in_minutes: 140
retry: *auto-retry

- label: "recovery test (madsim)"
key: "recovery-test-deterministic"
command: "TEST_NUM=12 KILL_RATE=1.0 BACKGROUND_DDL_RATE=0.0 timeout 70m ci/scripts/deterministic-recovery-test.sh"
command: "TEST_NUM=12 KILL_RATE=1.0 BACKGROUND_DDL_RATE=0.0 timeout 75m ci/scripts/deterministic-recovery-test.sh"
if: |
!(build.pull_request.labels includes "ci/main-cron/run-selected") && build.env("CI_STEPS") == null
|| build.pull_request.labels includes "ci/run-recovery-test-deterministic-simulation"
Expand All @@ -392,13 +392,13 @@ steps:
- docker-compose#v5.5.0: *docker-compose
# Only upload zipped files, otherwise the logs is too much.
- ./ci/plugins/upload-failure-logs-zipped
timeout_in_minutes: 75
timeout_in_minutes: 80
retry: *auto-retry

# Ddl statements will randomly run with background_ddl.
- label: "background_ddl, arrangement_backfill recovery test (madsim)"
key: "background-ddl-arrangement-backfill-recovery-test-deterministic"
command: "TEST_NUM=12 KILL_RATE=1.0 BACKGROUND_DDL_RATE=0.8 USE_ARRANGEMENT_BACKFILL=true timeout 70m ci/scripts/deterministic-recovery-test.sh"
command: "TEST_NUM=12 KILL_RATE=1.0 BACKGROUND_DDL_RATE=0.8 USE_ARRANGEMENT_BACKFILL=true timeout 75m ci/scripts/deterministic-recovery-test.sh"
if: |
!(build.pull_request.labels includes "ci/main-cron/run-selected") && build.env("CI_STEPS") == null
|| build.pull_request.labels includes "ci/run-recovery-test-deterministic-simulation"
Expand All @@ -408,7 +408,7 @@ steps:
- docker-compose#v5.5.0: *docker-compose
# Only upload zipped files, otherwise the logs is too much.
- ./ci/plugins/upload-failure-logs-zipped
timeout_in_minutes: 70
timeout_in_minutes: 80
retry: *auto-retry

- label: "end-to-end iceberg sink test (release)"
Expand Down
8 changes: 4 additions & 4 deletions ci/workflows/pull-request.yml
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,7 @@ steps:
plugins:
- docker-compose#v5.5.0: *docker-compose
- ./ci/plugins/upload-failure-logs
timeout_in_minutes: 25
timeout_in_minutes: 30
retry: *auto-retry

- label: "end-to-end test for opendal (parallel)"
Expand Down Expand Up @@ -562,7 +562,7 @@ steps:
environment:
- GITHUB_TOKEN
- ./ci/plugins/upload-failure-logs
timeout_in_minutes: 30
timeout_in_minutes: 35
cancel_on_build_failing: true
retry: *auto-retry

Expand All @@ -583,7 +583,7 @@ steps:
# - test-collector#v1.0.0:
# files: "*-junit.xml"
# format: "junit"
timeout_in_minutes: 35
timeout_in_minutes: 40
cancel_on_build_failing: true
retry: *auto-retry

Expand Down Expand Up @@ -748,7 +748,7 @@ steps:
run: ci-standard-env
propagate-environment: true
- ./ci/plugins/upload-failure-logs
timeout_in_minutes: 32
timeout_in_minutes: 37
retry: *auto-retry

# FIXME(kwannoel): Let the github PR labeller label it, if sqlsmith source files has changes.
Expand Down
Binary file added docs/dev/src/images/architecture_20250127.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
43 changes: 43 additions & 0 deletions e2e_test/batch/join/asof_join.slt
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
statement ok
SET RW_IMPLICIT_FLUSH TO true;

statement ok
create table t1 (v1 int, v2 int, v3 int primary key);

statement ok
create table t2 (v1 int, v2 int, v3 int primary key);

statement ok
insert into t1 values (1, 2, 3), (2, 3, 4), (1, 2, 9);

statement ok
insert into t2 values (1, NULL, 8), (1, 3, 4), (1, 2, 5), (1, 2, 6);

# asof inner join
query IIIIII
SELECT t1.v1 t1_v1, t1.v2 t1_v2, t1.v3 t1_v3, t2.v1 t2_v1, t2.v2 t2_v2, t2.v3 t2_v3 FROM t1 ASOF JOIN t2 ON t1.v1 = t2.v1 and t1.v2 < t2.v2 order by t1.v1, t1.v3;
----
1 2 3 1 3 4
1 2 9 1 3 4

# asof left join
query IIIIII
SELECT t1.v1 t1_v1, t1.v2 t1_v2, t1.v3 t1_v3, t2.v1 t2_v1, t2.v2 t2_v2, t2.v3 t2_v3 FROM t1 ASOF LEFT JOIN t2 ON t1.v1 = t2.v1 and t1.v2 < t2.v2 order by t1.v1, t1.v3;
----
1 2 3 1 3 4
1 2 9 1 3 4
2 3 4 NULL NULL NULL

# asof left join
query IIIIII
SELECT t1.v1 t1_v1, t1.v2 t1_v2, t1.v3 t1_v3, t2.v1 t2_v1, t2.v2 t2_v2, t2.v3 t2_v3 FROM t1 ASOF LEFT JOIN t2 ON t1.v1 = t2.v1 and t1.v2 > t2.v2 order by t1.v1, t1.v3;
----
1 2 3 NULL NULL NULL
1 2 9 NULL NULL NULL
2 3 4 NULL NULL NULL

statement ok
drop table t1;

statement ok
drop table t2;
41 changes: 41 additions & 0 deletions e2e_test/ddl/alter_table_column_issue_17121.slt
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# https://github.com/risingwavelabs/risingwave/issues/17121

statement ok
create table t(v int);

statement ok
insert into t values (1);

statement ok
alter table t add column now1 timestamptz default now();

# Epoch (then `now()`) will advance after `FLUSH`.
statement ok
flush;

statement ok
insert into t values (2);

statement ok
flush;

query I
select v from t order by now1;
----
1
2

# Add a new column again, causing the table to be replaced again.
statement ok
alter table t add column v2 varchar;

# We show that the "snapshot value" of `now1` does not get refreshed upon the above `ALTER TABLE`.
# Otherwise, the `now1` column of `v=1` would be greater than that of `v=2`.
query I
select v from t order by now1;
----
1
2

statement ok
drop table t;
Loading

0 comments on commit 0f5c4f6

Please sign in to comment.