[CELEBORN-1635] Introduce Blaze support document

### What changes were proposed in this pull request? Introduce Blaze support document. ### Why are the changes needed? [Blaze](https://github.com/kwai/blaze) supports Celeborn as remote shuffle service. It's recommened to Blaze support document for introduction of Blaze usage. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? No. Closes apache#2787 from SteNicholas/CELEBORN-1635. Authored-by: SteNicholas <[email protected]> Signed-off-by: Shuang <[email protected]>
SteNicholas · Oct 9, 2024 · 101c755 · 101c755
1 parent 87822f6
commit 101c755
Show file tree

Hide file tree

Showing 3 changed files with 50 additions and 5 deletions.
diff --git a/docs/developers/blazesupport.md b/docs/developers/blazesupport.md
@@ -0,0 +1,47 @@
+---
+license: |
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+      https://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+---
+
+# Blaze Support
+
+## Build Blaze
+
+[Blaze](https://github.com/kwai/blaze) supports Celeborn as remote shuffle service. Below introduction is used to enable this feature.
+
+First refer to [Build From Source](https://github.com/kwai/blaze/blob/master/README.md#build-from-source) or [Build With Docker](https://github.com/kwai/blaze/blob/master/README.md#build-with-docker) to build Blaze.
+
+## Blaze Configuration
+
+Currently, to use Blaze following configurations are required in `spark-defaults.conf`.
+
+```
+spark.shuffle.manager org.apache.spark.sql.execution.blaze.shuffle.celeborn.BlazeCelebornShuffleManager
+
+# celeborn master
+spark.celeborn.master.endpoints clb-master:9097
+
+spark.celeborn.client.spark.shuffle.writer hash
+spark.serializer org.apache.spark.serializer.KryoSerializer
+spark.sql.adaptive.localShuffleReader.enabled false
+```
+
+## Availability
+
+| Celeborn Version | Available in Blaze? | 
+|:----------------:|:-------------------:|
+|     < 0.5.0      |         No          |    
+|    \>= 0.5.0     |         Yes         |
diff --git a/docs/developers/glutensupport.md b/docs/developers/glutensupport.md
@@ -60,10 +60,6 @@ spark.shuffle.manager org.apache.spark.shuffle.gluten.celeborn.CelebornShuffleMa
 # celeborn master
 spark.celeborn.master.endpoints clb-master:9097
 
-# we recommend set `spark.celeborn.push.replicate.enabled` to true to enable server-side data replication
-# If you have only one worker, this setting must be false 
-spark.celeborn.client.push.replicate.enabled true
-
 spark.celeborn.client.spark.shuffle.writer hash
 # This is not necessary if your Spark external shuffle service is Spark 3.1 or newer
 spark.shuffle.service.enabled false

diff --git a/mkdocs.yml b/mkdocs.yml
@@ -97,5 +97,7 @@ nav:
       - Worker Exclusion: developers/workerexclusion.md
       - Integrating Celeborn: developers/integrate.md
       - SBT Build: developers/sbt.md
-      - Gluten Support: developers/glutensupport.md
+      - Native Engine Integration:
+        - Blaze: developers/blazesupport.md
+        - Gluten: developers/glutensupport.md
       - Helm Charts: developers/helm-charts.md