rocksdb 2台从服务器的sst文件区别很大

[hexuxu110]

Problem Type (问题类型)

None

Before submit

• 我已经确认现有的 Issues 与 FAQ 中没有相同 / 重复问题 (I have confirmed and searched that there are no similar problems in the historical issue and documents)

Environment (环境信息)

• Server Version: 1.0.0 (Apache Release Version)
• Backend: RocksDB
• OS: 64CPU 128Gmemory
• Data Size: xx vertices, xx edges

Your Question (问题描述)

第一台机器:
du -sh *
817G raft-log
1.3T rocksdb_data
0 rocksdb_log

find G -type f | wc -l
2w个sst文件

第二台机器:
732G raft-log
764G rocksdb_data
0 rocksdb_log

find G -type f | wc -l
1w个sst文件

我的是一主6从部署，为什么2个从服务器的占用磁盘大小不一致

Vertex/Edge example (问题点 / 边数据举例)

Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)

[imbajin]

what's your HG version?

Try to use 1.5.0+ for the community support, the previous multi-replica version has been archived and is no longer being maintained 🔢

[hexuxu110]

what's your HG version?

Try to use 1.5.0+ for the community support, the previous multi-replica version has been archived and is no longer being maintained 🔢

i'm using 1.0.0 version of hugegraph.
who is put like this issue before?

[imbajin]

what's your HG version?
Try to use 1.5.0+ for the community support, the previous multi-replica version has been archived and is no longer being maintained 🔢

i'm using 1.0.0 version of hugegraph. who is put like this issue before?

@dosu could you help the user to provide ways and methods for locating problems

[dosubot]

你能提供更多关于你的部署和配置的信息吗？例如，数据分片策略、复制因子、以及是否有任何自定义的配置参数。这些信息可能有助于更好地理解为什么两个从服务器的磁盘使用情况不同。

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[hexuxu110]

你能提供更多关于你的部署和配置的信息吗？例如，数据分片策略、复制因子、以及是否有任何自定义的配置参数。这些信息可能有助于更好地理解为什么两个从服务器的磁盘使用情况不同。

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other

2台机器都是这个配置,大部分都是1.0.0项目rocskdb默认配置

# gremlin entrance to create graph
# auth config: org.apache.hugegraph.auth.HugeFactoryAuthProxy
gremlin.graph=org.apache.hugegraph.HugeFactory

# cache config
#schema.cache_capacity=100000
# vertex-cache default is 1000w, 10min expired
vertex.cache_type=l2
vertex.cache_capacity=0
#vertex.cache_expire=600
# edge-cache default is 100w, 10min expired
edge.cache_type=l2
edge.cache_capacity=0
#edge.cache_expire=600


# schema illegal name template
#schema.illegal_name_regex=\s+|~.*

#vertex.default_label=vertex

backend=rocksdb
serializer=binary

store=octo_rocksdb

raft.mode=true
raft.path=/home/*/data1/octo-graph/raft-log
raft.safe_read=false
raft.use_replicator_pipeline=true
raft.election_timeout=10000
raft.snapshot_interval=21600
raft.backend_threads=48
raft.read_index_threads=8
raft.snapshot_threads=4
raft.snapshot_parallel_compress=true
raft.snapshot_compress_threads=16
raft.snapshot_decompress_threads=16
raft.read_strategy=ReadOnlyLeaseBased
raft.queue_size=16384
raft.queue_publish_timeout=60
raft.apply_batch=1
raft.rpc_threads=80
raft.rpc_connect_timeout=5000
raft.rpc_timeout=60
raft.install_snapshot_rpc_timeout=36000

search.text_analyzer=jieba
search.text_analyzer_mode=INDEX

# rocksdb backend config
rocksdb.data_path=/home/*/data1/octo-graph/rocksdb_data
rocksdb.wal_path=/home/*/data1/octo-graph/rocksdb_log

@dosu

[imbajin]

你能提供更多关于你的部署和配置的信息吗？例如，数据分片策略、复制因子、以及是否有任何自定义的配置参数。这些信息可能有助于更好地理解为什么两个从服务器的磁盘使用情况不同。
To continue talking to Dosu, mention @dosu.
Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other

2台机器都是这个配置,大部分都是1.0.0项目rocskdb默认配置

gremlin entrance to create graph

auth config: org.apache.hugegraph.auth.HugeFactoryAuthProxy

gremlin.graph=org.apache.hugegraph.HugeFactory

cache config

#schema.cache_capacity=100000

vertex-cache default is 1000w, 10min expired

vertex.cache_type=l2
vertex.cache_capacity=0
#vertex.cache_expire=600

edge-cache default is 100w, 10min expired

edge.cache_type=l2
edge.cache_capacity=0
#edge.cache_expire=600

schema illegal name template

#schema.illegal_name_regex=\s+|~.*

#vertex.default_label=vertex

backend=rocksdb
serializer=binary

store=octo_rocksdb

raft.mode=true
raft.path=/home/*/data1/octo-graph/raft-log
raft.safe_read=false
raft.use_replicator_pipeline=true
raft.election_timeout=10000
raft.snapshot_interval=21600
raft.backend_threads=48
raft.read_index_threads=8
raft.snapshot_threads=4
raft.snapshot_parallel_compress=true
raft.snapshot_compress_threads=16
raft.snapshot_decompress_threads=16
raft.read_strategy=ReadOnlyLeaseBased
raft.queue_size=16384
raft.queue_publish_timeout=60
raft.apply_batch=1
raft.rpc_threads=80
raft.rpc_connect_timeout=5000
raft.rpc_timeout=60
raft.install_snapshot_rpc_timeout=36000

search.text_analyzer=jieba
search.text_analyzer_mode=INDEX

rocksdb backend config

rocksdb.data_path=/home//data1/octo-graph/rocksdb_data
rocksdb.wal_path=/home//data1/octo-graph/rocksdb_log
@dosu

@dosu based on the context that user provided, give suggestion as much as possible

[dosubot]

To address the discrepancy in SST file sizes and counts between your two RocksDB replica servers, consider the following suggestions:

1. Check Data Consistency: Ensure that both replicas have the same data. Differences in data can lead to variations in SST file sizes and counts. Use tools or scripts to verify data consistency across replicas.

2. Review Compaction Settings: Compaction settings can significantly affect SST file sizes. Ensure that both servers have identical compaction configurations. Check parameters like rocksdb.compaction_style and rocksdb.level0_file_num_compaction_trigger.

3. Monitor Write Patterns: Different write patterns can lead to variations in SST file creation. Monitor the write load on each server to ensure they are receiving similar workloads.

4. Check for Disk Space Issues: Ensure that there are no disk space constraints on either server that might affect RocksDB's ability to perform compactions effectively.

5. Review Raft Configuration: Since you are using Raft, ensure that the Raft configuration is consistent across all nodes. Parameters like raft.snapshot_interval and raft.backend_threads can impact how data is replicated and stored.

6. Examine Logs for Errors: Check the logs on both servers for any errors or warnings that might indicate issues with data replication or compaction processes.

7. Upgrade Consideration: If possible, consider upgrading to a newer version of HugeGraph, as improvements and bug fixes in later versions might address some of these issues.

By following these steps, you should be able to identify and resolve the discrepancies in SST file sizes and counts between your RocksDB replica servers.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}