Question about link_num in Generated Topology (HPN 7.0)

[beatlesforever]

I have a question about the link_num in the generated HPN 7.0 topology. According to the generated topology file:

18 8 1 9 20 A100
8 9 10 11 12 13 14 15 16 17 
0 8 2400Gbps 0.000025ms 0
0 9 400Gbps 0.0005ms 0
1 8 2400Gbps 0.000025ms 0
1 10 400Gbps 0.0005ms 0
2 8 2400Gbps 0.000025ms 0
2 11 400Gbps 0.0005ms 0
3 8 2400Gbps 0.000025ms 0
3 12 400Gbps 0.0005ms 0
4 8 2400Gbps 0.000025ms 0
4 13 400Gbps 0.0005ms 0
5 8 2400Gbps 0.000025ms 0
5 14 400Gbps 0.0005ms 0
6 8 2400Gbps 0.000025ms 0
6 15 400Gbps 0.0005ms 0
7 8 2400Gbps 0.000025ms 0
7 16 400Gbps 0.0005ms 0
9 17 400Gbps 0.0005ms 0
10 17 400Gbps 0.0005ms 0
11 17 400Gbps 0.0005ms 0
12 17 400Gbps 0.0005ms 0
13 17 400Gbps 0.0005ms 0
14 17 400Gbps 0.0005ms 0
15 17 400Gbps 0.0005ms 0
16 17 400Gbps 0.0005ms 0

The file states that there are 20 links. However, based on the connection logic:

GPU to NVSwitch: There are 8 GPUs (0–7), each connected to one NVSwitch (node 8). This adds up to 8 links.
GPU to ASW: Each GPU (0–7) is also connected to an Aggregation Switch (ASW, nodes 9–16). This should also add 8 links.
ASW to PSW: Each ASW (nodes 9–16) is connected to the Pod Switch (PSW, node 17). This should add 8 links.

However, the topology file specifies only 20 links, which seems to leave out some connections. I may be misunderstanding the logic, so I’d appreciate clarification on the following points:

1.How is the link_num value of 20 calculated?
2.Are there specific links that are excluded in this count? If so, which ones and why?

[beatlesforever]

In the code, the total number of links is calculated using the following formula:

links = (int)((args.psw_switch_num / 2) * asw_switch_num + servers * asw_switch_num + servers * args.nv_switch_per_server * args.gpu_per_server)

I noticed that in the first term:

(args.psw_switch_num / 2) * asw_switch_num

Could you kindly clarify the reasoning behind this division? Does it represent a specific connection pattern, such as each ASW connecting to only half the PSWs? If this division is correct, the link_num of 20 makes sense, but I want to ensure I’m understanding the logic properly. Thanks so much!

[Yan824]

Hi, thanks for your question! For HPN7.0, dual plane is applied in the topology, so the input number of PSW is the total number of which in dual plane. And within single plane topology, the first half psw will be used. So here if you want to generate topology with 24 links, the -psn should be 2. There is some ambiguity, and we will make improvements in future versions.

[beatlesforever]

Thank you for the clarification! I ran the following command to generate the topology:

python3 ./astra-sim-alibabacloud/inputs/topo/gen_HPN_7.0_topo_mulgpus_one_link.py -g 8 -gt A100 -bw 400Gbps -apbw 400Gbps -nvbw 2400Gbps -l 0.0005ms -nl 0.000025ms -gps 8 -psn 2 -nsps 1 -er 0

The resulting topology file content is as follows:

19 8 1 10 24 A100
8 9 10 11 12 13 14 15 16 17 18
0 8 2400Gbps 0.000025ms 0
0 9 400Gbps 0.0005ms 0
1 8 2400Gbps 0.000025ms 0
1 10 400Gbps 0.0005ms 0
2 8 2400Gbps 0.000025ms 0
2 11 400Gbps 0.0005ms 0
3 8 2400Gbps 0.000025ms 0
3 12 400Gbps 0.0005ms 0
4 8 2400Gbps 0.000025ms 0
4 13 400Gbps 0.0005ms 0
5 8 2400Gbps 0.000025ms 0
5 14 400Gbps 0.0005ms 0
6 8 2400Gbps 0.000025ms 0
6 15 400Gbps 0.0005ms 0
7 8 2400Gbps 0.000025ms 0
7 16 400Gbps 0.0005ms 0
9 17 400Gbps 0.0005ms 0
10 17 400Gbps 0.0005ms 0
11 17 400Gbps 0.0005ms 0
12 17 400Gbps 0.0005ms 0
13 17 400Gbps 0.0005ms 0
14 17 400Gbps 0.0005ms 0
15 17 400Gbps 0.0005ms 0
16 17 400Gbps 0.0005ms 0

This command generates a topology that utilizes all links across the dual-plane configuration. The resulting link_num is 24, and all the PSWs and their associated links are correctly included. However, I have a question regarding the documentation and the behavior of the topology generator:

Documentation Clarity

• Should it be explicitly stated in the documentation that only the first link_num links will be used in the generated topology? This would help users understand that the topology generator may define more potential connections than are actively utilized, depending on the specified link_num.

Thank you for considering this feedback, and I look forward to further improvements in future versions! 🙏

[Huoyuan100861]

Thank you for the clarification! I ran the following command to generate the topology:

python3 ./astra-sim-alibabacloud/inputs/topo/gen_HPN_7.0_topo_mulgpus_one_link.py -g 8 -gt A100 -bw 400Gbps -apbw 400Gbps -nvbw 2400Gbps -l 0.0005ms -nl 0.000025ms -gps 8 -psn 2 -nsps 1 -er 0

The resulting topology file content is as follows:

19 8 1 10 24 A100
8 9 10 11 12 13 14 15 16 17 18
0 8 2400Gbps 0.000025ms 0
0 9 400Gbps 0.0005ms 0
1 8 2400Gbps 0.000025ms 0
1 10 400Gbps 0.0005ms 0
2 8 2400Gbps 0.000025ms 0
2 11 400Gbps 0.0005ms 0
3 8 2400Gbps 0.000025ms 0
3 12 400Gbps 0.0005ms 0
4 8 2400Gbps 0.000025ms 0
4 13 400Gbps 0.0005ms 0
5 8 2400Gbps 0.000025ms 0
5 14 400Gbps 0.0005ms 0
6 8 2400Gbps 0.000025ms 0
6 15 400Gbps 0.0005ms 0
7 8 2400Gbps 0.000025ms 0
7 16 400Gbps 0.0005ms 0
9 17 400Gbps 0.0005ms 0
10 17 400Gbps 0.0005ms 0
11 17 400Gbps 0.0005ms 0
12 17 400Gbps 0.0005ms 0
13 17 400Gbps 0.0005ms 0
14 17 400Gbps 0.0005ms 0
15 17 400Gbps 0.0005ms 0
16 17 400Gbps 0.0005ms 0

This command generates a topology that utilizes all links across the dual-plane configuration. The resulting link_num is 24, and all the PSWs and their associated links are correctly included. However, I have a question regarding the documentation and the behavior of the topology generator:

Documentation Clarity

• Should it be explicitly stated in the documentation that only the first link_num links will be used in the generated topology? This would help users understand that the topology generator may define more potential connections than are actively utilized, depending on the specified link_num.

Thank you for considering this feedback, and I look forward to further improvements in future versions! 🙏

Thank you for your valuable suggestions. We would be delighted if you could directly submit a Pull Request to our code repository based on your ideas. We will review it promptly.

[beatlesforever]

Thank you for the clarification! I ran the following command to generate the topology:

python3 ./astra-sim-alibabacloud/inputs/topo/gen_HPN_7.0_topo_mulgpus_one_link.py -g 8 -gt A100 -bw 400Gbps -apbw 400Gbps -nvbw 2400Gbps -l 0.0005ms -nl 0.000025ms -gps 8 -psn 2 -nsps 1 -er 0

The resulting topology file content is as follows:

19 8 1 10 24 A100
8 9 10 11 12 13 14 15 16 17 18
0 8 2400Gbps 0.000025ms 0
0 9 400Gbps 0.0005ms 0
1 8 2400Gbps 0.000025ms 0
1 10 400Gbps 0.0005ms 0
2 8 2400Gbps 0.000025ms 0
2 11 400Gbps 0.0005ms 0
3 8 2400Gbps 0.000025ms 0
3 12 400Gbps 0.0005ms 0
4 8 2400Gbps 0.000025ms 0
4 13 400Gbps 0.0005ms 0
5 8 2400Gbps 0.000025ms 0
5 14 400Gbps 0.0005ms 0
6 8 2400Gbps 0.000025ms 0
6 15 400Gbps 0.0005ms 0
7 8 2400Gbps 0.000025ms 0
7 16 400Gbps 0.0005ms 0
9 17 400Gbps 0.0005ms 0
10 17 400Gbps 0.0005ms 0
11 17 400Gbps 0.0005ms 0
12 17 400Gbps 0.0005ms 0
13 17 400Gbps 0.0005ms 0
14 17 400Gbps 0.0005ms 0
15 17 400Gbps 0.0005ms 0
16 17 400Gbps 0.0005ms 0

This command generates a topology that utilizes all links across the dual-plane configuration. The resulting link_num is 24, and all the PSWs and their associated links are correctly included. However, I have a question regarding the documentation and the behavior of the topology generator:
Documentation Clarity

• Should it be explicitly stated in the documentation that only the first link_num links will be used in the generated topology? This would help users understand that the topology generator may define more potential connections than are actively utilized, depending on the specified link_num.

Thank you for considering this feedback, and I look forward to further improvements in future versions! 🙏

Thank you for your valuable suggestions. We would be delighted if you could directly submit a Pull Request to our code repository based on your ideas. We will review it promptly.

Thank you for your kind response! I have submitted a Pull Request based on the discussed suggestions: PR #61.

Please feel free to review it at your convenience. Let me know if there are any further changes or clarifications needed. Thanks again for your support!

[Huoyuan100861]

Thank you for the clarification! I ran the following command to generate the topology:

python3 ./astra-sim-alibabacloud/inputs/topo/gen_HPN_7.0_topo_mulgpus_one_link.py -g 8 -gt A100 -bw 400Gbps -apbw 400Gbps -nvbw 2400Gbps -l 0.0005ms -nl 0.000025ms -gps 8 -psn 2 -nsps 1 -er 0

The resulting topology file content is as follows:

19 8 1 10 24 A100
8 9 10 11 12 13 14 15 16 17 18
0 8 2400Gbps 0.000025ms 0
0 9 400Gbps 0.0005ms 0
1 8 2400Gbps 0.000025ms 0
1 10 400Gbps 0.0005ms 0
2 8 2400Gbps 0.000025ms 0
2 11 400Gbps 0.0005ms 0
3 8 2400Gbps 0.000025ms 0
3 12 400Gbps 0.0005ms 0
4 8 2400Gbps 0.000025ms 0
4 13 400Gbps 0.0005ms 0
5 8 2400Gbps 0.000025ms 0
5 14 400Gbps 0.0005ms 0
6 8 2400Gbps 0.000025ms 0
6 15 400Gbps 0.0005ms 0
7 8 2400Gbps 0.000025ms 0
7 16 400Gbps 0.0005ms 0
9 17 400Gbps 0.0005ms 0
10 17 400Gbps 0.0005ms 0
11 17 400Gbps 0.0005ms 0
12 17 400Gbps 0.0005ms 0
13 17 400Gbps 0.0005ms 0
14 17 400Gbps 0.0005ms 0
15 17 400Gbps 0.0005ms 0
16 17 400Gbps 0.0005ms 0

This command generates a topology that utilizes all links across the dual-plane configuration. The resulting link_num is 24, and all the PSWs and their associated links are correctly included. However, I have a question regarding the documentation and the behavior of the topology generator:
Documentation Clarity

• Should it be explicitly stated in the documentation that only the first link_num links will be used in the generated topology? This would help users understand that the topology generator may define more potential connections than are actively utilized, depending on the specified link_num.

Thank you for considering this feedback, and I look forward to further improvements in future versions! 🙏

Thank you for your valuable suggestions. We would be delighted if you could directly submit a Pull Request to our code repository based on your ideas. We will review it promptly.

Thank you for your kind response! I have submitted a Pull Request based on the discussed suggestions: PR #61.

Please feel free to review it at your convenience. Let me know if there are any further changes or clarifications needed. Thanks again for your support!

Have merged, if any issues arise, please feel free to contact us again.

[gabrielecastellano]

Thank you for pointing out this. I struggled to understand why my simulation was crashing on a simple topology. Note that this behavior was not clear to me even if I was following the tutorial updated by @beatlesforever. I guess the main reason for the confusion is that script gen_HPN_7.0_topo_mulgpus_one_link.py assumes at the same time two opposite default behaviors:

• on one hand, it generates a single plane by default (see parameter --dp enable dual_plane, default single plane);
• on the other hand, it assumes that the value passed to parameter -psn refers to the desired number of PSW switches in the dual plane case.

I would suggest you document this behavior directly in the options description and make it also explicit in the tutorial (see PR #66).

Eventually, it would be ideal to have the two parameters behave in a congruent way (i.e., if --dp is false by default, then the -psn should refer to the single plane case).

I hope this can help others in the future.
Thank you for this great work!