diff --git a/high_performance_computing/hpc_mpi/05_collective_communication.md b/high_performance_computing/hpc_mpi/05_collective_communication.md index 99165a33..123df2e5 100644 --- a/high_performance_computing/hpc_mpi/05_collective_communication.md +++ b/high_performance_computing/hpc_mpi/05_collective_communication.md @@ -206,7 +206,7 @@ int main(int argc, char **argv) { ### Scatter -One way to parallelise processing amount of data is to have ranks process a subset of the data. +One way to parallelise processing an amount of data is to have ranks process a subset of the data. One method for distributing the data to each rank is to have a root rank which prepares the data, and then send the data to every rank. The communication could be done *manually* using point-to-point communication, but it's easier, and faster, to use a single collective communication. We can use `MPI_Scatter()` to split the data into *equal* sized chunks and communicate a different chunk to each rank, as shown in the diagram below. @@ -476,7 +476,7 @@ Not all collective operations support in-place operations, and the usage of `MPI :::::challenge{id=reductions, title="Reductions"} The following program creates an array called `vector` that contains a list of `n_numbers` on each rank. -The first rank contains the numbers from > 1 to n_numbers, the second rank from n_numbers to 2*n_numbers2 and so on. +The first rank contains the numbers from 0 to n_numbers, the second rank from n_numbers to 2*n_numbers and so on. It then calls the `find_max` and `find_sum` functions that should calculate the sum and maximum of the vector. These functions are not implemented in parallel and only return the sum and the maximum of the local vectors. diff --git a/high_performance_computing/hpc_mpi/06_non_blocking_communication.md b/high_performance_computing/hpc_mpi/06_non_blocking_communication.md index afdec2ab..8aedc740 100644 --- a/high_performance_computing/hpc_mpi/06_non_blocking_communication.md +++ b/high_performance_computing/hpc_mpi/06_non_blocking_communication.md @@ -370,7 +370,7 @@ int main(int argc, char **argv) if (num_ranks < 2) { printf("This example requires at least two ranks\n"); - MPI_Abort(MPI_COMM_WORLD, 1); + return MPI_Finalize(); } char send_message[MESSAGE_SIZE]; diff --git a/high_performance_computing/hpc_mpi/08_porting_serial_to_mpi.md b/high_performance_computing/hpc_mpi/08_porting_serial_to_mpi.md index e853451f..181e41c0 100644 --- a/high_performance_computing/hpc_mpi/08_porting_serial_to_mpi.md +++ b/high_performance_computing/hpc_mpi/08_porting_serial_to_mpi.md @@ -140,7 +140,7 @@ And finally, the state of the stick is set to the newly calculated values, and ` ```c // Overwrite u with the new field - for (int i = 1 ;i <= points; ++i) { + for (int i = 1; i <= points; i++) { u[i] = unew[i]; } @@ -398,10 +398,9 @@ Such an order might look like this (highlighting the odd ranks - only one in thi ![Communication strategy - odd ranks send to potential neighbours first, then receive from them](fig/poisson_comm_strategy_1.png) -So following the `MPI_Allreduce()` we've just added, let's deal with odd ranks first (again, put the declarations at the top of the function): +So, following the code that overwrites `u` with the new field, let's deal with odd ranks first (again, put the declarations at the top of the function): ```c - float sendbuf, recvbuf; MPI_Status mpi_status; // The u field has been changed, communicate it to neighbours @@ -410,12 +409,10 @@ So following the `MPI_Allreduce()` we've just added, let's deal with odd ranks f if ((rank % 2) == 1) { // Ranks with odd number send first - // Send data down from rank to rank - 1 - sendbuf = unew[1]; - MPI_Send(&sendbuf, 1, MPI_FLOAT, rank - 1, 1, MPI_COMM_WORLD); - // Receive data from rank - 1 - MPI_Recv(&recvbuf, 1, MPI_FLOAT, rank - 1, 2, MPI_COMM_WORLD, &mpi_status); - u[0] = recvbuf; + // Send data down from rank to rank-1 + MPI_Send(&u[1], 1, MPI_FLOAT, rank-1, 1, MPI_COMM_WORLD); + // Receive dat from rank-1 + MPI_Recv(&u[0], 1, MPI_FLOAT, rank-1, 2, MPI_COMM_WORLD, &mpi_status); if (rank != (n_ranks - 1)) { // Send data up to rank + 1 (if I'm not the last rank) @@ -431,7 +428,7 @@ We first send our newly computed leftmost value (at position `1` in our array) t Then, if the rank following us exists, we do the same, but this time we send the rightmost value at the end of our stick section, and receive the corresponding value from that rank. -These exchanges mean that - as an even rank - we now have effectively exchanged the states of the start and end of our slices with our respective neighbours. +These exchanges mean that - as an odd rank - we now have effectively exchanged the states of the start and end of our slices with our respective neighbours. And now we need to do the same for those neighbours (the even ranks), mirroring the same communication pattern but in the opposite order of receive/send. From the perspective of evens, it should look like the following (highlighting the two even ranks): diff --git a/high_performance_computing/hpc_mpi/10_communication_patterns.md b/high_performance_computing/hpc_mpi/10_communication_patterns.md index 4ee6d804..71cd38fc 100644 --- a/high_performance_computing/hpc_mpi/10_communication_patterns.md +++ b/high_performance_computing/hpc_mpi/10_communication_patterns.md @@ -362,7 +362,7 @@ To communicate the halos, we need to: 3. Using the derived types and neighbouring ranks, communicate the top row of the sub-domain to the bottom halo row of the neighbouring top domain. We also need to repeat the same process for the bottom row to the neighbouring sub-domain below and so on for the left and right columns of halo data. -To re-build the sub-domains into one domain, we can do the reverse of the hidden code exert of the function `scatter_sub_arrays_to_other ranks`. +To re-build the sub-domains into one domain, we can do the reverse of the the function `scatter_sub_arrays_to_other_ranks` shown [above](#extra-scattering-the-image-to-other-ranks) Instead of the root rank sending data, it instead receives data from the other ranks using the same `sub_array_t` derived type. :::: ::::: diff --git a/high_performance_computing/hpc_mpi/code/examples/poisson/poisson_mpi.c b/high_performance_computing/hpc_mpi/code/examples/poisson/poisson_mpi.c index 53b9b9c1..58e8b2ac 100644 --- a/high_performance_computing/hpc_mpi/code/examples/poisson/poisson_mpi.c +++ b/high_performance_computing/hpc_mpi/code/examples/poisson/poisson_mpi.c @@ -20,7 +20,6 @@ double poisson_step( int rank, int n_ranks ) { double unorm, global_unorm; - float sendbuf, recvbuf; MPI_Status mpi_status; // Calculate one timestep @@ -51,11 +50,9 @@ double poisson_step( // Ranks with odd number send first // Send data down from rank to rank-1 - sendbuf = unew[1]; - MPI_Send(&sendbuf, 1, MPI_FLOAT, rank-1, 1, MPI_COMM_WORLD); + MPI_Send(&u[1], 1, MPI_FLOAT, rank-1, 1, MPI_COMM_WORLD); // Receive dat from rank-1 - MPI_Recv(&recvbuf, 1, MPI_FLOAT, rank-1, 2, MPI_COMM_WORLD, &mpi_status); - u[0] = recvbuf; + MPI_Recv(&u[0], 1, MPI_FLOAT, rank-1, 2, MPI_COMM_WORLD, &mpi_status); if (rank != (n_ranks-1)) { // Send data up to rank+1 (if I'm not the last rank) diff --git a/high_performance_computing/hpc_mpi/code/solutions/09-poisson-mpi.c b/high_performance_computing/hpc_mpi/code/solutions/09-poisson-mpi.c deleted file mode 100644 index 457cf986..00000000 --- a/high_performance_computing/hpc_mpi/code/solutions/09-poisson-mpi.c +++ /dev/null @@ -1,156 +0,0 @@ -/* A serial code for Poisson equation - * This will apply the diffusion equation to an initial state - * untill an equilibrium state is reached. */ - -/* contact seyong.kim81@gmail.com for comments and questions */ - -#include -#include -#include -#include -#include - -#define MAX_ITERATIONS 25000 -#define GRIDSIZE 12 - - -/* Apply a single time step */ -double poisson_step( - float *u, float *unew, float *rho, - float hsq, int points, - int rank, int n_ranks -) { - double unorm, global_unorm; - - float sendbuf, recvbuf; - - MPI_Status mpi_status; - - // The u field has been changed, communicate it to neighbours - // With blocking communication, half the ranks should send first - // and the other half should receive first - - if ((rank%2) == 1) { - // Ranks with odd number send first - - // Send data down from rank to rank-1 - sendbuf = unew[1]; - MPI_Send(&sendbuf, 1, MPI_FLOAT, rank-1, 1, MPI_COMM_WORLD); - // Receive dat from rank-1 - MPI_Recv(&recvbuf, 1, MPI_FLOAT, rank-1, 2, MPI_COMM_WORLD, &mpi_status); - u[0] = recvbuf; - - - if (rank != (n_ranks-1)) { - // Send data up to rank+1 (if I'm not the last rank) - MPI_Send(&u[points], 1, MPI_FLOAT, rank+1, 1, MPI_COMM_WORLD); - // Receive data from rank+1 - MPI_Recv(&u[points+1], 1, MPI_FLOAT, rank+1, 2, MPI_COMM_WORLD, &mpi_status); - } - } else { - // Ranks with even number receive first - if (rank != 0) { - // Receive data from rank-1 (if I'm not the first rank) - MPI_Recv(&u[0], 1, MPI_FLOAT, rank-1, 1, MPI_COMM_WORLD, &mpi_status); - // Send data down to rank-1 - MPI_Send(&u[1], 1, MPI_FLOAT, rank-1, 2, MPI_COMM_WORLD); - } - - if (rank != (n_ranks-1)) { - // Receive data from rank+1 (if I'm not the last rank) - MPI_Recv(&u[points+1], 1, MPI_FLOAT, rank+1, 1, MPI_COMM_WORLD, &mpi_status); - // Send data up to rank+1 - MPI_Send(&u[points], 1, MPI_FLOAT, rank+1, 2, MPI_COMM_WORLD); - } - } - - // Calculate one timestep - for (int i = 1; i <= points; i++) { - float difference = u[i-1] + u[i+1]; - unew[i] = 0.5 * (difference - hsq*rho[i]); - } - - // Find the difference compared to the previous time step - unorm = 0.0; - for (int i = 1;i <= points; i++) { - float diff = unew[i]-u[i]; - unorm += diff*diff; - } - - // Overwrite u with the new field - for (int i = 1;i <= points; i++) { - u[i] = unew[i]; - } - - MPI_Allreduce(&unorm, &global_unorm, 1, MPI_DOUBLE, MPI_SUM, MPI_COMM_WORLD); - - return global_unorm; -} - - -int main(int argc, char** argv) { - int rank, n_ranks, rank_gridsize; - float *resultbuf; - - MPI_Init(&argc, &argv); - - // Find the number of slices calculated by each rank - // The simple calculation here assumes that GRIDSIZE is divisible by n_ranks - MPI_Comm_rank(MPI_COMM_WORLD, &rank); - MPI_Comm_size(MPI_COMM_WORLD, &n_ranks); - rank_gridsize = GRIDSIZE / n_ranks; - - // Test that the grid can be subdivided between the ranks properly - if (rank == 0) { - assert(GRIDSIZE % n_ranks == 0); - } - - // The heat energy in each block - float *u, *unew, *rho; - float h, hsq; - double unorm, residual; - int i; - - u = malloc(sizeof(*u) * (rank_gridsize + 2)); - unew = malloc(sizeof(*unew) * (rank_gridsize + 2)); - rho = malloc(sizeof(*rho) * (rank_gridsize + 2)); - - // Set up parameters - h = 0.1; - hsq = h*h; - residual = 1e-5; - - // Initialise the u and rho field to 0 - for (i = 0; i <= rank_gridsize+1; i++) { - u[i] = 0.0; - rho[i] = 0.0; - } - - // Create a start configuration with the heat energy - // u=10 at the x=0 boundary for rank 0 - if (rank == 0) - u[0] = 10.0; - - // Run iterations until the field reaches an equilibrium - // and no longer changes - for (i = 0; i < MAX_ITERATIONS; i++) { - unorm = poisson_step(u, unew, rho, hsq, rank_gridsize, rank, n_ranks); - if (sqrt(unorm) < sqrt(residual)) - break; - } - - // Gather results from all ranks - // We need to send data starting from the second element of u, since u[0] is a boundary - resultbuf = malloc(sizeof(*resultbuf) * GRIDSIZE); - MPI_Gather(&u[1], rank_gridsize, MPI_FLOAT, resultbuf, rank_gridsize, MPI_FLOAT, 0, MPI_COMM_WORLD); - - if (rank == 0) { - printf("Final result:\n"); - for (int j = 0; j < GRIDSIZE; j++) { - printf("%d-", (int) resultbuf[j]); - } - printf("\nRun completed in %d iterations with residue %g\n", i, unorm); - } - - MPI_Finalize(); -}