LargestSimulations

The aim of this page is to give a feeling of how large light scattering simulations can be done using ADDA. Hopefully, it will also present a variety of different applications, so that each example will tell an interesting short story by itself. Complexity of the light scattering simulation with ADDA depends on the many (connected) factors: scatterer size parameter, required discretization or accuracy, number of the dipoles in a computational grid, convergence speed of the iterative solver, and need for repeated calculations (e.g. orientation averaging). Therefore, this page consists of several categories (nominations) with several examples in each of them.

• Overview
• Largest size parameter
  • X1. Sphere with size parameter 320 and refractive index 1.05
  • X2. Aggregate of 1 million spheres
• Largest number of dipoles
  • N1. Truncated slab
  • N2. Red blood cell with very fine discretization
• Largest processor time
  • T1. Sphere with size parameter 130 and refractive index 1.2
  • T2. Aggregate of 10⁵ spheres, averaged over orientations

Overview

Currently there are only a very small number of examples. Users are encouraged to submit new examples based on their own work/experience. This can be done either by a comment to this wiki page or contacting the maintainer of this page directly. Before proceeding to the description of particular examples, we present a summary table.

Parameter	X1	X2	N1	N2	T1	T2
Shape	sphere	aggregate	slab	red blood cell	sphere	aggregate
Volume-equivalent size parameter	320	176	16	36	130	82
Refractive index (average)	1.05	1.5+10−4i	1.31+0.1i	1.03	1.2	1.5+10−4i
Dipoles per wavelength	10	10	257	93	12	10
Largest 1D grid size	1024	958	8192	1408	512	444
Number of dipoles in a grid, 106	1073	879	1073	1035	134	87
Number of occupied dipoles, 106	562	92	1073	619	70	9.2
Relative residual norm (-eps)	10−4	10−3	10−5	10−5	10−5	10−2
Iterative solver used	Bi-CGStab	QMR	QMR	QMR	Bi-CGStab	QMR
Number of iterations (average)	334	9200	23	10	29200	1100
Number of runs of iterative solver	1	2	2	1	1	116
Number of processor cores	512	1008	4096	560	64	240
Processor time, THz×hours	8.8	41	0.2	1.1	90	35
Total memory used, GiB	698	662	1093	772	71	65
ADDA version	0.79a2	1.3b4	1.2	0.79	0.75b	1.3b4
Supercomputer	MareNostrum	Sisu	Sisu	LISA	LISA	Sisu

The values in bold are the largest (the hardest to reach) ones for a particular parameter among all examples

Largest size parameter

X1. Sphere with size parameter 320 and refractive index 1.05

This was a benchmark simulation, performed to test the performance and extreme capabilities of ADDA on MareNostrum. It is published in Yurkin M.A. and Hoekstra A.G. The discrete-dipole-approximation code ADDA: capabilities and known limitations, J. Quant. Spectrosc. Radiat. Transfer 112, 2234–2247 (2011). ADDA was executed with a command line:

adda_mpi -m 1.05 0 -grid 1024 -size 640 -ntheta 1800 -iter bicgstab -eps 4

X2. Aggregate of 1 million spheres

The aggregates consisisted of 1 million spheres with size parameter 1.76 and refractive index 1.5 + 0.0001i, packed with volume fraction 20%. Outer shape of the aggregate is spherical with circumscribing size parameter 301. ADDA was executed with a command line:

adda_mpi -m 1.5 0.0001 -shape read n1000000_dpl10.geom -dpl 10.0003 -ntheta 1800 -eps 3

This is the maximum size we could reach, but it was only for a single aggregate in fixed orientation. Thus, it was not included in the paper in contrast to the results for 10⁵ spheres, where we could peform orientation averaging.

Largest number of dipoles

N1. Truncated slab

This was a part of a scalability study of ADDA on Sisu, performed by Antti Penttila. The overall parallel efficiency for the largest number of cores (4096) was estimated to be 63%, but it was 77% and 83% for 2048 and 512 cores respectively. ADDA was executed with a command line:

adda_mpi -size 100 -m 1.31 0.1 -grid 4096 32 8192 -shape box 0.0078125 2 

The size was chosen relatively small (corresponding to very large number of dipoles per wavelength) to keep the simulations fast. However, at least 10 times larger sizes seem feasible with such hardware. In particular, such simulations can be used to study optical properties of effectively infinite inhomogeneous (particulate) slabs.

N2. Red blood cell with very fine discretization

This simulation was a part of comparison of two methods for simulation of light scattering, published in Gilev K.V., Eremina E., Yurkin M.A., and Maltsev V.P. Comparison of the discrete dipole approximation and the discrete source method for simulation of light scattering by red blood cells, Opt. Express 18, 5681–5690 (2010).

Since both considered methods involve certain numerical errors, there was need for some reference, so that accuracy of the methods can be independently evaluated. Described simulation for very fine discretization was used as such a reference. ADDA was executed with a command line:

adda_mpi -lambda 0.4936 -m 1.03 0.0 -shape axisymmetric fung_100points.txt -size 7.5 -grid 1408 -scat_grid_inp scat_params_eryth.dat -yz

Moreover, convergence of DDA results with refining discretization was studied to estimate the accuracy of the reference itself.

Largest processor time

T1. Sphere with size parameter 130 and refractive index 1.2

This simulation was performed in frame of a benchmark study: Yurkin M.A., Maltsev V.P., and Hoekstra A.G. The discrete dipole approximation for simulation of light scattering by particles much larger than the wavelength, J. Quant. Spectrosc. Radiat. Transfer 106, 546–557 (2007). ADDA was executed 25 times in a row with a command line:

adda_mpi -m 1.2 0 -size 260 -grid 512 -ntheta 1440 -iter bicgstab -chpoint 17h [-chp_load]

where the last option was used for every incantation except the first one. Use of checkpoint system was necessary, since at that time no single job could use more than 1200 processor-hours on LISA.

T2. Aggregate of 10⁵ spheres, averaged over orientations

The aggregates consisisted of 100 000 spheres with size parameter 1.76 and refractive index 1.5 + 0.0001i, packed with volume fraction 20%. Outer shape of the aggregate is spherical with circumscribing size parameter 140. The orientation of the aggregate was averaged, by performing 58 independent ADDA simulations and using 512 scattering planes for each orientation. ADDA was executed with a command line:

adda_mpi -m 1.5 0.0001 -shape read 1h_10dpl.geom -dpl 10.0 -ntheta 1800 -eps 2 -orient avg avg-all.dat

The results were compared with other methods in Penttilä A., Markkanen J., Väisänen T., Räbinä J., Yurkin M.A., and Muinonen K. How much is enough? The convergence of finite sample scattering properties to those of infinite media, J. Quant. Spectrosc. Radiat. Transfer 262, 107524 (2021). One of the graphs is shown on another wiki page. Even larger aggregate has been simulated, but only in a fixed orientation.