Many simulations from science and engineering involve one or more grids and the discretizations (finite element, volume, or difference) of partial differential equations on this or these grids. Depending on the application we use regular, irregular, or block-structured grids. The latter form a hybrid type in between the other two. We create an irregular decomposition of the the domain into a set of blocks, but inside each block a regular grid is used. Block-structured grids are very popular for conservative discretizations in Computational Fluid Dynamics problems involving complex geometries.
We discuss the implementation and performance of HPF-based iterative solvers for each type of grid. For regular and irregular grids (and the resulting structured and unstructured matrices) we reported very good results in [1]. We extend these results and we also discuss implementations for block-structured grids. On the one hand the structure inside a block helps creating efficient implementations. On the other hand the two-level structure allows various choices in implementation that may yield large differences in performance for a given problem. The performance depends strongly on key parameters like the number of processors, the number of blocks, the sizes of the blocks in grid points or unknowns (which may vary significantly over the blocks), the interconnectivity of the blocks, and others. Therefore, a judicious choice in implementation is necessary, and we discuss the relevant issues. We show that for all types of grid-based problems, in principle, efficient HPF-based parallel programs are possible.