Reduce cost of 4th order hydro EOS and flux calculations

[felker]

Following a discussion with @Yurlungur, the high order method in Athena++ could be made a lot cheaper; we could nearly half the calls to EOS function and Riemann solver at the cost of additional ghost cell calculations and inter-MeshBlock communication.

Following McCorquodale and Collela (2011) and Guzik (2015), we have been computing the 4th order cell-averaged primitives by approach 1 in this sketch:

The previous authors chose this to reduce the stencil size, although I am not sure why that was the primary concern:

Approach 2 in the sketch is to communicate an additional layer of ghost cells and compute 4th order accurate pointwise U_i and W_i even in the first layer of ghost cells, but then you do not need to compute \bar{W}_i anywhere. This becomes less efficient for very small MeshBlocks and/or inexpensive EOS calls. And it brings corner ghost cells into the calculation, but we already have to worry about that in 2nd or 4th order CT.

• Why does Guzik say that "the stencil to compute W_i has radius 3"--- is it not "width 3"?
• Are we missing some disadvantage to switching to approach 2?

[Yurlungur]

One thing I'd add here, is I do think whether or not this is worth it depends on the cost of communication---at least for pure hydro. In the full 4th-order CT, I think you're right, we may already be paying the price.

My thinking is that it's a trade-space between computation and communication. If communication is much more expensive than computation, then a wider ghost halo (or additional comm steps) is basically always bad. But if communication is of comaprable cost, then the ratio real cells to ghost cells is probably what matters, which depends on halo size, dimension, and block size. I think that for 4th-order CT or for scenarios where con2prim is very expensive (as it is for multi-material hydro or GR), then probably it's better to avoid computing $\bar{W}_i$.