mpi: Add basic2 and basic3 modes

[georgebisbas]

mpi: Add basic2 and basic3 modes

Summary

This PR adds two new MPI halo-exchange schemes, both variants of basic
that avoid re-allocating gather/scatter buffers on every halo exchange
call:

• basic2 — basic, but the send/recv buffers are allocated once
  (via a pre-allocated MPIMsg-style C struct, MPIMsgBasic2) instead of
  malloc'd and freed on every single haloupdate() call.
• basic3 — basic2, further optimized the same way overlap2
  optimizes overlap: ranks and gather/scatter offsets are precomputed
  once in Python and read directly off the message struct
  (MPIMsgEnrichedBasic2) inside a single Iteration loop, instead of
  being recomputed via MPINeighborhood/symbolic-offset lookups at each
  of the 2*ndim unrolled sendrecv() call sites basic2 generates.

Both are selected the same way as the existing modes: DEVITO_MPI=basic2
/ DEVITO_MPI=basic3.

Why

basic's gather/scatter buffers are heap-allocated and freed on every
single halo exchange — i.e. every timestep, for every halo-touching
Function. For runs with many timesteps this is pure overhead: the buffer
shape for a given (Function, halo direction) pair never changes across
the run. basic2/basic3 allocate it once, mirroring what overlap/
overlap2/full already do for the asynchronous modes, but for basic's
synchronous, non-overlapped exchange pattern.

basic3 additionally cuts down haloupdate()'s own code size: basic/
basic2 emit one unrolled sendrecv() call per (dimension, side) pair,
so the generated code grows with the number of exchange directions.
basic3 collapses that into one small loop, the same way overlap2 is
more compact than overlap:

 static void haloupdate0(struct dataobj *restrict f_vec, MPI_Comm comm,
-                        struct neighborhood * nb, int otime, struct msg * msg0)
+                        struct msg * msg0, int ncomms, int otime)
 {
-  sendrecv0(f_vec,otime,f_vec->oofs[2],f_vec->hofs[4],otime,f_vec->hofs[3],
-            f_vec->hofs[4],nb->rc,nb->lc,comm,&msg0[0]);
-  sendrecv0(f_vec,otime,f_vec->oofs[3],f_vec->hofs[4],otime,f_vec->hofs[2],
-            f_vec->hofs[4],nb->lc,nb->rc,comm,&msg0[1]);
-  sendrecv0(f_vec,otime,f_vec->hofs[2],f_vec->oofs[4],otime,f_vec->hofs[2],
-            f_vec->hofs[5],nb->cr,nb->cl,comm,&msg0[2]);
-  sendrecv0(f_vec,otime,f_vec->hofs[2],f_vec->oofs[5],otime,f_vec->hofs[2],
-            f_vec->hofs[4],nb->cl,nb->cr,comm,&msg0[3]);
+  for (int i = 0; i <= ncomms - 1; i += 1)
+  {
+    MPI_Irecv(msg0[i].bufs,msg0[i].sizes[0]*msg0[i].sizes[1],MPI_FLOAT,
+              msg0[i].fromrank,13,comm,&msg0[i].rrecv);
+    if (msg0[i].torank != MPI_PROC_NULL)
+      gather0((float*)(msg0[i].bufg),msg0[i].sizes[0],msg0[i].sizes[1],
+              f_vec,otime,msg0[i].ofsg[0],msg0[i].ofsg[1]);
+    MPI_Isend(msg0[i].bufg,msg0[i].sizes[0]*msg0[i].sizes[1],MPI_FLOAT,
+              msg0[i].torank,13,comm,&msg0[i].rsend);
+    MPI_Wait(&msg0[i].rsend,MPI_STATUS_IGNORE);
+    MPI_Wait(&msg0[i].rrecv,MPI_STATUS_IGNORE);
+    if (msg0[i].fromrank != MPI_PROC_NULL)
+      scatter0((float*)(msg0[i].bufs),msg0[i].sizes[0],msg0[i].sizes[1],
+               f_vec,otime,msg0[i].ofss[0],msg0[i].ofss[1]);
+  }
 }

haloupdate0's code size is now independent of the number of exchange
directions — this matters more as dimensionality grows (3D, subdomains,
tensor/vector equations with several components), where basic2 would
otherwise keep emitting more unrolled calls. func_table drops from 4
entries (gather0, scatter0, sendrecv0, haloupdate0) under basic2
to 3 under basic3 (no separate sendrecv0).

What changed

• devito/mpi/routines.py:
  • MPIMsgBase/MPIMsg/MPIMsgBasic2/MPIMsgEnriched: buffer
    allocation now goes through infer_datasize() for overflow-safe
    sizing, and lazily pulls the allocator from args.allocator — the
    convention main moved to since this branch was first opened.
  • Basic2HaloExchangeBuilder: _make_haloupdate now delegates to the
    shared BasicHaloExchangeBuilder._make_haloupdate instead of
    duplicating its mapper/loop logic just to inject a haloid;
    BasicHaloExchangeBuilder._make_haloupdate threads haloid through
    generically (a no-op for plain basic), matching how
    DiagHaloExchangeBuilder already does it.
  • MPIMsgBasic2/MPIMsgEnrichedBasic2's _make_msg now filters
    hse.halos down to axis-aligned pairs only (diagonal corners are
    filled implicitly by basic's sequential per-axis exchange order and
    were previously counted towards npeers without ever being
    allocated).
  • New: Basic3HaloExchangeBuilder and MPIMsgEnrichedBasic2.
  • mpi_registry gains 'basic2' and 'basic3'.
• tests/test_mpi.py: basic2/basic3 added to the relevant existing
  parametrizations (test_trivial_eq_2d/3d, test_coupled_eqs_mixed_dims,
  test_cire, test_adjoint_F, test_min_code_size), plus new tests:
  • test_trivial_eq_2d_bundled — two same-pattern TimeFunctions
    forcing Devito's Bundle/Bag packing (a feature added to main after
    this branch first forked) through MPIMsgBasic2/MPIMsgEnrichedBasic2.
  • test_basic2_msg_fields — locks in MPIMsgEnrichedBasic2's field set.
  • test_basic2_comm_scheme / test_basic3_comm_scheme — white-box
    checks on the generated sendrecv/haloupdate structure for each
    mode, in the style of the existing test_diag_comm_scheme/
    test_poke_progress.
• benchmarks/user/README.md: mentions basic2 alongside basic/
  diag2/full as one of Devito's three most prevalent MPI modes.

Testing

• tests/test_mpi.py: 239 passed, 0 failed (local run, OpenMPI 4.1.6).
• pytest -m parallel tests/ (the broader suite CI's pytest-core-mpi
  workflow runs, not just test_mpi.py): 648 passed, 3 skipped,
  1 xpassed (pre-existing, unrelated xfail), 0 failed.
• DEVITO_MPI=1 mpirun -n 2 pytest examples/seismic/{acoustic,tti} (the
  workflow's MPI-examples step): 20/20 passed on both ranks — exercises
  the shared BasicHaloExchangeBuilder._make_haloupdate code path this
  PR touches, under plain basic.
• Numerical correctness cross-checked bit-for-bit against basic/diag/
  overlap/full at 1, 2, 3, 4, and 8 ranks; 2D and 3D; space_order 2
  and 4; with bundled multi-field targets; and on a real acoustic seismic
  example (norm(rec) matches to full float32 precision across every
  mode and rank count tested).
• Repeated stress runs (8+ repetitions at 4 and 8 ranks) show no crashes.

Not verified locally: CI's pytest-core-mpi job uses MPICH; this was
tested locally against OpenMPI 4.1.6 (MPICH wasn't installable
without sudo in this environment). Neither mode uses anything beyond
the portable MPI API already exercised identically by the working
overlap/diag2 paths (MPI_Isend/Irecv/Wait, MPI_PROC_NULL,
Comm_shift, Get_cart_rank), so this is expected to be a non-issue,
but CI will be the first real confirmation on MPICH.