Metadomain, subdomains and meshes

Relevant headers

• framework/domain/metadomain.h
• framework/domain/domain.h
• framework/domain/grid.h
• framework/domain/mesh.h

The main object which contains all the data and configuration of the simulation, including domain decomposition, the mesh discretization, the metric, the fields and the particles is the Metadomain<S,M> class (where S is the simulation engine template argument, and M is the metric class). From the metadomain object, we can access all the subdomains Metadomain<S,M>::subdomain(idx) -> const Domain<S,M>& (::subdomain_ptr(idx) -> Domain<S,M>*; idx is the index of the subdomain), the global mesh Metadomain<S,M>::mesh() -> const Mesh<M>&, and the global metric Metadomain<S,M>::mesh().metric -> const Metric<D>&. Each subdomain also contains a mesh and a metric object, which can be accessed with Domain<S,M>::mesh and Domain<S,M>::mesh.metric. Fields and particle species are stored for each subdomain: Domain<S,M>::fields and Domain<S,M>::species.

Thus, the hierarchy of data objects is as follows:

Metadomain<S,M>
├── Mesh<M> : g_mesh
└── [Domain<S,M>, Domain<S,M>, ...] : g_subdomains
     ├── Mesh<M> : mesh
     │   └── Metric<D> : metric
     ├── Fields<D,S> : fields
     └── [Particles<D,S>, ...] : species

Remember, that the local code-unit coordinates in Entity go from \(0\) to \(n_i\) (where \(n_i\) is the number of cells in the \(i\)-th direction). Because of that, metrics and coordinate systems on all subdomains can be different. However, the code guarantees that the transitions between the subdomains is continuous, and no vector transformations are needed when passing from one subdomain to another (coordinate transformations still have to be done).

Local vs non-local subdomains

While the metadomain object which exists on all MPI ranks contains information on all the subdomains of the global simulation, the data for fields and particles is only allocated for the so-called "local" subdomains, owned by the current MPI rank. To get the indices of the subdomains on the local MPI rank, use Metadomain<S,M>::local_subdomain_indices() -> std::vector<unsigned int>. To run a specific function on all the local subdomains use the following lambda construct:

metadomain.runOnLocalDomains([&](auto& loc_dom) {
  // do something with loc_dom
});

If you do not plan to modify the local subdomains, you should use the const version of this function:

metadomain.runOnLocalDomainsConst([&](auto& loc_dom) {
  // do something with loc_dom without modifying it
});

Non-local subdomains are also referred to as placeholders, and one could check whether a given subdomain is a placeholder using the built-in is_placeholder() method.

The diagram below demonstrates the structure of the metadomain object, the subdomains, and the mesh with all the contained variables and methods. Fields and particles are described in the following section, while the metrics are discussed here.

classDiagram
  class MetricBase~Dimension~ {
    see metrics...*
  }
  class Fields~Dimension, SimEngine~ {
    see fields...*
  }
  class Particles~Dimension, Coord~ {
    see particles...*
  }
  class Metadomain~SimEngine, class~ {
    +Dimension D$
    -uint g_ndomains
    -vector~int~ g_decomposition
    -vector~uint~ g_ndomains_per_dim
    -vector~vector~uint~~ g_domain_offsets
    -map~vector~uint~, uint~ g_domain_offset2index
    -vector~Domain~S, M~~ g_subdomains
    -vector~uint~ g_local_subdomain_indices
    -Mesh~M~ g_mesh
    -const map~string, real_t~ g_metric_params
    -const vector~ParticleSpecies~ g_species_params
    -stats\:\:Writer g_stats_writer
    +initialValidityCheck() void
    +finalValidityCheck() void
    +metricCompatibilityCheck() void
    +createEmptyDomains() void
    +redefineNeighbors() void
    +redefineBoundaries() void
    +CommunicateFields(Domain~S, M~ &, CommTags ) void
    +SynchronizeFields(Domain~S, M~ &, CommTags ) void
    +CommunicateParticles(Domain~S, M~ &) void
    +RemoveDeadParticles(Domain~S, M~ &) void
    +InitStatsWriter(SimulationParams &, bool ) void
    +WriteStats(SimulationParams &, timestep_t , timestep_t , simtime_t , simtime_t ) bool
    +setFldsBC(bc_in &, FldsBC &) void
    +setPrtlBC(bc_in &, PrtlBC &) void
    +ndomains() uint
    +ndomains_per_dim() vector~uint~
    +subdomain(uint idx) const Domain~S, M~&
    +subdomain_ptr(uint idx) Domain~S, M~*
    +mesh() const Mesh~M~&
    +species_params() const vector~ParticleSpecies~&
    +l_subdomain_indices() vector~uint~
    +l_npart_perspec() vector~npart_t~
    +l_maxnpart_perspec() vector~npart_t~
    +l_npart() size_t
    +l_ncells() size_t
    +species_labels() vector~string~
  }
  class Domain~SimEngine, class~ {
    +Dimension D$
    +Mesh~M~ mesh
    +Fields~D, S~ fields
    +vector~Particles~D, M\:\:CoordType~~ species
    +random_number_pool_t random_pool
    -uint m_index
    -vector~uint~ m_offset_ndomains
    -vector~ncells_t~ m_offset_ncells
    -dir\:\:map_t~D, uint~ m_neighbor_idx
    -int m_mpi_rank
    +index() uint
    +offset_ndomains() vector~uint~
    +offset_ncells() vector~ncells_t~
    +neighbor_idx_in(dir\:\:direction_t~D~ & dir) uint
    +is_placeholder() bool
    +set_neighbor_idx(dir\:\:direction_t~D~ & dir, uint idx) void
  }
  class Grid~Dimension~ {
    #vector~ncells_t~ m_resolution
    +i_min(in i) ncells_t
    +i_max(in i) ncells_t
    +n_active(in i) ncells_t
    +n_active() vector~ncells_t~
    +num_active() ncells_t
    +n_all(in i) ncells_t
    +n_all() vector~ncells_t~
    +num_all() ncells_t
    +rangeActiveCells() range_t~D~
    +rangeAllCells() range_t~D~
    +rangeCells(box_region_t~D~ &) range_t~D~
    +rangeCells(tuple_t~list_t~int, 2~, D~ &) range_t~D~
    +rangeActiveCellsOnHost() range_h_t~D~
    +rangeAllCellsOnHost() range_h_t~D~
    +rangeCellsOnHost(box_region_t~D~ &) range_h_t~D~
  }
  class Mesh~class~ {
    +bool is_mesh$
    +Dimension D$
    +M metric
    -boundaries_t~real_t~ m_extent
    -dir\:\:map_t~D, FldsBC~ m_flds_bc
    -dir\:\:map_t~D, PrtlBC~ m_prtl_bc
    +Intersection(boundaries_t~real_t~ box) boundaries_t~real_t~
    +Intersects(boundaries_t~real_t~ box) bool
    +ExtentToRange(boundaries_t~real_t~ box, boundaries_t~bool~ incl_ghosts) boundaries_t~ncells_t~
    +extent(in i) real_t,real_t
    +extent() boundaries_t~real_t~
    +flds_bc() boundaries_t~FldsBC~
    +prtl_bc() boundaries_t~PrtlBC~
    +flds_bc_in(dir\:\:direction_t~D~ & direction) FldsBC
    +prtl_bc_in(dir\:\:direction_t~D~ & direction) PrtlBC
    +set_flds_bc(dir\:\:direction_t~D~ & direction, FldsBC & bc) void
    +set_prtl_bc(dir\:\:direction_t~D~ & direction, PrtlBC & bc) void
  }

  Domain --* Mesh : contains
  Grid <|-- Mesh : inherits
  Mesh --* MetricBase : contains
  Metadomain --* Domain : contains many
  Metadomain --* Mesh : contains
  Domain --* Fields : contains
  Domain --* Particles : contains many

  note "+: public\n-: private\n#: protected\nunderline: static constexpr\nitalic: virtual"