Ember: OTF-2 motif with Merlin's mesh topology

[SBA0486]

I have been using the OTF-2 trace replay functionality of Ember to simulate workloads of benchmark applications with Merlin. I though the condition for trace replay is that the number of SST nodes should match the number of ranks from the trace, however the simulations tend to hang when the mesh network topology is used, for example when replaying an 8-rank trace that was collected from a torus network into a 2x4 mesh network. The same trace is replayed fine when selecting 8 nodes from a dragonfly topology. I would like to ask if this is an issue with the OTF-2 trace replay or is it an issue specific to mesh networks?

[feldergast]

There's no way to know for sure without digging in a little deeper, but I would suspect it's an issue with the mesh topology. Probably something in the routing algorithm is quite right. Can you try a bigger mesh and just use 8 of the nodes? I'm wondering if there's something wrong with the dimension size of 2 routing.

[SBA0486]

The issue does not seem to be limited to the dimension size of 2. In fact all mesh topologies have the same problem, except when ranks are allocated using nodes on the same raw/column. Other topologies seem fine.

[SBA0486]

Setting up vcs_per_vn[i] = 2 instead of vcs_per_vn[i] = 1 in merlin/topology/mesh.h solved the issue .

[feldergast]

That would make sense. That was definitely an error in the mesh code. I'm going to reopen the issue to track this bug until we get the fix into the devel branch.