Kratos: How does the Metis reorder for MPI?

[philbucher]

Hi @roigcarlo
I have the following problem: I want to write a test that works both in serial and in MPI. I am using the Node-Ids for checking the results. Now the problem is that the ids are different in serial and MPI. I know that they are being reordered, which is why I also use the reorder-IO in serial
I though this would reorder the IDs in the same way, and hence they would be the same in both cases, but this unfortunately is not the case.
To me it seems like there is some additional reordering happening inside the partitioning (I tried both file-based and in-memory)
Do you have an idea what is happening? Or am I missing sth?
Thx
FYI @jcotela

[roigcarlo]

Hi @philbucher

If I am right (please someone confirm) the reorder-io does a reordering prior to the metis process, and has nothing to do with partitions. It only ensures that the id's of the entities are in order and consecutive.

Metis itself as far as I know does not do by itself any reordering, but some processes associated with the partitioning do. I am specially aware of the re-balancing process doing it for example, which does another reordering which will have a different result.

[philbucher]

thanks for the info.
do you think this reordering could be somehow excluded such that it can also be used standalone?
From what you say it seems to be rather specific right?

[roigcarlo]

I can look into it, could you give me a more specific example of what you are trying to accomplish? (maybe a WIP branch?)

[philbucher]

Thanks!
see #4319
I want to create a test that can be run in serial and MPI. For this I am trying to create a reference-file, which requires the node-ids to be the same in serial and MPI.

[pooyan-dadvand]

As far as I remember, the reorderer should not change anything if all the numberings are consecutive and starting from 1.

[philbucher]

I am pretty convinced the reorder-IO does not change the numbering if the Ids are already consecutive

However there seems to be some reordering happening inside the partitioning, thats what @roigcarlo wants to look into

If this can be figured out, then writing tests that work for serial and MPI will become quite easy :tada:

[philbucher]

I played around a bit and came up with a minimal example:
mdpa before partitioning:
~~~
Begin Properties 0

End Properties

Begin Nodes
1 -0.25 -0.75 0
2 -0.25 0.75 0
3 1.25 -0.75 0
4 1.25 0.75 0
End Nodes

Begin Elements Element2D3N
1 0 1 4 2
2 0 1 3 4
End Elements
~~~

partitioning into two partitions (partition 0 gets nothing, which I think is related to the example, with bigger examples it gets more evenly distributed)
partition 0:
~~~
Begin Properties 0

End Properties
Begin Nodes
End Nodes
Begin Elements Element2D3N
End Elements
Begin NodalData PARTITION_INDEX
End NodalData
Begin CommunicatorData
NEIGHBOURS_INDICES 4
NUMBER_OF_COLORS 1
Begin LocalNodes 0
End LocalNodes
Begin GhostNodes 0
End GhostNodes
End CommunicatorData
~~~

partition 1:
~~~
Begin Properties 0

End Properties
Begin Nodes
1 -0.25 -0.75 0
3 -0.25 0.75 0
4 1.25 -0.75 0
2 1.25 0.75 0
End Nodes
Begin Elements Element2D3N
1 0 1 2 3
2 0 1 4 2
End Elements
Begin NodalData PARTITION_INDEX
1 0 1
2 0 1
3 0 1
4 0 1
End NodalData
Begin CommunicatorData
NEIGHBOURS_INDICES 4
NUMBER_OF_COLORS 1
Begin LocalNodes 0
1
2
3
4
End LocalNodes
Begin GhostNodes 0
End GhostNodes
End CommunicatorData
~~~

As you can see the numbering in of the nodes changes. I had a close look at the functions involved, but I could not figure it out, it is unfortunately not straight forward.

One place though where potentially sth happens is in the ModelPartIO when the nodal graph is constructed

I used a small script to test it, you can easily reproduce the example above. Either you pass as argument the name of the mdpa file, or otherwise it creates a modelpart inside (default), writes it to file and uses it for the partitioning. Just have a look, I think it is quite easy to understand
test_partitioning.zip
It then outputs if the node/element/condition-ids changed in the partitioned case, which is currently happening.
I think we can use this script then later as a test to check that this won't happen again in the future

Another finding I made is that the IDs of Elements and Conditions does NOT change! Their connectivity does, as seen in the example above,

[philbucher]

BTW it is the same, independent if I partition in memory or not

[jcotela]

I will take a look too, assigning myself so that I don't loose track of the issue.

[jcotela]

OK I think I understand the problem...

@pooyan-dadvand is right that the order of the nodes should be preserved if the nodes are read first (which is what usually happens when reading a modelpart).

But (there is always a but):

The partitioners generate the connectivity for metis using ModelPartIO::ReadNodalGraph method, which reads the lists of Elements and Conditions only, so that the nodes end up reordered to the order in which they appear in the Elemental connectivity

The ReadNodalGraph method then does its own validation to ensure that nodes are numbered consecutively, which should not be necessary when using the ReorderConsecutiveModelPartIO.

An option would be to give the ReorderConsecutiveModelPartIO needs its own implementation of ReadNodalGraph which starts by processing the node list. In this case, the node list would already be automagically consecutive, so I would expect that the "validation" part of the ModelPartIO is no longer needed for the derived class.

I will do a PR implementing this.

[pooyan-dadvand]

Thanks @jcotela for resolving this mystery!!

[jcotela]

Actually, my proposed solution would require making quite a few of the internals of ModelPartIO protected... (Things like ResetInput and ReadWord....) so maybe it is not a great idea.