Kratos: [MappingApplication] Is there a missing synchronize in MPI?

[marcnunezc]

Description
Regarding #7728, I have been trying to do the following.

All run in MPI:

• Run a FluidDyn simulation (initial_model_part)
• Remesh with ParMmg (getting a main_model_part)
• Map the velocity and pressure fields from initial model_part to the main_model_part
• Run the FluidDyn simulation with the new mesh and fields (main_model_part).

What I am experiencing is that after mapping the fields, the new simulation genereates weird features in the field. This are caused due to a lack of synchornization of the values of the velocity across ranks.

This is the correct mapped field.

Field after some seconds of simulation (see the peaks on the velocity field).

If after performing the mapping I run the lines:

main_model_part.GetCommunicator().SynchronizeVariable(KratosMultiphysics.VELOCITY)
main_model_part.GetCommunicator().SynchronizeVariable(KratosMultiphysics.PRESSURE)

The weird effects do not happen.

If the velocity is compared among ranks, it can be seen that without the synchronization, different ranks have different values of VELOCITY on the same node.

The mapping parameters that I use are:

mapper_settings = KratosMultiphysics.Parameters("""{
"mapper_type": "nearest_element",
"interface_submodel_part_origin": "Parts_Fluid",
"interface_submodel_part_destination": "Parts_Fluid",
"search_radius" : 0.01,
"echo_level" : 0
}""")

# creating a mapper for distributed memory
mpi_mapper = KratosMapping.MapperFactory.CreateMPIMapper(
    initial_main_model_part,
    main_model_part,
    mapper_settings)
mpi_mapper.Map(KratosMultiphysics.VELOCITY, KratosMultiphysics.VELOCITY)
mpi_mapper.Map(KratosMultiphysics.PRESSURE, KratosMultiphysics.PRESSURE)

@philbucher Have you experienced this? Is there anything am I doing wrong? Is it expected that the user synchronizes the variables after mapping?

Scope

• Mapping application
• Fluid Dynamics application

To Reproduce
Find the script in this zip (run with 4 procs on master branch).

mapping_sync_cylinder_test.zip

Lines 126 and 127 are commented (which perform the sync). MMG/ParMMG is not required

Expected behavior
That variables are synchronized after mapping.

Environment
Please specify your operating system, kratos branch, python version or any other library versions that you consider relevant.
E.g.:

• OS: Ubuntu 18.04
• Branch: master
• Python 3.6.9

[philbucher]

Hi @marcnunezc
Yes this is known and has been discussed before with @roigcarlo and @RiccardoRossi. We were not sure if the sync should be done in the Mapper or not.
Since it can cause confusion I can also add it, I don't mind

any objections @roigcarlo @RiccardoRossi ?

[roigcarlo]

Not from my side, also ping @ddiezrod . They use it in their code so if you add it they must remove it.

[RiccardoRossi]

I didn't remember the discussion...please go ahead for me

Riccardo

On Tue, Nov 10, 2020 at 7:34 PM Carlos Roig [e-mail]
wrote:

Not from my side, also ping @ddiezrod https://github.com/ddiezrod .
They use it in their code so if you add it they must remove it.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/KratosMultiphysics/Kratos/issues/7780#issuecomment-724887086,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB5PWEL42TOBV2FTLM5G4L3SPGBUPANCNFSM4TQYTADQ
.

[philbucher]

Ok seems like we are in agreement.
I will add the sync in the mapper then, I also don#t remember why we didn't do it in the first place...

I will leave this issue open until I fixed it

[rubenzorrilla]

Just mentioning that I'd also do the sync in the map. I think that the user will assume this is done in the vast majority of cases.