Kratos: [CFD] Simulation crashes - pontential cause: Normals

[mpentek]

I have been experiencing simulations crashing since a certain commit. The same model with input was working before. It is a CFD case where I use Slip boundary conditions. @AndreasWinterstein experienced similar, switching to NoSlip or going back one PR seems to solve the issue.

Still works: #6867
Does not work since: #6866

This seems to happen only in MPI. The typical symptom is, that the CFD with Fractional Step and Slip BC will run normally, converging in 2-3 iterations per time step. Then a certain time step starts already with NaNs. I repeated the tests for 2 different cases. They all fail at a very specific step since #6866 got merged. Before they ran through without a problem.

I can provide the models, but they need circa 15-30 minutes run on 144 cores, as they have around 6 million elements each and need 200-300 simulation steps to get to the point where they fail. I have yet to come up with a minimal solution. Also, I can only pinpoint the PR, but not the exact cause.

@AndreasWinterstein tested and observed similar
@sunethwarna might be affected, as RANS relies heavily on Slip condition

pontentially @rubenzorrilla could be affected as well

Adding @roigcarlo and @philbucher as you are definitely more suited to judge where this problem and solution needs to go to.

Again to summarize:

• Crashes with MPI (OpenMP works)
• Crashes when Slip (or Wall function which relies on Slip -> in my case I specifically use the FSGeneralizedWallCondition -> afaik this relies on neighbour elements so could it be related to #7000 -> the usage and dependency is made clearer here #6119) is used
• Crashed without showing any problems with convergence when running CFD with Fractional Step, but the convergence starts with NaNs for the fluid solve from a certain time step
• Happens with more complex examples (6 million elements, running on 3 nodes - 144 cores MPI)

Here is a snippet with the simulation failing:

            Fluid Dynamics Analysis: STEP:  315
            Fluid Dynamics Analysis: TIME:  6.299999999999954
            Fractional Step Strategy : : CONVERGENCE CHECK:
            Fractional Step Strategy : : FRAC VEL.: ratio = 0.00320083; exp.ratio = 1e-05 abs = 24.5245
            Fractional Step Strategy : : CONVERGENCE CHECK:
            Fractional Step Strategy : : FRAC VEL.: ratio = 2.44525e-06; exp.ratio = 1e-05 abs = 0.0187353
            Fractional velocity converged in 2 iterations.
            Calculating Pressure.
            Updating Velocity.
            Fluid Dynamics Analysis: STEP:  316
            Fluid Dynamics Analysis: TIME:  6.319999999999953
            Fractional Step Strategy : : CONVERGENCE CHECK:
            Fractional Step Strategy : : FRAC VEL.: ratio = nan; exp.ratio = 1e-05 abs = nan
            Fractional Step Strategy : : CONVERGENCE CHECK:
            Fractional Step Strategy : : FRAC VEL.: ratio = nan; exp.ratio = 1e-05 abs = nan

Here is a snippet with the simulation running:

            Fluid Dynamics Analysis: STEP:  315
            Fluid Dynamics Analysis: TIME:  6.299999999999954
            Fractional Step Strategy : : CONVERGENCE CHECK:
            Fractional Step Strategy : : FRAC VEL.: ratio = 0.00320083; exp.ratio = 1e-05 abs = 24.5245
            Fractional Step Strategy : : CONVERGENCE CHECK:
            Fractional Step Strategy : : FRAC VEL.: ratio = 2.44525e-06; exp.ratio = 1e-05 abs = 0.0187353
            Fractional velocity converged in 2 iterations.
            Calculating Pressure.
            Updating Velocity.
            Fluid Dynamics Analysis: STEP:  316
            Fluid Dynamics Analysis: TIME:  6.319999999999953
            Fractional Step Strategy : : CONVERGENCE CHECK:
            Fractional Step Strategy : : FRAC VEL.: ratio = 0.00320616; exp.ratio = 1e-05 abs = 24.6515
            Fractional Step Strategy : : CONVERGENCE CHECK:
            Fractional Step Strategy : : FRAC VEL.: ratio = 2.40194e-06; exp.ratio = 1e-05 abs = 0.0184679
            Fractional velocity converged in 2 iterations.
            Calculating Pressure.

[roigcarlo]

Well I may have fould the reason.... can you try that case with #7110 ?

[mpentek]

With #7110 seems to work. Could somebody else confirm? @AndreasWinterstein