Kratos: [AMGCL] is less robust than the one with release 5.1

[pooyan-dadvand]

After updating the AMGCL from the one in release 5.1 of Kratos to the newer ones we observed that our solutions become unstable and examples that are working with AMGCL 5.1 start crashing with newer versions.

We have tested two different versions which gave different behaviors;

• AMGCL which is packed with release 6.0 after some time steps starts to diverge from the results given by the older solver and finally crashes both in fluid and thermal problems
• AMGCL in master ( dcc5cd85ba7e58c4a2e41a30ee0537a22e102850 ) gives very similar results comparing to the previous version but crashes randomly in some cases

I have attached the matrices of a solution in which in step 6 the solver start diverging and in step 8 it just crashes. The lgmres solver and ilu0 preconditioner combining with multigrid is used for this solution

@ddemidov would you please take a look at it.

[ddemidov]

I'll have a look at the matrices.

[ddemidov]

To make it clear, how do you count the steps in amgcl? Just by setting maxiter?

[pooyan-dadvand]

Sorry, I didn't explain it well.

I mean the time steps. In the zip file you can find A and b for each time step. So the sixth one is in time 0.0128735 corresponding to (A_0.0128735_1.mm and b_0.0128735_1.mm.rhs) LHS and RHS and the crashing step is the last one in time 0.019034.

[ddiezrod]

I add here the settings in case it is necessary for you

"linear_solver_settings": {
                    "solver_type"                    : "AMGCL",
                    "max_iteration"                  : 500,
                    "tolerance"                      : 1e-6,
                    "provide_coordinates"            : false,
                    "smoother_type"                  : "ilu0",
                    "krylov_type"                    : "lgmres",
                    "gmres_krylov_space_dimension"   : 25,
                    "use_block_matrices_if_possible" : false,
                    "coarsening_type"                : "aggregation",
                    "scaling"                        : true,
                    "verbosity"                      : 1,
                    "coarse_enough"                  : 1000
                    },

[ddemidov]

So git bisect points to ddemidov/amgcl@9d45397b4aeda9af8797018b397faad8cc863cf8 as the first bad commit. The commit message says "Fix LGMRES for the case of left preconditioning", so I would say that this should only affect LGMRES solver. I'll look into what is happening in the commit, but could you check that other solvers work?

For example, BICGSTAB seems to work:

./solver_master -A problems/kratos-3066/A_0.0128735_1.mm -f problems/kratos-3066/b_0.0128735_1.mm.rhs -p solver.type=bicgstab solver.maxiter=500 precond.relax.type=ilu0 precond.coarsening.type=aggregation
Solver
======
Type:             BiCGStab
Unknowns:         17304
Memory footprint: 946.31 K

Preconditioner
==============
Number of levels:    2
Operator complexity: 1.10
Grid complexity:     1.12
Memory footprint:    9.45 M

level     unknowns       nonzeros      memory
---------------------------------------------
    0        17304         217358      8.52 M (90.51%)
    1         1997          22779    955.32 K ( 9.49%)

Iterations: 52
Error:      1.12312e-09

[Profile:       1.190 s] (100.00%)
[  reading:     0.324 s] ( 27.27%)
[  setup:       0.032 s] (  2.72%)
[  solve:       0.833 s] ( 69.97%)

[pooyan-dadvand]

The problem is that bicg fails to converge in some of our examples (maybe due to some similar error) so we have chosen the gmres.
In fact last month I gave the bicg another try in our other solver and it crashes in some cases more or less the same manner that 6.1 is crashing even with the one in 5.1. So we turned back to lgmres of 5.1.

[ddemidov]

Ok, another possible workaround is using left preconditioner with lgmres (solver.pside=left, right is the default one, and is the one that apparently fails):

./solver -A problems/kratos-3066/A_0.0128735_1.mm -f problems/kratos-3066/b_0.0128735_1.mm.rhs -p solver.type=lgmres solver.pside=left solver.maxiter=500 precond.relax.type=ilu0
coarsening:          smoothed_aggregation
relaxation:          ilu0
Number of levels:    2
Operator complexity: 1.27
Grid complexity:     1.12

level     unknowns       nonzeros
---------------------------------
    0        17304         217358 (78.73%)
    1         1997          58735 (21.27%)

Iterations: 150
Error:      1.12996e-09

[Profile:       1.645 s] (100.00%)
[  reading:     0.259 s] ( 15.75%)
[  setup:       0.036 s] (  2.20%)
[  solve:       1.349 s] ( 82.02%)

[ddemidov]

LGMRES was ported from scipy and I am seeing a lot of changes and bug fixes there since then. So I am thinking of updating the amgcl implementation of lgmres to match the recent version in scipy.

[ddiezrod]

@ddemidov I will need some help to test what you mentioned about the preconditioner side. Can I do this modification through the settings?

[ddemidov]

I believe @pooyan-dadvand was going to test this. The easiest way to try would be to change 'right' to'left' in this line:

https://github.com/KratosMultiphysics/Kratos/blob/master/external_libraries/amgcl/solver/lgmres.hpp#L145

[RiccardoRossi]

nope you cannot.

i will check and tell you smthg

Riccardo

On Tue, Oct 16, 2018, 5:41 PM Daniel Diez [e-mail] wrote:

@ddemidov https://github.com/ddemidov I will need some help to test
what you mentioned about the preconditioner side. Can I do this
modification through the settings?

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/KratosMultiphysics/Kratos/issues/3066#issuecomment-430287974,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHr7EYp-c2OspVeXMmNrZbDSFg2ZdNmVks5ulf5AgaJpZM4Xd1kL
.

[pooyan-dadvand]

@ddiezrod you cannot change it with parameters.... You should change the source code of AMGCL as @ddemidov mentioned

[ddiezrod]

I have tried what @ddemidov mentioned and it apparently works fine using the left preconditioner and the results are correct. I cannot be 100% sure because the solver crashed quite randomly with the right preconditioner but I would say it is working now.

[ddemidov]

There is a nicer fix for the problem, see my comment here: https://github.com/KratosMultiphysics/Kratos/pull/3106#issuecomment-431290373

[ddemidov]

Just a note: with #3110, smoothed aggregation works twice better for this matrix than non-smoothed aggregation, used in https://github.com/KratosMultiphysics/Kratos/issues/3066#issuecomment-430176726:

• smoothed:

./solver -A problems/kratos-3066/A_0.0128735_1.mm -f problems/kratos-3066/b_0.0128735_1.mm.rhs -p solver.{type=lgmres,maxiter=500} precond.{relax.type=ilu0,coarsening.type=smoothed_aggregation}
Solver
======
Type:             LGMRES(30,3)
Unknowns:         17304
Memory footprint: 5.03 M

Preconditioner
==============
Number of levels:    2
Operator complexity: 1.27
Grid complexity:     1.12
Memory footprint:    11.17 M

level     unknowns       nonzeros      memory
---------------------------------------------
    0        17304         217358      9.67 M (78.73%)
    1         1997          58735      1.51 M (21.27%)

Iterations: 65
Error:      7.04333e-09

[Profile:       0.948 s] (100.00%)
[  reading:     0.309 s] ( 32.56%)
[  setup:       0.089 s] (  9.35%)
[  solve:       0.551 s] ( 58.04%)

• non-smoothed:

./solver -A problems/kratos-3066/A_0.0128735_1.mm -f problems/kratos-3066/b_0.0128735_1.mm.rhs -p solver.{type=lgmres,maxiter=500} precond.{relax.type=ilu0,coarsening.type=aggregation}
Solver
======
Type:             LGMRES(30,3)
Unknowns:         17304
Memory footprint: 5.03 M

Preconditioner
==============
Number of levels:    2
Operator complexity: 1.10
Grid complexity:     1.12
Memory footprint:    9.45 M

level     unknowns       nonzeros      memory
---------------------------------------------
    0        17304         217358      8.52 M (90.51%)
    1         1997          22779    955.32 K ( 9.49%)

Iterations: 131
Error:      9.43495e-09

[Profile:       1.470 s] (100.00%)
[  reading:     0.288 s] ( 19.59%)
[  setup:       0.048 s] (  3.24%)
[  solve:       1.134 s] ( 77.13%)