Ballerina-lang: [Testerina] AssertEquals function fails when checking array equality

[ldclakmal]

Description:
Please refer to the following sample codes. These test cases are failing with the misleading error messages.

@test:Config {}
function test1() {
    string[] keys = ["A"];
    test:assertEquals(keys, ["A"]);
}

@test:Config {}
function test2() {
    int[] arr = foo();
    test:assertEquals(arr, [1]);
}

function foo() returns int[] {
    int[] arr = [1];
    return arr;
}

_Output_

[fail] test1:
    Assertion Failed!: expected 'A' but found 'A' 
        at ballerina.test:createBallerinaError(assert.bal:41)
        ballerina.test:assertEquals(assert.bal:82)

[fail] test2:
    Assertion Failed!: expected '1' but found '1' 
        at ballerina.test:createBallerinaError(assert.bal:41)
        ballerina.test:assertEquals(assert.bal:82)

Affected Versions:
Checked with Ballerina 1.2.0

[MaryamZi]

Seems like improving the failure message is being tracked with https://github.com/ballerina-platform/ballerina-lang/issues/12892.

As for the tests themselves though the following would pass.

@test:Config {}
function test1() {
    string[] keys = ["A"];
    var expected =  ["A"]; // no contextually expected type, so the type is set as `[string]`
    test:assertEquals(keys, expected);
}

But IMO getting a comparison with an in-line definition to pass would be useful.

The current logic for test:assertEquals is as follows.

public function assertEquals(any|error actual, any|error expected, public string msg = "Assertion Failed!") {
    boolean isEqual = false;
    if (actual is anydata|error && expected is anydata|error) {
        isEqual = actual == expected;
    } else {
        isEqual = actual === expected;
    }

    if (!isEqual) {
        string expectedStr = io:sprintf("%s", expected);
        string actualStr = io:sprintf("%s", actual);
        string errorMsg = string `${msg}: expected '${expectedStr}' but found '${actualStr}'`;
        panic createBallerinaError(errorMsg, assertFailureErrorCategory);
    }
}

For something like test:assertEquals(keys, ["A"]) the second parameter's type any|error is set as the contextually expected type for ["A"]. Thus ["A"]'s type is set as any[].

This results in expected is anydata|error to evaluate to false which in turn results in a reference equality check === for this test (which then results in false).

IMO, we can improve this a bit by introducing separate test functions to assert based on value equality vs reference equality.

For example,

1. public function assertValueEquals(anydata|error actual, anydata|error expected, public string msg = "Assertion Failed!") - test:assertValueEquals(keys, ["A"]) will pass.
2. public function assertReferenceEquals(any|error actual, any|error expected, public string msg = "Assertion Failed!")

Maybe we can have the current assertEquals also as a convenience function for those who want the current behaviour.

@hevayo @azinneera FYA please.

[janihur]

At minimum I'd like to see the value vs. reference equality explained in:

• Assertions chapter in https://ballerina.io/v1-2/learn/how-to-test-ballerina-code/
• https://ballerina.io/v1-2/learn/by-example/testerina-assertions.html
• https://ballerina.io/v1-2/learn/api-docs/ballerina/test/functions.html#assertEquals

When one knows that's the problem then one might be able to find equality comparison explanation in https://ballerina.io/v1-2/learn/by-example/equality.html

[Dilhasha]

Following doc[1] contains an analysis on how the AssertEquals function implementation works against data types available in Ballerina 1.2.x with examples.

[1] [https://docs.google.com/document/d/11CVryMHclAX3KMX7_GLP0099cUeNKvvxawHqrFUewec/edit?usp=sharing](https://docs.google.com/document/d/11CVryMHclAX3KMX7_GLP0099cUeNKvvxawHqrFUewec/edit?usp=sharing)

[Dilhasha]

In other test frameworks such as Junit5, following functionalities are available.

• AssertEquals → class loading to support all available data types(deep value checks)
• AssertArrayEquals → class loading to support arrays of all available data types
• AssertSame → Assert whether it’ s the same object.

Based on these frameworks, the usual expectation from an "AssertEquals" function is to have deep value equality check.
I suggest that we have the following options.

assertEquals - deep value equality for (any|error) i.e. for all available types in Ballerina

Also, as @MaryamZi mentioned we can have the following too.

assertReferenceEquals - check whether expected and actual refer to same object/record/table etc.

@hevayo @azinneera @Ibaqu
Appreciate your thoughts.

[Dilhasha]

Based on the current specification, we are unable to support value equality for behavioral values. So, we are planning to introduce the following functions instead of "assertEquals".

1. public function assertValueEquals(anydata|error actual, anydata|error expected, public string msg = "Value Assertion Failed!")

2. public function assertExactEquals(any|error actual, any|error expected, public string msg = "Exact Assertion Failed!")

Until there is a way to support value equality for behavioral values, we plan to remove the "assertEquals" function. See the discussion at [1].

Update: We decided to change the function signature for "assertEquals" without introducing a new "assertValueEquals"

Following is the list of tasks.

• [x] Implement "assertExactEquals", "assertExactNotEquals" and change function signature for "assertEquals", "assertNotEquals" based on expected behavior.
• [x] Write test cases to cover every data type
• [x] Update the website test guide [2]
• [x] Update BBE for testing [3]
• [x] Update modules.md
• [x] Update API docs [4]
• [x] Update the spec [5]

[1] https://groups.google.com/g/ballerina-dev/c/hNlQLAu3b4M/m/wOiRvJhvBAAJ
[2] https://ballerina.io/learn/testing-ballerina-code/
[3] https://ballerina.io/learn/by-example/testerina-assertions.html
[4] https://ballerina.io/learn/api-docs/ballerina/test/index.html
[5] https://github.com/ballerina-platform/ballerina-spec/blob/master/test/test-framework-spec.md

[Dilhasha]

Test specification updated with https://github.com/ballerina-platform/ballerina-spec/pull/587