Testcafe: Sending custom data to reporter

[oreporan]

What is your Test Scenario?

We are using testcafe to measure performance regressions. Our test case does some setup, tests something, and makes sure its under some X time.
I would like to send the time it took (currently just a console.log) to our custom reporter so we can track this in a graph and see how performance increases/decreases over time.

Currently this is not possible, the closest solution i saw was to use meta like here but I need custom information after the test has passed, not before.

What are you suggesting?

Some way to pass data, either by log or by the test object, into my reporter.
The time of the test is not good enough because it includes the setup/teardown of the test.

What alternatives have you considered?

Right now I need to log the results to a file, and pick them up in another step of my CI disconnected completely from testcafe.

[sijosyn]

I would also like to send custom data to reporter. My use-case scenario is as follows:

I grouped bunch of test-scenarios together in one single test function (this is done in this way for test execution time saving purpose). Each test-scenario is considered to be a single test case and has associated testcase ID in TestRail test management system (https://www.gurock.com/testrail) and I use TestRail API in reporter plugin (https://www.npmjs.com/package/testcafe-reporter-html-testrail) to update the status of the test cases. Since I grouped the tests, I want to be able to capture the status during course of the test execution and make that info available in test reporter. For e.g

test("My Test", async (t) => {
    // Test scenario for caseID_123
    await t.click(<DOM element found>);
    await t.report({caseID_123: "PASS"},/*  set the status of  caseID_123 as PASS*/);

    // Test scenario for caseID_234  
    await t.click(<DOM element found>);
    await t.report({caseID_234: "PASS"},/*  set the status of  caseID_234 as PASS*/);

    // Test scenario for caseID_456
     await t.click(<DOM element not found>);
     await t.report({caseID_456: "FAIL"},/*  set the status of  caseID_456 as FAIL or NOT TESTED */)
});

I don't think meta helps. So how can we send custom data to reporter?

[AlexKamaev]

Currently, TestCafe does not have such functionality. However, I think it'll be useful.
I discussed the issue with the team and we decided that since we already have meta in reporters, we can add some API for modifying meta at runtime.
I marked this issue as enhancement, however, I cannot give you estimates as to when it will be implemented.
Of course, your PR will be welcomed.

[sivanisentinel]

Was this enhancement implemented? I really need this feature

[Dmitry-Ostashev]

Currently, this feature is still not implemented, but we keep it in mind. You are welcome to submit your PR and contribute to our project.

[danlomantoSamsungNext]

I'd love to try and create a PR for this. Is there any direction or tips I should know before starting? Digging into a completely new codebase can get confusing sometimes. :)

[AndreyBelym]

@danlomantoSamsungNext, I think we need to discuss API extensions first. From what it seems to me, simply adding t.log(...args) that will handle its arguments similarly to console.log and append log lines to test reports will be enough. To implement such function, we need to introduce changes to TestController and Reporter subsystems, plus implement support for this feature for in all reporter packages. I'm afraid this feature is too big to get started with contributing to the TestCafe codebase.

[danlomantoSamsungNext]

@AndreyBelym I think people are looking for more than just a t.log(...args) ability that's similar to console.log(). I believe there were discussions for using t.meta(key, value) so that people could save different kinds of test information during a test run that would be accessible via a reporter. There are a couple of referenced tickets that demonstrate this.

I believe the thinking was to use the meta() functionality because it's already accessible when building a custom reporter as well.

[AndreyBelym]

@danlomantoSamsungNext yes, I've got your point. I discussed it in private with some other team members, and we agreed that it is easier to add metadata modification functions to the test controller.
What do you think about such API for this feature:

type Meta = { [name: string]: string };

interface TestControllerMetaExtensions {
    /**
     * Returns the test metadata as an key-value object.
     * @example
     * test.meta('foo', 'bar')('test', async t => {
     *     console.log(t.meta);
     * });
     * // { 'foo': 'bar' }
     */
    readonly meta: Meta;

    /**
     * Adds an additional metadata entry to the test metadata.
     * If an entry with the specified name already exists, overrides its value.
     * @param name -  The name of the metadata entry to be added.
     * @param value - The value of the metadata entry to be added.
     * @example
     * test.meta('foo', 'bar')('test', async t => {
     *     await t.addMeta('ans', '42');
     *     console.log(t.meta);
     *     await t.addMeta('foo', 'foobar');
     *     console.log(t.meta);
     * });
     * // { 'foo': 'bar', 'ans': '42' }
     * // { 'foo': 'foobar', 'ans': '42' }
     */
    addMeta(name: string, value: string): TestController;

    /**
     * Adds an additional metadata entries to the test metadata.
     * If an entry with the specified name already exists, overrides its value.
     * @param meta -  A set of metadata entries to be added.
     * @example
     * test.meta('foo', 'bar')('test', async t => {
     *     await t.addMeta({ 'foo': 'foobar', 'ans': '42' });
     *     console.log(t.meta);
     * });
     * // { 'foo': 'foobar', 'ans': '42' }
     */
    addMeta(meta: Meta): TestController;

    /**
     * Removes metadata entries with the specified names from the test metadata;
     * @param names - A list of metadata names to be removed
     * @example
     * test.meta({ 'foo': 'bar', 'ans': '42' })('test', async t => {
     *     await t.removeMeta('foo');
     *     console.log(t.meta);
     * });
     * // { 'ans': '42' }
     * test.meta({ 'foo': 'bar', 'ans': '42' })('test', async t => {
     *     await t.removeMeta('foo', 'ans');
     *     console.log(t.meta);
     * });
     * // { }
     * test.meta({ 'foo': 'bar', 'ans': '42' })('test', async t => {
     *     await t.removeMeta(['foo', 'ans']);
     *     console.log(t.meta);
     * });
     * // { }
     */
    removeMeta(...names: (string | string[])[]): TestController;
}

[danlomantoSamsungNext]

@AndreyBelym That looks absolutely fantastic to me!!!! Thank you so much for digging into that!

[sivanisentinel]

It seems to me like this solution is “abusing” meta data. It is strange that this is the only way to send data to the report.
I feel like we need dedicated api for that.
I also thinks that we should have the option to send general data to the test and not only fixture or test related one.
For example if I want to send environment details or username who triggered the run, do we really have to send it in all fixtures again and again?

On 31 Jul 2019, at 15:51, Dan Lomanto [e-mail] wrote:

@AndreyBelym https://github.com/AndreyBelym That looks absolutely fantastic to me!!!! Thank you so much for digging into that!

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/DevExpress/testcafe/issues/3584?email_source=notifications&email_token=AENHQ2HBHMNMIMG42KZRLDDQCGDDJA5CNFSM4G7QQYGKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3HELHI#issuecomment-516834717, or mute the thread https://github.com/notifications/unsubscribe-auth/AENHQ2FW2YQPHM7X5Y3UXM3QCGDDJANCNFSM4G7QQYGA.

[danlomantoSamsungNext]

@sivanisentinel I think I see your point, but I think your request is a bit different than the original intention of this ticket.

I think it would be good to have some general way to associate data/info to an entire test run, so that you don't have to repeat yourself in all your beforeEach() and afterEach() methods (ie. Test Environment, User Info, etc.). I think this would be a different enhancement though.

The reason why I think using the meta() capability is valid here is because the data/info I'd like to set it truly test-specific data/info that I'd like to have access to in a custom reporter.

[AndreyBelym]

@danlomantoSamsungNext thank you for warming words :)

@sivanisentinel I also thought about this way first, as you can see in my comment. I would like to see it implemented someday, but we need to invest a lot more time and powers into it.

[danlomantoSamsungNext]

I see that this is marked as "Planned". Does that mean, by any chance, that this has a potential release date? Or an estimate on when this might be worked on?
Thanks!!!!

[miherlosev]

The "Planned" milestone means that the TestCafe team has plans to implement this feature someday. At present, there is no precise time estimation for that. If the feature is very important for you, the pull request would be appreciated.

[Evilweed]

@AlexKamaev this feature is in Your best interest as having this functionality allows creating much powerful Test Reporters like the one that prints Error Stack, URL of the website during error, and console errors from the browser, which is currently impossible to do. Debugging errors that occured on CI on complicated websites, without this information is much harder.

[AndreyBelym]

@Evilweed error stacks should be printed with all built-in reporters for both server-side and client-side errors automatically by TestCafe. If there is no error stack in a report, likely it means that your browser or Node.js failed to capture it.

To properly integrate reporting current URLs and browser console logs in reports, we should also solve a problem with delivering this data consistently from a browser to the TestCafe. Otherwise, it will be not a big difference from using console.log and TestCafe APIs like ClientFunction and getBrowserConsoleMessages that can work unreliably in cases of fatal errors on a tested page.

[Evilweed]

My current solution which is rarely used by any developer because it is not presented in Allure report and it's not presented as a part of console error.

Forgot to wrap getBrowserConsoleMessages in try/catch.

// Try catch was added as a precaution in order not to break Test Runner
// TODO(m) Create out of this proper Test Reporter plugin to Test Cafe
export async function printInformationFromBrowsersConsole(t: TestController): Promise<any> {
    const { log, warn, info, error } = await t.getBrowserConsoleMessages();
    let url = "";
    let testName = "";
    let fixtureName = "";
    try {
        url = await getUrl();
    } catch(e) {}

    try {
        // Do not remove - those objects are not defined in typings,
        // and can potentially break in future Test Cafe versions.
        // @ts-ignore
        testName = t.testRun.test.name;
        // @ts-ignore
        fixtureName = t.testRun.test.fixture.name;
    } catch(e) {}

    if ((log && log.length > 0) ||
        (warn && warn.length > 0) ||
        // (info && info.length > 0) ||
        (error && error.length > 0)) {
        console.log("\n");
        console.log("--------------------------------");
        console.log(`Test information:`);
        console.log("--------------------------------\n");
        console.log(`Fixture: ${fixtureName}`);
        console.log(`Test: ${testName}`);
        console.log(`🌍 Page URL: ${url}`);

        printEachIfExists(LogType.LOG, log);
        printEachIfExists(LogType.WARNING, warn);
        //printEachIfExists(LogType.INFO, info);
        printEachIfExists(LogType.ERROR, error);

        console.log("--------------------------------");
    }

}

[Farfurix]

One more request: https://github.com/DevExpress/testcafe/issues/4370.