Flank: Save flank output as json file

[adamfilipow92]

Author the user story for this feature

As a Flank user, I want to have the output saved to the file so I can easily parse this file on CI or use it in Flank integration tests.

Describe the solution

1. Add command --output-file-format to determine the output file type.
2. Save output file to matrix directory
3. Upload to gcs
4. Use it in integration tests it could help with the easiest tests update

What team think about this proposal?
@bootstraponline @piotradamczyk5 @jan-gogo @pawelpasterz @Sloox @axelzuziak-gogo

[piotradamczyk5]

I think that we should prepare some output json with information about tests results

[bootstraponline]

What output do we need that’s not already captured in junit xml?

[adamfilipow92]

I thinking about something like Flank console output but in json format. That output could contain

1. args
2. weblinks
3. test results
4. cost
5. errors if exist

[bootstraponline]

That makes sense. Maybe we should look at how gcloud CLI supports saving output to JSON and implement that?

The default output from describe commands is YAML format, but you can use the --format flag to choose between JSON, YAML, and text output formats. JSON formatted output can be useful if you are parsing the output, while text formatted output puts each property on a separate line.

https://cloud.google.com/compute/docs/gcloud-compute/tips

[adamfilipow92]

That makes sense. Maybe we should look at how gcloud CLI supports saving output to JSON and implement that?

The default output from describe commands is YAML format, but you can use the --format flag to choose between JSON, YAML, and text output formats. JSON formatted output can be useful if you are parsing the output, while text formatted output puts each property on a separate line.
https://cloud.google.com/compute/docs/gcloud-compute/tips

Yes, I will check how it's looks.

[piotradamczyk5]

Save Flank output as json file

Save information displayed during executing Flank process as easily parsed json report.

References

gcloud cli similar solution:

The default output from describe commands is YAML format, but you can use the --format flag to choose between JSON, YAML, and text output formats. JSON formatted output can be useful if you are parsing the output, while text formatted output puts each property on a separate line.

Motivation

Currently parsing output is not easy. The user gets exit code and plain console output.
Optional json report will allow users to easily parse run information and use them with their needs.
The additional motivation is to improve integrations tests of Flank project by having easily parsed JSON report instead of comparing regex-based files with console output.

Goals

• new option to enable saving report is available
• report with designed information is stored in the results directory
• report is optionally uploaded to gcloud storage

Non-Goals

After finishing this task parsing integrations tests output could be improved.

Design

Implementation steps

1. Add option to Flank to save output report (proposed name enableOutputReport)
2. Create Report file class (proposed name OutputReport) which generate json report with the following information:
   
   
   
   • args
   
   
   • weblinks
   
   
   • test results
   
   
   • cost
   
   
   • errors if exist
   
   
3. Use newly created report object to fill data and generate it at the end of Flank run process

Risks

~There could be a problem with store errors because Report is designed to use in class ReportManager which runs after finished tests.~

API

Option

In class CommonFlankConfig add parsing new option:

    @set:CommandLine.Option(
        names = ["--enable-output-report"],
        description = ["Saves output results as parsable json file and optionally upload it to Gcloud."]
    )
    @set:JsonProperty("enable-output-report")
    var enableOutputReport: Boolean? by data

Report

Create new class OutputReport class under ftl.reports package which will be able to take objects

typealias OutputData = Map<String, Any>

object OutputReport {
    private val outputData: OutputData = mutableMapOf()
    fun addToOutputReport(key: String, reportNode: Any) {..}


    private fun generateReport(): File { 
         val report = outputData.toJson() 
         report <save to file>
    }

    private fun OutputData.toJson() {...}
    private fun File.uploadToGcloud(args: IArgs) {...}
}

OutputReport will be used in any place in the app and at the end it will be generated, stored and send to gcloud

Json report

    {
        "args":<object>,
        "weblinks":<string array>,
        "testresults": [
            "matrix_id": {
                "test_axis_value":<String>,
                "passed" : <int>,
                "failed": <int>,
                "flaky":<int>,
                "error" <int>,
                "outcome": <String>,
                "test_details" : <String>
           },
           ...
           ],
           "cost":<float>,
           "error":<String>
        }

Results

Flank's run output is saved as JSON file if the user enables this feature. Task for improving parsing integrations tests will be created.

Dependencies

Currently, there are not any blockers for this task. However, the addition of this task should be included in research results for task #1317

Testing

1. Run Flank as usual
2. Check if the output file is stored in the results directory
3. Compare output file from the console output

[jan-gogo]

I would love to see also the proposition of structured JSON or YAML output. This could help to identify the best way for implementation.