Packit-service: Schemathesis

[lachmanfrantisek]

I've tried to use kiwicom/schemathesis to test our api endpoint.

• It uses the https://prod.packit.dev/api/swagger.json definition of our API and uses hypothesis to create various inputs to tests the API.
• We can try that also to write tests for our API. WDYT?
• btw, it gives me a following output:

$ schemathesis run https://prod.packit.dev/api/swagger.json
======================================================================================================= Schemathesis test session starts ======================================================================================================
platform Linux -- Python 3.7.6, schemathesis-0.23.5, hypothesis-5.3.1, hypothesis_jsonschema-0.11.0, jsonschema-3.2.0
rootdir: /home/flachman
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/flachman/.hypothesis/examples')
Schema location: https://prod.packit.dev/api/swagger.json
Base URL: https://prod.packit.dev
Specification version: Swagger 2.0
Workers: 1
collected endpoints: 12

GET /api/copr-builds .                                                                                                                                                                                                                  [  8%]
GET /api/copr-builds/{id} .                                                                                                                                                                                                             [ 16%]
GET /api/healthz .                                                                                                                                                                                                                      [ 25%]
HEAD /api/healthz .                                                                                                                                                                                                                     [ 33%]
GET /api/installations E                                                                                                                                                                                                                [ 41%]
GET /api/installations/{id} .                                                                                                                                                                                                           [ 50%]
GET /api/tasks E                                                                                                                                                                                                                        [ 58%]
GET /api/tasks/{id} .                                                                                                                                                                                                                   [ 66%]
POST /api/testing-farm/results .                                                                                                                                                                                                        [ 75%]
POST /api/webhooks/github .                                                                                                                                                                                                             [ 83%]
GET /api/whitelist .                                                                                                                                                                                                                    [ 91%]
GET /api/whitelist/{login} .                                                                                                                                                                                                            [100%]

============================================================================================================== HYPOTHESIS OUTPUT ==============================================================================================================
Unreliable test timings! On an initial run, this test took 2338.67ms, which exceeded the deadline of 500.00ms, but on a subsequent run it took 52.12 ms, which did not. If you expect this sort of variability in your test timings, consider turning deadlines off for this test by setting deadline=None.
Unreliable test timings! On an initial run, this test took 1749.83ms, which exceeded the deadline of 500.00ms, but on a subsequent run it took 207.65 ms, which did not. If you expect this sort of variability in your test timings, consider turning deadlines off for this test by setting deadline=None.
==================================================================================================================== ERRORS ===================================================================================================================
___________________________________________________________________________________________________________ GET: /api/installations ___________________________________________________________________________________________________________
hypothesis.errors.Flaky: Tests on this endpoint produce unreliable results: 
Falsified on the first call but did not on a subsequent one


Run this Python code to reproduce this failure: 

    requests.get('https://prod.packit.dev/api/installations')

Or add this option to your command line parameters: --hypothesis-seed=233614955431053425846825088152675660077
_______________________________________________________________________________________________________________ GET: /api/tasks _______________________________________________________________________________________________________________
hypothesis.errors.Flaky: Tests on this endpoint produce unreliable results: 
Falsified on the first call but did not on a subsequent one

Query           : {'page': -81}

Run this Python code to reproduce this failure: 

    requests.get('https://prod.packit.dev/api/tasks', params={'page': -81})

Or add this option to your command line parameters: --hypothesis-seed=64027480291566386922747730227404210915
=================================================================================================================== FAILURES ==================================================================================================================
=================================================================================================================== SUMMARY ===================================================================================================================

not_a_server_error            806 / 919 passed          FAILED 

======================================================================================================== 10 passed, 2 errored in 92.98s ======================================================================================================

[jpopelka]

+1 once we move the data from redis to postgres, because now it's slow as hell

[dhodovsk]

What are the test cases that are tested by this tool?

[Stranger6667]

Hello folks! :)

What are the test cases that are tested by this tool?

At the moment - random data, that fits the API schema + examples specified directly in the schema. The main purpose to check if the app behavior matches the schema spec even though that in practice there are many validation rules outside of the schema

From the next version, it will also be random data that does NOT fit the API schema (as a separate test type) + we might add optional instrumentation to have coverage guided random data generator (a hill-climbing algorithm to maximize the number of lines hit by the test)

Feel free to let us know if you need any adjustments on the Schemathesis side to fit your use case better :) Feedback is much appreciated

[lachmanfrantisek]

@Stranger6667 thanks for the info!

[lachmanfrantisek]

Some update:

It can be used to create our tests as well:

• running directly via wsgi: https://github.com/kiwicom/schemathesis#wsgi-applications-support
• custom checks: https://github.com/kiwicom/schemathesis#in-code
• link to docs: https://schemathesis.readthedocs.io/en/latest/

[lachmanfrantisek]

And there is a episode of Test & Code podcast about property-based testing mentioning hypothesis and schemathesis.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Thank you for your contributions.
We are doing this to be sure that the issue is still relevant. Anyone can comment to remove the stale state. (The issues marked with pinned, security, bug or EPIC label
are not considered stale.)

[lachmanfrantisek]

I still want to take a look at it...;)

[stale[bot]]

This issue has been marked as stale because it hasn't seen any
activity for the last 60 days.

Stale issues are closed after 14 days, unless the label is removed
by a maintainer or someone comments on it.

This is done in order to ensure that open issues are still relevant.

Thank you for your contribution! :unicorn: :rocket: :robot:

(Note: issues labeled with pinned, security, bug or EPIC are
never marked as stale.)

[Stranger6667]

As a side note, the generated API schema contains strings like "HTTPStatus.OK" inside of the "responses" keyword, which violates the spec. Probably in cases like this one here it should be HTTPStatus.PARTIAL_CONTENT.value instead of HTTPStatus.PARTIAL_CONTENT so the str call will produce values appropriate for this regex - ^([0-9]{3})$|^(default)$ (as per spec). This change enables other Schemathesis checks (available via -c all CLI option) and may uncover some more potential issues like missing response definitions for 404 or responses with text/html content type when only application/json is declared (this endpoint, for example - GET /api/koji-builds/{id})

[lachmanfrantisek]

@Stranger6667 thanks for trying that!
I want to set this up finally. (I still have other things with higher priority..;(

@IceWreck do you want to take a look at this specific problem @Stranger6667 mentioned above?

[IceWreck]

Probably in cases like this one here it should be HTTPStatus.PARTIAL_CONTENT.value instead of HTTPStatus.PARTIAL_CONTENT so the str call will produce values appropriate for this regex...

Thanks. Will do.
https://docs.python.org/3/library/http.html

[lachmanfrantisek]

@IceWreck thanks!

[stale[bot]]

This issue has been marked as stale because it hasn't seen any
activity for the last 60 days.

Stale issues are closed after 14 days, unless the label is removed
by a maintainer or someone comments on it.

This is done in order to ensure that open issues are still relevant.

Thank you for your contribution! :unicorn: :rocket: :robot:

(Note: issues labeled with pinned, security, bug or EPIC are
never marked as stale.)

[stale[bot]]

This issue has been marked as stale because it hasn't seen any
activity for the last 60 days.

Stale issues are closed after 14 days, unless the label is removed
by a maintainer or someone comments on it.

This is done in order to ensure that open issues are still relevant.

Thank you for your contribution! :unicorn: :rocket: :robot:

(Note: issues labeled with pinned or EPIC are
never marked as stale.)

[lachmanfrantisek]

Looks like a good Xmas work..;)

[TomasTomecek]

since it's been a year since this was created, is it a real expectation that someone would work on this in coming months?

[stale[bot]]

This issue has been marked as stale because it hasn't seen any
activity for the last 60 days.

Stale issues are closed after 14 days, unless the label is removed
by a maintainer or someone comments on it.

This is done in order to ensure that open issues are still relevant.

Thank you for your contribution! :unicorn: :rocket: :robot:

(Note: issues labeled with pinned or EPIC are
never marked as stale.)

[TomasTomecek]

I am going to be the jerk and close this one. Please reopen if anyone is planning to work on this.