V8-archive: API Functional Tests

[benhaynes]

Feature Request

Create API Functional Tests

What problem does this feature solve?

Knowing that the API is working as expected; more thoroughly testing the API after updates/fixes... and before releasing.

How do you think this should be implemented?

TBD. @shealavington has been working on this for a while, but we need to align on exactly _how_ these tests will be structured. Here is what Shea wrote:

I've got some tests for the Items endpoint running. There's a few changes I'd like to make _(Such as outputting successful tests to the console)_, but I guess the first iteration is almost ready to be master-merged. Though on that note, I'm currently testing the 'Demo' directly, therefore some fail _(deletions)_ as it doesn't spawn a new instance every time haha.

• 'Items' Tests:
  
  
  
  • Create: Create a single item & Create multiple items in one payload.
  
  
  • Read: Get item, Get multiple items, Get all items _(by default should max out at 200 items)_
  
  
  • Update: Update item, Update multiple items with the same data, Update multiple items with different data.
  
  
  • Delete: Delete item, delete multiple items.
  
  
• Server: Ping should Pong
• Project: Expected keys should exist when authed, and shouldn't when not authed.

Status

Authentication

• [ ] Tokens
• [ ] Get Auth Token
• [ ] Refresh Auth Token
• [ ] Password Reset Request
• [ ] Password Reset
• [ ] SSO

Query Parameters

• [ ] Metadata
• [ ] Fields
• [ ] Limit
• [ ] Offset
• [ ] Single
• [ ] Sorting
• [ ] Status
• [ ] Filtering
• [ ] Language
• [ ] Search Query
• [ ] Groups
• [ ] Skip Activity Logging
• [ ] Activity Comment

Items

• [X] Get Item
• [X] Get Multiple Items
• [X] Get All Items
• [X] Create Item
• [X] Create Items
• [X] Update Item
• [X] Update Items
• [X] Delete Item
• [X] Delete Items
• [ ] Get Item Revision
• [ ] Get Item Revisions
• [ ] Revert Item
• [ ] Revert Items?

System

• [ ] Activity
• [ ] Collections
• [ ] Collection Presets
• [ ] Fields
• [ ] Files
• [ ] Folders
• [ ] Permissions
• [ ] Relations
• [ ] Revisions
• [ ] Roles
• [ ] Settings
• [ ] Users

Utilities

• [ ] Hash String
• [ ] Match Hashed String
• [ ] Generate Random String

Mail

• [ ] Send Email

Extensions

• [ ] Get Extensions
• [ ] Get Interface
• [ ] Get Page
• [ ] Get Custom Endpoint

Server

• [ ] Information
• [x] Ping

Projects

• [ ] Information
• [ ] Update
• [ ] Create Project
• [ ] Projects Auth Config
• [ ] Projects CORS Config
• [ ] Projects Cache Config
• [ ] Projects Storage Config
• [ ] Projects Mail Config
• [ ] Projects Rate Limit Config

Field Types

• [ ] Webhooks

Directus Objects

• [ ] Activity Object
• [ ] Activity Seen Object
• [ ] Collection Object
• [ ] Collection Preset Object
• [ ] Field Object
• [ ] File Object
• [ ] Folder Object
• [ ] Permission Object
• [ ] Relation Object
• [ ] Revision Object
• [ ] Role Object
• [ ] Setting Object
• [ ] User Object
• [ ] User Role Object

SCIM

• [ ] Overview
• [ ] Create SCIM User
• [ ] Get SCIM Users
• [ ] Get SCIM User
• [ ] Update SCIM User
• [ ] Create SCIM Group
• [ ] Get SCIM Groups
• [ ] Get SCIM Group
• [ ] Update SCIM Group
• [ ] Delete SCIM Group

[benhaynes]

We all want to keep this as simple as possible, and I know @shealavington had some very clean PoCs... but @WoLfulus raised a few points for using a proprietary cli command instead of relying on "off the shelf" stuff from the mocha docs:

• Use of docker was important
• So you could run the tests from anywhere and not just the test directory (global install)
• So you could create tests for your own use cases

We need to discuss this and find a good balance so that we can keep writing tests. Can we keep the core tests suite super simple (off the shelf) and then allow for more complex tests to happen through the CLI?

[rijkvanzanten]

I think we should start with a very KISS MVP

IMO having a single start command with a flag to set the API URL should be enough for now

[benhaynes]

Cool — simple is good!

@shealavington — does this work for you? Any recommendations to simplify things further?

[shealavington]

My recommendation for simplification is the confusion around directest which doesn't seem to want to work on one of the computers I use. I wonder if there's a way @WoLfulus could make this easier. I don't truly understand the intricacies of what was done with that.

Other than that, I agree... We should KISS :wink: to start with.

[WoLfulus]

I'm ok with whatever you guys decide, but having a way to parametrize the target url/api installation would be awesome (since I can use it to test docker images later)

[sayore]

If nobody is currently on to it, i would be making some tests for the API. With that in mind i would go this route:

To make a simple Testsuite, there probably should be some kind of way to make a collection via API Calls, and destroying it after testing.

I think a testing Script via NodeJS or plain JS, using node-fetch could already give a very quick and dirty way to test everything.

The steps should simply be:

1. Create a Testcollection
2. Run each API Call in a simple configuration
3. Delete Testcollection

The Tests can just be
[Partial Pseudocode]

var url = args[1]; //unsure rn what the correct way of fetching the args was

var Tests = [ { query, result, [testname, testgroup, description]:optional }... ]
Tests.map(e => { fetch(e.url).then((result)=>{ e.result = result; return e; } ) })
Tests.foreach(e => { /* Handle Results*/ });

The results could be printed in console, or as an HTML File, using a simple UI Framework to make a Clickable SlideDown Menu.

Edit: Note to self:
If working on this maybe "query" should be either string or function OR add validation-function into array.

[benhaynes]

@edenprojectde — that would be awesome! We would love all the help we can get! Right now @shealavington (with guidance from our Tech lead, @rijkvanzanten) has been leading this effort.

Shea, any thoughts on this approach? Could we get all of this standardized and started so that it's easier for contributors to add more tests one-by-one?

[rijkvanzanten]

Also feel free to reach out on Slack, is a little easier to discuss over there 🙂

[shealavington]

Starting Thursday, I'll have some time free in the evenings and on the weekends, I'm hoping to finalise a standard that can be used.

Feel free to message me on Slack if you'd like to help or work with me on this. @edenprojectde

[benhaynes]

That'd be awesome @shealavington! Even if we can just get the basic structure finalized, that's enough for others to start helping out.

[jooola]

What is the status of this issue ? Have some work been done ?

IMO this is a high priority ticket. I might have time to build something for this.

[benhaynes]

We're refactoring the Directus API into Laravel and adding tests as we go. No ETA though.

[jooola]

@benhaynes I guess this will involve a new major release v9 ? Is there any eta on the Laravel side ? How could I be usefull ?

[benhaynes]

We're looking for full API feature parity with v8, so it _might_ not be a major release... but we also want to take this opportunity to clean things up, so maybe there will be breaking changes. We have a new branch on the API repo, and will organize a place for those conversations. Once that happens we would love the communities feedback on decisions along the way! So far we've just been looking into the ideal libraries to utilize so that as much of our core API logic is covered (less custom code to manage).

🔔 @directus/api

[jooola]

What about lumen framework ? Based on Laravel ?

Anyway I'll wait on the new discussion place before I start suggesting anything :wink:

[WoLfulus]

I thought about that, but this really depends on what path we'll take (how users will extend directus, how a directus project should look like, and things like that)