Botocore: High level explanation for data/*.jsons

[fxfactorial]

Hi,

Is there some documentation that can describe the meaning of the fields in the json used by loaders.py.

For example, botocore/botocore/data/aws/s3/2006-03-01.normal.json the meaning of shapes field.

Is it some kind of convention in a data modeling way? Related to this? http://w3c.github.io/data-shapes/data-shapes-primer/

Thank you

[jamesls]

There's no documentation (yet) about the keys in the JSON params.

The shapes key defines structures that are referenced in the "operations" key that define how the input parameters and output response should be formatted. If there's any other specific questions you have about the model before we get documentation published, feel free to ask.

cc @trevorrowe

[fxfactorial]

@jamesls Thanks for the quick reply. So just to make sure I understand this, here's an example from S3's normal json file

list(a['shapes']['GetObjectRequest']['members'])
Out[103]: 
['ResponseContentType',
 'Range',
 'Key',
 'IfModifiedSince',
 'IfNoneMatch',
 'ResponseContentDisposition',
 'VersionId',
 'IfMatch',
 'IfUnmodifiedSince',
 'ResponseCacheControl',
 'SSECustomerKeyMD5',
 'RequestPayer',
 'ResponseContentEncoding',
 'ResponseExpires',
 'SSECustomerKey',
 'Bucket',
 'SSECustomerAlgorithm',
 'ResponseContentLanguage']

If I understand correctly, these are all the things that S3 understands for a get that is specifically a GetObjectRequest action, and if I do a['shapes']['GetObjectRequest']['required'], then I get the fields of this previously shown json that are actually required. (Which seems to imply that most stuff is optional)

Also, is there a way to know ahead of type which fields are nullable? or rather don't always show up, like documentation and documentation_url.

[trevorrowe]

You are correct about the required trait. One thing to consider, some services will have conditionally required elements, such that field b is required when field a is present. Another funny one is exclusively required fields, such that a or b, but not both must be set. These types of conditions are not described or tracked by the required trait.

The list of fields that must be set in a json document depend mostly on the context. The majority of fields though are optional. For example, when you are looking in the dictionary of shapes, the type field is always set. If the shape is a scalar shape, this is the only field that must be set. If you are looking at a structure shape then the members dictionary must be set. The required list is an optional trait, do is documentation, etc.

I have a work-in-progress, and very much not yet complete JSON schema for the API models here:
https://github.com/awslabs/aws-model-validators/blob/master/schemas/api_v2.json

[fxfactorial]

@trevorrowe Much appreciated. Question on the conditionally required fields...if its not documented, then how does botocore/boto do it correctly? Some hard coded values?

[trevorrowe]

It doesn't do anything special. It relies on API reference documentation and when a user makes a request that does not satisfy the conditionally required constraint, the service returns an error. Its less efficient that catching it client-side with a helpful error, but it captures the error regardless.

[kyleknap]

Closing. Please reopen if you have any more questions.

[fxfactorial]

@kyleknap Thank you, it explained everything I needed.