Pydantic: valid urls not being accepted by pyndantic

[majorgilles]

For bugs/questions:

• OS: osx mojave
• Python version 3.6
• Pydantic version 0.18.1

The following (ugly) url is valid, but not accepted by pydantic url validator (tested in the browser without any issues, but I've replaced the host in this case)

https://dev.somesite.io/offer/pdf/url_checker?deal_id=622374008&to=https%3A%2F%2Flizy-offer-pdf-dev.s3.amazonaws.com%2FFR%2520-%2520test_company_name%2520-%2520test_last_name%2520-%2520Ibiza%25205D%2520-%25202019-03-21%3FX-Amz-Algorithm%3DAWS4-HMAC-SHA256%26X-Amz-Credential%3DAKIAIEV3CYKSCFTOUTMA%252F20190321%252Feu-central-1%252Fs3%252Faws4_request%26X-Amz-Date%3D20190321T163250Z%26X-Amz-Expires%3D604800%26X-Amz-SignedHeaders%3Dhost%26X-Amz-Signature%3D5160868362c313c3456b13d83a41cf0ba761387497df0844a837f46ffdd37e52%27

[samuelcolvin]

In [1]: from django.core.validators import URLValidator                                                                                      

In [2]: validate = URLValidator()                                                                                                            

In [3]: validate('<the url above>')
...
ValidationError: ['Enter a valid URL.']

That was using django 2.1.7 final.

Our url validation logic was taken if i remember correctly from django. If you really think this is a bug, please report it to django and let me know here if you find a fix.

[Gr1N]

Validation logic is taken from Marshmallow https://github.com/marshmallow-code/marshmallow/blob/298870ef6c089fb4d91efae9ca4168453ffe00d2/marshmallow/validate.py#L37.

>> from marshmallow import Schema, fields
>>> class Foo(Schema):
...     bar = fields.Url()
...
>>> schema = Foo()
>>> schema.load({'bar': 'https://dev.some_site.io/offer/pdf/url_checker?deal_id=622374008&to=https%3A%2F%2Flizy-offer-pdf-dev.s3.amazonaws.com%2FFR%2520-%2520test_company_name%2520-%2520test_last_name%2520-%2520Ibiza%25205D%2520-%25202019-03-21%3FX-Amz-Algorithm%3DAWS4-HMAC-SHA256%26X-Amz-Credential%3DAKIAIEV3CYKSCFTOUTMA%252F20190321%252Feu-central-1%252Fs3%252Faws4_request%26X-Amz-Date%3D20190321T163250Z%26X-Amz-Expires%3D604800%26X-Amz-SignedHeaders%3Dhost%26X-Amz-Signature%3D5160868362c313c3456b13d83a41cf0ba761387497df0844a837f46ffdd37e52%27'})
UnmarshalResult(data={}, errors={'bar': ['Not a valid URL.']})

Your URL is invalid because of underscore in domain. @majorgilles you can find some details at https://www.quora.com/Domain-Name-System-2/Why-are-underscores-not-allowed-in-DNS-host-names

[samuelcolvin]

Thanks @Gr1N, very helpful.

[majorgilles]

actually I added that underscore manually, this isn't the actual url.
the original url is fully working.

just edited to avoid any confusion.

Will contact the django team to see if they're willing to do anything, otherwise, I'll just avoid using the validator for that particular url.

[Gr1N]

@majorgilles I tried your URL with latest version of pydantic and everything works as expected:

~/.virtualenvs/tmp-288412f32340648 via tmp-288412f32340648 took 17m 48s
➜ pip install pydantic
Collecting pydantic
  Using cached https://files.pythonhosted.org/packages/0c/34/26797d017fc20907186cba125b1f1e5ea2c1dc2b4326399c8242844f18bf/pydantic-0.21-py36.py37.py38-none-any.whl
Installing collected packages: pydantic
Successfully installed pydantic-0.21

~/.virtualenvs/tmp-288412f32340648 via tmp-288412f32340648
➜ python
Python 3.7.2 (default, Feb 24 2019, 19:05:45)
[Clang 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from pydantic import BaseModel, UrlStr
>>>
>>> class Foo(BaseModel):
...     bar: UrlStr
...
>>> Foo(bar='https://dev.somesite.io/offer/pdf/url_checker?deal_id=622374008&to=https%3A%2F%2Flizy-offer-pdf-dev.s3.amazonaws.com%2FFR%2520-%2520test_company_name%2520-%2520test_last_name%2520-%2520Ibiza%25205D%2520-%25202019-03-21%3FX-Amz-Algorithm%3DAWS4-HMAC-SHA256%26X-Amz-Credential%3DAKIAIEV3CYKSCFTOUTMA%252F20190321%252Feu-central-1%252Fs3%252Faws4_request%26X-Amz-Date%3D20190321T163250Z%26X-Amz-Expires%3D604800%26X-Amz-SignedHeaders%3Dhost%26X-Amz-Signature%3D5160868362c313c3456b13d83a41cf0ba761387497df0844a837f46ffdd37e52%27')
<Foo bar='https://dev.somesite.io/offer/pdf/url_checker?deal_id=622374008&to=https%3A%2…'>
>>>

[majorgilles]

Ok nice to know. Will update and see if this makes any difference. Thanks @Gr1N

[Gr1N]

I see only one difference between pydantic's code and marshmallow's. 5 months ago they added a small fix for basic auth part: https://github.com/marshmallow-code/marshmallow/commit/cef251e4b9056031b406029a564dc13b17a10adf. Let me know if it can help you, I can fix it quickly.

[majorgilles]

Currently encountering an unrelated issue when updating to latest pydantic.

  @classmethod
    def _get_value(cls, v: Any, by_alias: bool, skip_defaults: bool) -> Any:
        if isinstance(v, BaseModel):
>           return v.dict(by_alias=by_alias, skip_defaults=skip_defaults)
E           TypeError: dict() got an unexpected keyword argument 'skip_defaults'

Will be able to test this once I've determined what's going awry.

[samuelcolvin]

please keep the discussion relevant to the issue.