Moto: httpretty Python 3 missing in 0.8.12 release and with new requests version

[tmylk]

Thanks a lot for moto - smart_open wouldn't have been possible without it.

We have had a couple of httpretty Python 3 issues recently in smart_open. It is a dependency of moto. I am hesitant to raise them in httpretty project as it doesn't officially support Python 3, but moto does. What is the best way to report them?

• Python 3 builds are broken in httpretty 0.8.12 . We would really like to upgrade to 0.8.12 in smart_open to enable LC_ALL=C mode
• The version of requests used by httpretty in testing is pinned at requests=2.8.1 which uses a broken version of urllib3. Python 3 build requests=2.9.0 and httpretty=2.8.10 versions gives UnicodeDecodeError .

[spulec]

Thanks for the kind words.

Unfortunately, if httpretty isn't willing to support Python3 anymore, I'm not sure how Moto can. I don't have the bandwidth to fork and support a python3-compatible version of httpretty.

I'm open to alternate ideas. We could look at other mocking libraries. As of now, I will probably have to lock down the old version of httpretty.

[spulec]

It looks like there may be some hope: https://github.com/gabrielfalcao/HTTPretty/issues/278#issuecomment-172536064

Does that solve your issues?

[nikicc]

Do you agree with replacing httpretty with some other mocking library? If I try to replace it with mock and send a PR, would you consider merging it?

[spulec]

If we can get all of the same functionality, yes (in other words, all the existing tests work). We want to support at least boto and boto3.

I think it will be a challenging endeavor, but I would likely merge.

[nikicc]

Maybe I rushed it a bit, without understanding the code base first. I will check if this is something that I could do and if I manage to do something better I will send a PR 😄

[PatrickDRusk]

Based on https://github.com/gabrielfalcao/HTTPretty/issues/278#issuecomment-172536064, are you possibly able to change your dependency to httpretty>=0.8.14?

[spulec]

It still doesn't seem that HTTPretty is going to officially support Python3 at this time.

[tmylk]

Suggest the pin to httpretty=0.8.10 can be updated to httpretty=0.8.14.

The reason is that smart_open tests pass with httpretty=0.8.14 and moto==0.4.30.

BTW some work on Python 3 might happen in httpretty in the future https://github.com/gabrielfalcao/HTTPretty/issues/307

[smarlowucf]

I have the same result. IAM tests written using moto==0.4.30 with httpretty==0.8.14 all pass as expected. However, attempting to run moto's test suite against httpretty 0.8.14 yields over 100 errors and takes over 2 hours to finish.

Edit: To clarify, I wrote unit tests for another app and ran them with moto==0.4.30 and httpretty==0.8.14 successfully. But, if I run the unit test suite for Moto with httpretty==0.8.14 it fails as mentioned above.

[spulec]

If anyone is interested in really getting involved here, I would be open to looking into moving to something like Responses: https://github.com/getsentry/responses

I have done little research though so don't know what is possible/necessary.

[shaform]

@smarlowucf I've investigated a little bit. The only reason moto's test suite fails against httpretty 0.8.14 is because of this behavior change: https://github.com/gabrielfalcao/HTTPretty/commit/19ed56efc5ab5a574c03189f30853f32f52d170f. Originally when an unimplemented feature is encountered in moto, a NotImplemented Exception is raised and caught by the test. Now, it would return empty results, which could ultimately and raise a BadStatusLine or RemoteDisconnected exception.

When this change is reverted, moto tests pass as before: https://travis-ci.org/shaform/moto/builds/185614323.

For responses, it only mocks the requests module and not the underlying sockets. So it might not be useful for moto, as some libraries might not send all of their HTTP requests through requests library.

[spulec]

@shaform thanks for digging into that!

[spulec]

We now use Responses in the current master branch, which supports Python 3. Feel free to open issues for any specific issues.

[smarlowucf]

Great to hear! @spulec Can you cut a release with the latest changes?

[spulec]

Due to the number of backward-incompatible changes, this will be a major release (likely 1.0). I'm hoping to get it out in the next week or so.

See more here if you are curious: https://github.com/spulec/moto/issues/848