Pyro: Allow configurable error-checking level

[fritzo]

As of https://github.com/pytorch/pytorch/pull/5358 PyTorch supports globally configurable error checking of distribution parameters via Distribution.set_default_validate_args(-). It would be helpful (e.g in diagnosing #875) if Pyro similarly had a globally configuration function like pyro.set_validation_level(-) to throttle model/guide sanity checks (check_model_guide_match, check_site_shape, etc.) in Trace*_ELBO (and possibly to also throttle Distribution checks, though @jpchen feels this is cumbersome).

It would also be nice to implement some checks with higher false-positive rate, e.g. detecting possible violation of irange/iarange variable dependency in Trace*_ELBO implementations (as suggested by @martinjankowiak). Ideally we not log these warnings by default, but only if the user requested extra validation logic.

Possible policies:

• allow configuration levels 0, 1, 2, 3, 4.
• allow configuration frequencies "never", "once", "always". (maybe it's fine to let users control this across iterations)

Benchmarks

| validation_level = | 0 | 1 | 2 | 3 |
| -: | - | - | - | - |
| test_valid_models.py | ? | ? | ? | ? |
| test_enum.py | ? | ? | ? | ? |
| test_inference.py::test_exponential_gamma | ? | ? | ? | ? |

Possible Tasks

• [x] update to PyTorch after https://github.com/pytorch/pytorch/pull/5358
• [x] add a global variable and setter for validation configuration
• [x] guard checks in Trace*_ELBO by validation level
• [x] add death tests to test_valid_models.py to test high validation level
• [ ] start using Distributions.set_default_validate_args(True) in some tests
• [x] turn on validation in all of our smoke-tested tutorials
• [ ] turn on validation in most tests (maybe only the first iteration of our expensive inference tests) - Not required yet.
• [x] informally benchmark a couple tests to determine overhead (see above table) - Done in #913

[fritzo]

@jpchen @neerajprad Which do you think is cleanest?

1. Separate validation toggles
   py torch.distributions.Distribution.set_default_validate_args(True) pyro.set_validate_model(True)
2. A combined validation knob
   py pyro.set_validation_level(2)
3. Per-module validation toggles
   py torch.distributions.Distribution.set_default_validate_args(True) pyro.infer.set_validation(True) pyro.param.set_validation(False)
4. Something else?

(feel free to edit this comment)

[neerajprad]

I am more in favor of 2, since you would typically interact with all three at the same time, unless there is a strong need to toggle one and not the other.

[eb8680]

I'd vote for implementing 2 on top of 3

[jpchen]

im ok with 2 if we can replace the number with an enum or something descriptive like pyro.set_validation_level('all')