Pyro: Update tutorials/examples

[martinjankowiak]

we're collecting some best practices for tutorial/example rewrites in the main body of this issue. stay tuned for refinements! feel free to add stuff and refine.

Pyro stuff

• make sure you have pyro.enable_validation(True) and smoke_test = ('CI' in os.environ) somewhere at the top
• no need to specify requires_grad=True in pyro.param
• use Pyro constraint constructs, e.g. pyro.param(..., constraint=constraints.positive)
  see here: http://pytorch.org/docs/master/distributions.html?highlight=constraints#module-torch.distributions.constraints
• ensure proper shaping using iarange and .reshape
• for expensive iterative computations, set num_iters = 2 if smoke_test else 1000 or similar

PyTorch stuff (a number of these borrowed from @fehiepsi !)

@neerajprad @fehiepsi and others in the know, please check!

• everything should use torch.tensor; no Variable or variable
• initialize FloatTensors with floats and LongTensors with ints, e.g. torch.tensor(0.) versus torch.tensor(0)
• replace .data[0] with .item() to get float value of a scalar
• use .tolist() to convert a tensor to Python list.
• use .detach() to make requires_grad=False.
• use .numpy() on a non-grad non-cuda tensor to convert it to numpy array. they use the same memory. A good practice is to use .cpu().data.numpy() or .cpu().detach().numpy().
• no need to use Pyro's utils ng_zeros, ng_ones (they no longer exist). use torch.ones, torch.zeros instead.
• use x.new_zeros(...) rather than torch.zeros(...).type_as(x) etc. Similar for x.new_ones(...).
• use x.new_empty(...) to create a new empty tensor (same behavior with the legacy x.new(...))
• use torch.tensor(..., dtype=..., requires_grad=...) to create a tensor with the corresponding type and requires_grad attribute. the previous ways torch.FloatTensor(...), torch.cuda.DoubleTensor(...) also work.
• b = torch.tensor(...).type_as(a) is the same as b = torch.tensor(..., dtype=a.dtype). However, it is better to use a.new_tensor(...) which also corrects gpu device.
• .sum() always return a scalar tensor.

## Jupyter stuff

• runnable code should be in a code box
• code you don't want run should be in a text box escaped with backticks
• link to documentation, e.g.[SVI](http://docs.pyro.ai/en/dev/inference_algos.html#svi)
• figures should be embedded images, not execution output (for website rendering and scrubbinb)
• link to github example at the bottom of the tutorial
• make -C tutorial html locally to see that your tutorial renders correctly in html

Miscellaneous

• PyTorch is capitalized PyTorch

Tutorials/examples to be updated

(tentative assignees in parentheses)

• [x] SVI I/II/III (martin)
• [x] VAE/VAE comparison (neeraj)
• [x] Bayesian regression (JP)
• [x] inclined plane (martin)
• [x] GMM (fritz)
• [x] #992 contrib.named (fritz)
• [x] Intro I/II (eli)
• [x] DMM (martin)
• [x] AIR (martin/paulH)
• [x] SS-VAE (jp)
• [x] schelling (eli)
• [x] gp (martin/du phan)

[eb8680]

Should we try to standardize the code style and idioms in core Pyro anchor examples/tutorials? That is, should we make the same choices throughout about things like using pyro.condition vs pyro.observe or pyro.sample(..., obs=...), writing models as nn.Modules with model and guide methods vs separate functions, etc.?

[martinjankowiak]

maybe some things could be fruitfully standardized but in my opinion the two particular things you mentioned should be allowed to vary from case to case

[fritzo]

Re: pyro.observe we might be better off removing it before 0.2 release; I've added this to the renaming issue #827.

[fehiepsi]

@martinjankowiak As pointed out by @neerajprad , .data is similar .detach() (confirmed by Adam Paske that they are very similar except some additional checks in .detach()). Some minor things (numpy-ic) are we can use .shape and .reshape(...) now.

torch.set_default_tensor_type(http://pytorch.org/docs/master/torch.html#torch.set_default_tensor_type) is also useful.

[fehiepsi]

Related to Pytorch: new() is legacy, use new_empty() for the old behavior. To create a new tensor with the same data type, gpu device, use .new_tensor(...) like torch.tensor(...).

[martinjankowiak]

@fehiepsi if you have suggested best practices, please put them in main text at the top of the issue under the correct heading

[fehiepsi]

@martinjankowiak I don't have the privilege to edit.

[martinjankowiak]

haha whoops, sorry, i didn't realize that, never mind

[jpchen]

@fehiepsi i gave you edit permissions

[katherine-h]

It's my understanding that in contrast to iarange, the distributions from which samples are drawn inside irange must have batch_shape = (). It would be useful to note this in the irange documentation and/or in tutorials (and also add a warning).

[martinjankowiak]

@jpchen you're still planning more ss-vae updates right? do you need help?
@eb8680 what's the status on intro i/ii tutorial updates?

[jpchen]

@martinjankowiak yes, ill push a PR later, ive fixed the model sample as we discussed but i havent investigated into newer ways to do the clipped non-linearities

[eb8680]

The schelling examples were removed in #1044 and there's not much to do with the language intro tutorials except update them to match #1019

[eb8680]

Closing now that #1076 is merged, we can follow up in new targeted issues if necessary