Flux.jl: Transition examples in docs to doctests

[KristofferC]

Doctests in the Julia documentation has IMHO been a great success, it is very often that documentation that would be wrong has to be updated before a PR is merged. Doing the same for Flux would likely be a good idea.

As an example, just trying to run the very first example in the gradient docs gives:

julia> d2f(2) # 6.0 (tracked)
ERROR: Use `gradient(...; nest = true)` for nested derivatives

[ayan-b]

I would like to work on this issue. :smile:

[MikeInnes]

Please do! I'll happily take a PR to the docs to update them. You'll probably want to look at Documenter.jl, if you haven't already.

[gxyd]

Hi when I do, I get a lot of errors are those intentionally kept as such? (Trimmed output present below)

julia> using Flux, Documenter

julia> doctest(Flux)
[ Info: SetupBuildDirectory: setting up build directory.
[ Info: Doctest: running doctests.
┌ Error: doctest failure in ~/.julia/dev/Flux/src/optimise/train.jl:126-132
│
│ ```jldoctest
│ julia> Flux.@epochs 2 println("hello")
│ [ Info: Epoch 1
│ hello
│ [ Info: Epoch 2
│ hello
│ ```
│
│ Subexpression:
│
│ Flux.@epochs 2 println("hello")
│
│ Evaluated output:
│
│ ERROR: LoadError: UndefVarError: Flux not defined
│ in expression starting at none:1
│
│ Expected output:
│
│ [ Info: Epoch 1
│ hello
│ [ Info: Epoch 2
│ hello
│
│   diff =
│    [ Info: Epoch 1
│    hello
│    [ Info: Epoch 2
│    helloERROR: LoadError: UndefVarError: Flux not defined
│    in expression starting at none:1
└ @ Documenter.DocTests ~/.julia/packages/Documenter/6vUwN/src/DocTests.jl:369
┌ Error: doctest failure in ~/.julia/dev/Flux/src/layers/conv.jl:539-547
│
│ ```jldoctest
│ julia> xs = rand(Float32, 100, 100, 3, 50);  # batch of 50 RGB images
│
│ julia> AdaptiveMaxPool((25, 25))(xs) |> size
│ (25, 25, 3, 50)
│
│ julia> MaxPool((4,4))(xs) ≈ AdaptiveMaxPool((25, 25))(xs)
│ true
│ ```
│
│ Subexpression:
│
│ MaxPool((4,4))(xs) ≈ AdaptiveMaxPool((25, 25))(xs)
│
│ Evaluated output:
│
┌ Error: Doctesting failed
│   exception =
│    `makedocs` encountered a doctest error. Terminating build
│    Stacktrace:
│     [1] error(::String) at ./error.jl:33
│     [2] runner(::Type{Documenter.Builder.Doctest}, ::Documenter.Documents.Document) at /Users/gaurav/.julia/packages/Documenter/6vUwN/src/Builder.jl:217
│     [3] dispatch(::Type{Documenter.Builder.DocumentPipeline}, ::Documenter.Documents.Document) at /Users/gaurav/.julia/packages/Documenter/6vUwN/src/Utilities/Selectors.jl:170
│     [4] #2 at /Users/gaurav/.julia/packages/Documenter/6vUwN/src/Documenter.jl:247 [inlined]
│     [5] cd(::Documenter.var"#2#3"{Documenter.Documents.Document}, ::String) at ./file.jl:104
│     [6] makedocs(; debug::Bool, format::Documenter.Writers.HTMLWriter.HTML, kwargs::Base.Iterators.Pairs{Symbol,Any,NTuple{6,Symbol},NamedTuple{(:root, :source, :sitename, :doctest, :modules, :doctestfilters),Tuple{String,String,String,Symbol,Array{Module,1},Array{Regex,1}}}}) at /Users/gaurav/.julia/packages/Documenter/6vUwN/src/Documenter.jl:246
│     [7] (::Documenter.var"#all_doctests#35"{Bool,Array{Regex,1},Array{Module,1}})() at /Users/gaurav/.julia/packages/Documenter/6vUwN/src/Documenter.jl:811
│     [8] macro expansion at /Users/gaurav/.julia/packages/Documenter/6vUwN/src/Documenter.jl:828 [inlined]
│     [9] macro expansion at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115 [inlined]
│     [10] doctest(::String, ::Array{Module,1}; fix::Bool, testset::String, doctestfilters::Array{Regex,1}) at /Users/gaurav/.julia/packages/Documenter/6vUwN/src/Documenter.jl:828
│     [11] doctest(::Module; manual::Bool, testset::Nothing, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /Users/gaurav/.julia/packages/Documenter/6vUwN/src/Documenter.jl:767
│     [12] doctest(::Module) at /Users/gaurav/.julia/packages/Documenter/6vUwN/src/Documenter.jl:754
│     [13] top-level scope at REPL[2]:1
│     [14] eval(::Module, ::Any) at ./boot.jl:331
│     [15] eval_user_input(::Any, ::REPL.REPLBackend) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:134
│     [16] repl_backend_loop(::REPL.REPLBackend) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:195
│     [17] start_repl_backend(::REPL.REPLBackend, ::Any) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:180
│     [18] run_repl(::REPL.AbstractREPL, ::Any; backend_on_current_task::Bool) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:292
│     [19] run_repl(::REPL.AbstractREPL, ::Any) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:288
│     [20] (::Base.var"#807#809"{Bool,Bool,Bool,Bool})(::Module) at ./client.jl:399
│     [21] #invokelatest#1 at ./essentials.jl:710 [inlined]
│     [22] invokelatest at ./essentials.jl:709 [inlined]
│     [23] run_main_repl(::Bool, ::Bool, ::Bool, ::Bool, ::Bool) at ./client.jl:383
│     [24] exec_options(::Base.JLOptions) at ./client.jl:313
│     [25] _start() at ./client.jl:506
└ @ Documenter ~/.julia/packages/Documenter/6vUwN/src/Documenter.jl:821
Doctests: Flux: Test Failed at /Users/gaurav/.julia/packages/Documenter/6vUwN/src/Documenter.jl:828
  Expression: all_doctests()
Stacktrace:
 [1] macro expansion at /Users/gaurav/.julia/packages/Documenter/6vUwN/src/Documenter.jl:828 [inlined]
 [2] macro expansion at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115 [inlined]
 [3] doctest(::String, ::Array{Module,1}; fix::Bool, testset::String, doctestfilters::Array{Regex,1}) at /Users/gaurav/.julia/packages/Documenter/6vUwN/src/Documenter.jl:828
Test Summary:  | Fail  Total
Doctests: Flux |    1      1
ERROR: Some tests did not pass: 0 passed, 1 failed, 0 errored, 0 broken.

julia>

Sorry for a really long output, I can pastebin somewhere if you want me to..

[DhairyaLGandhi]

Can I request you to compress the output. You might want to check how our ci does doc tests, you need to set the docsmeta to set some preprocessing :)

[gxyd]

I tried using by copying the commands from https://github.com/FluxML/Flux.jl/runs/1478546117 , I still seems to be getting the same warnings, even the logs in the link do have those warnings. The ones against the command julia --project=docs docs/make.jl.
Below is my versionInfo:

Julia Version 1.5.3
Commit 788b2c77c1 (2020-11-09 13:37 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin18.7.0)
  CPU: Intel(R) Core(TM) i7-4770HQ CPU @ 2.20GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, haswell)
Environment:
  JULIA_PROJECT = @.

the output I'm getting includes warnings like:

┌ Warning: unable to get the binding for '[`Flux.binarycrossentropy`](@ref)' in src/models/losses.md from expression ':(Flux.binarycrossentropy)' in module Flux.Losses
│   exception = UndefVarError: Flux not defined

Also, I'm not really sure if running the doctests this way would run the docstrings present in FluxML/src/layers/basic.jl, does it do that?

[gxyd]

As I can see the makedocs here, doesn't have path to src/layers/basic.jl, or am I missing something?

[gxyd]

Hi @DhairyaLGandhi , please if have any input on the above let me know..