Stack: `stack hoogle` command

[chrisdone]

I want this! It should:

1. Run hoogle with the right database according to the stack config in the directory in which it's run (so it's easily ran from a terminal or by IDEs).
2. Automatically download the hoogle database from stackage.org e.g. https://www.stackage.org/snapshot/lts-2.1/db.hoo for lts-2.1.
3. Put it in some reasonable place like ~/.stack/hoogle-databases/lts-2.1.hoo.
4. Or when running inside docker, use the hoogle database provided by the docker.

The last docker point means we probably need to sync up some things.

With this command I could provide some nice Emacs support with a live hoogle search buffer, and I'd basically never have to leave Emacs to open my browser to read docs again. This closes off two big pain points for me: (1) always having to open the browser and leave my editor environment, (2) needing an internet connection to view docs. This will also basically make my whole haskell-docs package redundant which I'll be able to deprecate in favor of stack.

Pinging @manny-fp and @snoyberg for feedback/opinions.

[chrisdone]

Second thoughts, seems hoogle's -d expects a directory. So it'd probably be better as:

~/.stack/hoogle-databases/lts-2.1/lts-2.1.hoo

This has some potential for putting other databases in there, such as ones generated from your local projects, which could be neat for megarepos.

[ndmitchell]

We should probably be careful about doing too much work with Hoogle 4, when I want to push forward on Hoogle 5, which changes most of this stuff quite dramatically. It's a bit of an awkward middle situation though.

[ndmitchell]

I've just pushed something to Hoogle about 10 patches before the minimum-viable product, but it's enough to see where the holes are. Given Hoogle HEAD, you can do hoogle generate --local and it will use ghc-pkg to figure out which packages are installed and where there docs are, then hoogle map and it will find map or hoogle server and it will launch a server at localhost which you can search from. Clicking links will still go to the remote docs, but it is based on the local Hoogle info.

When doing that under Stack there are 122 packages, but only 23 have docs (those shipped with GHC), because Stack doesn't include docs by default. To make stack hoogle work reasonably then stack hoogle would have to generate docs for the immediate dependencies of a module (at least), either always as it was going, or when you typed stack hoogle.

[borsboom]

Cool! stack build --haddock (aka stack haddock) already generates docs incrementally (i.e. it will build them for dependencies that don't already have them), so doing that implicitly before stack hoogle (with an option to skip for the impatient) would probably work.

[ndmitchell]

That works as a solution - it's just a shame that you have to reconfigure all the packages and that it's so slow. Also it fails on Windows, but I'm sure that's fixable: #1266.

[borsboom]

One thing that will help with this is that we're planning to support settings defaults for build options on the configuration file. That means you could configure Stack to build haddocks for dependencies by default.

It may also be possible to implement just adding haddocks to a dependency without having to rebuild the library. I looked into this a while back and it seemed feasible, but tricky to integrate into the current build plan construction/execution pipeline.

[ndmitchell]

Given things in their current state, if stack hoogle did:

• Build Hoogle from https://github.com/ndmitchell/hoogle (easy with #1278)
• stack haddock (and ideally, if that worked better, see #1317)
• stack exec -- hoogle generate --local --database=.stack-work/hoogle
• stack exec -- hoogle $ARGS --database=.stack-work/hoogle

Then we'd have enough that someone could do stack hoogle filter or stack hoogle server and it would work.

[chrisdone]

Implemented! Made a document here for anyone who wants to try it out: https://gist.github.com/chrisdone/78c1c1c052a853697cf3b902de086172

[ndmitchell]

I tried it last night, works nicely. Any chance of a stack hoogle --server which runs stack hoogle -- --server --local? I find that a very nice way to use Hoogle via Stack.

[chrisdone]

@ndmitchell That makes sense to me!

[chrisdone]

@ndmitchell Made an issue here: https://github.com/commercialhaskell/stack/issues/2310

[krisajenkins]

Works perfectly for me. 👍

[creswick]

I think I may be missing something... this doesn't seem to index the local packages. e.g.:

$ stack hoogle -- -i myLocalFunctionNameThatIsIndeedExported
No results found.
$ stack hoogle -- map
Prelude map :: (a -> b) -> [a] -> [b]
...

All the local packages haddock successfully. This should index the local packages, right?

[ndmitchell]

Try stack hoogle -- is:package which should give you a good idea of which packages Hoogle has indexed. Is it your dependencies? Or all of Stackage?

[creswick]

@ndmitchell it looks like it's indexed our local packages, and many (likely all) of our dependencies:

$ stack hoogle -- is:package --count=900000 | wc
272 544 5208

Grepping that out put for our packages also shows matches, (our packages are in that list)

[ndmitchell]

What about grepping for the package that exports myLocalFunctionNameThatIsIndeedExported?

[creswick]

I did do that, and it is in the list.

That function is exported from local package Foo, which is in the list, and used by package Bar, which is also in the list.

[ndmitchell]

If it's in the list suggest searching for package:Foo a and see what comes up. Is it a case of not indexing anything in Foo? Or just that this particular function is not being indexed for some reason.

[creswick]

It doesn't appear to be indexing anything in that package... where can I
find the index files? it may be useful to look for / at those.

On Tue, Apr 10, 2018 at 1:44 PM Neil Mitchell [e-mail]
wrote:

If it's in the list suggest searching for package:Foo a and see what
comes up. Is it a case of not indexing anything in Foo? Or just that this
particular function is not being indexed for some reason.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/commercialhaskell/stack/issues/55#issuecomment-380240390,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAPusHNswbX7DTgdfy5oGYzKnyOCXKLdks5tnRmdgaJpZM4EnrGz
.

[ndmitchell]

Don't know... If you look at the haddock output you might find a file Foo.txt in that directory or one up. Is there output in the Haddock docs?

[creswick]

I'm not sure... I'll try and reproduce this problem with a public package
so we can talk about specifics (and work towards a minimal test case)

On Tue, Apr 10, 2018 at 1:48 PM Neil Mitchell [e-mail]
wrote:

Don't know... If you look at the haddock output you might find a file
Foo.txt in that directory or one up. Is there output in the Haddock docs?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/commercialhaskell/stack/issues/55#issuecomment-380241504,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAPusNCFqIhMOdWKDhghaRVcdWkEFS8Mks5tnRqDgaJpZM4EnrGz
.

[creswick]

Ok, I think anyone can reproduce this in the following way:

$ git clone https://github.com/creswick/chatter.git
$ cd chatter
$ stack setup
$ stack hoogle -- -i trainStr
<snipped lots of output>
No results found
$ stack hoogle -- -i map
map :: (a -> b) -> [a] -> [b]
...

Chatter haddock's successfully, and defines trainStr :: Tag t => POSTagger t -> String -> IO (POSTagger t) in NLP.POS, and exports that module in chatter.cabal

[ndmitchell]

I get:

C:\Neil\chatter>stack hoogle -- -i trainStr
trainStr :: Tag t => POSTagger t -> String -> IO (POSTagger t)
chatter NLP.POS
Train a tagger on string input in the standard form for POS tagged
corpora:


trainStr tagger "the/at dog/nn jumped/vbd ./."

I am on 64 bit Windows with:

C:\Neil\chatter>stack --version
Version 1.6.5, Git revision 24ab0d6ff07f28276e082c3ce74dfdeb1a2ca9e9 (5514 commits) x86_64 hpack-0.20.0

C:\Neil\chatter>hoogle --version
Hoogle 5.0.17.2, http://hoogle.haskell.org/

It's possible my Hoogle is actually Hoogle HEAD (I don't really remember what/when I compiled it), so if the latest versions don't work, try Hoogle HEAD.

[creswick]

Ok, I was somewhat behind on Stack (1.5.x), but updating that didn't seem
to help. I'll need to get an up-to-date GHC to build hoogle head, but I can
try that in the next few days.

On Wed, Apr 11, 2018 at 3:17 PM Neil Mitchell [e-mail]
wrote:

I get:

C:\Neil\chatter>stack hoogle -- -i trainStr
trainStr :: Tag t => POSTagger t -> String -> IO (POSTagger t)
chatter NLP.POS
Train a tagger on string input in the standard form for POS tagged
corpora:

trainStr tagger "the/at dog/nn jumped/vbd ./."

I am on 64 bit Windows with:

C:\Neil\chatter>stack --version
Version 1.6.5, Git revision 24ab0d6ff07f28276e082c3ce74dfdeb1a2ca9e9 (5514 commits) x86_64 hpack-0.20.0

C:\Neil\chatter>hoogle --version
Hoogle 5.0.17.2, http://hoogle.haskell.org/

It's possible my Hoogle is actually Hoogle HEAD (I don't really remember
what/when I compiled it), so if the latest versions don't work, try Hoogle
HEAD.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/commercialhaskell/stack/issues/55#issuecomment-380613444,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAPusDANmRcilW76Vth3p9Wb0jXdy1Tdks5tnoDngaJpZM4EnrGz
.

[creswick]

Ahha! This does work with:

• Stack 1.6.5
• Ghc 8.4.1
• Hoogle head (a49b83cde500d250c81afc5edb90cbe416517c8a or newer)

[ndmitchell]

Not sure if the latest Hoogle would have worked or not, but I'll upload a new one just to be sure (once Hackage recovers). With that I suggest we close this issue?

[creswick]

the latest hoogle was not sufficient -- something in head made the
difference.

Closing the issue is fine by me, it definitely works, just took more recent
code than I'd expected :)

On Fri, Apr 13, 2018 at 1:54 AM Neil Mitchell [e-mail]
wrote:

Not sure if the latest Hoogle would have worked or not, but I'll upload a
new one just to be sure (once Hackage recovers). With that I suggest we
close this issue?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/commercialhaskell/stack/issues/55#issuecomment-381070657,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAPusOScHzQ11UyivPIYD-xLd2z9lZd4ks5toGfJgaJpZM4EnrGz
.

[ndmitchell]

I just released Hoogle 5.0.17.3 which has all the changes from HEAD, so should work.