Delve: IDE interfaces

[dlsniper]

Hi @derekparker,

First of all, Thank you for this!

While I haven't played with it yet, I plan to do so this weekend.

If all things work good enough, I'd be interesting in integrating it with the Go plugin for IntelliJ IDEA.

As I haven't seen anything that could help out IDEs do the interfacing part so I'd like to ask if there's any plans for this. I could potentially help out by defining an HTTP interface for it and just pipe all the commands to the debugger via HTTP calls / JSON responses. A side effect of this, then it could allow creating web interfaces for this.

What do you think? Does it make sense to you? If you'd like to chat, please drop me a mention on Gitter.

Kind regards

[reflog]

+1 for this. love the IDEA plugin (thanks @dlsniper !) and would love to see delve integrated!

[derekparker]

Hey - Thanks for checking it out!

I definitely would like to support editor integration with Delve, that's certainly on the roadmap.

As far as implementation goes, I imagine communication between Delve and the editor happening via a unix socket or websocket rather than exposing a full HTTP interface. This approach will satisfy editor integration as well as allowing web interface integration as well.

[eidmanna]

+1 for this.
liteide integration would be awesome.

gdb debug support is broken since golang 1.3 :-(

[dlsniper]

@derekparker well I'm very serious about making this happen so I'd like to have a chat with you on it before coming up with a PR. If you prefer something else that Gitter, please let me know (my e-mail address is in my Github profile and you can contact me via it or Hangouts).

Thanks.

[goen]

+1 this issue

[nkev]

+1000 for Delve integration into Go plugin for Intellij IDEA! That would make IDEA the only real GO IDE. I'll be happy to test it extensively on MAC/PC/Linux too.

[szabba]

Would hate to hijack the discussion, but how generic would this interface be? gocode, gorename etc are not IDE-specific, and thats a great boon for a lot of people.

[dlsniper]

@szabba I don't plan to do it specific to IDEA or other IDE. Think of it like the CLI version only over HTTP. The reasons I want it over HTTP are:

• everyone can make a nice interface for it in whatever they want (js+css)
• you can use it over remote machines in your deployment (if you want that) and have the UI on your machine
• it's cross-platform
• WebSockets are not the best thing just yet as they could be a very efficient vector for attack and I wouldn't want to have my name tied to a big security hole just yet :)

This being said, I'm going to draft it this weekend and when @derekparker and other will be happy about it, I'll send a PR to implement it.

[szabba]

I think it might be nice to have the possibility of using the same data exchange format - be it JSON, XML or whatever - irrespective of the medium. HTTP sure would be useful too, but it seems that when debugging locally a popen to the debugger could be a little nicer. Just a thought.

[derekparker]

I really like the idea of having a browser connected to a remote debugging session. An HTTP server would be easy to implement without requiring any outside dependencies, however I feel that it would be overkill for local editor communication. Really, that's my main hesitation with an HTTP a service.

The more I think about it though, the more excited I get about a browser based debug session, as I think it could be very useful and powerful for a lot of different situations. I'm certainly open to suggestions on the interface, and I look forward to your draft @dlsniper and I'm all ears for anybody else who has written plugins for other editors (vim, emacs, sublime text).

[goen]

here is a http interface https://github.com/simon-whitehead/go-debug

http user interface may also offer a rich visual debugging

[sbinet]

there is also this one:
https://github.com/sirnewton01/godev

with the GDB interface:
https://github.com/sirnewton01/godbg

[nkev]

@dlsniper "WebSockets are not the best thing just yet as they could be a very efficient vector for attack and I wouldn't want to have my name tied to a big security hole just yet".
Can I suggest you reconsider this? A websocket offers 2-way communication and will be more appropriate when the server needs to initiate messages. For example, you could always be running in debug mode while developing and the server would intercept errors and break at the line containing the error (like Visual Studio). That would be hard to implement without a 2-way socket.

As for security, we're talking about a debugging tool for developers, not a generic websocket library so it will be disabled most of the time in production. We could also make it only work over TLS for remote communications. What do you think?

[nkev]

BTW, I met and spoke with Rob Pike in the Sydney Go Meetup last week and asked him about the status of an official Go debugger. He said it was very hard to create one properly but said they are looking into it. He also said he knew about this repo and said he was watching it... I couldn't actually remember the name of the project but he reminded me that it was "Delve" :)

[pavelnikolov]

Completely agree with @NuriKevenoglu
99% of the cases you would run this tool on your local machine - web socket is the way to go if you are using a browser. And you can still use some form of authentication token (JWT for example). TLS should be mandatory for remote use. I cannot think of a case where I would debug remote code...
I have used extensively node-inspector for debugging node.js apps in browser since its creation - it is very convenient and it uses web sockets.

[derekparker]

@NuriKevenoglu I agree, a socket based implementation is ideal, and what I had in mind when this issue was opened.

That's fantastic that it's on his radar! I have a lot planned for Delve, and a lot of improvements that I hope to have merged in soon. Also I really appreciate the input and work put in so far from the community.

[nkev]

@derekparker I told you you had more fans than you know... ;)

[dlsniper]

Unfortunately I haven't been able to do anything relevant this weekend due to some personal issues. However, it does seem that majority of people would like to use websockets (and I can see the benefits for that as well). As such, I'll update my proposal and come back with something based on those. Thanks for holding on and for the input on this one.

[nkev]

@dlsniper Any progress on this? Can I perhaps help with input on the proposal?

[dlsniper]

Sorry, I've been away with some personal issues. I'll have more time in the next weeks as I'll be off-duty from my job.
I'm cc-ing @ignatov here to keep an eye on this.

[derekparker]

Any more ideas on this?

I think it could be interesting and benefitial to modify the current architecture to be more client / server, with communication happening over channels. That way the backend "server" could remain unchanged, and different clients could potentially just be plugged in. I may work on some of this initial rearchitecting, which could simplify the proposals for IDE integration clients.

[grafov]

+1 for websockets. It is easy way for send messages from each side and easy to integrate with browsers to build webUI. I more interested with integration with GNU Emacs, it has libs for websockets support.

[jacque006]

+1 for this feature, would be very useful for developers where I work.

[ironcladlou]

@derekparker

I just reviewed @dlsniper's websockets branch. Have you taken a look? What are the chances you'll eventually accept a PR based on the approach? It's not how I would have gone about the refactor, but it looks viable.

I ask because I was interested in starting to integrate delve with my own Sublime tooling, and am trying to decide where to most effectively spend my time (e.g. contributing to @dlsniper's branch, contributing to some other more acceptable branch you know of, starting my own client/server refactor, a stopgap interface to control the CLI directly, etc.)

Thanks for this tool!

[dlsniper]

I'll do the promised rebase this weekend as I finally have time for it, sorry for disappearing for a while.

[markturansky]

I volunteer to help test the IDEA plugin, as I'm using it already and would love to help improve it. A working Go debugger in IDEA is nirvana.

[derekparker]

@ironcladlou I haven't taken a look recently, I've been polishing the OS X stuff and mucking around with the core. I'd like to take another look after the rebase has happened since there has been a lot of changes on master recently that may or may not affect this branch.

So to answer your question, I'm certainly open to more discussion around this, and if you were interested in working through a pure refactor I'd absolutely be open to that. There are no competing branches for this feature that I am aware of.

There are some high level refactors I would love to work on / accept PRs for, namely moving towards a client/server implementation with the _existing_ cli, i.e. dogfood first with what exists and build out from there, essentially making the CLI a client of the server, preventing it from using proctl directly, instead sending commands with arguments to a core server which actually handles process manipulation.

Then with that refactor done, this PR could be simply adding a new client to the core debug server, with a standard interface that has been defined via usage by the CLI. If you'd be interested in working on that ( @ironcladlou or anyone else) I'd happily help move it along and merge it, as currently I'm spending a majority of my time working on the guts of the thing.

@dlsniper thanks for your work on this and I will promptly take a look at the branch post-rebase.

[ironcladlou]

@derekparker @dlsniper

So, for discussion I thought it'd be faster to just start a POC of the client/server separation I had in mind. Here's my WIP branch:

https://github.com/derekparker/delve/compare/master...ironcladlou:http-api?expand=1

What I've done there is:

• Started a public API in api/types.go. For now, and for simplicity's sake, this is unstable. Later, you could subpackage _public_ types api/v1beta1 or something, use api/types.go as the _internal_ types, and convert to/from internal/external with registered API converters. This would allow IDE integrators to have stable versioned APIs on which to build tools, while allowing you the flexibility to advance the API internally and eventually mint new public versions while maintaining backwards compatibility.
  
  
  
  • For an example of this in practice, check out Google's Kubernetes or OpenShift Origin; I work on both and so I have this stuff on the brain lately.
  
  
• Built a net/http server in proctl/server on top of proctl/DebuggedProcess. This abstracts access to the debugging session. It's not yet websocket based.
• Defined a Go client interface in client/client.go which provides Go consumers with an interface to the server. The default implementation uses the net/http library to make server requests.
• Changed the responsibility of the dlv command to be:
  
  
  
  1. Start and attach to the process
  
  
  2. Spin up the server
  
  
  3. Instantiate a Go client to the server
  
  
  4. Create a CLI which uses the client to interface with the server (it also gets a pointer to the process for backwards compatibility to allow piecemeal replacement of commands)
  
  
• Decoupled the CLI (terminal) from the session bootstrapping; the CLI is now able to be incrementally converted to a pure Go client implementation

There's still a ton of work to do:

• Finish defining the API
• Finish implementing the server interface
• Convert the web server to a websockets implementation
• Abstract encoding/decoding
• Abstract server handling boilerplate
• Support public API versioning/conversion
• Godocs
• API documentation

Other random stuff that could be done:

• Take full advantage of websockets by implementing something like syscall.Wait4 or something to allow client breakpoint notification subscriptions.

Have a look and let me know how you feel. If you like the direction, I'd be happy to work on it in my spare time.

[dlsniper]

@ironcladlou I've looked at the changes you've done and it looks interesting. The ideas you have for how to approach the API are good. However, before you can do that, I believe a separate refactoring of the internals should be done as currently none of the internal commands are returning the output but rather use fmt.Print* directly. I'd be happy to have a look every now and then over the progress but I can't do any sustainable effort to first to the internal changes then implement something on top of them.

[ironcladlou]

@dlsniper

However, before you can do that, I believe a separate refactoring of the internals should be done as currently none of the internal commands are returning the output but rather use fmt.Print* directly.

We might be having a terminology mismatch here. I already changed the command interface and two command implementations (breakpoints and threads) to use the new API client. I think some confusion is coming from the mixed terms at play here.

I have other ideas for how the CLI should be refactored to draw better distinctions (e.g. daemonized server mode, interactive client mode, foreground server/interactive client), but I'd rather discuss those in a PR.

[ironcladlou]

Moving the discussion to #88 to keep this thread clean.

[ldornin]

+1, and I'd like to add Idea is the only real IDE... period.

[dlsniper]

with #88 being closed, I think I can close this one as well. If not, please feel free to reopen this.