Specflow: [Discussion] Future Roadmap for IDE Integrations

[SabotageAndi]

Because of #651 and a lot of feature request/bugs in the current Visual Studio integration, I think it is best, that we work on a new IDE integration, as the current one is based on the old Gherkin parser.

@gasparnagy started at TechTalk with a new editor which is based on the new Gherkin parser.
Current features:

• Syntax highlighting
• Table formating

I suggest that the new IDE Integration is based on this, as I am allowed to publish it BSD licensed.

As I remember following features are included in the current VS integration:

• file templates
• step skeleton generation
• jump from step in feature to step in code
• generation of code behind file

As for the generation of the code behind file, I am for moving it completely into the build process. We have here already the MSBuild generation. For .Net Core we also need a solution for generating the code-behind file at build time, because there is currently no such feature like CustomTool available.
I think the code generation is one of the most difficult feature, we could save time and effort, if we skip it.

So we do not have to load the SpecFlow dlls that are used in the opened project.

As an additional goal I would like to design the integration in a way, that it is easier to reuse it in other IDEs like Xamarin Studio. AFAIK Microsofts Roadmap is, that Xamarin Studio is the main IDE for Mac OS X.

As first step I would create a new repository named SpecFlow.IdeIntegration and push the current code there.

[gasparnagy]

Thx @SabotageAndi! Good news. My feelings/comments:

• The new editor that I did based on the new parser is pretty much VS specific. I don't think it will really be useful for other IDEs. As far as I have seen, the IDEs are pretty much different, so it would be really hard to make a generic IDE support, especially not in a single git repo, so that we would need to synchronize the changes with the different IDEs. So my suggestion is to make a simpler, more modular VS integration, and based on that make another IDE integration, and once this is done think about what are those aspects that can be generalized.
• The current VS integration also does step intellisense and "run/debug scenario" from the editor
• The biggest problem/complexity of the current VS integration related to three things:
  
  
  
  1. As the parser was too slow, the syntax coloring had to be implemented async and had to support partial re-parse -- this is eliminated with my new editor that uses the new parser
  
  
  2. The VS extension was trying to discover a step-map (a big registry of steps and step definitions). This step map is used for the navigation and for the step intellisense. The discovery is currently able to collect step definitions from the uncompiled c# files (once saved) and also from compiled assemblies. The discovery from uncomplied c# was using the code navigation infrastructure of VS that is very hard to use and quite unreliable.
  
  
  3. To further increase the problems, re-discovery from c#/feature files that were changed _outside_ of VS (e.g. with a git branch switch) is a problem. I could not find any working VS infrastructure to get notified about these, so currently the re-discovery is triggered by the combination of the VS events and a FileSystemWatcher. Combining these is pretty complex and brittle.
  
  
  4. We also need to track the changes of the feature files (because of the step map) and also the configuration file (because of various reasons).
  
  
• for the second problem, my suggestion would be to simplify the step-map discovery process, and only discover step definitions from the compiled assemblies. This would simplify the process, as we would only need to listen to the build event, but on the other hand this would mean that until you compile your code, the new steps would be still displayed as undefined (purple). I don't know...
• we could maybe further simplify the step-map discovery process by not processing the feature files (just the step definitions). This means that for jumping back from a step definition to a feature file, we would need to re-parse and re-match all feature files in the solution. The new parser is very fast, but we would need to verify if it is fast enough for this... I am not sure.
• I also would like to get rid of the design-time generation, but I would separate this topic from the IDE rewrite. The generation is one of the simplest parts of the current IDE integration (well, except the automatic regen popup, but that anyway does not work properly). IMHO we would not win so much by throwing it out. (Not even mentioning the backwards compatibility.) Actually I would like to get rid of the code generation entirely, but this is another topic...
• However we do this, this is big chunk of work, so to be able to complete it we need to setup a model where many people can easily contribute (document, tests, build, dependencies, etc) and we need to keep the scope as small as possible.
• I would make a SpecFlow.VisualStudio2 repo with the codebase I did earlier, and keep it until it reaches the feature complete state and then replace the current SpecFlow.VisualStudio repo with that.

[stajs]

I'd love to see .NET Core support.

For .Net Core we also need a solution for generating the code-behind file at build time, because there is currently no such feature like CustomTool available.

I hacked around this by hooking in to the prebuild and shelling out to the specflow EXE to generate. It could be a way to get initial support, with the hope that a better way is available in the future.

Good luck!

[samholder]

@SabotageAndi that all sounds good, though like @gasparnagy said sometimes trying to aim for a generalized solution from the start is difficult to do, sometimes its better to have a working version for one, then try a second integration and see what bits can be shared. By the time you have done 3 integrations you probably have something which is quite generalized.

Knowing how we can unit test the code used in the VS integration is going to be key for getting people involved as one of the things I find difficult with the existing VS integration is not knowing if a change has broken anything.

Also having documentation about how VS extensions work in general would also be really useful, at least for me!

[SabotageAndi]

@gasparnagy I forgot the intellisense. _ups_.
Are the stepmaps the Visual Studio way of doing intellisense?

I was aware, that we cannot write an IDE integration for multiple different IDEs on the first try. But we could have a more deeper look at writing it reusable.

On all other points I am with you two.

Probably we can copy some parts simply from the current integration. The step skeleton generation is the first I am thinking about.

I created the SpecFlow.VisualStudio2 repo and will push the sources there.

For building we could set up an AppVeyor build and MyGet supports VS Extension Gallery feeds. So we can get early versions out for us and users.

About the code generation:
Lets create a separate issue for discussing that.

@stajs Have a look at PR https://github.com/techtalk/SpecFlow/pull/649 and branch https://github.com/techtalk/SpecFlow/tree/DotNetCore

[gasparnagy]

@SabotageAndi no. stepmaps is our own solution.

the problem with the generalized solution is that every IDE has its own concept about language support, and since this is a very performance intensive part (for every keystroke we have to calculate, parse and create many objects), putting to much abstraction around is dangerous.

@samholder I really recommend for everyone to look at the SpecFlowVisualStudio2 to get the feeling. It is not a big codebase. I recommend starting from the syntax coloring (https://github.com/techtalk/SpecFlow.VisualStudio2/tree/master/SpecFlow.VisualStudio.Editor/Classification) that uses the VS specific parser wrappers at https://github.com/techtalk/SpecFlow.VisualStudio2/tree/master/SpecFlow.VisualStudio.Editor/Parser.

I am also happy to give an intro to anyone in one of our online meetings.

[samholder]

@gasparnagy thanks. I grabbed the code last week and started to have a look. Hopefully I'll get a bit more time this weekend to play.

[jmezach]

@gasparnagy Couldn't the stepmaps problem be fairly easily and elegantly solved by using Roslyn? It should provide us with an in-memory representation of the current code in the solution so I guess it shouldn't be that hard to determine the available steps from that?

Edit: To prove my point I made a very crude implementation of it that utilizes Roslyn to figure out the available step definitions within the current solution. At the moment the entire set is rebuild every time something changes which is probably not the most efficient, but that could be improved by keeping track of where the step definitions are located in source and then invalidating those when the source file changes. Before I take this any further I would love your thoughts on this @gasparnagy. I left the code here

[gasparnagy]

@jmezach Yes, that would be definitely useful! And your code looks really a good start. Yes, please keep on doing it. We would need to reach a point where the perf can be measured (for a bigger project), so we can make the decision about when to trigger the change and whether we really need to persist the cache (which causes many issues). /cc @samholder @SabotageAndi

[jmezach]

First of all, apologies for the late reply. I've been away on holiday so I had some other things on my mind ;).

Glad you like my attempt at working something out @gasparnagy. I just had a quick look at the performance and it doesn't seem to be that bad right now actually. I have a solution with three projects only one of which has a reference to the TechTalk.SpecFlow assembly (which I filter on right now which I think is okay since if it doesn't have a reference to that assembly it can't contain step definitions anyway, can you confirm?).

That project currently has 251 step definitions (at least that is what my code finds at the moment, but it sounds about right) and it only takes 20 to 40ms to find them with my current implementation.

But we do have to consider the fact that this code is run every time I type something in a file so it will run very very often so it seems a bit wasteful to discover the step definitions every time a single character is typed somewhere. Would love your input on this though.

[dasMulli]

better late than never: @stajs

I hacked around this by hooking in to the prebuild and shelling out to the specflow EXE to generate. It could be a way to get initial support, with the hope that a better way is available in the future.

Just like microsoft moves .net build targets to a NuGet package (Microsoft.NET.Sdk) that only ships msbuild targets, one could build a NuGet package that contains all targets needed to perform the necessary build steps.
This would also remove any dependency on VS-integrated tooling to at least build & run SpecFlow based test-projects on any platform. Plus one could try to move the generated source away from the gherkin files (e.g. to obj/) so no one accidentally forgets to exclude them in .gitignore files. Just like the project.lock.json is replaced by a obj/project.assets.json and the nuget generated props and targets files also land in obj/.

[dasMulli]

cont:

An MSBuild target could potentially participate in the project system and even incremental compilation by evaluating a custom item type.
A "new-world" csproj could potentially have:

<ItemGroup>
    <Compile Include="**\*.cs" />
    <SpecFlow Include="**\*.feature" />
    <PackageReference Include="SpecFlow.Sdk.XUnit" Version="1.0.0" PrivateAssets="All" />
    <PackageReference Include="Microsoft.NET.Sdk" Version="1.0.0" PrivateAssets="All" />
</ItemGroup>

The sample SpecFlow.SDK NuGet package could then inject a before-build-target that evaluates the SpecFlow items, generates source code and emits Compile items that the standard build-targets then pick up for compilation.
It could also include a target that any other language tool could use to inspect the SpecFlow aspect of a project. e.g. an msbuild my.csproj /t:ListSpecFlowBindings could be called by a language service that can then drive IntelliSense for VSCode.

[stajs]

The MSBuild option will be a no-brainer when the next Visual Studio tooling drops that converts project.json back to MSBuild based .csproj. It's also quite attractive for VSCode.

[SabotageAndi]

FYI - for generating the code behind file there is already a MSBuild integration http://www.specflow.org/documentation/Generate-Tests-from-MsBuild/

[SabotageAndi]

A lot of changed in the last 2 years and I think this is not more up to date.
So I am closing this issue.

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.