Cordova-sqlite-storage: Support building this plugin in Visual Studio [2017 RC] out of the box

[vladimir-kotikov]

Visual Studio '15' Preview is now publicitly available and we'd like to make sure that building apps using this plugin will work with VS15 just out of the box. However as of now the plugin requires VS 2015 to be installed as it specifies <PlatformToolset>v140</PlatformToolset> in SQLite3.UWP.vcxproj and will not build from VS15 which ships with toolset v141

So, to support VS15 I propose to:

• bump toolset version in SQLite C++ project
• to make sure that the users of previous versions of VS are not broken, ensure that the version of the plugin with updated toolset specifies new requirement "cordova-windows": ">=4.4.1" in package.json (see more about plugin requirements here: Specifying Cordova dependencies).

The support for build with VS 2015 then might be moved to Cordova-sqlite-legacy-build-support (just as in #579) so users would still be able to build the plugin with previous version of VS.

[brodybits]

Hi @vladimir-kotikov,

I have a couple questions:

• Is there any way we could get rid of the <PlatformToolset> dependency altogether?
• Would the proposed "cordova-windows": ">=4.4.1" requirement be a reliable indicator of whether or not the developer has installed VS 15 [Preview]?

In case we cannot get rid of the <PlatformToolset> dependency I would rather support VS15 preview in a special plugin version until VS15 is actually released. I am undecided whether to participate in this trial myself. If not I will ask if you can do this in a fork until VS15 is officially released.

Thanks!

[vladimir-kotikov]

Is there any way we could get rid of the dependency altogether?

Unfortunately this wouldn't work - and even if it worked it would have a significan drawback - one app compiled from two different versions of VS would have different binaries because of the different compilers, which IMO is wrong.

Would the proposed "cordova-windows": ">=4.4.1" requirement be a reliable indicator of whether or not the developer has installed VS 15 [Preview]?

I'd say that it the user have cordova-windows <4.4.1 then he must have VS 2015 (with toolset v140) or less because these versions of cordova-windows are missing some critical fixes, required to work with VS15.

[brodybits]

Thanks @vladimir-kotikov. I will probably make a new "vsnext" version next week to support testing with VS15. It will _not_ support PhoneGap Build. (I can support PhoneGap Build based on the legacy version if you need it for any reason.) I also do not expect to publish it on nom unless you need it for some reason.

As I write this I wonder if there could be a way to make this <PlatformToolset> setting dependent on a configuration item in config.xml?

[mbraude]

Hey @brodybits. I'm the dev lead for the Cordova tools in Visual Studio. I think it would help if I gave you context. Can you email me at [email protected] so we can set up a time to talk? Thanks.

[brodybits]

Hey @vladimir-kotikov @mbraude I tried installing VS 15 preview 5 and using it to build and run https://github.com/brodybits/Cordova-sqlite-bootstrap-test with and without applying the first proposed fix (replace v140 with v141) to this plugin. With or without the first proposed fix the referenced test app does not work when built by VS15. In general I get no response if I do a sqlite string test and it crashes if I try the native alert test. If I try a very simple dialog test with no sqlite plugin it works fine.

Unfortunately I cannot tell whether the problem lies within this plugin, VS15 preview, or both. I wish there were a sample Cordova C++ UWP plugin that can be supported without the SQLite3 part. I would be happy to submit a sample if necessary.

[vladimir-kotikov]

Odd. It works for me out of the box. String test gives me the following alert:

I suspect that there might be something wrong with the cordova project itself - at least symptoms sound like the cordova-plugin-dialogs plugin hasn't been installed into the project and that causes navigator.notification.alert to crash the application since there is no navigator.notification object.

[brodybits]

My bad. If I build and run the Cordova-sqlite-bootstrap-test in VS 15 preview 5 with the plugins installed properly it works fine.

I think the problem now is that this plugin will not build if someone has _only_ VS 15 installed. Can you confirm that this is correct?

[brodybits]

@vladimir-kotikov I took the liberty to reach out to you in https://github.com/phonegap/build/issues/562#issuecomment-259182139 in case you can help us with PhoneGap Build. This is a blocking issue for an existing customer.

@vladimir-kotikov @mbraude I would like to propose the following to prepare for the upcoming VS 15 release:

• remove support for Windows 8.1/Windows Phone 8.1 from this version (#579)
• make a new release with the proposed changes: reference to platform toolset v141 and "cordova-windows": ">=4.4.1" requirement
• document this in README.md in case of manual Cordova CLI users
• then until VS 15 is released: for each set of changes make two incremental plugin releases:
  
  
  
  • incremental plugin release with old toolset v140 and no cordova-windows requirement, working for VS 2015
  
  
  • another incremental plugin release with new toolset v141 and new cordova-windows requirement
  
  

I think this should solve the problem with VS 15 previews while keeping VS 2015 users up to date.

Please report if this sounds OK or not, thanks!

[vladimir-kotikov]

@brodybits, I will take a look at that phonegap-build problem

[vladimir-kotikov]

The migration plan looks good! Just a couple of questions/suggestions:

1. Do you consider this as a breaking change? It sounds like it is, so probably it will be a 2.0.0 version, right? Major version change also would be an additional reminder to evaluate migration issues and IMO could simplify maintenance of package.json because you would have the same cordovaDependencies (see below) in both versions and would not need to update it at every release.

json cordovaDependencies: { "1.4.x": { "cordova-windows": "<4.4.1"}, "2.x": { "cordova-windows": ">=4.4.1"} }

1. Also as an additional protection (though I'm not sure if it's really needed), you could hide _2.x ("edge")_ version (with VS15 support) behind a separate dist-tag on NPM so regular npm install call would fetch _"latest"_, not _"edge"_. This way
   
   
   
   • users of cordova <= 6.1 (which doesn't know anything about 'cordovaDependencies' and advanced plugin fetching) would always get _"stable"_ with VS 2015 support, so they will not be broken unless they explicitly install cordova-sqlite-storage@edge
   
   
   • users of cordova > 6.1 with would be able to get _"edge"_ because cordova plugin fetching logic will use 2.x as described in cordovaDependencies of 1.4.x
   
   
2. > document this in README.md in case of manual Cordova CLI users

You could also use <info> element in plugin.xml to draw an additional attention, just as whitelist plugin does

@brodybits, @mbraude, what do you think?

I will be happy to help with any issues related to this migration and could send a PR with all necessary changes.

[brodybits]

Yeah I think this should be considered a breaking change.

For suggestion 1, I can think of the following alternatives for handling plugin updates with the proposed cordovaDependencies idea:

• older cordova-windows users do not get future plugin updates
• use multiple branches and merge periodically (there could still be an update delay for certain users)
• alternate between VS 2015 (1.4.x) and VS 15 (2.x) as described above when making future releases

For suggestion 2: I must admit I have not used dist-tag(s) before and just read a nice article at http://jbavari.github.io/blog/2015/10/16/using-npm-tags/. This might be good in cases where a someone uses an updated version of Cordova CLI with VS 2015.

I will probably need a little time to decide which would work best for me. I will keep you posted and let you know if I need any help with the changes needed.

I wish there was a way to configure the <PlatformToolset> value in config.xml or build.json, but it is probably not worth fixing since there are not so many plugins using C++ UWP projects for Windows.

For PhoneGap Build they do not seem so helpful. (I did get responses from people I had met at PhoneGap Day but not from someone who can fix it.) Here are the discussion/issue links:

• https://forums.adobe.com/thread/2230161
• https://github.com/phonegap/build/issues/562
• https://forums.adobe.com/message/9123283

A possible approach may be to build a sample Cordova plugin (with echo functionality for example) with a C++ UWP project (Windows 10 only), submit a test app with that sample plugin, and work with PhoneGap Build to sort it out. Just an idea. _UPDATE: Partial answer given by @vladimir-kotikov in https://forums.adobe.com/message/9123283, directions/pointers still needed._

Thanks for your efforts here.

[vladimir-kotikov]

older cordova-windows users do not get future plugin updates

Yeah, this might be a good idea to reduce maintenance cost once 2.x became stable

use multiple branches and merge periodically (there could still be an update delay for certain users)

This is very close to what I proposed - you can have 1.4.x and 2.x branches with the same cordovaDependencies, periodically merge/cherry-pick changes from one to another and do releases of 1.4.x and 2.x in parallel from different branches

I wish there was a way to configure the value in config.xml or build.json

Yeah, I agree, It would be helpful addition, especially looking at the android platform which already has a way to provide custom arguments to build system (--gradleArg)

[mbraude]

@brodybits this plan sounds good to me. Thanks!

[brodybits]

Progress and questions so far

My goals

• First get it working on the desktop, ideally in both x86 and x64 builds
• then deal with ARM mobile device
• keep it as simple and easy as possible for plugin users on Windows 10

Progress

VS 15 preview 5

I was able to replace v140 with v141 in SQLite3.UWP.vcxproj and get it working with VS 15 preview 5 on the desktop (x86) (see PR #586). (This is with sqlite updated to 3.15.1 which should not make a difference here.) I had trouble with building for my ARM mobile device. (I may have got it working with sqlite 3.8.10.2 before but do not remember for sure. Again I don't think the sqlite version matters here.)

VS 2017 RC

I then tried replacing VS 15 preview 5 with VS 2017 RC. During the installation of VS 2017 RC I selected the Universal Windows Platform development and Desktop development with C++ Windows "Workloads". I did _not_ select .NET desktop development since I did not think it would be relevant here. When I tried to open a newly generated project with the modified plugin in VS RC I got a message that some Universal Windows component was missing. I went through some dialogs to deal with the missing component then stopped it. (I discovered with preview 5 that it was much better to reopen the installer GUI and install the missing components from there.)

Looking through the installer GUI I found a "Visual C++ runtime for UWP" component that was not already selected. I tried selecting this component and installing it. Then I made another newly generated project with the modified plugin, tried opening it with VS RC and got the same missing component dialog again.

At this point I tried generating a project with no sqlite plugin and it worked on both desktop and mobile device.

Ideas for next steps

Some other ideas I can think of to try installing:

• "C++/CLI support" component under Desktop development with C++ on the right side of the installation GUI (also under Compilers, build tools, and runtimes if I click Individual components). If I hover over this component in either place it shows me that this is for the .NET framework.
• .NET desktop development under Windows "Workloads"
• Mobile development with C++ workload (apparently for cross platform Android/iOS/Windows development)
• "Visual C++ compilers and libraries for ARM" under Compilers, build tools, and runtimes if I click Individual components

NOTE: If I click the Mobile development with C++ workload it does not seem to automatically select "Visual C++ compilers and libraries for ARM".

Questions

@vladimir-kotikov can you give me some guidance which components I should try installing to get the build working on the desktop?

Is there any way to get this working without the full .NET desktop development or Mobile development with C++ workloads?

I suspect that I will have to explicitly add the "Visual C++ compilers and libraries for ARM" to get it working on my ARM mobile device. Does this sound right?

Thanks!

[vladimir-kotikov]

@brodybits, sadly I haven't tried VS 2017 RC yet, going to install it now and test your changes. I will let you know once i have some results.

[brodybits]

Thanks @vladimir-kotikov. In principle my change was just to replace v140 with v141. If you see anything else that has to be changed a new PR would be great (I would probably cherry-pick it and fix some change info).

[vladimir-kotikov]

Okay, I have just installed VS 2017 with "Universal Windows Platform development" workload, and two optional components for this workload - "C++ UWP Support" and "Windows 10 SDK (10.0.10586.0)" and tried the following:

cordova.cmd plugin add https://github.com/brodybits/cordova-sqlite-storage#cb-vs2017-wip --save
cordova.cmd platform add windows -d --save
set VSINSTALLDIR=c:\Program Files (x86)\Microsoft Visual Studio\2017\Community
cordova.cmd build windows -- --archs="x86 x64 arm" --appx uap

Build succeeded and I can see built packages in platforms/windows/AppPackages folder. Notice that to use MSBuild that is shipped with new Visual Studio you'll need to set VSINSTALLDIR as I did above - this is a mechanism that Cordova-windows uses to detect where VS 2017 is installed.

So in theory you wouldn't need to install workloads other than "Universal Windows Platform development" with some additional components.

If you're still having build issues, could you share an error output from 'cordova build windows'

Upd. I'm also going ot install "Mobile development with Javascript" workload to test how this works in TACO

[brodybits]

Thanks @vladimir-kotikov. I will try this early next week and report.

[brodybits]

Trying without arm:

Setting:

λ set VSINSTALLDIR="c:\Program Files (x86)\Microsoft Visual Studio\2017\Community"

Build command:

λ cordova build windows -- --release --archs="x86 x64" --appx uap                                                       

Build output:

Building project: C:\Users\Chris\Documents\git\dee\platforms\windows\CordovaApp.Windows10.jsproj                        
        Configuration : debug                                                                                           
        Platform      : x86                                                                                             
  Patching 10 in prebuild event...                                                                                      
  Injected base.js reference to the /www/index.html                                                                     
  Removing /( *)(<script\s+(?:type="text\/javascript"\s+)?src="\/\/Microsoft.WinJS.2.0\/js\/base.js">\s*<\/script>)(\s* 
  )/ from /www/index.html                                                                                               
  Removing /( *)(<script\s+(?:type="text\/javascript"\s+)?src="\/\/Microsoft.Phone.WinJS.2.1\/js\/base.js">\s*<\/script 
  >)(\s*)/ from /www/index.html                                                                                         
C:\Program Files (x86)\MSBuild\14.0\bin\Microsoft.Common.CurrentVersion.targets(724,5): error : The OutputPath property is not set for project 'SQLite3.UWP.vcxproj'.  Please check to make sure that you have specified a valid combination of Configuration and Platform for this project.  Configuration='debug'  Platform='Win32'.  You may be seeing this message because you are trying to build a project without a solution file, and have specified a non-default Configuration or Platform that doesn't exist for this project. [C:\Users\Chris\Documents\git\dee\plugins\cordova-sqlite-storage\src\windows\SQLite3-Win-RT\SQLite3\SQLite3.UWP\SQLite3.UWP.vcxproj]                                                             
Error: C:\Program Files (x86)\MSBuild\14.0\bin\msbuild.exe: Command failed with exit code 1

[vladimir-kotikov]

Hmm.. I think this is because you're setting VSINSTALLDIR with quotes. It should be set VSINSTALLDIR=c:\Program Files (x86)\Microsoft Visual Studio\2017\Community

[brodybits]

I got it running on desktop (x86) and mobile ARM with some trial and error and expect to release the "edge" trial within the next 1-2 days or so.

The following commands worked for me (desktop build, got it running with the test suite):

set VSINSTALLDIR=c:\Program Files (x86)\Microsoft Visual Studio\2017\Community
cordova build windows -- --release --archs="x86 x64" --appx uap # probably not needed
cordova run windows -- --release --archs="x86" --appx uap

Cordova did not seem to see my --release flag but ok, good enough for now.

When I tried the following command:

cordova build windows -- --release --archs="x86 arm" --appx uap

I got the following output (x86 part omitted):

Building project: C:\Users\Chris\Documents\git\dee\platforms\windows\CordovaApp.Windows10.jsproj
        Configuration : debug
        Platform      : arm
C:\Program Files (x86)\MSBuild\15.0\Microsoft.Common.targets\ImportBefore\Microsoft.NetNative.ImportBefore.targets(25,3 ): warning MSB4011: "C:\Program Files (x86)\MSBuild\15.0\.Net\.NetNative\15.0.24208\Microsoft.NetNative.Settings.target s" cannot be imported again. It was already imported at "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\ MSBuild\15.0\Microsoft.Common.targets\ImportBefore\Microsoft.NetNative.ImportBefore.targets (25,3)". This is most likel y a build authoring error. This subsequent import will be ignored. [C:\Users\Chris\Documents\git\dee\platforms\windows\ CordovaApp.Windows10.jsproj]
C:\Program Files (x86)\MSBuild\15.0\Microsoft.Common.targets\ImportAfter\Microsoft.Net.CoreRuntime.ImportAfter.targets( 17,3): warning MSB4011: "C:\Program Files (x86)\MSBuild\15.0\.Net\CoreRuntime\Microsoft.Net.CoreRuntime.targets" cannot  be imported again. It was already imported at "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\1 5.0\Microsoft.Common.targets\ImportAfter\Microsoft.Net.CoreRuntime.ImportAfter.targets (17,3)". This is most likely a b uild authoring error. This subsequent import will be ignored. [C:\Users\Chris\Documents\git\dee\platforms\windows\Cordo vaApp.Windows10.jsproj]
C:\Program Files (x86)\MSBuild\15.0\Microsoft.Common.targets\ImportAfter\Microsoft.NetNative.ImportAfter.targets(16,3):  warning MSB4011: "C:\Program Files (x86)\MSBuild\15.0\.Net\.NetNative\15.0.24208\Microsoft.NetNative.targets" cannot b e imported again. It was already imported at "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15. 0\Microsoft.Common.targets\ImportAfter\Microsoft.NetNative.ImportAfter.targets (16,3)". This is most likely a build aut horing error. This subsequent import will be ignored. [C:\Users\Chris\Documents\git\dee\platforms\windows\CordovaApp.Wi ndows10.jsproj]
  Patching 10 in prebuild event...
  pch.cpp
  Constants.cpp
  Database.cpp
C:\Users\Chris\Documents\git\dee\plugins\cordova-sqlite-storage\src\windows\SQLite3-Win-RT\SQLite3\Database.cpp(48): wa rning C4244: 'return': conversion from 'sqlite3_int64' to 'int', possible loss of data [C:\Users\Chris\Documents\git\de e\plugins\cordova-sqlite-storage\src\windows\SQLite3-Win-RT\SQLite3\SQLite3.UWP\SQLite3.UWP.vcxproj]
  Statement.cpp
  sqlite3.c
arm\debug\Constants.obj : fatal error LNK1112: module machine type 'X86' conflicts with target machine type 'ARM' [C:\U sers\Chris\Documents\git\dee\plugins\cordova-sqlite-storage\src\windows\SQLite3-Win-RT\SQLite3\SQLite3.UWP\SQLite3.UWP. vcxproj]
Error: c:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Bin\msbuild.exe: Command failed with exit code 1

I tried adding the "Visual C++ compilers and libraries for ARM" individual component in the installer but it didn't seem to help.

Then I tried on a clean project with the test suite (with the modified plugin and windows platform installed):
```
cordova build windows -- --release --archs="arm" --appx uap # probably not needed here

failed cordova run windows attempts omitted here

cordova run windows --device --archs="arm" -- -phone --appx uap
```
and it installed on my mobile. I had to start it manually.

[vladimir-kotikov]

Cordova did not seem to see my --release flag but ok, good enough for now

You need to pass this flag _before_ --, i.e. cordova run windows --release -- --archs="x86" --appx uap

I had to start it manually

Yeah, I think that's the issue with cordova-windows - deploy util was failing to start app on device so we had to disable automatic start after installation.

The build with mixed 'x86' and 'arm' architectures works perfectly for me. Perhaps you have some additional linker options set up globally, that causes that. I found a good guide to diagnose this sort of problems on SO

[brodybits]

I just published [email protected] with edge tag. I verified that cordova plugin add cordova-sqlite-storage@edge installs the new version and the normal usage installs 1.5.0. Thanks for the help!

[vladimir-kotikov]

Hey, Chris. Sorry for annnoyance but this still needs some tweaking to work as it expected to be. Specifying exact version (or dist tag) for cordova plugin add works perfectly, but we'd really need to install 2.0.0 version _by default_ with cordova-windows>4.4.1. This is not the case right now due to

• 1.5.0 version doesn't have cordovaDependencies in its' package.json, pointing to 2.0.0 that has a [email protected] requirement
• when cordova tries to determine version to fetch it _ignores all versions that have any 'prerelease' prefix_ (-dev, -edge, etc.) - this is by design and there doesn't seem to be any appropriate workarounds.

So i'd like to ask you to add this to package.json

  "engines": {
    "cordovaDependencies": {
      "2.0.0": { "cordova-windows": ">4.4.1" }
    }
  },

to both 1.5.x and 2.x branches, so cordova would be able to fetch proper version based on installed cordova-windows. This will also require to republish both 1.5x (will became 1.5.1) and 2.0.0-edge1 (will became just 2.0.0)

[brodybits]

I will fix this within the next 1-2 days or so.

Also do you have any suggestions that can help me with the problems in https://github.com/litehelpers/Cordova-sqlite-legacy-build-support/issues/10 (Windows 10 SQLite3 C++ library on PhoneGap Build)?

[vladimir-kotikov]

Thanks. Chris!
Yeah, I'm going to take a look at those problems soon

[brodybits]

@vladimir-kotikov if I publish newer versions such as 2.0.1, 2.1.0, ... would I have to update the cordovaDependencies part as well?

[vladimir-kotikov]

No, you wouldn't. Having "2.0.0": { "cordova-windows": ">4.4.1" } entry basically means that any version >=2.0.0 would satisfy this constraint.

[brodybits]

"2.0.0": { "cordova-windows": ">4.4.1" }

Should be "2.0.0": { "cordova-windows": ">=4.4.1" } right?

[vladimir-kotikov]

Hmm... 4.4.1 had a bug that was preventing it to work w/ VS 15/2017 properly, which has been fixed in 4.4.2 (see CB-11458 in RELEASENOTES ), so I'd better stick to 4.4.2

[brodybits]

@vladimir-kotikov I just published 1.5.1 and 2.0.0 with the tweaks you requested.

I asked a couple more questions on litehelpers/Cordova-sqlite-legacy-build-support#10. Can you or someone else from cordova-windows give me a quick answer?

[vladimir-kotikov]

Thanks Chris, I just verified it works perfectly, e.g. for [email protected]:

➜  test-sqlite-plugin cordova plugin add cordova-sqlite-storage -d
No version specified for cordova-sqlite-storage, retrieving version from config.xml
No version for cordova-sqlite-storage saved in config.xml
Attempting to use npm info for cordova-sqlite-storage to choose a compatible release
...
Unmet project requirements for latest version of cordova-sqlite-storage:
    cordova-windows (4.3.2 in project, >4.4.1 required)
Fetching highest version of cordova-sqlite-storage that this project supports: 1.5.1 (latest is 2.0.0)
Calling plugman.fetch on plugin "[email protected]"

[brodybits]

Thanks for the confirmation!

[brodybits]

When installing the latest version of Visual Studio 2017 RC: in addition to the big Universal Windows Platform development item I had to check "C++ Universal Windows Platform tools" on the right hand side.

I had also checked the big Desktop development with C++ item but unchecked it since I do not think it is necessary for this plugin. I will test and confirm here. _CONFIRMED_

[brodybits]

From the original description:

The support for build with VS 2015 then might be moved to Cordova-sqlite-legacy-build-support (just as in #579) so users would still be able to build the plugin with previous version of VS.

I will probably drop support for VS 2015 from the next release, and direct VS 2015 users to Cordova-sqlite-legacy-build-support. Please give a shout ASAP if this will be a problem, thanks.

[brodybits]

Support for Visual Studio 2017 is now part of the default storage-master branch. VS 2015 is now supported in litehelpers / Cordova-sqlite-legacy-build-support, dropped from this project to avoid any further confusion ref: #615 #599 and others

[andreujuanc]

@brodybits @vladimir-kotikov Everything works just fine! +1

[brodybits]

Something strange is that if I would install Visual Studio 2015 Update 3 on a new system it seems to include a version of Windows 10 SDK with platform toolset v141 by default. (I had to install some other components to get it to work with cordova-sqlite-legacy which supports old platform toolset v140.) But if I would try running a project with this plugin on the Visual Studio 2015 Update 3 installation it would still complain that I need to install a certain Windows 10 SDK component version. It would be very interesting if VS 2015 (Update 3) would actually work with this plugin if I would install the needed Windows 10 SDK component version.

Another side note is that in https://github.com/litehelpers/cordova-sqlite-ext/issues/60#issuecomment-302972686 @tuanbs contributed the following pointer to a workaround for using the old platform toolset v140 on VS 2017, which I still have not tried myself: https://developercommunity.visualstudio.com/content/problem/48806/cant-find-v140-in-visual-studio-2017.html