Ferdi: No community recipes get shown anymore

[vantezzen]

It looks like Franz has had a server-side update the last days.
Whenever I try to create create a new service using my Ferdi test account, I only get a very limited number of recipes that I can choose from (see video below).

Looking further into this, Franz's API is now returning the setting isCommunityRecipesIncludedInCurrentPlan: false. I have just published d06c47a which resets this feature on the client-side, but it looks like Franz's API also has a protection that only returns recipes that are "included in the current plan".

Removing all community recipes (i.e. most recipes that Ferdi has) drastically limits Ferdi's use.

Does anyone have an idea on how we could fix this?
An idea would be to setup a custom Ferdi server at getferdi.com to replace the Franz API and rebuilding a list of community packages, but this would require a server and maintenance to keep all user data save.

[Gaboris]

Hmmm... yeah that was expected I think. To be fair Franz's premium content should all rely on stuff tied to their servers and let all other functionality unhindered.

Couldn't Ferdi just include all the recipes by default and not rely on a server? How often are those things changed? Couldn't it be handled manually like that?

[Makazzz]

I am using the 5.3.4-beta.1 and I still see all the service with the default server of franz for the moment.

[vantezzen]

@Makazzz this change is currently only visible when using the Franz Development API, not the normal API server, but I guess that this change will roll out to the normal server sometime in the near future.
This is why I think we should already prepare and think of possible solutions so that we are prepared when it does roll out.

[Makazzz]

Good to know.

[kytwb]

@vantezzen What about taking a snapshot of the response payload and hardcode it for now or use it as a fallback if the API doesn't return anything? Until we figure a longer-term plan and see if it's worth having to maintain a private server.

[vantezzen]

Good idea.
My only concern with this is that Ferdi will first fetch the list of available recipes, then download the single recipes separately. I don't currently know if the recipe download is also restricted or if the protection is only implemented into the recipe listing.
I have downloaded the current list of all recipes, we'll have to test if this works.

Edit: I have now also downloaded the recipe files - just in case.

[vantezzen]

Couldn't Ferdi just include all the recipes by default and not rely on a server? How often are those things changed? Couldn't it be handled manually like that?

My original argument would have been that all these files would be too large, but it looks like all 72 current recipes combined are only around 2.3MB, 1.4MB if we ship them compressed. Adding to the ca. 90MB Ferdi already has this wouldn't make too much of a difference.
I'll look into how hard it is to implement this into the code.

[vantezzen]

I have now added this functionality via 64ab923.
Recipes are now stored inside getferdi/recipes and will be included as a submodule inside this repository. It works fine in development but we'll have to see if building it breaks things.

I've also added console output indicating if it uses the build-in recipe files or fetches them from the server - if it outputs "[ServerApi::getRecipePackage] Using internal recipe file" when creating a new, unused service everything works.

Overall, this should now add ca. 1.4MB to the package size.

[vantezzen]

I do however still think that (if we get enough money through OpenCollective) we should run our own Ferdi server as I don't think Franz's creators will be too happy about this fork using their servers without paying for premium accounts.

[kytwb]

@vantezzen Agreed, we should remove the dependency on Franz resources usage as soon as possible. I think https://github.com/getferdi/ferdi/issues/50 and #5 will bring us closer to it. We will still need to make sure new users with an existing Franz account trying to transition to Ferdi can still import their workspaces and such.

In the meantime, I'm happy to provide access to an EC2 instance if you want to start taking a look at it, just let me know.

[kytwb]

Recipes are now stored inside getferdi/recipes and will be included as a submodule inside this repository. It works fine in development but we'll have to see if building it breaks things.

In Travis it should be fine (see this line). In AppVeyor it is not ok I think as the build shows no mention of the submodules being taken in consideration:

Build started
git clone -q --branch=master https://github.com/getferdi/ferdi.git C:\projects\ferdi
git checkout -qf 80fa8e0971d2caf707e3abab29447948ba2ba9c7
APPVEYOR_CACHE_SKIP_RESTORE is set to 'true' - skipping cache restore
Running Install scripts
Install-Product node 10
Uninstalling node 8.16.0 (x86)...
Installing node 10.16.0 (x86)...
npx lerna bootstrap

Should be fixed via https://github.com/getferdi/ferdi/commit/5d2a1b816f1f668e02a8b49aec55b8dba104d213.

Edit: https://ci.appveyor.com/project/kytwb/ferdi/builds/27677667

Build started
git clone -q --branch=master https://github.com/getferdi/ferdi.git C:\projects\ferdi
git checkout -qf 5d2a1b816f1f668e02a8b49aec55b8dba104d213
APPVEYOR_CACHE_SKIP_RESTORE is set to 'true' - skipping cache restore
Running Install scripts
git submodule update --init --recursive
Submodule 'recipes' (https://github.com/getferdi/recipes) registered for path 'recipes'
Cloning into 'C:/projects/ferdi/recipes'...
Submodule path 'recipes': checked out 'bd8c25be7961d167d315e76276627306b2379dc2'
Install-Product node 10
Uninstalling node 8.16.0 (x86)...
Installing node 10.16.0 (x86)...
npx lerna bootstrap

[vantezzen]

In the meantime, I'm happy to provide access to an EC2 instance if you want to start taking a look at it, just let me know.

Yes, I could set up the EC2 instance.