Arduino-cli: Inform user when the indexes are outdated

[endorama]

Hello team :)

Today I was fighting against a compilation error when using a library (ArduinoIoTCloud to be precise). For some reason a type that should have been defined wasn't.

After some debugging the issue was discovered: my library was outdated. I was using the 0.8.0 while the latest available was 0.11.1.

But until someone told me about running arduino-cli lib updated-index I wasn't able to discover that. I was using arduino-cli lib search to search for the library and check if my version was the latest, and nowhere there was an hint about the fact that my library index was outdated.

So I'm here proposing a small feature request that may help in such cases: include a log when running any lib related command to inform the user if their library index is out of date in relation to the latest published. Is this possible? Would have greatly helped in my case.

Once that was done I tried compiling and had another error (missing SNU.h). Again I was left in the dark, without any clue. I tried running core upgrade, but it didn't work. Guessed why? Yes, you got it: I had the core index outdated. Everything was fixed with core update-index and core upgrade, but the experience was a real pain.

I would have loved something that alerted me that not only I had outdated things installed but that I had outdated indexes, so searching for newer version was incomplete due to this.

Thank you and keep up the amazing work :heart:

PS: I have no idea (as I don't know how to downgrade my library index now) if the outdated command is affected by the same behaviour: does that command show outdated in relation to the latest "online" library index or does it show outdated based on the local library index?

[ubidefeo]

@endorama
we recently introduced arduino-cli update command which also accepts --show-outdated.
Like for a package manager you have to manually launch it and see what needs an upgrade.

a small feature request that may help in such cases: include a log when running any lib related command to inform the user if their library index is out of date in relation to the latest published.

this could be possible, but consider that we'd need to send a query every time a user uses the lib command. I'm not sure about that.
can you detail a specific workflow?

why would a user's due diligence arduino-cli update --show-outdated at launch not be enough?
I believe such command (had you known about it) would have alerted you (had you known about it :D )

[matthijskooijman]

A lighter solution might be to print the date of the last index update on various lib commands, so a user could at least notice that he's looking at old information (without really needing to know whether any new information actually exists) and realize that he should update the indexes.

[ubidefeo]

@matthijskooijman
I like this idea
how would you see this being printed out?
Also this information might get lost in the terminal...

We have a release field in the JSON, but not sure what's that for since it's always empty
@per1234 ?

{
    "library": {
      "name": "WiFi101",
      "author": "Arduino",
      "maintainer": "Arduino \[email protected]\u003e",
      "sentence": "Network driver for ATMEL WINC1500 module (used on Arduino/Genuino Wifi Shield 101 and MKR1000 boards)",
      "paragraph": "This library implements a network driver for devices based on the ATMEL WINC1500 wifi module",
      "website": "http://www.arduino.cc/en/Reference/WiFi101",
      "category": "Communication",
      "architectures": [
        "*"
      ],
      "install_dir": "/data/Dropbox/AppsDocuments/Arduino/libraries/WiFi101",
      "source_dir": "/data/Dropbox/AppsDocuments/Arduino/libraries/WiFi101/src",
      "real_name": "WiFi101",
      "version": "0.16.0",
      "license": "Unspecified",
      "location": 1,
      "layout": 1,
      "examples": [
        "/data/Dropbox/AppsDocuments/Arduino/libraries/WiFi101/examples/AP_SimpleWebServer",
        "/data/Dropbox/AppsDocuments/Arduino/libraries/WiFi101/examples/CheckWifi101FirmwareVersion",
        "/data/Dropbox/AppsDocuments/Arduino/libraries/WiFi101/examples/ConnectNoEncryption",
        "/data/Dropbox/AppsDocuments/Arduino/libraries/WiFi101/examples/ConnectWithWEP",
        "/data/Dropbox/AppsDocuments/Arduino/libraries/WiFi101/examples/ConnectWithWPA",
        "/data/Dropbox/AppsDocuments/Arduino/libraries/WiFi101/examples/FirmwareUpdater",
        "/data/Dropbox/AppsDocuments/Arduino/libraries/WiFi101/examples/MDNS_WiFiWebServer",
        "/data/Dropbox/AppsDocuments/Arduino/libraries/WiFi101/examples/Provisioning_WiFiWebServer",
        "/data/Dropbox/AppsDocuments/Arduino/libraries/WiFi101/examples/ScanNetworks",
        "/data/Dropbox/AppsDocuments/Arduino/libraries/WiFi101/examples/ScanNetworksAdvanced",
        "/data/Dropbox/AppsDocuments/Arduino/libraries/WiFi101/examples/SimpleWebServerWiFi",
        "/data/Dropbox/AppsDocuments/Arduino/libraries/WiFi101/examples/WiFiChatServer",
        "/data/Dropbox/AppsDocuments/Arduino/libraries/WiFi101/examples/WiFiPing",
        "/data/Dropbox/AppsDocuments/Arduino/libraries/WiFi101/examples/WiFiSSLClient",
        "/data/Dropbox/AppsDocuments/Arduino/libraries/WiFi101/examples/WiFiUdpNtpClient",
        "/data/Dropbox/AppsDocuments/Arduino/libraries/WiFi101/examples/WiFiUdpSendReceiveString",
        "/data/Dropbox/AppsDocuments/Arduino/libraries/WiFi101/examples/WiFiWebClient",
        "/data/Dropbox/AppsDocuments/Arduino/libraries/WiFi101/examples/WiFiWebClientRepeating",
        "/data/Dropbox/AppsDocuments/Arduino/libraries/WiFi101/examples/WiFiWebServer"
      ],
      "provides_includes": [
        "WiFi101.h"
      ]
    },
    "release": {}
  }

[matthijskooijman]

I wasn't thinking so much about the JSON output, but the human-readable output. E.g. something like:

matthijs@grubby:~$ arduino-cli lib search lmic
Searching library index, last updated at 2020-01-03 12:21:33
Name: "IBM LMIC framework"
  Author: IBM
  Maintainer: Matthijs Kooijman <[email protected]>
  Sentence: Arduino port of the LMIC (LoraWAN-in-C, formerly LoraMAC-in-C) framework provided by IBM.
  Paragraph: Supports SX1272/SX1276 and HopeRF RFM92/RFM95 tranceivers
  Website: http://www.research.ibm.com/labs/zurich/ics/lrsc/lmic.html
  Category: Communication
  Architecture: *
  Types: Contributed
  Versions: [1.5.0+arduino-2, 1.5.0+arduino-0, 1.5.0+arduino-1]
... more results ...

Or maybe the message could be more prominent and at the end of the results, but then only when the last update is more than, say 3 days ago or so? So something like:

matthijs@grubby:~$ arduino-cli lib search lmic
Name: "IBM LMIC framework"
  Author: IBM
  Maintainer: Matthijs Kooijman <[email protected]>
  Sentence: Arduino port of the LMIC (LoraWAN-in-C, formerly LoraMAC-in-C) framework provided by IBM.
  Paragraph: Supports SX1272/SX1276 and HopeRF RFM92/RFM95 tranceivers
  Website: http://www.research.ibm.com/labs/zurich/ics/lrsc/lmic.html
  Category: Communication
  Architecture: *
  Types: Contributed
  Versions: [1.5.0+arduino-2, 1.5.0+arduino-0, 1.5.0+arduino-1]
... more results ...

Note: these results are based on your local library index, which was last updated on 2020-01-03. Run "arduino-cli update" to update to the latest results.

OTOH, I guess that always showing the entry might be useful, to prevent the case where someone has recently updated, changes his own library and can't figure out why the new version isn't showing up somehow.

[ubidefeo]

I like the second approach since we're mainly talking about the library index and not versions of the libraries.
Actually thinking in this direction we might output the message every time an index is older than X days for both core and library
Thank you for the thought trigger, @matthijskooijman 👍
@endorama @rsora @cmaglie what do you think?

[per1234]

We have a release field in the JSON, but not sure what's that for since it's always empty

@ubidefeo the [].release field contains the index data for the latest version of the library when the index contains a newer version than the currently installed one. When the latest version of the library is installed, that field is empty.

[ubidefeo]

thank you @per1234
I think we could "enhance" this field expanding its capabilities.
Needs some thinking

[endorama]

TL;DR: @ubidefeo I like the direction the fix is going, that would be a sufficient mitigation in my view.

this could be possible, but consider that we'd need to send a query every time a user uses the lib command. I'm not sure about that.
can you detail a specific workflow?

Yes, automatically update the index on lib search or core search (or around any other search functionality), with non-blocking errors and automatic fallback to the local copy.

I would consider what's the use case for having a not-up-to-date index. Is there a use case?

why would a user's due diligence arduino-cli update --show-outdated at launch not be enough?
I believe such command (had you known about it) would have alerted you (had you known about it :D )

How am I, as a user, expected to know (or discover) it? The text proposed by @matthijskooijman (Note: these results are based on your local library index, which was last updated on 2020-01-03. Run "arduino-cli update" to update to the latest results.) feels very appropriate to help.

Side note: my laptop and servers have auto updates enabled since years 😉 and yours too, probably

[ubidefeo]

@endorama

Yes, automatically update the index on lib search or core search (or around any other search functionality), with non-blocking errors and automatic fallback to the local copy.

interesting and doable

I would consider what's the use case for having a not-up-to-date index. Is there a use case?

There's no interest, of course. An updated set of index files is always good

The reason why I talk about "due diligence" is because I and many HomeBrew users out there, but also Linux shell users, manually issue a brew update+brew outdated or apt update + apt list --upgradable

Adding such automated update when searching makes sense in matter of UX, so I'm adding this to our backlog :)