Shaarli: archive.org - ensure the linked page is present in the archive

[virtualtam]

Depends on #283

@virtualtam:

There is a usage caveat: the plugin only adds a link to the archived link on archive.org _(yup, I'm aware it's the exact intended usage ^^)_

Consequently, the following use case is broken:

• a link to http://websi.te is archived
• no one really cares about that particuliar site, thus it is never archived on archive.org
• the site goes down
• some time later, the Shaarli user attempts to retrieve the archive... which doesn't exist

Is it possible for a plugin to add postAdd, postEdit, postDelete... hooks that would be called when a link is modified?

In this case, this would mean:

• request the page from the archive.org cache
  
  
  
  • it exists: that's a good start, maybe we can ask for it to be refreshed?
  
  
  • it does not: ask for it to be added to the archive
  
  

This could be done through the JSON API or the Advanced Search form

@ArthurHoaro:

It's possible because there is a save_link and delete_link hooks (called before the action though). It requires a bit of work though.

[nodiscc]

Shouldn't this be optional? Users of this plugin may not want to send every bookmarked page URL to archive.org (specifically private links). It can be either

• A global configuration option (Send bookmarked link URLs to archive.org)
• A per-link option in the save dialog.

or both?

[virtualtam]

Both seem necessary: a default config option + a per-link checkbox

The use case may also depend on how long it takes for archive.org to cache a page -the operation takes some time if there is multimedia content, included JS & CSS, etc.

[mro]

did you check, that's compliant to archive.org terms of use?

[nodiscc]

archive.org's TOS can be found here https://archive.org/about/terms.php and I did not find anything preventing us from accessing it's services in an automated way.

[mro]

@nodiscc awesome. Didn't read the legalese however, but it's good to not violate it knowingly and have the terms at hand.

[virtualtam]

did you check, that's compliant to archive.org terms of use?

Yup.

• the TOS do not discourage (nor encourage) programatically adding content to the Wayback Machine
• there is a set of APIs to query content and snapshot information
  
  
  
  • JSON API
  
  
  • Wayback API with Memento support
  
  
• the FAQs extensively cover the possible usages of the Wayback Machine; the closest information regarding content addition is:

How can I help the Internet Archive and the Wayback Machine?

The Internet Archive actively seeks donations of digital materials for preservation. If you have digital materials that may be of interest to future generations, please let us know by sending an email to info at archive dot org. The Internet Archive is also seeking additional funding to continue this important mission. You can click the donate tab above or click here. Thank you for considering us in your charitable giving.

From a functional point of view, I see no difference between:

• manually entering an URL on the site and asking for its archival
• having an automated tool perform this for a batch of links, with a small delay between requests not to flood the site

From an ethical / community point of view, it would be nice to make small donations to the archiving platform if you frequently sollicit its services: bandwidth, storage space, web TARDISes, _und so weiter_ :)

[qwertygc]

Just question, if we make a backup archive on own server, it's not more resiliant ? we can use archive.org process with the local backup. Streisand preventive effect :-)

[Aldarone]

I made a plugin that does just that : https://framagit.org/Alda/shaarli-autoarchiveorg

[ArthurHoaro]

Neat! Feel free to add it in the wiki. Also, I may suggest a file_get_contents fallback function if cURL isn't available.