The docs website Java Agent release notes RSS feed includes a guid field. This guid field is used by some RSS feed readers to uniquely identify new posts and prevent re-downloading already seen posts.
In this specific case we're using Huginn and it's built in RSS agent to monitor the Java Agent Release Notes RSS feed.
The guid field in the XML feed is changing/being regenerated on a regular basis. My suspicion would be when some kind of build event occurs, but that's just a guess.
Because the guid is changing, the RSS reader thinks there are new posts available for download. Because every single guid for every single post is changing, it re-downloads hundreds of posts each time.
I do not know what is triggering the guid to change, but whatever it is occurring daily at some time in the evening UTC (approx 1800-2000 UTC).
Expected that guid do not change so that posts can be uniquely identified and prevent re-downloading.
Link to specific feed:
https://docs.newrelic.com/docs/release-notes/agent-release-notes/java-release-notes/feed.xml
N/A
@dogsbody-josh Thanks so much for submitting this issue! Looks like we (falsely) assumed that the guid was a stable value between the builds of the site 馃槵 as we use whatever Gatsby assigns as the id to that node. We'll see what we can do to make this value more stable to prevent this from happening.
@dogsbody-josh thanks for the detail on this issue like @jerelmiller our team will take a look at this problem. Do you have any work around at the moment to get around this problem?
@jpvajda I have switched to using a different 'agent' in Huginn to parse the XML directly. There's no workaround in the standard Huginn RSS agent as there is no way to alter which value it uses as the 'source of truth' for uniquely identifying already seen posts.
It may be possible to alter what other RSS readers use as the 'source of truth', perhaps to use post titles or pubDate, but that wasn't possible for us.
Switching to parsing the XML feed 'manually' allowed us to uniquely identify on title and pubDate, so others in this position might try those same actions to work around this issue.
Adding to Sprint 1, to see if we can resolve this issue.
Most helpful comment
@dogsbody-josh Thanks so much for submitting this issue! Looks like we (falsely) assumed that the
guidwas a stable value between the builds of the site 馃槵 as we use whatever Gatsby assigns as theidto that node. We'll see what we can do to make this value more stable to prevent this from happening.