Dsmr-reader: Gas meter reading failed to upload to mindergas on 08JAN

[mkruiver]

While the mindergas.nl output has been working for two days, and DSMR reader is still displaying gas usage, the gas meter reading of JAN08 was never uploaded to mindergas.nl.

Is there any logging available to check what might have gone wrong here?

Next export:
Jan. 10, 2017
Timestamp of the next export. Automatically updated by application.
SAVESave and continue editing

Maybe this?

Data processing is on schedule: Jan. 8, 2017

(While GAS STATISTICS
FOR DASHBOARD DISPLAY.
Data processing is on schedule: an hour ago)

[dennissiemensma]

I think that it's a timing bug. The gas positions are uploaded right at midnight, but the gasmeter sometimes takes a few minutes to report it's new hourly value to the smart meter.

This might be fixed by accident with #204, which is part of a patch release that just happens to be released in the next hour.

You could check your own logs to see when the gas meter positions should be uploaded:
(as sudo/root user)
zgrep MinderGas /var/log/supervisor/dsmr_backend.log.* -B 10

[dennissiemensma]

The patch is released. You should:

• Update to v1.5.2 first .
• Then force a re-upload of your missing gas positions:
  sudo -u postgres psql dsmrreader -c "update dsmr_mindergas_mindergassettings set next_export = '2016-01-09 12:00:00+01' where id = 1;"
• It should be updated within a few seconds. And you should see the next export in the configuration pages pushed forward to tomorrow again.

In the /var/log/supervisor/dsmr_backend.log logfile you should see something like:

2017-01-09 21:21:41.199798+01:00: Calling backend services
 - Attempting to export gas meter position to MinderGas.
 - Exporting gas meter position to MinderGas: 1234.567
 - Delaying the next export to MinderGas until: 2017-01-10 00:47:00+01:00

Please let me know whether this resolves your problem.

[mkruiver]

Thanks. I have updated.

After setting the next export (to Jan 9 at noon(!)) it did not "update within a few seconds" however.

It is still at:

Next export:
Jan. 9, 2016, noon
Timestamp of the next export. Automatically updated by application.

[mkruiver]

I think my DSMR-reader instance is now hammering mindergas.nl every minute...
Also, I seem to be unable to stop the upload, because if I remove the checkmark from "upload to mindergas" and SAVE settings the application seems to crash.. Is there a command line option to temporarily disable mindergas uploads?

 File "/home/dsmr/dsmr-reader/dsmr_mindergas/services.py", line 70, in export
    raise AssertionError('MinderGas upload failed: %s (HTTP %s)'.format(response.text, response.status_code))
  File "/home/dsmr/dsmr-reader/dsmr_backend/management/commands/dsmr_backend.py", line 29, in run
    raise current_response

  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/dispatch/dispatcher.py", line 227, in send_robust
    response = receiver(signal=self, sender=sender, **named)

  File "/home/dsmr/dsmr-reader/dsmr_mindergas/apps.py", line 20, in _on_backend_called_signal
    dsmr_mindergas.services.export()

  File "/home/dsmr/dsmr-reader/dsmr_mindergas/services.py", line 70, in export
    raise AssertionError('MinderGas upload failed: %s (HTTP %s)'.format(response.text, response.status_code))
CommandError: ['  File "/home/dsmr/dsmr-reader/dsmr_backend/management/commands/dsmr_backend.py", line 29, in run\n    raise current_respo$
Starting INFINITE command loop...
2017-01-10 20:22:37.762427+01:00: Calling backend services
 - Attempting to export gas meter position to MinderGas.
 >>> Uncaught exception :: MinderGas upload failed: %s (HTTP %s)
 >>> <bound method AppConfig._on_backend_called_signal of <AppConfig: dsmr_mindergas>> ::   File "/home/dsmr/dsmr-reader/dsmr_backend/mana$
    raise current_response

  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/dispatch/dispatcher.py", line 227, in send_robust
    response = receiver(signal=self, sender=sender, **named)

  File "/home/dsmr/dsmr-reader/dsmr_mindergas/apps.py", line 20, in _on_backend_called_signal
    dsmr_mindergas.services.export()

 File "/home/dsmr/dsmr-reader/dsmr_mindergas/services.py", line 70, in export
    raise AssertionError('MinderGas upload failed: %s (HTTP %s)'.format(response.text, response.status_code))
  File "/home/dsmr/dsmr-reader/dsmr_backend/management/commands/dsmr_backend.py", line 29, in run
    raise current_response

  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/dispatch/dispatcher.py", line 227, in send_robust
    response = receiver(signal=self, sender=sender, **named)

  File "/home/dsmr/dsmr-reader/dsmr_mindergas/apps.py", line 20, in _on_backend_called_signal
    dsmr_mindergas.services.export()

  File "/home/dsmr/dsmr-reader/dsmr_mindergas/services.py", line 70, in export
    raise AssertionError('MinderGas upload failed: %s (HTTP %s)'.format(response.text, response.status_code))
CommandError: ['  File "/home/dsmr/dsmr-reader/dsmr_backend/management/commands/dsmr_backend.py", line 29, in run\n    raise current_respo$
Starting INFINITE command loop...
2017-01-10 20:22:42.160058+01:00: Calling backend services
 - Attempting to export gas meter position to MinderGas.
 - Exporting gas meter position to MinderGas: 99.854
 >>> Uncaught exception :: MinderGas upload failed: %s (HTTP %s)
 >>> <bound method AppConfig._on_backend_called_signal of <AppConfig: dsmr_mindergas>> ::   File "/home/dsmr/dsmr-reader/dsmr_backend/mana$
    raise current_response

  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/dispatch/dispatcher.py", line 227, in send_robust
    response = receiver(signal=self, sender=sender, **named)

  File "/home/dsmr/dsmr-reader/dsmr_mindergas/apps.py", line 20, in _on_backend_called_signal
    dsmr_mindergas.services.export()

  File "/home/dsmr/dsmr-reader/dsmr_mindergas/services.py", line 70, in export
 File "/home/dsmr/dsmr-reader/dsmr_mindergas/services.py", line 70, in export
    raise AssertionError('MinderGas upload failed: %s (HTTP %s)'.format(response.text, response.status_code))
  File "/home/dsmr/dsmr-reader/dsmr_backend/management/commands/dsmr_backend.py", line 29, in run
    raise current_response

  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/dispatch/dispatcher.py", line 227, in send_robust
    response = receiver(signal=self, sender=sender, **named)

  File "/home/dsmr/dsmr-reader/dsmr_mindergas/apps.py", line 20, in _on_backend_called_signal
    dsmr_mindergas.services.export()

  File "/home/dsmr/dsmr-reader/dsmr_mindergas/services.py", line 70, in export
    raise AssertionError('MinderGas upload failed: %s (HTTP %s)'.format(response.text, response.status_code))
CommandError: ['  File "/home/dsmr/dsmr-reader/dsmr_backend/management/commands/dsmr_backend.py", line 29, in run\n    raise current_respo$

[dennissiemensma]

Can you try setting the next_export field a bit forward into the future then? You can stop dsmr_backend in Supervisor temporarily to stop the spam, until you've set the datetime for in the future.

It seem that MinderGas rejects the reading, but due to a small typo of me the actual error isn't displayed. And the hammering is something that I will fix as well.

Can you check your API key you've entered?

Also, the info message states: - Exporting gas meter position to MinderGas: 99.854
And I know that MinderGas validates/compares any readings with the previous one as well. Is it lower than the last one entered on the website?

[mkruiver]

Thanks for your quick response.
I already set the Next export to:
Jan. 11, 2016, midnight

But it seems to still be hammering.

I have verified and re-entered the API key and that is correct. Which makes sense as the 09JAN meter reading (99,854) did upload correctly last night.

[dennissiemensma]

Hmmm, can you try to disable the Exporteer gegevens naar MinderGas checkbox in the settings for now? I think it's best to stop the hammering as they might restrict API calls at some time.

I will make sure that the hammering stops and that a decent error message is displayed later.

[mkruiver]

I seem to be unable to stop the upload, because if I remove the checkmark from "upload to mindergas" and SAVE settings the application seems to crash.. Is there a command line option to temporarily disable mindergas uploads?

[dennissiemensma]

Sure, this should do it:
sudo -u postgres psql dsmrreader -c "update dsmr_mindergas_mindergassettings set export = False where id = 1;"

[mkruiver]

That worked, thanks.

[dennissiemensma]

It weird that the application crashes on the settings pages though.

I will continue to investigate this tomorrow and also create a patch to stop the hammering, as well as displaying the error from MinderGas. Thanks for reporting this though, as it's very nasty bug.

[dennissiemensma]

Aangezien ik er (pas) gisteren achter kwam dat ik tegen dezelfde persoon zowel in NL als EN praatte in verschillende ticket, zal ik hier NL doen voor het gemak.

Ik heb een nieuwe patch uitgebracht nu, v1.5.3. Daarin zou nu zichtbaar moeten zijn waarom MinderGas de upload weigert (status code en hopelijk ook foutmelding).
En het hammeren zou niet meer voor moeten komen. Bij fouten probeert die het dan pas over een uur weer, ipv continu.

Wil je kijken of je dit meer inzicht geeft in waarom de upload faalt?

[mkruiver]

1.5.3 geeft inderdaad meer inzicht, in eerste instantie faalt de upload door een foute API key (401), ik had namelijk nog een foute API-key staan omdat ik die express naar een random value had verandert om de upload te 'stoppen'. Maar de hammering lijkt er ook nog te zijn.

2017-01-12 19:56:37.408320+01:00: Calling backend services
Command completed. Sleeping for 5 second(s)...

2017-01-12 19:56:42.475430+01:00: Calling backend services
Command completed. Sleeping for 5 second(s)...

2017-01-12 19:56:47.547122+01:00: Calling backend services
Command completed. Sleeping for 5 second(s)...

2017-01-12 19:56:52.623669+01:00: Calling backend services
 - MinderGas | Attempting to upload gas meter position.
 - MinderGas | Uploading gas meter position: 112.366
 [!] MinderGas upload failed (HTTP 401):
 - MinderGas | Delaying the next upload until: 2017-01-12 01:00:00+01:00
Command completed. Sleeping for 5 second(s)...

2017-01-12 19:56:57.969407+01:00: Calling backend services
 - MinderGas | Attempting to upload gas meter position.
 - MinderGas | Uploading gas meter position: 112.366
 [!] MinderGas upload failed (HTTP 401):
 - MinderGas | Delaying the next upload until: 2017-01-12 01:00:00+01:00
Command completed. Sleeping for 5 second(s)...

2017-01-12 19:57:03.257120+01:00: Calling backend services
 - Processed reading: 2017-01-12 19:56:23+01:00.
 - Processed reading: 2017-01-12 19:56:40+01:00.
 - Processed reading: 2017-01-12 19:56:51+01:00.
 - MinderGas | Attempting to upload gas meter position.
 - MinderGas | Uploading gas meter position: 112.366
 [!] MinderGas upload failed (HTTP 401):
 - MinderGas | Delaying the next upload until: 2017-01-12 01:00:00+01:00
Command completed. Sleeping for 5 second(s)...

2017-01-12 19:57:08.732695+01:00: Calling backend services
 - MinderGas | Attempting to upload gas meter position.
 - MinderGas | Uploading gas meter position: 112.366
 [!] MinderGas upload failed (HTTP 401):
 - MinderGas | Delaying the next upload until: 2017-01-12 01:00:00+01:00
Command completed. Sleeping for 5 second(s)...

2017-01-12 19:57:14.023236+01:00: Calling backend services
 - MinderGas | Attempting to upload gas meter position.
 - MinderGas | Uploading gas meter position: 112.366
 [!] MinderGas upload failed (HTTP 401):
 - MinderGas | Delaying the next upload until: 2017-01-12 01:00:00+01:00
Command completed. Sleeping for 5 second(s)...

2017-01-12 19:57:19.321179+01:00: Calling backend services
 - MinderGas | Attempting to upload gas meter position.
 - MinderGas | Uploading gas meter position: 112.366
 [!] MinderGas upload failed (HTTP 401):
 - MinderGas | Delaying the next upload until: 2017-01-12 01:00:00+01:00
Command completed. Sleeping for 5 second(s)...

Na het invullen van de goede API key werkt de upload weer:

2017-01-12 20:31:47.327292+01:00: Calling backend services
 - MinderGas | Attempting to upload gas meter position.
 - MinderGas | Uploading gas meter position: 112.366
 - MinderGas | Delaying the next upload until: 2017-01-13 00:40:00+01:00
Command completed. Sleeping for 5 second(s)...

2017-01-12 20:31:52.834689+01:00: Calling backend services
Command completed. Sleeping for 5 second(s)...

[dennissiemensma]

Ik zie het al, zoals eerder gezegd was ik niet fit, maar ook vooral niet wakker zie ik. Het was de bedoeling dat hij bij falen het een uur vooruit schuift. Dat doet die ook, maar vanaf de meest recent middernacht ipv 'nu': * facepalm *

Je kunt het instellen in het DB-veld auth_token:

sudo -u postgres psql dsmrreader -c "update dsmr_mindergas_mindergassettings set auth_token = 'API-SLEUTEL' where id = 1;"

[dennissiemensma]

Ik heb het laatste bugje zojuist opgelost in patch v1.5.4. Erg bedankt voor het melden en telkens uitproberen.

Mocht je nog benoemd willen worden in de 'hall of fame', laat dan even wat achter in #231.

En laat het ook vooral weten wanneer de MinderGas upload toch niet meer blijkt te werken.

[mkruiver]

Top, het werkt nu en ik zal me weer melden als er toch nog iets fout mocht gaan.