Core: Error when return a json as payload_template in automation actions

[vassilis-panos]

The problem

 action:
    service: mqtt.publish
    data_template:
      topic: "topic/test"
      payload_template: >
        {% set payload = {'value1': 'test', 'value2': 'test' } %}
        {{ payload|to_json }}
      qos: 0
      retain: true

Environment

• Home Assistant Core release with the issue: 0.117.0b1
• Last working Home Assistant Core release (if known): 0.116

Traceback/Error logs

Error executing script. Invalid data for call_service at pos 1: value should be a string for dictionary value @ data['payload_template']

[probot-home-assistant[bot]]

Hey there @phracturedblue, @tetienne, mind taking a look at this issue as its been labeled with an integration (template) you are listed as a codeowner for? Thanks!
_{^{(message by CodeOwnersMention)}}

[VDRainer]

I think this is related?

# RTL 433 MQTT Demultiplexer
  - alias: 'rtl_433 demultiplexer'
    initial_state: true
    mode: parallel
    max: 20
    trigger:
      platform: mqtt
      topic: home/rtl_433

    action:
      service: mqtt.publish
      data_template:
        topic: "{{ 'home/rtl_433_demux/' + trigger.payload_json.model|string + '_' + trigger.payload_json.id|string }}"
        payload: "{{trigger.payload}}"
        retain: true

2020-10-22 19:13:45 ERROR (MainThread) [homeassistant.components.automation.rtl_433_demultiplexer] rtl_433 demultiplexer: Error executing script. Unexpected error for call_service at pos 1: payload must be a string, bytearray, int, float or None.
Traceback (most recent call last):
  File "/srv/ha/lib/python3.8/site-packages/homeassistant/helpers/script.py", line 253, in _async_step
    await getattr(
  File "/srv/ha/lib/python3.8/site-packages/homeassistant/helpers/script.py", line 460, in _async_call_service_step
    await service_task
  File "/srv/ha/lib/python3.8/site-packages/homeassistant/core.py", line 1448, in async_call
    task.result()
  File "/srv/ha/lib/python3.8/site-packages/homeassistant/core.py", line 1483, in _execute_service
    await handler.job.target(service_call)
  File "/srv/ha/lib/python3.8/site-packages/homeassistant/components/mqtt/__init__.py", line 581, in async_publish_service
    await hass.data[DATA_MQTT].async_publish(msg_topic, payload, qos, retain)
  File "/srv/ha/lib/python3.8/site-packages/homeassistant/components/mqtt/__init__.py", line 782, in async_publish
    msg_info = await self.hass.async_add_executor_job(
  File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/srv/ha/lib/python3.8/site-packages/paho/mqtt/client.py", line 1259, in publish
    raise TypeError(
TypeError: payload must be a string, bytearray, int, float or None.

[frenck]

Yes

[tdejneka]

Does it work if you cast it to string?

        payload: "{{ trigger.payload | string }}"

[VDRainer]

Thanks @tdejneka

Does it work if you cast it to string?

No, error remains.

[Mariusthvdb]

please let me add 2 issues, on longstanding automations, which seem to be related? on 117.0b2 meanwhile. legacy_templates: truemakes them behave as before.

seeing this:

Last logged: 11:17:03 PM

Marijn Presence Update from Router or Bluetooth: Error executing script. Unexpected error for call_service at pos 1: payload must be a string, bytearray, int, float or None.
While executing automation automation.marijn_presence_update_from_router_or_bluetooth

.....
TypeError: payload must be a string, bytearray, int, float or None.

for this action in an automation:

    action:
      service: mqtt.publish
      data:
        topic: location/marijn
        retain: true
        payload: >-
          {
            "latitude": "{{state_attr('zone.home','latitude')}}",
            "longitude": "{{state_attr('zone.home','longitude')}}",
            "battery_level": {{states('sensor.calltheboss_battery_level')|int}},
            "gps_accuracy": 0
          }

and this error:

voluptuous.error.MultipleInvalid: template value should be a string for dictionary value @ data['attributes_template']

for this action:

      - service: variable.set_variable
        data:
          variable: last_motion
          attributes_template: >
            {
             "history_1":"{{states('variable.last_motion')}}",
             "history_2":"{{state_attr('variable.last_motion','history_1')}}",
             "history_3":"{{state_attr('variable.last_motion','history_2')}}",
             "history_4":"{{state_attr('variable.last_motion','history_3')}}",
             "history_5":"{{state_attr('variable.last_motion','history_4')}}",
             "history_6":"{{state_attr('variable.last_motion','history_5')}}",
             "history_7":"{{state_attr('variable.last_motion','history_6')}}",
             "history_8":"{{state_attr('variable.last_motion','history_7')}}",
             "history_9":"{{state_attr('variable.last_motion','history_8')}}",
             "history_10":"{{state_attr('variable.last_motion','history_9')}}"
            }
          value: >
            {{trigger.to_state.name|replace(' sensor motion','')}}:
            {{as_timestamp(trigger.from_state.last_changed)|timestamp_custom('%X')}}

or:

error:

TypeError: encoding without a string argument

using script:

  set_hue_threshold:
    alias: Set Hue Indoors threshold
    mode: restart
    icon: mdi:cog
    sequence:
      service: rest_command.set_hue_command
      data:
        command: config
        type: sensors
        data: >
          {"tholddark": {{states('input_number.slide_hue_threshold')|int}} }
        id: >
          {% set mapper =
            {'Laundry':'25',
             'Dining table':'53',
             'Auditorium':'45',
             'Frontdoor':'195',
             'Library':'21',
             'Dorm':'57',
             'Front room':'9',
             'Guest room':'35',
             'Attic':'13'} %}
          {% set state = states('input_select.select_hue_motion_sensor') %}
          {% set id = mapper[state] if state in mapper %}
          {{id}}

as a final note, all off the above templates return being of type String in the template editor, which is what the errors complain about being incorrect:

none of the formats suggested here https://community.home-assistant.io/t/template-rebuilds-after-117-x-needed/238153/12?u=mariusthvdb

work. though the yaml would be correct either way

[brianhanifin]

The template editor reports it as a "dict" for me, even after storing it in a variable then outputting it. I am running 117.0b2 with the new template engine. (I did not enable legacy_templates.) It still reports as a dict when trying to convert it to a string {{ output | string }}.

It seems the template only returns a dict in 117.0b2. If I hard code the dict as a string in the services tool mqtt.publish works as expected. Not wrapping a dict in a quote marks does errors with TypeError: payload must be a string, bytearray, int, float or None.

Working service data:

topic: "custom/sensor/last_message/attributes"
payload: '{"attribute1": "value1", "attribute2": "value2"}'
retain: true

Error service data:

topic: "custom/sensor/last_message/attributes"
payload: {"attribute1": "value1", "attribute2": "value2"}
retain: true

[tdejneka]

Interesting. I thought the string filter would cast the dict to string. I wonder why that doesn't work?

So does this return a float or a string?

{% set x = '125.45' %}
{{ x | float }}

Fingers crossed it's float.

[frenck]

We are going way off subject here. If you want to raise a different issue than the one is listed in the opening issue description, please raise a new issue. If you have questions in general, please use or community forum - or better yet - use the #beta channel on our Discord chat server.

Thanks 👍

[tdejneka]

Sorry about that. I'll wait for the release version.

[brianhanifin]

UPDATE: I now see @frenck's off-topic comment was directed at another user. Nevermind.

---

What I wrote was about an error when calling mqtt.publish with a dict payload (which I believe is the same as @VDRainer's issue above). This did not occur until 117.0b1 & b2. I provided additional, I believe related information that the template engine reports the value is a dict instead of a string. I am also reporting that attempting to use the | string filter on the dict variable does not resolve the problem.

I assumed because this issue was already raised that I could add to the conversation rather than following the issue format for submitting the first report. Am i off topic? I love Home Assistant, and I am trying hard not to feel discouraged. Please help me understand how to do better in the future.

[frenck]

I am also reporting that attempting to use the | string filter on the dict variable does not resolve the problem.

That is expected behavior. The result of a template is evaluated and that determines the result. Any typecasting within the template only applies within the template scope itself. This has always been the case and hasn't changed.

This means, this template will result in a float: {{ 1.1 | string }}

[Mariusthvdb]

not sure if there is some development in this issue, I was wondering @brianhanifin if you can check if your working service data:

topic: "custom/sensor/last_message/attributes"
payload: '{"attribute1": "value1", "attribute2": "value2"}'
retain: true

also works you have a template in the value fields? Because that's what I am trying to fix, and cant just yet. Seems varying I try either breaks the dictionary, or the templates for that matter... see my post above on the attributes_template setting the variable.

also rewriting:

    sequence:
      service: rest_command.set_hue_command
      data:
        command: config
        type: sensors
        data: >
          {"tholddark": {{states('input_number.slide_hue_threshold')|int}} }

to

    sequence:
      service: rest_command.set_hue_command
      data:
        command: config
        type: sensors
        data:
          tholddark: >
           {{states('input_number.slide_hue_threshold')|int}}

which would be correct formatted yaml and type, doenst work, because the command to the Hue api needs to be completely as posted

{"tholddark": {{states('input_number.slide_hue_threshold')|int}} }

I dont see how I can do that an another way.
So the question remains, is this a bug, or should the yaml be written in a different way. If we are waiting for a fix in the core code, it would be nice to get that as feedback, because we can stop trying to fix our own.
thanks

[brianhanifin]

@Mariusthvdb here is a test script I have been working with (script.mqtt_dict_test). I included 2 mqtt.publish calls. The first publishes a simple string, which succeeds. The second attempts to publish the dict value, which fails with an error. No matter how I try to pass the attributes value (directly, or passed through a variable first) it gives the same error.

Variables are not currently allowed in this use case. The only way to get it to succeed is to hard code payload: '{"test":"test"}', or use legacy_templates: true.

@frenck I use this to create attributes of a MQTT Sensor. This is not a mission critical use case, and I could do without it if necessary. But since I came across it I thought I should share it before the stable release. I am assuming this is an unintended side effect of the template data type change, which I think will be nice.

variables:
  name: "mqtt_dict_test"
  attributes: >
    { 
      "name": "{{ name }}"
    }
  #attributes: '{"name":"{{ name }}"}'

sequence:
  # This one succeeds.
  - service: mqtt.publish
    data:
      topic: "custom/sensor/{{ name }}/state"
      payload: "{{ name }}"
      retain: true

  # This one errors: "payload must be a string, bytearray, int, float or None."
  - service: mqtt.publish
    data:
      topic: "custom/sensor/{{ name }}/attributes"
      payload: "{{ attributes }}"
      #payload: '{"name":"{{ name }}"}'
      #payload: '{"test":"test"}'
      retain: true

[Mariusthvdb]

ok thanks, glad you can confirm what you did here, no way to get the dict value. Nor the templated variation.
Not sure I can agree with you it is not mission critical though, in my setup it is, because several automations rely on getting this right, on a daily base, and it is not something I want to do manually on the Hue hub each day... Not are the motion sensors history buildups.
So, hope to get some directions here, or instruction to wait for a fix (if it is a bug)

also, still confused with the types:

it's a number (True), but type: string?

[brianhanifin]

@Mariusthvdb Franck joined Drzz's live stream today and said that he will probably work on this issue tomorrow.

It sounds like the problem is that mqtt.publish expects JSON, but the template is outputting a Python Dict data type. So, that makes me wonder if all integrations that expect JSON values will need to be updated to convert Dict values? Either way Franck recognizes this as an unintended consequence of this change and will be working on a fix soon.

[Mariusthvdb]

that's great to hear! thanks to both of you. And yes, as ive tried to add to the issues for this, its not only mqtt.publish.

so really glad this is been recognized, and am sure F will fix it ;-)

[frenck]

Alright, Paulus and I are going to crack this one today. We had a talk this morning on the approach and made a couple of tasks we are going to complete today.

[balloob]

Fixed in #42398

[Mariusthvdb]

thanks for all the extra and speedy effort that went into this. much appreciated!

is it allowed to ask some additional info on what to change now in the original services/templates posted above now we should be able to use the new type functionality?

think it might be a good idea to have some transitional examples in the release notes/documentation on this, and the posted ones could be of good use?

this will be one of the more difficult changes, so some extra guidance would be useful