Tibber data unstable and possibly wrong

[natterangell]

I've been using the amshan integration for well over a year, and its been mostly a great experience. But something changed within the last month or so (due to lower electricity prices overall this summer, I didn't pay much attention to it initially).

My setup
-HomeAssistant in a VM on Proxmox, with amshan 2023.6.0
-Mosquitto MQTT server running as a docker container on a separate system
-Tibber Pulse publishing locally to mqtt
-Kaifa MA304H3E meter

As mentioned, this has been working well until sometime within the last month or so.

Sensor unavailable after restarting HA
Restarting HA i.e. after an update will reliably cause the sensor to become unavailable ("This entity is no longer being provided by the amshan integration. If the entity is no longer in use, delete it in settings.")

Restarting the mosquitto container makes no difference. Neither does it help to remove the amshan integration and re-add it; when selecting MQTT during setup and adding "pulse/publish" as topic it results in a timeout.

I can verify that Tibber Pulse is still publishing to the MQTT server when this is happening by connecting with MQTT Explorer, or by listening to "pulse/publish" from the MQTT integration in /config/integrations/integration/mqtt

The only way to "fix" the problem is by physically disconnecting the ethernet cable from Tibber Pulse, wait for a little while until it shuts off and then reconnecting. AmsHan integration will then pick up the MQTT-messages again, and the integration works until the next time HA is restarted.

This leads me to believe there must be some sort of message/data being transmitted when tibber pulse starts up and connects to the MQTT-server that allows AmsHan to find it (?). Makes little sense to me. I've been wondering if issue 66 was related, but the issue was closed again without further details.

Tibber Pulse data update frequency is too low
While AmsHan now "works", there is something wrong with the data being recieved

Look at the curve:

There are several minutes, up to an hour, between updated readings. The sensors doesn't seem to pick up on continuous readings, but almost appears to display "snapshot" values over time. Again, I can see in MQTT Explorer that Tibber Pulse reliably publishes messages every few seconds, but they are not displayed by AmsHan.

Consequently, the Riemann sum integral sensor I have set up to convert watts to kilowatts and feed into the Energy dashboard is incorrect too, the kwh values per hour are all over the place.

[grahamsanders]

I've got the same issue too with a very similar setup:

-HomeAssistant in a VM
-Mosquitto sever running within HomeAssistant
-Tibber Pulse sending locally
-Kaifa MA304H3E meter

-AMSHAN v2023.6.0
-HA v2023.9.2
-Mosquitto Broker Addon v6.3.1

Has been working for over a year with no issue before and I've only noticed the issue this last couple of weeks. The only difference I can think of would be a HA or Mosquitto update.

Can confirm that the "fix" works for me too after physically disconnecting then reconnecting Tibber Pulse. However, it doesn't take a HA restart before it breaks again for me. It will work fine for the first 30 minutes or so (readings once per second as expected) but it will eventually break randomly and go into the "Unavailable" state, with the message:

"This entity is no longer being provided by the amshan integration. If the entity is no longer in use, delete it in settings."

Unfortunately this makes the integration unusable for me as I have to keep unplugging then replugging Tibber every half our or so.

Edit: Also just to add, I can monitor the MQTT messages and they are definitely still being sent to my broker every second.

[toreamun]

I have just studied the MQTT message stream.

Kaifa should send list 1 (short) every two second, and list 2 (long) every tenth second, and list 3 (long + aggregated) 10 second past each hour. Most common pattern repeated every 10 second is then list 2 followed by four list 1 messages.

Every list message would normally result in a separate MQTT message, but the Pulse starts to bundle them up as a single crippled message every tenth second (list 2). The start of each message is the last part of a previous message. Similar every hour.

Do other observe the same? I used this command to debug the MQTT message stream:
mosquitto_sub -F '@Y-@m-@dT@H:@M:@S@z : %t : %x' -t 'pulse/#'

Another way of watching the stream of messages can be found here:
https://github.com/toreamun/amshan-homeassistant/wiki/Feils%C3%B8king#mqtt

This change must be from a subtle change in messages from Kaifa that throws the Pulse into an invalid state. Maybe the Pulse needs a firmware upgrade to overcome this? This can probably be done be connecting the pulse with Tibber.

I could look at making changes to have the integration skip the crippled start of the message, but some messages would then be lost and the integration would only update every tenth second (when Pulse sends bundled messages).

[toreamun]

There are several minutes, up to an hour, between updated readings. The sensors doesn't seem to pick up on continuous readings, but almost appears to display "snapshot" values over time. Again, I can see in MQTT Explorer that Tibber Pulse reliably publishes messages every few seconds, but they are not displayed by AmsHan.

Are you sure messages are published every 2nd sec and not every 10th sec?

[natterangell]

It does indeed look like messages are pushed every 10 seconds:

[natterangell]

@grahamsanders, can you double check the frequency on your broker add-on?

[natterangell]

This change must be from a subtle change in messages from Kaifa that throws the Pulse into an invalid state. Maybe the Pulse needs a firmware upgrade to overcome this? This can probably be done be connecting the pulse with Tibber.

When I set the pulse up, I retained the firmware OTA url that came with it as stock, hoping that would help keep firmware updated, but maybe that doesn't work. I can try to set it up again with the Tibber app (tomorrow, travelling today).

I could look at making changes to have the integration skip the crippled start of the message, but some messages would then be lost and the integration would only update every tenth second (when Pulse sends bundled messages).

every 10 seconds would still be acceptable to me, if that can work reliably.

I'll let you know if connecting through Tibber changes anything, should probably do that first so you don't need to do anything unnecessary.

[grahamsanders]

Sorry, you're absolutely correct, the MQTT messages are every 10s when it's in it's failure state. When it's in it's working state though, it's every 2 seconds.

I've attached a debug log of the last day of messages if it helps. I unplugged Tibber at the 2023-09-19 20:42:13.735 which is why it starts to work again.

As for the Tibber firmware, I let the app configure it and update it last week after I saw this issue (so this issues exists for both old and new firmware). My original firmware was v1.1.10, and it's now v1.2.3.

amshan.log

[grahamsanders]

I use my meter readings mostly for hourly-resolution trends and automations so updates every 10 seconds would be perfectly acceptable for me too, especially considering I can't use the integration at all at the moment. It can always be reverted back again once Tibber or Kaifa fix the issue.

And to update my previous comment, I've attached a new log which shows the meter going from working to broken just now at the 2023-09-19 20:56:28.405 timestamp (meaning it worked for only 15 minutes before breaking).

amshan_good_to_bad.log

[MyGithubUser01]

Same thing happened to me as Tibber Pulse overnight (~30th of August) updated it's firmware to version 1.2.3.
From tibber support: "Ingen begrensinger ang. MQTT, men siste firmware gjør at Pulsen batcher etter ca 15 minutter oppetid."

"Så her ser det ut som noe er endret, men de mener det kan være noe annet som er årsaken. Muligens AMSHAN ikke støtter denne "batchingen"?"

AIDON

[natterangell]

Same thing happened to me as Tibber Pulse overnight (~30th of August) updated it's firmware to version 1.2.3. From tibber support: "Ingen begrensinger ang. MQTT, men siste firmware gjør at Pulsen batcher etter ca 15 minutter oppetid."

"Så her ser det ut som noe er endret, men de mener det kan være noe annet som er årsaken. Muligens AMSHAN ikke støtter denne "batchingen"?"

AIDON

Yeah, updating the firmware doesn't resolve anything for me either. I wonder what they mean when they say "batching". Do they mean that is sends snapshot values after 15 minutes? Sort of defeating the purpose of the device.

Does anyone know if the firmware can be downgraded?

[MyGithubUser01]

Are you on 1.2.3 now, or is there a newer version? Mine was also stable for about year.
Be sure to mention this issue to Tibber support! I've done it many times already :)

[natterangell]

1.2.3, yes.

I've given up for now and went back to the Tibber integration in Home Assistant. It used to be a bit unstable but looks like it is working well again.

[mgartin]

I am experiencing the same, I was pointed to this thread from the norwegian discussion here.

Would it be possible to decode the new batched messages from the Tibber Pulse?
I can collect a set of messages and send them, but am not comfortable to send them all in the open in case they contain sensistive information.
Essentially, all the batched messages starts with 08dcd9... for me.

I have two AMS devices, and two Tibber Pulse's and the first one has worked without problems for a few years now, but the second one is probably of newer firmware or got updated, and started to behave strangely in August this year.

I have already bought a new Tibber Pulse, but realize that it probably won't solve my problem if it is already on version 1.2.3. So, I'd be happy to provide any information needed in order to decode the new messages. Please let me know.

[ronped]

I just bought a Tibber Pulse and have a Kaifa meter. Quickly got the same issue as described here. I found an ugly workaround by assuming meter messages have fixed offsets in this new bundled message. I pushed my workaround here: https://github.com/ronped/amshan-homeassistant/tree/tibber-pulse-workaround. What would be better would be to get Tibber to disclose the specifics on how they do the bundling, but I assume that would be something for @toreamun to try to find out in order to implement a proper fix.

[docstalek]

I just noticed something interesting. When opening the Tibber app, their backend will send a message to the Tibber Pulse, deactivating the batching for 15 minutes.
Payload: batching_disable 900

I have now a workaround running which sends this message to the Tibber Pulse receive topic every 15 minutes. Seems to be working flawlessly. Better than sending a reboot command every 15 minutes.

[natterangell]

Why on earth did Tibber make this change? Is it simply to give users real time view of consumption when actually opening the app, and lowering the overall amount of data traffic when no one is looking?

[docstalek]

Probably to reduce cost. They pay AWS based on traffic on their MQTT broker.

[Lanjelin]

I just noticed something interesting. When opening the Tibber app, their backend will send a message to the Tibber Pulse, deactivating the batching for 15 minutes.

Payload: batching_disable 900

I have now a workaround running which sends this message to the Tibber Pulse receive topic every 15 minutes. Seems to be working flawlessly. Better than sending a reboot command every 15 minutes.

Sending batching_disable true deactivates it for an hour, it returns

{
  "batching": "disabled",
  "timeout": 3600
}

Added the following automation to send the message every 30min.

alias: Tibber-fix
description: Disable Batching
trigger:
  - platform: time_pattern
    minutes: /30
condition: []
action:
  - service: mqtt.publish
    data:
      topic: pulse/subscribe
      payload: batching_disable true
mode: single

[atus]

@Danielhiversen @maggud @finnmich @flygare @yasasKahawala found you on Tibber's org page
Bringing this to your attention in case you get an opportunity to fix this by disabling batching when mqtt is enabled and pulse isn't publishing to Tibber's AWS endpoint. Allowing the pulse to be used locally with MQTT is a fantastic feature that's broken with the latest batching change and requires this workaround.

[tfriberg]

I seems it might be possible to send batching_disable -1 to the mqtt_topic_sub (in my case rebbit) which should give the reply:

{
"batching": "disabled",
"timeout": -1
}

Too soon to tell if it lasts, but it's been working longer than the batching_disable true so far.

[atus]

It's been 7d. Did it work as a long term solution? And does it survive reboots?

[ronped]

I seems it might be possible to send batching_disable -1 to the mqtt_topic_sub (in my case rebbit) which should give the reply:

{
"batching": "disabled",
"timeout": -1
}

Too soon to tell if it lasts, but it's been working longer than the batching_disable true so far.

Yeah, I also tried that. It eventually reverted back to the batching, but it took a while.

[tfriberg]

I believe it reverted back after a few days after which I had to send the command again.

I got so annoyed with this so I replaced my Tibber Pulse for an AMSreader Pow-P1. It doesn't require the AMSHan-HA integration (so it becomes a little OT), but I am worried about the Tibber Pulse being degraded over time with destructive software/firware patches and API restrictions, and don't want to "play that game" any longer.

Also note that I am not a paying Tibber customer, I just liked their hardware. Initially.