Incorrect summary data

[shtrom]

I have installed AuroraPlusHA via HACS and, for all intent and purposes, the connection seems functional, and I'm getting some data out.

I am not certain the data makes sense, though, as it doesn't quite line up with what I get in Aurora+:

• consumption and feed in only appear once a day, seemingly for the full amount
• those full amounts don't match what the dashboard shows.

Out of curiosity, I also checked the usage in HA for the next day, in case the daily reports are aggregates of the previous day, but that doesn't seem to be the case either.

Is it possible to get hourly usage+feed-in data?

Playing around with https://pypi.org/project/auroraplus/ and my account, I see a number of weird things, such as

• 'HasSolarTariff': False,
• 'KilowattHourUsage': None,
  so I suspect there might be something weird in my Aurora+ account setup, rather than with the HA integration.

I'm not sure where to look next to try to fix that. Would you have any suggestions?

[shtrom]

Actually, looking more at the data from the auroraplus Python module, I see data that matches what the Aurora+ dashboards GETs from their API, e.g.,

{'DollarValueUsage': None, 'KilowattHourUsage': {'T140': 0.0, 'T93OFFPEAK': 2.885}, 'KilowattHourUsageAEST': {'T140': 0.0, 'T93OFFPEAK': 1.51}, 'StartTime': '2023-05-27T09:00:00Z', 'EndTime': '2023-05-27T10:00:00Z', 'TimeMeasureCount': 1, 'HasSubstitudedData': False, 'TimeMeasureUnit': 'Hour'},
{'DollarValueUsage': None, 'KilowattHourUsage': {'T140': 0.0, 'T93OFFPEAK': 1.51}, 'KilowattHourUsageAEST': {'T140': 0.0, 'T93OFFPEAK': 1.582}, 'StartTime': '2023-05-27T10:00:00Z', 'EndTime': '2023-05-27T11:00:00Z', 'TimeMeasureCount': 1, 'HasSubstitudedData': False, 'TimeMeasureUnit': 'Hour'},
{'DollarValueUsage': None, 'KilowattHourUsage': {'T140': 0.0, 'T93OFFPEAK': 1.582}, 'KilowattHourUsageAEST': {'T140': 0.0, 'T93OFFPEAK': 0.834}, 'StartTime': '2023-05-27T11:00:00Z', 'EndTime': '2023-05-27T12:00:00Z', 'TimeMeasureCount': 1, 'HasSubstitudedData': False, 'TimeMeasureUnit': 'Hour'},

Reading the HA integration a bit more, it looks like it is just not reading those values.

[shtrom]

Ok, this looks like an inherent issue with HA. Even though the hourly data is available from the Aurora+ API, it is for the previous day. And HA doesn't support historical data ingestion https://community.home-assistant.io/t/energy-usage-is-delayed-or-shifted-in-time-in-the-graphs/362875/19

[shtrom]

There is one more problem with the sensor reporting daily summaries.

I noticed there always was a down-bar followed by an up-bar, and assume that was solar feed-in. It didn't match other readings, which is what made me doubt in the first place.

It's not actually solar feed-in. It is the negated consumption from the previous day.

This is due, I think, to how Aurora+ present the data, which go down to 0 for a bit, before going back up to the new daily summary.

The drop to 0 is recognised by HA as a negative consumption, hence the down bar. I think this is because the kWh usage is a TOTAL sensor class, which kinda makes sense. https://github.com/LeighCurran/AuroraPlusHA/blob/main/custom_components/auroraplus/sensor.py#L101-L107

However, due to the way Aurora+ present the data, the integration should manually control the last_reset attribute whenever it drops back to 0 for the day https://developers.home-assistant.io/blog/2021/09/20/state_class_total/

The data will still lag by a day when ingested in HA, and still not be at an hourly granularity, but at least it should be representative of the aggregated consumption (though it may be worth breaking it down by tariff, which is also available from the Summary data).

[shtrom]

Actually, maybe we can get away with setting the sensor to be TOTAL_INCREASING https://developers.home-assistant.io/blog/2021/09/20/state_class_total/#state-class-total_increasing. I'll give that a go.

[shtrom]

Ok, with #7, this looks a bit nicer

This is still incorrect for the energy dashboard, because it's yesterday's consumption summary vs. today's solar production, but it's better than nothing!

[shtrom]

Nope, still buggy. Unlike the total power usage, it seems the tariff usages don't reset to 0, so the TOTAL_INCREASING does't work if the new reading is higher than the previous one.

[shtrom]

Yep, looks like MEASUREMENT is the best way to have the right data show up, once per day. I find it counter intuitive, but after testing, that seems to work.

[shtrom]

This almost looks the part if you forget that it's yesterday's data!

[shtrom]

Ok, @LeighCurran I think #7 is as good as it gets.

All data is a TOTAL with a last_reset that will be set on every value change (which risks missing data if the daily value is the same as the previous day). I think it's as good as we can get if we want to use only the summary data from auroraplus.

It's good enough for some fancy, but of course that still doesn't work well in the HA Energy Dashboard, because the data shows up too late.

[8RedEight]

Guys - complete novice - how do I add this in HACS?

[slothking87]

Actually, looking more at the data from the auroraplus Python module, I see data that matches what the Aurora+ dashboards GETs from their API, e.g.,

{'DollarValueUsage': None, 'KilowattHourUsage': {'T140': 0.0, 'T93OFFPEAK': 2.885}, 'KilowattHourUsageAEST': {'T140': 0.0, 'T93OFFPEAK': 1.51}, 'StartTime': '2023-05-27T09:00:00Z', 'EndTime': '2023-05-27T10:00:00Z', 'TimeMeasureCount': 1, 'HasSubstitudedData': False, 'TimeMeasureUnit': 'Hour'},
{'DollarValueUsage': None, 'KilowattHourUsage': {'T140': 0.0, 'T93OFFPEAK': 1.51}, 'KilowattHourUsageAEST': {'T140': 0.0, 'T93OFFPEAK': 1.582}, 'StartTime': '2023-05-27T10:00:00Z', 'EndTime': '2023-05-27T11:00:00Z', 'TimeMeasureCount': 1, 'HasSubstitudedData': False, 'TimeMeasureUnit': 'Hour'},
{'DollarValueUsage': None, 'KilowattHourUsage': {'T140': 0.0, 'T93OFFPEAK': 1.582}, 'KilowattHourUsageAEST': {'T140': 0.0, 'T93OFFPEAK': 0.834}, 'StartTime': '2023-05-27T11:00:00Z', 'EndTime': '2023-05-27T12:00:00Z', 'TimeMeasureCount': 1, 'HasSubstitudedData': False, 'TimeMeasureUnit': 'Hour'},

Reading the HA integration a bit more, it looks like it is just not reading those values.

Hi Shtrom. Just started playing around with this myself. I love the work you have already done to help this!.

With the hourly data - Is the problem that HA doesnt accept it? or is it not available via the API.

[slothking87]

Actually, looking more at the data from the auroraplus Python module, I see data that matches what the Aurora+ dashboards GETs from their API, e.g.,

{'DollarValueUsage': None, 'KilowattHourUsage': {'T140': 0.0, 'T93OFFPEAK': 2.885}, 'KilowattHourUsageAEST': {'T140': 0.0, 'T93OFFPEAK': 1.51}, 'StartTime': '2023-05-27T09:00:00Z', 'EndTime': '2023-05-27T10:00:00Z', 'TimeMeasureCount': 1, 'HasSubstitudedData': False, 'TimeMeasureUnit': 'Hour'},

{'DollarValueUsage': None, 'KilowattHourUsage': {'T140': 0.0, 'T93OFFPEAK': 1.51}, 'KilowattHourUsageAEST': {'T140': 0.0, 'T93OFFPEAK': 1.582}, 'StartTime': '2023-05-27T10:00:00Z', 'EndTime': '2023-05-27T11:00:00Z', 'TimeMeasureCount': 1, 'HasSubstitudedData': False, 'TimeMeasureUnit': 'Hour'},

{'DollarValueUsage': None, 'KilowattHourUsage': {'T140': 0.0, 'T93OFFPEAK': 1.582}, 'KilowattHourUsageAEST': {'T140': 0.0, 'T93OFFPEAK': 0.834}, 'StartTime': '2023-05-27T11:00:00Z', 'EndTime': '2023-05-27T12:00:00Z', 'TimeMeasureCount': 1, 'HasSubstitudedData': False, 'TimeMeasureUnit': 'Hour'},

Reading the HA integration a bit more, it looks like it is just not reading those values.

Hi Shtrom. Just started playing around with this myself. I love the work you have already done to help this!.

With the hourly data - Is the problem that HA doesnt accept it? or is it not available via the API.

Oh - I just re-read what you wrote. The problem is Aurora doesn't publish data live which seems dumb.

[slothking87]

And is that something like this could help with?

https://github.com/ldotlopez/ha-historical-sensor

[slothking87]

Or spook

Service: Import statistics

Call it using: recorder.import_statistics

Advanced service to directly inject historical statistics data into the recorder long-term stats database. #easy

https://github.com/frenck/spook

[shtrom]

The problem is Aurora doesn't publish data live which seems dumb.

Yep. Though this seems common. Hourly data is available, too, but given the lag, I didn't bother adding support for it.

I came across Spook before, but the warnings... spooked me.

ldotlopez/ha-historical-sensor might be worth a shot. But it looks like we'd need to add some sort of dependency for AuroraPlusHA.

EDIT: Yeah, ha-historical-sensor seems easy enough to implement https://github.com/ldotlopez/ha-ideenergy/blob/c9592079df7734798fdd00d07c01ef1dba3074a6/custom_components/ideenergy/sensor.py#L390-L403

I'd probably try to make another PR based on #7 to keep the changes separable, but it could work.

[slothking87]

Thanks for that. Even with the delay I think it would be worthwhile adding the data into HA to be able to report on it a bit more and to be able to trace energy back to when we were home or not.

I have never written python before - but ill see what I can figure out

[shtrom]

👆🏼 Don't get your hopes up, it doesn't work just yet.

[slothking87]

👆🏼 Don't get your hopes up, it doesn't work just yet.

Starting to sound plausible - How is it looking?

[shtrom]

It's looking like this at the moment

But I don't seem to be able to get it to show in the Energy dashboard.

[ldotlopez]

Hi, Historical Sensor author here.

Energy dashboard doesn't use sensors, it uses statistics (it's a bit confusing, yes).
Statistics usually have the same name (id) of the source sensor, hence the confusion.

If your sensor doesn't show up in energy panel options its because it's not generating statistics. For a standard sensor Home Assistant does that job but HistoricalSensor it's not.

HistoricalSensor basically inserts states into the database, using almost raw sql INSERT stamens, so any internal process of HomeAssistant doesn't apply.

Generating statistics it's not an easy generalizable job, HistoricalSensor provides some support and helpers but you have to write some code:

• Define a statistic_metadata property
• Define how to calculate statistics data with the async_calculate_statistic_data method
• I recommend using just "sum" or "mean" statistics, not both, the one which applies to your sensor. Delorian integration shows both just as example.

Statistics will be calculated once new historical data comes in, then it will show up in energy panel.

PS. Please update me once this code it's merge and I will add your integration to users of Historical Sensors 😉

[shtrom]

Hey @ldotlopez,

I'm not gonna lie, I was hoping you might look this way.

I suspected as much about statistics. Thanks for the details.

I'll probably end up sending a PR your way to update the README, as I found a couple more inconsistencies.

[ldotlopez]

I'll probably end up sending a PR your way to update the README, as I found a couple more inconsistencies.

I will appreciate it a lot, it's a very experimental and under-commented code 😄

[shtrom]

Woooo!

Gotta clean the code, now.

@ldotlopez I got a bit confused about the statistic_id. The delorian example just uses the entity_id, but doing so, I had the following error

homeassistant  |   File "/usr/src/homeassistant/homeassistant/components/recorder/statistics.py", line 2140, in async_import_statistics                                 
homeassistant  |     raise HomeAssistantError("Invalid statistic_id")                                                                                                   
homeassistant  | homeassistant.exceptions.HomeAssistantError: Invalid statistic_id 

Checking the validation RE, it looks like it needs a lowercase-only string, prefixed by sensor:, so 'sensor:' + self._name.replace(' ', '_').lower() fixed it, but I couldn't really find anything confirming that it was the needed format. Perhaps you know?

[slothking87]

What entity did you use for your energy dashboard? I tried a few and got this message below in the energy dashboard setup for each one

Entity not defined
Check the integration or your configuration that provides:
sensor.power_usage_kilowatt_hour_usage_tariff_t41

[shtrom]

Ok, I rebased this a bit to make the commits more sensible.

Still testing on my end, but looking good. One outstanding issue is that I'd like to throttle the calls to the data coordinator, but this somehow wasn't working before. I'll need to get back to it.

@slothking87 depending on what versions of the code you tested before, it's likely that it created duplicate entities with very similar IDs. I'd suggest:

1. disable the plugin, restart
2. remove all the unavailable entities
3. reenable the plugin, restart

You're likely to lose some data, but you should only have the sensors with the latest ID available. I still have 2 of each, which I suspect is due to the naming of the statistics, but it's easy to pick one and see if it works, and if not, pick the other one. I'll also have to revisit the statistics ID to make sure I do it right. For the moment I just sanitised all the entity_ids.

[slothking87]

Bingo. That was some effort though (none of it your fault). Turns out you need to remove the sensors from entities, restart, remove the plugin., restart, remove the entities that remain, then remove the redundant statistics from under the statistics section in Dev tools, restart, then start the install again.

[slothking87]

Another oddity - Not sure if it is related. But the new sensors (t31 and T41 for me) when I add them to the energy dashboard, I can only select to not track costs, or to use an entity to track total costs. It wont let me manually enter the charge to track down the $ value manually, or let me select an entity to use with the current price.

[shtrom]

@slothking87 did you still have two similar entities after all your cleanup?

[shtrom]

I seem to be able to add any sort of pricing info with no issue here.

[slothking87]

@slothking87 did you still have two similar entities after all your cleanup?

No - that got rid of all mine. Took a few goes as sometimes they came back disabled after a reboot. And cleaning out the statistics helped

[slothking87]

I seem to be able to add any sort of pricing info with no issue here.

Does it let you select either of the bottom two? I can sometimes edit it but it doesn't seem to save

[shtrom]

Does it let you select either of the bottom two? I can sometimes edit it but it doesn't seem to save

Yep, seems to work. Though, even when editing an existing entry, I still need to re-select the source entity in addition to the price before I can save.

[slothking87]

Hmmmm, I think not being able to select the static price, or entity with current price has something to do with the historical sensor. If I try to use T31 or T41, it wont let me select either of the bottom two options. T31 and T41 both have an unknown status when you check the sensor (as expected from what I read). Other sensors such as the kilowatt_hour_usage allow me to use the options.
No big problem though. Im sure I can do some manual sensor calculation.

[shtrom]

OK, I managed to fix the Throttling. Unfortunately, I don't seem to be able to get my data out of AuroraPlus today, for yesterday (even on their dashboard). Hopefully it's not because I was hammering them with unthrottled update requests 😅

Another caveat is that I see all the data having KilowattHourUsage and KilowattHourUsageAEST. The data is currently in the former, but I suspect it will switch to the latter comes DST, which is would currently confuse the integration. We'll see in time, I guess...

[shtrom]

Barring the lack of data today, it's really looking good!

[slothking87]

Interesting - Mine doesn't show usage for 09/08 either - but it does show in the app now.

I cant figure out the costings for T31 and T41. I can add the usage in, get stuck with the costing. I cant make the energy dashboard calculate it (it ignores any value I put in the "use static price" and it wont let me use an entity with the current price for some reason. I can add in a "Use entity to track total costs" but that just takes the kw usage and turns it into $. I cant make a manual sensor to calculate it because of the unknown status of the sensor. Not the end of the world by any means!.

[shtrom]

Data for the 9th has showed up for me in HA, too. It will generally only show up sometime in HA the next day, as the AuroraPlus Python module can only fetch the previous day from the API.

But I have also noticed that it takes a variable amount of time into the next day before the data shows up in HA, and I'm not sure why. We'll see how it goes in the next few days.

Data from the 8th is definitely gone, I reckon. It's not even in the Aurora+ web dashboard.

[shtrom]

Oh, btw, I also added price-tracking entities, and I too see $0. I suspect it may be due to the way we put historical data into the recorder. We might be bypassing the cost calculation. @ldotlopez did you see the same thing with your own energy tracking? Can you use HA to calculate costs with HistoricalStates?

[ldotlopez]

@shtrom Hi,

I don't calculate costs in my setup and I don't think it will work without additional code.

Maybe the energy websocket API can be useful:
https://github.com/home-assistant/core/blob/master/homeassistant/components/energy/websocket_api.py

[slothking87]

Data for the 9th has showed up for me in HA, too. It will generally only show up sometime in HA the next day, as the AuroraPlus Python module can only fetch the previous day from the API.

But I have also noticed that it takes a variable amount of time into the next day before the data shows up in HA, and I'm not sure why. We'll see how it goes in the next few days.

Data from the 8th is definitely gone, I reckon. It's not even in the Aurora+ web dashboard.

Im not sure if I have broken something in my instance - But I havent had any download since 08/08. I just updated all the files with your latest versions in case I have missed something

[shtrom]

I broke something in the fetching. I pushed a potential fix yesterday, updating the AuroraPlus lib to 1.5.0. -- Olivier Mehani ***@***.***> Sent from my mobile, please excuse my brevity.

[LeighCurran]

Hey all, not sure if it will help what you are trying to achieve but I've made the The PyPi package repo public: https://github.com/LeighCurran/AuroraPlus

[shtrom]

Ok, so looking at the debug logs, all the throttling and the fetching work, but the data doesn't seem to go all the way to the recorder 🤔

[slothking87]

Seems like now that aurora is enforcing MFA (according to a recent email) - this no longer is able to retrieve the information.

I suspect this would sadly be the end of this integration unless there has also been a change to the API

[shtrom]

I suspect this would sadly be the end of this integration unless there has also been a change to the API

For the benefit of the random bystander landing here, no, it's not the end. PR #9 has now been updated to support logging in via a token obtained after MFA. It's rough, but it works.

[slothking87]

@shtrom - Looks like there might be a problem when moving to HA 2024.2

Logger: homeassistant.config
Source: config.py:1518
First occurred: 9:12:54 AM (1 occurrences)
Last logged: 9:12:54 AM

Platform error: sensor - No module named 'auroraplus.get_token'
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/config.py", line 1518, in async_process_component_config
platform = p_integration.get_platform(domain)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/loader.py", line 847, in get_platform
cache[full_name] = self._import_platform(platform_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/loader.py", line 864, in _import_platform
return importlib.import_module(f"{self.pkg_path}.{platform_name}")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/importlib/init.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1387, in _gcd_import
File "", line 1360, in _find_and_load
File "", line 1331, in _find_and_load_unlocked
File "", line 935, in _load_unlocked
File "", line 994, in exec_module
File "", line 488, in _call_with_frames_removed
File "/config/custom_components/auroraplus/sensor.py", line 6, in
import auroraplus
File "/usr/local/lib/python3.12/site-packages/auroraplus/init.py", line 13, in
from .get_token import get_token
ModuleNotFoundError: No module named 'auroraplus.get_token'

[shtrom]

Rah, no. I must have botched the import of the token-fetching tool, and your system pulled the update with the HA upgrade. i'll have a look over the weekend. -- Olivier Mehani ***@***.***> Sent from my mobile, please excuse my brevity.

[shtrom]

Well... That was easy to fix... Turns out I forgot to git add the new token-getting script... I just pushed a commit that adds it to AuroraPlus, so that might work for you now (+/- the dance to clear the cache of previous versions).

[slothking87]

Edit: After a HA upgrade it seemed to fix its self....