pyPowerwall erroring/timing out for 15-30 minutes

[ThePnuts]

While I have been working on trying to convert to Influx 2.6, I have noticed multiple times where data stops feeding into the database from my powerwall

Looking at Telegraf, I would get time out errors, This would repeat in the log:

2023-03-03T04:12:11Z E! [inputs.http] Error in plugin: [url=http://<pyPowerwall Docker IP>:8675/alerts/pw]: Get "http://<pyPowerwall Docker IP>:8675/alerts/pw": EOF
2023-03-03T04:12:11Z E! [inputs.http] Error in plugin: [url=http://<pyPowerwall Docker IP>:8675/temps/pw]: Get "http://<pyPowerwall Docker IP>:8675/temps/pw": EOF
2023-03-03T04:12:11Z E! [inputs.http] Error in plugin: [url=http://<pyPowerwall Docker IP>:8675/freq]: Get "http://<pyPowerwall Docker IP>:8675/freq": EOF
2023-03-03T04:12:11Z E! [inputs.http] Error in plugin: [url=http://<pyPowerwall Docker IP>:8675/pod]: Get "http://<pyPowerwall Docker IP>:8675/pod": EOF
2023-03-03T04:12:18Z E! [inputs.http] Error in plugin: [url=http://<pyPowerwall Docker IP>:8675/temps/pw]: Get "http://<pyPowerwall Docker IP>:8675/temps/pw": EOF
2023-03-03T04:12:19Z E! [inputs.http] Error in plugin: [url=http://<pyPowerwall Docker IP>:8675/alerts/pw]: Get "http://<pyPowerwall Docker IP>:8675/alerts/pw": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2023-03-03T04:12:19Z E! [inputs.http] Error in plugin: [url=http://<pyPowerwall Docker IP>:8675/soe]: Get "http://<pyPowerwall Docker IP>:8675/soe": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2023-03-03T04:12:19Z E! [inputs.http] Error in plugin: [url=http://<pyPowerwall Docker IP>:8675/pod]: Get "http://<pyPowerwall Docker IP>:8675/pod": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2023-03-03T04:12:19Z E! [inputs.http] Error in plugin: [url=http://<pyPowerwall Docker IP>:8675/aggregates]: Get "http://<pyPowerwall Docker IP>:8675/aggregates": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2023-03-03T04:12:19Z E! [inputs.http] Error in plugin: [url=http://<pyPowerwall Docker IP>:8675/strings]: Get "http://<pyPowerwall Docker IP>:8675/strings": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2023-03-03T04:12:19Z E! [inputs.http] Error in plugin: [url=http://<pyPowerwall Docker IP>:8675/freq]: Get "http://<pyPowerwall Docker IP>:8675/freq": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2023-03-03T04:12:21Z E! [inputs.http] Error in plugin: [url=http://<pyPowerwall Docker IP>:8675/aggregates]: parsing metrics failed: invalid character 'T' looking for beginning of value
2023-03-03T04:12:21Z E! [inputs.http] Error in plugin: [url=http://<pyPowerwall Docker IP>:8675/soe]: parsing metrics failed: invalid character 'T' looking for beginning of value
2023-03-03T04:12:21Z E! [inputs.http] Error in plugin: [url=http://<pyPowerwall Docker IP>:8675/strings]: parsing metrics failed: invalid character 'T' looking for beginning of value

While this was going on, the pypowerwall website would not be accessible and timeout as expected from the above errors.

Looking at the log from pyPowerwall:

----------------------------------------
03/02/2023 07:11:21 PM [proxy] [ERROR] Socket broken sending response [doGET]
  File "/usr/local/lib/python3.10/socketserver.py", line 683, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/local/lib/python3.10/socketserver.py", line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)
Exception occurred during processing of request from ('<Docker IP>', 33506)
  File "/usr/local/lib/python3.10/socketserver.py", line 747, in __init__
    self.handle()
  File "/usr/local/lib/python3.10/http/server.py", line 433, in handle
    self.handle_one_request()
  File "/usr/local/lib/python3.10/http/server.py", line 421, in handle_one_request
    method()
  File "/app/server.py", line 201, in do_GET
    for device in vitals:
  File "/usr/local/lib/python3.10/socketserver.py", line 747, in __init__
    self.handle()
  File "/usr/local/lib/python3.10/http/server.py", line 433, in handle
    self.handle_one_request()
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/socketserver.py", line 683, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/local/lib/python3.10/socketserver.py", line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/local/lib/python3.10/http/server.py", line 421, in handle_one_request
    method()
Traceback (most recent call last):
  File "/app/server.py", line 222, in do_GET
    for device in vitals:
  File "/usr/local/lib/python3.10/socketserver.py", line 747, in __init__
    self.handle()
03/02/2023 07:11:21 PM [urllib3.connectionpool] [WARNING] Connection pool is full, discarding connection: <Tesla Gateway IP>. Connection pool size: 10
TypeError: 'NoneType' object is not iterable
  File "/usr/local/lib/python3.10/http/server.py", line 433, in handle
    self.handle_one_request()
  File "/usr/local/lib/python3.10/http/server.py", line 421, in handle_one_request
    method()
TypeError: 'NoneType' object is not iterable
  File "/app/server.py", line 179, in do_GET
    temps = pw.temps()
  File "/usr/local/lib/python3.10/socketserver.py", line 683, in process_request_thread
    self.finish_request(request, client_address)
----------------------------------------
03/02/2023 07:11:21 PM [urllib3.connectionpool] [WARNING] Connection pool is full, discarding connection: <Tesla Gateway IP>. Connection pool size: 10
  File "/usr/local/lib/python3.10/socketserver.py", line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)
----------------------------------------
  File "/usr/local/lib/python3.10/site-packages/pypowerwall/__init__.py", line 544, in temps
    for device in devices:
  File "/app/server.py", line 192, in do_GET
    alerts = pw.alerts()
  File "/usr/local/lib/python3.10/site-packages/pypowerwall/__init__.py", line 566, in alerts
    for device in devices:
Exception occurred during processing of request from ('<Docker IP>', 54522)
----------------------------------------
TypeError: 'NoneType' object is not iterable
TypeError: 'NoneType' object is not iterable
  File "/usr/local/lib/python3.10/socketserver.py", line 747, in __init__
    self.handle()
  File "/usr/local/lib/python3.10/http/server.py", line 433, in handle
    self.handle_one_request()
----------------------------------------
----------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/http/server.py", line 421, in handle_one_request
    method()
  File "/app/server.py", line 192, in do_GET
    alerts = pw.alerts()
  File "/usr/local/lib/python3.10/site-packages/pypowerwall/__init__.py", line 566, in alerts
    for device in devices:
  File "/usr/local/lib/python3.10/socketserver.py", line 683, in process_request_thread
    self.finish_request(request, client_address)
TypeError: 'NoneType' object is not iterable
  File "/usr/local/lib/python3.10/socketserver.py", line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)
----------------------------------------
  File "/usr/local/lib/python3.10/socketserver.py", line 747, in __init__
    self.handle()
  File "/usr/local/lib/python3.10/http/server.py", line 433, in handle
    self.handle_one_request()
  File "/usr/local/lib/python3.10/http/server.py", line 421, in handle_one_request
    method()
  File "/app/server.py", line 179, in do_GET
    temps = pw.temps()
  File "/usr/local/lib/python3.10/site-packages/pypowerwall/__init__.py", line 544, in temps
    for device in devices:
TypeError: 'NoneType' object is not iterable
----------------------------------------
03/02/2023 07:11:22 PM [urllib3.connectionpool] [WARNING] Connection pool is full, discarding connection: <Tesla Gateway IP>. Connection pool size: 10
03/02/2023 07:11:22 PM [proxy] [ERROR] Socket broken sending response [doGET]

This is the end of the log, while the issue was going on, the above errors would be scrolling until it self resolved.
Restarting the docker did not resolve the issue while it was going on.

I tried to search here but didnt see anything similar reported before.

Anything I can look at to try to resolve this?

[jasonacox]

I've seen that before. PyPowerwall is having trouble connecting to your Powerwall. I had this occur when I had my Powerwall on WiFi. It showed a strong signal but would still periodically drop off and be unresponsive. I eventually ran an ethernet cable and hardwired it in to my network switch. I haven't seen issue since. Others have reported the same thing.

The only odd bit about your case is that it seems to connect but then hang. I have seen that happen during high CPU load on the Powerwall during an upgrade or when I had another monitoring tool polling it too frequently (which was part of the reason I wrote pypowerwall).

I wish you had the dashboard running. It would be interesting to see if you are getting Powerwall alerts that would indicate a problem. See if you can log in tot he Powerwall console directly and see if it shows any alerts. Alternatively, try http://pypowerwall:8675/alerts (where pypowerwall is the system running pypowerwall container).

[ThePnuts]

I have been tempted to just spin your dashboard up in new dockers and use influx 1.8 just for it, but I worry if I have something working, I'll end up finding something else to work on instead of influx 2.

here is the alerts page:

[
    "PodCommissionTime",
    "FWUpdateSucceeded",
    "SystemConnectedToGrid",
    "GridCodesWrite",
    "POD_w110_SW_EOC",
    "PINV_a067_overvoltageNeutralChassis",
    "SYNC_a001_SW_App_Boot",
    "SYNC_a044_IslanderDisconnectWithin2s"
]

I did try connecting directly to the lan IP as well and it timed out during this period. I guess next time I notice it happen, I can run out and connect to its local WiFi AP and see if its still accessible that way as well.

Just googled where the ethernet point is and its in the gateway, which is outside for me vs the powerwall+ being in the garage.

That might be a separate project for another day if I can confirm this is just a WiFi issue. Its silly since I have a mesh AP like 15' away from it. I think I can also look if it has any network activity next time this happens. The Tesla app is still showing live data, so unless its failing over to its cell connection, its still using\on my WiFi.