New provider: National Grid

[disforw]

Let's gooooo!
I'd love to jump onboard! This library looks good...
There's a lot of articles talking about national Grid and Opower relations but I can't seem to find a login page. Can you direct me in the right direction?

[tronikos]

Do you see requests to opower.com when going to your utility website? If yes, it's up to you or the community to add some code in this library to convert the login credentials to an opower access token. Existing utility implementations might help you.

[danieljkemp]

Some network calls I saw when logging into national grid:

scope openid https://login.nationalgridus.com/opower-uwp/opower profile offline_access
redirect_uri https://myaccount.nationalgrid.com/s/opower-widget

  https://login.nationalgridus.com/loginnationalgridus.onmicrosoft.com/b2c_1a_nationalgrid_convert_merge_signin/oauth2/v2.0/authorize

[disforw]

Nice find!! Let me see if I can line this up with any of the existing providers…

[danieljkemp]

Page source also has these urls:

https://ngbk.opower.com
https://ngny.opower.com
https://ngma.opower.com
https://ngri.opower.com
https://ngli.opower.com
https://ngny-gas.opower.com

[iamkirkbater]

I won't realistically be able to get to this for a few days but I can take a stab at this potentially some time within the next week if it's not already in progress. Just happened to come across this library trying to figure out the same problem, so this directly relates to my interests haha

[disforw]

This seems like it would be very similar to ConEd but maybe we should split it up like Exelon because of the different subdomains.

[disforw]

Here’s what I found for PSEG LI GAS:

initial login endpoint (questionable?)
https://login.nationalgrid.com/loginnationalgridus.onmicrosoft.com/oauth2/v2.0/authorize?client_id=88d004b4-3d39-4599-b410-093849907ee5&p=B2C_1A_UWP_NationalGrid_convert_merge_signin

initial login form payload:
signInName: [email protected]
password: xxxxxxxxxxxxxxxxx
Signin-forgotPassword: FORGOT_PASSWORD_FALSE
rememberUserName: true
request_type: RESPONSE

Get opower access_token:
https://login.nationalgrid.com/loginnationalgridus.onmicrosoft.com/b2c_1a_nationalgrid_convert_merge_signin/oauth2/v2.0/token

the response will include the access_token

the token seems to require a code, I see it in the payload of the request, just not sure where it comes from. There also looks to be many opower subdomains as indicated above, that is only relevant once we have the access_token

[yigitgungor]

This is also one of the calls they make to create the "Green Button Report" https://ngli.opower.com/ei/edge/apis/DataBrowser-v1/cws/utilities/ngli/customers/<customer_id>/usage_export/download?format=csv it starts a download of a csv with all account data, however even though the browser is logged in it gives:

{ "error": { "httpStatus": 401, "serviceErrorCode": "UNAUTHORIZED", "details": "Expected logged in customer but received NOT_LOGGED_IN_WEB_USER" } }

so perhaps the call should explicitly pass the user information too.

[disforw]

I need help here.
To get the access_token needed for this integration seems to be a simple call to the following URL with 3 fields in the body of the request. I just cant seem to find where to locate the code or the code_verifier. They change every time I login, so I know it’s being generated upon login, I just cant find them.
https://login.nationalgrid.com/loginnationalgridus.onmicrosoft.com/b2c_1a_nationalgrid_convert_merge_signin/oauth2/v2.0/token
grant_type: authorization_code
code: xxxxxxxx
code_verifier: xxxxxx

[disforw]

Looks like a user on another project built a login to an Azure app exactly like we need to do here!
LeighCurran/AuroraPlus#2 (comment)

also, adding the following blob for reference
https://github.com/blue-army/zync/blob/03f652bbac4d43be92ed5969d1c22df60f78fcc5/jibe/src/web/assets/docs/token.txt#L18

[X-sam]

Ngrid uses a standard Azure AD B2C provider
MS provides a python library implementation. Shouldn't need to reinvent the wheel here.
Probably all the info you need to instantiate a valid PublicClientApplication:

accountNumber: "" //account-specific
authority: "https://login.nationalgrid.com/loginnationalgridus.onmicrosoft.com/B2C_1A_NationalGrid_convert_merge_signin"
clientId: "36488660-e86a-4a0d-8316-3df49af8d06d"
coreJsUrl:"https://ngny.opower.com/ei/x/embedded-api/core.js?auth-mode=oauth&locale=en_US"
customerType: "Residential"
envurl: "https://myaccount.nationalgrid.com/s/opower-widget"
fuelType: "ELEC"
isOpowerSessionForCurrentUser: false
region: "UNY" //region-specific. UNY is 'upstate new york'
scope: "https://login.nationalgridus.com/opower-uwp/opower"
servicePostalCode: "" //account-specific
source: "CSS"
tenant: "loginnationalgridus"

edit: looks like they use two different client ids for authentication (i.e. login page) (88d004b4-3d39-4599-b410-093849907ee5) and authorization (i.e. getting the token for opower) (see above json data)

[disforw]

from msal import PublicClientApplication

authority = "https://login.nationalgrid.com/loginnationalgridus.onmicrosoft.com/B2C_1A_NationalGrid_convert_merge_signin"
app = PublicClientApplication("88d004b4-3d39-4599-b410-093849907ee5", authority=authority)

result = app.acquire_token_by_username_password("36488660-e86a-4a0d-8316-3df49af8d06d", username="{username}", password="{password}")

if "acess_token" in result:
    print("Login successful!")
    access_token = result["access_token"]
    # Use the access token for further operations
else:
    print("Login failed. Check your credentials.")

[X-sam]

Several notes, most of which can be summed up by "read up on the links above, as well as authentication & authorization flows with OIDC"-

• you'll need to login with the login client, then use the received JWT to obtain an authorization against the authorization client.
• the login client doesn't appear to support authentication non-interactively

    result['error_description']
    'AADB2C90224: Resource owner flow has not been enabled for the application.'

  you'll either have to solve that, provide a pop-up to the user, or build a request-based flow to authenticate, whereupon you can pass the response to a MSAL client on the authorization side. (You can follow along with how national grid instantiates the web version of MSAL, it's not minified so it's easy to find with debug tools in the post-login browser session)

[1ockwood]

Hey, all. After recently getting a National Grid "smart meter" installed, I've been going down the path of trying to integrate the data into Home Assistant, eventually arriving here. I've been digging in a bit myself to see if I can get anything working, but think I may need some help. Based on the discussion so far, and from my experience, it seems like the National Grid auth flow is somewhat troublesome to get working correctly with this project. However, one potential solution that occurred to me would be to instead try to leverage the token endpoint itself.

It would be more roundabout to setup initially, but theoretically, if a user were to log into the site using their browser and manually grab the refresh_token, we could then use that to get an access_token for opower. I'm not that well versed in Python, so I'm a bit out of my depth in adjusting the code to get something like this to work, but here's what I'm thinking if anyone has any feedback or thoughts:

• Log in to your account at https://myaccount.nationalgrid.com/ using your browser.
• Once logged in, you should see some POST requests to https://login.nationalgrid.com/login.nationalgridus.com/b2c_1a_nationalgrid_convert_merge_signin/oauth2/v2.0/token in your Network panel. In the response, you'll see the access_token as well as a refresh_token (among other things).
• If you grab that refresh_token, you can now make a GET request to https://login.nationalgrid.com/login.nationalgridus.com/b2c_1a_nationalgrid_convert_merge_signin/oauth2/v2.0/token with the following:
  • grant_type=refresh_token
  • refresh_token=abc... (this would be the refresh token you pulled from the site)
• That will return your access_token as well as all of the other data, so that I think the refresh token could potentially be stored since it will expire, as noted by the refresh_token_expires_in key in the response.

I'm curious to hear if this is feasible in the context of this project. I have successfully modified the code to roughly work like this, but the main bumps I'm hitting are:

• I'm not sure how to implement support for passing in the refresh_token instead of username and password.
• I'm not sure how to handle updating the refresh token when it expires.
• I'm not sure how to handle the various subdomains, as noted above (such as ngny, ngri, ngli, etc.).

Below is my nationalgrid.py, for reference. Mind you this is a rough proof-of-concept, but using this you can successfully run the demo command, such as:

python src/demo.py --utility nationalgrid --username="anything" password="abcdef" --start_date 2024-04-01 --end_date 2024-04-07 --aggregate_type day

where password is actually the refresh_token you grabbed.

You'll also need to modify the subdomain to match your region.

"""National Grid"""

from typing import Optional

import aiohttp

from ..exceptions import InvalidAuth
from .base import UtilityBase

import logging

class NationalGrid(UtilityBase):
    """National Grid utility class."""

    @staticmethod
    def name() -> str:
        return "National Grid"

    @staticmethod
    def subdomain() -> str:
        return "ngny"

    @staticmethod
    def timezone() -> str:
        return "America/New_York"

    @staticmethod
    def uses_refresh_token() -> bool:
        """Check if utility uses refresh token for authorization."""
        return True

    @staticmethod
    async def async_login(
        session: aiohttp.ClientSession,
        username: str,
        password: str,
        optional_mfa_secret: Optional[str],
    ) -> Optional[str]:
        #
        logging.debug("LOGIN");
        token_url = "https://login.nationalgrid.com/login.nationalgridus.com/b2c_1a_nationalgrid_convert_merge_signin/oauth2/v2.0/token"
        async with session.post(
            token_url,
            data={
                "grant_type": "refresh_token",
                "refresh_token": password
            }
        ) as response:
            if response.status == 200:
                data = await response.json()
                access_token = data.get("access_token")
                return access_token
            else:
                # If login fails, raise an exception
                raise InvalidAuth("Invalid username or password")

[X-sam]

Sorry, I haven't had time to put everything together into a PR for opower. But it's not hard to get through the initial user OAuth, here's how I did it for my region.

import re
import requests
import json

def authSession():
    username=""
    password=""
    session = requests.Session()
    session.verify = False
    session.cookies.set("USRW","r=nyupstate&ct=home",domain=".nationalgridus.com")
    session.headers['User-Agent']='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36'
    req = session.get("https://myaccount.nationalgrid.com/services/auth/sso/NGP_SignIn_NY_Upstate_Home")
    sets = re.search("var SETTINGS = ({.*?});",req.text,re.DOTALL)
    settings = json.loads(sets.group(1))
    # var f = r.hosts.tenant + "/" + r.api + "?tx=" + r.transId + "&p=" + r.hosts.policy
    url = f"https://login.nationalgrid.com/{settings['hosts']['tenant']}/{settings['api']}?tx={settings['transId']}&p={settings['hosts']['policy']}"
    headers = {'X-CSRF-TOKEN':settings['csrf']}
    authed=session.post(url,headers=headers,data=[("signInName",username),("password",password),("Signin-forgotPassword","FORGOT_PASSWORD_FALSE"),("request_type","RESPONSE")])

    completed = f"https://login.nationalgrid.com/{settings['hosts']['tenant']}/api/{settings['api']}/confirmed?tx={settings['transId']}&p={settings['hosts']['policy']}&csrf_token={settings['csrf']}"
    res = session.get(completed)
    breakpoint()
    r = res.text
    session.close()
    
    return r

[1ockwood]

That's great to hear @X-sam! Let me know if I can help in any way. Like I said, I'm a little out of my area of expertise here, but happy to contribute if I'm able.

[disforw]

Don't forget about me, what can I do to help?

[mag2352]

Are there any updates on this? I'm going to take a look this week, but am interested if any further developments have been made here.

[disforw]

Nothing really, I believe we have all the components here but couldn't seem to put the pieces together.

[disforw]

from msal import PublicClientApplication

# Replace with your own values
tenant_id = "your-tenant-id"
client_id = "your-application-id"
authority = f"https://login.microsoftonline.com/{tenant_id}/B2C_1_susi"

# Initialize the MSAL client
app = PublicClientApplication(client_id, authority=authority)

# Acquire a token silently (if user is already authenticated)
result = app.acquire_token_silent(["https://your-api-resource-uri"])

if not result:
    # If silent acquisition fails, prompt the user to sign in interactively
    result = app.acquire_token_interactive(["https://your-api-resource-uri"])

if "access_token" in result:
    access_token = result["access_token"]
    print(f"Access token: {access_token}")
else:
    print("Authentication failed.")

# Now you can use the access token to call the Opower API
# Remember to handle token expiration and refresh as needed

[SplicedNZ]

I've submitted a pull request for a different utility company that uses Azure B2C OAuth: https://github.com/tronikos/opower/pull/87/files Might help here if you can replace values for those specific to National Grid

[X-sam]

Thanks for that @SplicedNZ, worked for this as well. #91 should cover National Grid in the Upstate NY region.

[tronikos]

Can people from other regions try this out? I expect you will need to update the opower subdomain (ngny) and judging from the posts above, at least the POLICY (B2C_1A_UWP_NationalGrid_convert_merge_signin).

[disforw]

tested, failed. seems like I will need a subdomain of NGLI. Should I just submit a PR to change the name and subdomain?

[tronikos]

You need to clone the repo, setup a python development environment, copy the existing nationalgridnyupstate.py to a new file, rename the file and class to match your region, update constants in the file and try it out with:

python -m pip install .
python -m opower

Whenever you make changes to the file you need to rerun both of these commands to test your changes. Once it's working, submit a PR.

[shaunburdick]

Similar to home-assistant/core#122658 and now that #91 is merged, do we need to bump opower in HA?

[tronikos]

Bumping in home-assistant/core#124475
@X-sam could you create a virtual integration? I think a single National Grid virtual integration should be fine. I don't think we really need one for each region.

[X-sam]

Happy to give it a go, where might I find some context for that?

…

On Fri, Aug 23, 2024, 6:41 AM tronikos ***@***.***> wrote: Bumping in home-assistant/core#124475 <home-assistant/core#124475> @X-sam <https://github.com/X-sam> could you create a virtual integration? I think a single National Grid virtual integration should be fine. I don't think we really need one for each region. — Reply to this email directly, view it on GitHub <#45 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABSQJ56DM554DQBURPEJI43ZS4GT3AVCNFSM6AAAAAA4RVII56VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMBWHAYTKNBSGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[tronikos]

You need a total of 3 PRs similar to:
home-assistant/core#122650
home-assistant/home-assistant.io#33978
home-assistant/brands#5705

[disforw]

To be clear, one virtual integration for National Grid, but we still need a utility for each region breaking out and changing the subdomains. Right?

[tronikos]

Right. People from other regions need to send a PR with a utility class with the right domain and constants.

[disforw]

Shall I just run through the list and make all 6?

[tronikos]

Shall I just run through the list and make all 6?

Is the only difference in the domain? If everything else is the same, can we just have a single class and after the login use the appropriate domain based on the result of some endpoint?

[disforw]

Small tiny little issue here, changing the subdomain successfully logs me in and adds the integration to HA, but no entities come up.
No errors either

[tronikos]

For many utilities there are no entities. Are there any statistics? You use the statistics in the energy dashboard. See documentation with screenshots.

[EffortlessHome]

Hi, I created a Nat Grid integration for NGMA. I have tested and it is working well with my energy data. No sensors, but usage and costs show in the energy dashboard. @tronikos would you please review and add to Pypi?
nationalgridma.py.txt