Improve WiFi-Provisioning

[DerPicknicker]

Hi @CarlosDerSeher ,

in the other Issue we discussed the Wifi-provisoning. I think two options would increase the usability a lot.

The current Situation doesn't allow changing Wifi-SSIDs / Passwords without flashing the board again.

Option 1:
Using the current implementation and start the Provisioning again after 3-5 failed connection attempts.

Option 2:
Use this Library: https://github.com/jnthas/Improv-WiFi-Library and support the Improv-Protocol. Big benefit is that you don't need the Espressif app and you can flash it with your browser and setup the wifi as well in your browser. I did a small research and I can't find a pure C-Version of the Library. I guess a lot of Code should be working also in pure C. Only the Vectors might be Problem if I am informed correctly.

[CarlosDerSeher]

I'll have a look at it as soon as my development machine is up and running again. I installed some new drivers yesterday and it won't boot anymore since then :(

[CarlosDerSeher]

Also I saw you can also use serial connection with Improv lib to provision wifi

[DerPicknicker]

Yes that's the main usecase. The webflasher supports that. So you can flash it and directly set up the WiFi.

[CarlosDerSeher]

Ok I'll check it out. Something new to learn here :) I've not done much mixing of C and C++ code but seems it is possible to write a C wrapper for C++ functions. I'll see how much effort this will take and report here.

[DerPicknicker]

I think the benefit is that we can build the image with the GitHub action (I will try to build them after syncing is merged). And the webinstaller will automatically have the latest versions online.

[CarlosDerSeher]

Let's see what @anabolyc thinks about the espressif implementation and if he wants to use that before we put any effort into this. It can also do serial provisioning

[anabolyc]

I think the benefit is that we can build the image with the GitHub action (I will try to build them after syncing is merged). And the webinstaller will automatically have the latest versions online.

I tried to do that in the other project with artifacts taken from the release, which in turn were generated by GitHub action using the release tag. The problem is that esp-web-tools would only allow firmware to be pulled from the same domain. So I had to place binaries in the repository itself. It is possible to commit them into the repo using GitHub action, it is a bit clumsy but should work.

In any case, until WiFi provisioning is in place, it would not work anyway. I'll try to look into that before EOW.

[anabolyc]

@CarlosDerSeher @DerPicknicker I just tried example implementation of WiFi Provisioning. Book example have only BLE and SoftAP methods implemented, and they both require installing an app on your phone (an extra device) and not particularly elegant.

The Console option seems to be functionally equivalent to improv-wifi but it is not supported by ESP Web Tools. I asked folks if they are planning to add it, but I would not expect this to happen soon.

So I still think improv-wifi is the best option, as long as it is possible to wrap it up into esp-idf component.

[anabolyc]

@CarlosDerSeher here is the ESPHome implementation demo. Just to give you an idea how easy this is to flash and configure wifi in the same UI

[CarlosDerSeher]

@anabolyc they use BLE as a transport though don't they?

[anabolyc]

@anabolyc they use BLE as a transport though don't they?

If you're asking about ESPHome, they use Serial-console provisioning. Espressif's implementation has 3 options, BLE is one of them. Bear in mind, it is not available for all ESP32 versions.

[CarlosDerSeher]

I've been looking at this example and it seems to me it is for BLE provisioning. Do you have a link where serial provisioning using improv is shown?

[CarlosDerSeher]

Ok, found some more documentation. Still, an example would be great.

[DerPicknicker]

Ok, found some more documentation. Still, an example would be great.

What is the problem by using the library which I linked? Seems fairly easy to port to pure c. Only the integration into the snapcast code is not that straightforward because we must change the current logic.. checking from time to time if WiFi provisioning data is available or if connection failed multiple times

[CarlosDerSeher]

Hahaha ok my brain completely forgot about the arduino implementation :)

[DerPicknicker]

@CarlosDerSeher out of sync => hardsync 😉

The only thing I can't say is how much effort it is to mix c++ code in IDF or port it to pure c. Which way is the best. That depends on the code structure. You have definitely more insights ;-)

[CarlosDerSeher]

@DerPicknicker @anabolyc ok, so I am trying implement improv Wifi but I am unsure how to test if the responses are right... Any pointers / scripts / web sites which enable me to send improv packets or at least generate improv packets which I can send over the serial port?

[DerPicknicker]

@CarlosDerSeher .. Great to hear.

I guess here if you connect you should be able to provision the device.

[CarlosDerSeher]

Ok that's a start, but I guess there is no possibility to debug with this approach... I have no option to see what's going on on the serial port. If there is no way to generate defined packets so I can step manually through the provisioning process I guess my only option is to attach another (two) usb2serial RX line to my dev board... Any other suggestions how to proceed smarter here?

How is the checksum calculated here?

[DerPicknicker]

You can use putty to connect to the port and send them manually or you could check the browser developer settings if there is some output what is going on. I guess there should be some logging.

Alternatively print out each packet which is sent to your dev-board like a loopback device.

[DerPicknicker]

See this taken from here (282):

 void ImprovSerialComponent::send_response_(std::vector<uint8_t> &response) {
   std::vector<uint8_t> data = {'I', 'M', 'P', 'R', 'O', 'V'};
   data.resize(9);
   data[6] = IMPROV_SERIAL_VERSION;
   data[7] = TYPE_RPC_RESPONSE;
   data[8] = response.size();
   data.insert(data.end(), response.begin(), response.end());
 
   uint8_t checksum = 0x00;
   for (uint8_t d : data)
     checksum += d;
   data.push_back(checksum);
 
   this->write_data_(data);
 }

[CarlosDerSeher]

Yeah, already saw this right after my post. thanks though :)

[CarlosDerSeher]

@anabolyc @DerPicknicker just a little teaser, I got improv service working and I am able to get credentials through https://web.esphome.io/ into my esp32. What's missing now is integration into wifi_interface component. Hopefully I'll be able to push the branch by the end of the week :)

[DerPicknicker]

Really Cool! This would be great after that I will do my best to create an GitHub action to build automatically binary's for the web-installer

[anabolyc]

@CarlosDerSeher that's really cool. Can't wait to try it out. @DerPicknicker I will try to help there, as I want to integrate it into my web installer

[DerPicknicker]

If I pressed the button 3x will also start the Espressif provisioning and the improv WiFi? Or only the last one?

The only problem I see is if a board doesn't have the button you won't be able to restart the provisioning. That's why I thought about to check from time to time.

[CarlosDerSeher]

You got a point. Some time ago I found a component which enables logging to a network port. Maybe ghis will be a possible route to go. So disable logging to serial but enable network logs?!

[DerPicknicker]

@CarlosDerSeher ...

See here https://github.com/VedantParanjape/esp-wifi-logger

Or alternatively display the logs on the HTML-Page? Safe them to the NVS and place a button which downloads the text-file.

[CarlosDerSeher]

I'd prefer to not wear out the flash by writing all those logs to it. I think it will be better put to RAM and presented on a udp port or similar. I'll see how much RAM I am willing to sacrifice for that :)

[DerPicknicker]

Good point. Do we really need logs?
I think not really . If the code is stable enough I would just disable them completely.

[CarlosDerSeher]

@DerPicknicker ok I pushed some changes to the branch because I found this whole discussion about logging obsolete. I tried provisioning and logs always on, and what can I say... it just works together. The improv header "IMPROV" ensures all other data sent is discarded by the web config interface.

[DerPicknicker]

@CarlosDerSeher .. That sounds great. When do you think it will be merged? After syncing is merged?

[CarlosDerSeher]

Yes we can merge this together or merge beforehand to sync branch doesn't matter

[anabolyc]

Hey @CarlosDerSeher @DerPicknicker I can confirm that wifi-provisioning works via web-installer. Great work!
I will try integrating it into the pipeline and let you know how it goes

[CarlosDerSeher]

@anabolyc I think the implementation might change at some point though as I don't like how I've done it currently. For now I guess it is good enough.

[DerPicknicker]

@anabolyc ... I can support you. If you want. Contact me on discord. I would love to see if we can build binary's for each supported board (just copying the SDKCONFIGS)...

[anabolyc]

I mostly have it working, just one thing concerns me. Do I need to configure PSRAM or is it assumed on every board?
Upd: found it, will enable and retest everything ;)

[DerPicknicker]

@anabolyc .. Could you share how you did it? Because it would be Cool if we can do that for all supported boards (lyrat, Lyrat Mini, etc.)

[anabolyc]

@CarlosDerSeher @DerPicknicker

Here is the pipeline

The idea is that whenever code is committed to the feature branch, the pipeline fires up and builds firmware for defined boards sdkconfigs (I added three Esparagus boards for now). Built images are committed to the same branch and manifest files are updated.

Then I need to pull the latest changes into my feature branch for testing locally and it can be merged to master, where the pipeline is not enabled.

Earlier I added a web installer that defines UI for the web installer app. Built artifacts and manifests are published to docs folder, which is automatically published to GitHub pages.

Comitting binaries approach is not very neat, but esp-web-tools require firmware files to be served from the same domain (allowed cross-domain sources to be precise), so I cannot pull binaries from build artifacts or release files, they have to be published together with GitHub pages. An alternative approach is to host binaries outside GitHub, but I don't want to maintain and pay for external infra.

Example run here

[anabolyc]

@CarlosDerSeher I think I mentioned before, that I can share the Esparagus or Louder-ESP board with you if you plan to support and use it. I appreciate your work here and that's the least I can do to support you. Obviously, @DerPicknicker already has one :)

[DerPicknicker]

Thank you for the detailed explanation - maybe @CarlosDerSeher can also share his thoughts. I think the Snapcast-Repo should include the Webinstaller for all boards as it's the place where all changes are happening.

So maybe we can add a GitHub page to this repo where we can get the builded binary's. That includes all Louder-Board and also the Lyrat boards.

That's only an strategic discussion;-)

@anabolyc I use your board quite often for testing it's great.

[CarlosDerSeher]

@anabolyc thanks I appreciate your offer. As I already have my own line level solution the Louder-ESP would be of interest to me. Maybe it is able to replace my justboom amp hat rpi player. I doubt it though as it probably hasn't enough juice for outside playback in my garden. But for sure there will be a spot for it :)

You are selling these players commercially in Europe? How do you handle CE?

[CarlosDerSeher]

Comitting binaries approach is not very neat, but esp-web-tools require firmware files to be served from the same domain (allowed cross-domain sources to be precise), so I cannot pull binaries from build artifacts or release files, they have to be published together with GitHub pages.

I don't really get this part. Normally I would think the best way is to use github releases to host pre-built binaries

[anabolyc]

I don't really get this part. Normally I would think the best way is to use github releases to host pre-built binaries

I started with that approach. Problem is releases are served from github.com domain. Github pages (and thus installer) from github.io domain. JS code cannot fetch files cross-domain unless CORS headers are present, and they are not in fact.

[anabolyc]

@anabolyc thanks I appreciate your offer. As I already have my own line level solution the Louder-ESP would be of interest to me. Maybe it is able to replace my justboom amp hat rpi player. I doubt it though as it probably hasn't enough juice for outside playback in my garden. But for sure there will be a spot for it :)

Pls drop me a line to [email protected], I'll need your address for shipping. Just to be sure, Louder-ESP is a bare board, while Louder Esparagus is a cased solution, but both have external wifi antenna.

You are selling these players commercially in Europe? How do you handle CE?

I worked with Crowd Supply on CE/UKCA compliance, they have good lawyers :)

[DerPicknicker]

Thank you for the detailed explanation - maybe @CarlosDerSeher can also share his thoughts. I think the Snapcast-Repo should include the Webinstaller for all boards as it's the place where all changes are happening.

@CarlosDerSeher .. What do you think about this topic?

[CarlosDerSeher]

You mean the source code only or also a hosted version on github.io where actions push latest binaries?

We can do that. How should we handle the generic boards though?

[DerPicknicker]

The second part - the changes happen here. So why not build news binary's when changes are made. Build the binary's for each supported board and publish them to the web installer.

[CarlosDerSeher]

We could do that if someone is willing to put in the effort to make it work. What are the requirements for this exactly?

[CarlosDerSeher]

Also shouldn't this discussion continue in #72?

[anabolyc]

We could do that if someone is willing to put in the effort to make it work. What are the requirements for this exactly?

I can do that. Just give me a list of boards that you're willing to support, I'll add them to my wen installer and push changes upstream

[DerPicknicker]

@anabolyc

Lyrat 4.3 @CarlosDerSeher used this (if I remember correctly...) Let's move this discussion to the issue Carlos linked ;-)

Lyrat Mini 1.3 I used this board.

I would also recommend creating a binary for generic boards like those boards which doesn't require any special configuration (i2c, psram) ..