lcec driver: Include in Machinekit distribution?

[zultron]

I want to propose distributing binary packages of @sittner's EtherCAT HAL driver in the Machinekit package streams. Considerations for the project are at least the (complex) questions about licensing and simply whether it's in the project's scope.

The IgH EtherLab Master source code is licensed GPLv2, which makes sense especially because it runs in a kernel module. There has been a question (some time back in the LCNC list) about whether the GPL is incompatible with the Beckhoff EtherCAT Master License Agreement, since the latter restricts someone from distributing an EtherCAT Master implementation without first obtaining license, but the GPL explicitly forbids any additional restrictions other than those within the GPL itself.

My thinking has changed over time; at this point in time, (IANAL, of course) I think that the main risk is Beckhoff tells us to stop, and we have to remove the packages. Gerd Hoppe explained to me personally the major concern is a broken implementation could hurt the public of the EtherCAT brand; the license does indeed emphasize this. I don't see how rebuilding IgH's software could change anything in that regard, and if for some reason there were a problem, this suggests that we might get back into compliance by removing the branding, or worst case, simply expunge the packages from project repos.

The other question is whether distributing lcec driver packages is in the project's scope. The source comes from an external project, but IMO, supporting this widely-used, industry standard fieldbus out of the box adds a lot of value to Machinekit, a situation comparable to the EMC application.

While I haven't worked out the technical details, my proposal would probably involve mirroring @sittner's linuxcnc-ethercat driver along with the IgH Master and @sittner's Debian packaging repos in the Machinekit GH org and adding CI configurations to build them and push packages into this org's Cloudsmith repo.

So, first thoughts? Questions? Any support for or objection to this proposal?

[cerna]

My interest in law was always limited to the criminal portion of it only. So I have never been particularly interested in the deep mechanics of why the EtherCAT driver cannot be incorporated into the Machinekit proper even though I started several times to read through the aforementioned (and others here in Machinekit organization) discussion.

So the last time I was thinking about it, my mind went with distributing EtherCAT module (and really pretty much all HAL modules - probably not that great idea) as separate C/C++/whatever packages with something like Conan.io for local build (or separate .deb packages for binary Debian format) - the idea was that if the user built it individually himself, it would solve the distribution legal issue. (Or at least it would be no different from how it is done now.)

Gerd Hoppe explained to me personally the major concern is a broken implementation could hurt the public of the EtherCAT brand; the license does indeed emphasize this.

This is understandable, I think. They are running enterprise and have to think about how they look.

The other question is whether distributing lcec driver packages is in the project's scope. The source comes from an external project, but IMO, supporting this widely-used, industry standard fieldbus out of the box adds a lot of value to Machinekit, a situation comparable to the EMC application.

I think there are two viewpoints being at odds with each other (maybe even at rapier's length). One can argue that the Machinekit-HAL project should only comprise the framework and everything else should be pluggable third party. But then the drivers for Mesa hardware, different PCI cards, PRUs and all other should be expunged too.

Then we have the realpolitik, which says that endeavour can be advanced however it wants but it is still going to fail if the concept is too complex or unsustainable for the end user or the input knowledge needed for the system to get it running (or keep it running) is too high (or too steep in the long run).

So, in other words, for the ease of use and all that, I think the Machinekit EtherCAT driver should be incorporated. Question is in which form? (On another note, the EtherCAT driver does not have to be the only one. I, for example, liked the SERCOS III work from @aschiffler.)

BTW, I agree about the value sentiment.

Another question is if Machinekit organization should just mirror the code or try to coordinate the development somehow. For example in the EtherCAT Driver repository there are unmerged pull requests (which is not very C4-ish) and also the discussion there doesn't seem very active.

Basically, I am all in. I had a problem in the past with how LinuxCNC (more than Machinekit one) community is becoming more and more Mesa oriented exclusively. (With the fixation of reasonable parport configuration on RTAI, which is now dead horse and no amount of beating will make it running again. Plus the Pico Systems is also hanging on thread with shimmering user base.) And I always thought that the EtherCAT option has too high entrance barrier. And I am all in anything which will help with Machinekit's marketing.

Anyway, @zultron, do you want access to Machinekit's Cloudsmith organization? I probably shouldn't be the only one with it.

[zultron]

Then we have the realpolitik, which says that endeavour can be advanced however it wants but it is still going to fail if the concept is too complex or unsustainable for the end user or the input knowledge needed for the system to get it running (or keep it running) is too high (or too steep in the long run).

Very good point.

So, in other words, for the ease of use and all that, I think the Machinekit EtherCAT driver should be incorporated. Question is in which form?

Good news for me!

Another question is if Machinekit organization should just mirror the code or try to coordinate the development somehow. For example in the EtherCAT Driver repository there are unmerged pull requests (which is not very C4-ish) and also the discussion there doesn't seem very active.

Good question. I guess that if folks start using the packages from the MK project, that could attract pull requests and then we'd want to merge them. I can't think of any excuse C4 wouldn't apply to that repo, too. I do need a little guidance from you on how your LCEC "mirroring" works, given that the MK project's version has local changes, and mirroring isn't as simple as a cron job running git fetch upstream; git push --mirror mirror.

Anyway, @zultron, do you want access to Machinekit's Cloudsmith organization? I probably shouldn't be the only one with it.

Sure, good idea.

Right now, I'm just trying to figure out how to build the EtherLab Master in GH Actions, using your config in the MK-HAL repo as an example. It's complicated!

Does your MK CI build push any Docker images anywhere that can be reused for these purposes?

[zultron]

I have a very basic GH Actions configuration based on @sittner's Debianization that builds the EtherLab Master in a Docker container. It doesn't actually do anything useful, like build for all the distros and architectures that the HAL CI builds for, or push packages anywhere. TBH I'm a little fearful of replicating that from the Machinekit examples, since it's so frightfully complicated, but I'll do my best.

Once that works, the lcec HAL driver is next.

[cerna]

Right now, I'm just trying to figure out how to build the EtherLab Master in GH Actions, using your config in the MK-HAL repo as an example. It's complicated!

Sorry? I think that the most important questing is if the EtherLab Master code can be cross compiled, or some portion of its build-system prevents it and it needs to be compiled natively in QEMU. (Of course, if Machinekit organization says that it will not be simply mirrored, then nothing is preventing from patching it.)

Does your MK CI build push any Docker images anywhere that can be reused for these purposes?

Yes, it was meant to. There was a problem that the only way how to store repository specific (i.e. not same across all the forks) information is the secrets vault. And that started causing problems when one of the secrets was "machinekit" - because that is a normally used word in the workflow. So, Github started burning the secrets. I think I wrote about it in #268.

I created a Quay Machinekit organization as the Docker Hub name was already taken (it looks like someone from here created it, but @luminize didn't have the keys and the Quay looks better anyway).

I was thinking that I will repair it by storing the required information in secrets vault in form of JSON string. But as nobody was really using it, I pushed it on the back burner. I have almost finished the pull request for the problem mentioned here, so I will do it next.

I do need a little guidance from you on how your LCEC "mirroring" works, given that the MK project's version has local changes, and mirroring isn't as simple as a cron job running git fetch upstream; git push --mirror mirror.

I don't have LCEC "mirroring", I have the EMCApplication mirroring (not yet applied, as I needed break from it). But generally, I think you need to merge the changes only (to not dirty the history too much). Then you can look if the merge will be fast-forward or if there are some conflicts and then run tests on it.

The tests are the main concern for me there. How can one test it without the hardware?

[cerna]

TBH I'm a little fearful of replicating that from the Machinekit examples, since it's so frightfully complicated, but I'll do my best.

Hmm, I thought it was simple. Or at least simple for what it does - building and testing software in isolated containerized environment. If you want, keep it as simple (one system/architecture) proof of concept and I will do it. To keep it in the same limitations, it should be fairly easy.

[zultron]

Hmm, I thought it was simple.

I took another look, not so bad, and anyway it is worth learning, since I need it for other projects. I'll holler if I need help. Thanks!

[cerna]

Taking deeper look at the state of EtherCAT in Machinekit/LinuxCNC, I wonder what was the main reason why choosing the IgH EtherCAT master over SOEM? I am not saying one is better than other or anything like it. Just genuinely interested why this decision was made.

Reading through manuals they both look to support the RT_PREEMPT kernel extension and Xenomai. So they look (at least at the first glance) similar.

[zultron]

I wonder what was the main reason why choosing the IgH EtherCAT master over SOEM?

@sittner might tell us. I bet it's that linuxcnc-ethercat has been around a long time, longer than SOEM.

I got the etherlabmaster and linuxcnc-ethercat packages building on GH Actions now. The linuxcnc-ethercat build revealed that the machinekit-hal packaging has Multi-Arch: problems, unsurprisingly, so it only builds for amd64 architecture for now.

https://cloudsmith.io/~zultron/repos/etherlabmaster/packages/

[cerna]

Nice.

The linuxcnc-ethercat build revealed that the machinekit-hal packaging has Multi-Arch: problems, unsurprisingly, so it only builds for amd64 architecture for now.

Yes, I haven't done much work on bringing the packaging rules up to modern standards. Quite the opposite in fact, I did the minimal amount to make the Machinekit-HAL packaging useable.

I like your composite run steps actions. This will go nicely in reusability of build flow through all Machinekit's projects. When I started with Github Actions, I had a feeling that it lacks quite a lot of functionality, but now I am starting to realize that it is one of the most advanced public CI system out there.

---

BTW, it looks like the linuxcnc-ethercat package has a problem with dependencies:

apt install linuxcnc-ethercat
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 linuxcnc-ethercat : Depends: linuxcnc but it is not installable or
                              linuxcnc-sim but it is not installable or
                              linuxcnc-uspace but it is not installable or
                              machinekit but it is not installable
E: Unable to correct problems, you have held broken packages.

(Require machinekit package, which is deprecated.)

Maybe (just maybe) it would be good to rename linuxcnc-ethercat to something which is useable both by Machinekit organization and LinuxCNC organization. I am afraid that this name will cause problems and confusion down the line. Something like HAL EtherCAT Driver, HALECd for short (taking notes from LITTLE.big marketing).

[zultron]

the linuxcnc-ethercat package has a problem with dependencies

Thanks, fixed! That saves me annoyance later today when I discover the problem and have to wait through another build.

I haven't done much work on bringing the packaging rules up to modern standards

I'm not used to having someone else care about this besides me! I just submitted #310 about this.

Maybe (just maybe) it would be good to rename linuxcnc-ethercat

Agreed, but @sittner needs to do that; I don't want to fork the project. See sittner/linuxcnc-ethercat#91.

I like your composite run steps actions. This will go nicely in reusability of build flow through all Machinekit's projects.

Thanks! That was a real PITA to get working. However, I got it to build packages for EtherLab Master using the ec-debianize packaging and linuxcnc-ethercat packages, and it also worked to build a few (MK-related) ROS packages. It's still pretty delicate but it's improving, and one of those packages even built successfully on the first try. :P Of course credit for the original work all goes to you!

[cerna]

[...] I don't want to fork the project [...]

Hmm, I understood you incorrectly, then. Based on your answer here I though that fork is exactly what you want.

[...] That was a real PITA to get working [...]

Enjoy then! Do you plan the machinekit_ci repository switch to Machinekit organization? And if so (this is just my own personal opinion) would you mind renaming it machinekit-ci (or something like it), so it is in line with other repositories like machinekit-hal or machinekit-cnc?

[zultron]

[...] I don't want to fork the project [...]

Hmm, I understood you incorrectly, then. Based on your answer here I though that fork is exactly what you want.

You're right, I'm conflicted. If @sittner doesn't want to merge pulls (no problem, there are plenty of legit reasons not to want to), then it does make sense to fork the project into MK, and the MK project would be free to rename.

[...] That was a real PITA to get working [...]

Enjoy then! Do you plan the machinekit_ci repository switch to Machinekit organization? And if so (this is just my own personal opinion) would you mind renaming it machinekit-ci (or something like it), so it is in line with other repositories like machinekit-hal or machinekit-cnc?

If machinekit_ci gets used by the MK org, then it should be pulled in, yes. And of course we can rename it.

[cerna]

You're right, I'm conflicted. If @sittner doesn't want to merge pulls (no problem, there are plenty of legit reasons not to want to), then it does make sense to fork the project into MK, and the MK project would be free to rename.

I read on a local forum that somebody had to spend 40 hours to get the master kernel module to work on 5.4 and arm architecture. Here, I don't know which master he was using, but as it was thread about LinuxCNC, I would presume he was using linuxcnc-ethercat drives with IgH EtherLab EtherCAT master. I also don't know which version it was, as the master code is still developed (last commit two days ago by Florian Pose) - it cannot be designated actively developed, but it is not dead.

However, if this indeed will be so, then some way how to repair these problems in-house (or at least without begging the upstream) and distribute will be needed. Maybe some form of abstraction in linuxcnc-ethercat driver to support both IgH EtherLab EtherCAT master and SOEM would not go amiss.

[cerna]

If machinekit_ci gets used by the MK org, then it should be pulled in, yes. And of course we can rename it.

I still need to solve the automatic building and package upload in EMCApplication. More so now because of the Python problem (machinekit/EMCApplication#2). I was thinking to just copy it there, but if Machinekit starts to use the Machinekit-CI separate repository, then I should do it that way.