What's the priority?

[shykes]

Sorry for the vague title. I'm hoping to continue here a conversation that started live. The topic is: what should Project Gale prioritize? I see two options:

1. A better alternative to nektos/act. This means: drop-in compatibility for an entire Github Actions workflow, with the goal of testing it locally. Just like Act - but better in some way. Under the hood it leverages Dagger to be 10x faster, or simpler to use, or easier to customize, etc. But in this scenario you don't need to know about Dagger to use Gale - it's just an ingredient. The biggest challenge is compatibility: even a small incompatibility can ruin the experience ("why bother to test on Gale if I'll have to test again on Github?"), and the compatibility surface area is quite large. First 80% is easy; last 20% is very hard. The biggest benefit of this option, is that the potential audience is huge - lots of people use Github Actions, many of them use Act, many of those don't love it and would welcome an alternative.

2. A Github Actions loader for Dagger developers. This means: engineers already using Dagger to develop their pipelines, and looking for a way to incorporate some of their legacy Github Actions into their new Dagger project with minimal effort (for example to overcome objections by their team), get an easy way to do this. In just a few lines of code, boom, they can run the same Github Action with the same inputs, from code instead of YAML. This can greatly accelerate adoption of Dagger. This is what @kjuulh is looking for, for example. pro: compatibility is easier - less surface area (only run one step at a time; possibly one job? only need to run it from code; stakes are lower too: if only 50% of the my actions can be run, I can use that 50% for my Dagger project). con: smaller audience - you need to be a Dagger developer to enjoy the benefits of Gale. At the moment there are less Dagger developers than Github Actions users (although we hope to change that one day :)

NOTE: this may not change the core implementation of Gale. But it can change how it is packaged, documented, and marketed - quite a bit!

I would love to hear everyone's thoughts on this!

Option	What is it for?	Who is it for?	Pros	Cons
1	Like Act, but better. Run your Github Actions workflow locally, for easier local testing	Github Actions users	Bigger audience	Unclear differentiation; Compatibility is a big problem
2	Run Github Actions from your Dagger pipeline. Makes it easier to adopt Dagger incrementally when already using Github Actions.	Teams who use Github Actions, and want to adopt Dagger	Solves a clear problem for Dagger developers; compatibility is less of a problem	Smaller audience

[aweris]

Hey @shykes, thank you for initiating this discussion and creating this issue. I understand your points and mostly agree with them.

A better alternative to nektos/act. This means: drop-in compatibility for an entire Github Actions workflow, with the goal of testing it locally.

Our ultimate objective is to be able to execute GitHub actions locally without any discrepancies from our actual pipelines. However, as you previously mentioned, the last 20% of the task is quite challenging to predict in terms of the effort required.

A Github Actions loader for Dagger developers.

This could be a significant milestone on our journey. Despite the audience being small, the effort required is also relatively minimal.

Prioritizing option 2 would be a beneficial decision for everyone involved.

• It would make it simpler for the community to gradually incorporate Dagger into their projects if they are already using Github Actions.
• Additionally, opting for option 2 would allow for earlier community involvement and the establishment of a feedback loop, resulting in faster and improved compatibility support for option 1.

[shykes]

Prioritizing option 2 would be a beneficial decision for everyone involved.

• It would make it simpler for the community to gradually incorporate Dagger into their projects if they are already using Github Actions.
• Additionally, opting for option 2 would allow for earlier community involvement and the establishment of a feedback loop, resulting in faster and improved compatibility support for option 1.

That makes sense to me. Making it a gradual evolution from "option 2" to "option 1", where the audience expands over time, and the compatibility matures over time, seems natural.

Wdyt @gerhard @tiborvass @kjuulh ?

[kjuulh]

Option 2 would definitely be awesowe. I'd be happy to give it a shot as well. Getting support for individual actions would allow a huge catalogue of prebuilt tools from the github marketplace. Tbh it seems to fit quite well with the extension tooling, and could initially just be part of this tooling.

This could also be the same for other Ci tools: drone, Travis, you name it. Being able to support those would smooth the migration experience quite a bit.

And again it is something I've got on my personal road map so I'd be happy to contribute here

[tiborvass]

@shykes makes sense to do 2 first, and then 1.

[c-p-b]

I also like 2. I had mentioned to Dagger team a couple of times that we were also looking at implementing a shim of sorts to help drive adoption in our org, so centralizing that effort makes sense IMO.

I also want to also amplify the notion that completely emulating GHA locally is tough. A brief glance of the issues on nektos/act really highlights the long tail of issues and whack a mole required to do so properly. Though I do think it makes sense as a long term goal!

[aweris]

As we seem to favor option 2, I would like to elaborate on it further. Specifically, I want to clarify the meaning of "A Github Actions loader for Dagger developers."

In my opinion, there are several options available:

1. Implement it as a standalone CLI. This solution is the easiest but may not offer adequate support for development or a smooth transition to Dagger.
2. Use it as a Dagger entry point. This option enables a more user-friendly experience with gale like dagger do gale:run. However, it is the same as option 1, not adding additional value.
3. Develop it as a library that wraps Dagger in a transparent manner and provides additional functionality for utilizing custom actions or pipelines created by workflow jobs.
4. Develop it as a library that wraps Dagger in a transparent manner, loads the Github Actions environment into Dagger containers, and acts as a proxy between Github Actions and Dagger container.

Combining options 3 and 4 would be optimal. For instance:

gale.FromContainer(dagger.Container)
 .LoadGithubEnv()
 .WithStepAction("actions/setup-go@v4", ...options)
 .WithStepRun("go build -o ./build/gale")
 .Exec()

or

gale.FromWorkflow(".workflows/test.yaml")
 .WithInputs({some: "inputs"})
 .WithEvent(<some event>)
 .Container()

This approach enables us to utilize local execution support and better utilization on Running or Converting Github action runs to dagger pipelines.

These are just a few options that come to mind, and we can support one or more of them. Please feel free to provide feedback or add additional suggestions.

WDYT @kjuulh @cpdeethree @tiborvass @shykes ?

[tiborvass]

I don't see the added value of the gale.FromContainer example, I was imagining something more like the gale.FromWorkflow example.

I do think that the dagger entry point would be really great. It would show that "by default" dagger supports GHA yaml and that one can add additional non-GHA workflows if needed. Then as a next step in customization, if one wants to augment a GHA workflow with dagger, that's when they could use gale.FromWorkflow("...").Action("...") and continue building the graph (or removing/modifying parts of it if that's even possible).

[c-p-b]

Conversely, I more see the value in the first example. The idea being that our workflows themselves are fairly easily rewritten but reimplementing all of the actions from the marketplace are not. I do think from Dagger’s selling point perspective the second example has a lot of value, in that someone can get immediate loose-grained benefits with a tiny change, but without properly going in and declaring inputs and outputs one won’t be able to reap the biggest benefits. Perhaps the second example is what you do first to demonstrate the value, then move towards doing the first example for marketplace actions that aren’t trivial to reimplement as simple code as a next step.

Another note: in order to best preserve the abstraction of GitHub, loadGithubEnv should probably force the end user to explicitly load the environment variables in instead of implicitly taking them from the host environment (not sure if that was actually the thought process in the example). I feel like it is better to let the end user construct and control the object that defines the GitHub Actions Environment so that CI remains portable eg if someone were to switch to Jenkins they wouldn’t have to manipulate the environment variables on the Jenkins runner to make this solution work, they would simply change how they construct that object.

[gerhard]

@shykes I imagine Option 1, where, the gale CLI:

1. Recognises a GitHub Actions workflow YAML
2. Is able to run the most popular actions (e.g. checkout, setup-go, etc.)

I'm thinking:

gale run .github/workflows/test.yaml

I imagine this running the inline progress UI view dagger/dagger#5078

In my mind, GALE's main priority is to convert existing GitHub Actions workflow YAML to something that Dagger is able to run. For example, the checkout action would become Git()..., etc.

The second priority is figuring out how to make env, secrets & context work locally.

Being able to understand syntax such as if & matrix is third on my list of priorities.

Not needing to worry about converting existing GitHub Actions YAML to something else - regardless how simple - is the lowest-friction adoption that I think everyone will want. My mental model is "make the hard thing simple, then do the simple thing".

Note
I am wondering if we could use something like https://www.gharchive.org/ or https://ossinsight.io/explore/ to gain more insight into all public GitHub Actions workflows.

[aweris]

Another note: in order to best preserve the abstraction of GitHub, loadGithubEnv should probably force the end user to explicitly load the environment variables in instead of implicitly taking them from the host environment (not sure if that was actually the thought process in the example). I feel like it is better to let the end user construct and control the object that defines the GitHub Actions Environment so that CI remains portable eg if someone were to switch to Jenkins they wouldn’t have to manipulate the environment variables on the Jenkins runner to make this solution work, they would simply change how they construct that object.

Yes, definitely we need to be more flexible about loading env however, forcing users to configure env to themselves will make an initial investment too much. I think we can find a middle-ground load environment implicitly and then allow users to override values by configuration or some other mechanism.

[aweris]

I don't see the added value of the gale.FromContainer example, I was imagining something more like the gale.FromWorkflow example.

In my opinion, both of these options will serve different purposes. The first example aims to minimize the expenses associated with moving GitHub action workflows to Dagger while simultaneously providing users with more choices. On the other hand, the second example offers users the ability to run workflows with Dagger entirely. Nevertheless, both of these instances are simply brief codes that provide examples of potential approaches. We can accommodate both of them, and the only distinction will be how we present the API.

[aweris]

Hey 👋

Regarding option two, I have generated a PR that establishes an interface similar to the one I mentioned earlier #23.

I still need to work on that PR but do think is it useful for option 2?

cc: @kjuulh @cpdeethree

[kjuulh]

It looks quite promising I will need to test it out to get a feel for the api.

It looks to do the job quite well.

Awesome work!

[aweris]

Just a warning, it's still hacky, I'm still working on it. Probably it's not ready to test but I think I can fix and add some examples today

[c-p-b]

Nice. will leave feedback on the other pr

[aweris]

I am closing this issue since it's not meaningful post Zenith gale structure.