kubectl iter8 plugin CLI

[kalantar]

For discussion of the CLI needed for a kubectl plugin for iter8. An iter8 CLI can be used to run experiments locally (as a local process). The kubectl plugin extends this function to include the ability to run experiments in a container in a kubernetes cluster. In particular, we envision it running in a kuberneties Job.

Support is needed throughout the experiment lifecycle. Some of these are common to local experiments, others unique to remote (kubernetes) experiments. A design goal is to use common commands where ever possible.

The proposed commands are intentionally simple. Defaults are assumed and minimal options are expressed. This represents the simplest form of the commands. Extensions can be considered in the future. A few are mentioned at the end.

A suggested priority for each command is documented. If not specified, it is high.

1. Authoring Experiments

[kubectl] iter8 verify - verifies that the experiment (defined in experiment.yaml) is valid. In particular, that it is compatible with any expressed iter8version. It is expected that an implementation will start simple (and incomplete) and be developed over time. medium priority

[kubectl] iter8 lint - Implements the verification done by iter8 verify) and suggests best practices that might improve an experiment. For example, it is valid to have multiple assess-version tasks. However, only the last such task will be used by Iter8. iter8 lint would identify multiple tasks as inconsistent with best practice. As with iter8 verify this is expected start simple and be extended over time. low priority

2. Running Experiments

[kubectl] iter8 run - Runs the experiment (defined in experiment.yaml) locally.

kubectl iter8 generate [-t target] - Generates a manifest (yaml string) defining the Kubernetes resources that should be deployed to run the experiment (defined in experiment.yaml) in the cluster. To generate and deploy an experiment, use : kubectl iter8 generate | kubectl apply -f -.

The -t option identifies a target of the generation. By default, this is standard kubernetes. An alternative allows for objects specific to Code Engine.

3. Observing Experiments

kubectl get - list experiments found in the current Kubernetes context. This list is displayed in tabular format with the following headers:

NAME      COMPLETED      FAILED      WINNER FOUND     WINNER      MODEL VERSION

Fields COMPLETED, FAILED, and WINNER FOUND are all boolean. If no winner is found, the WINNER is empty.
medium priority

[kubectl] iter8 report [-o output_format] - generates information about the current state of the most recent experiment (in the current Kubernetes context). The set of valid output formats, the default format, and the details of each is TBD.

[kubectl] iter8 assert [-t timeout] (-c condition)+ - Evaluates the specified conditions against the most recent experiment (in the current Kubernetes context). Valid conditions are: completed, failed, winnerfound, and winner=<label>.

kubectl iter8 manifest - retrieves the yaml manifest of all kubernetes objects related to the most recent experiment (in the current Kubernetes context).

kubectl iter8 logs - display the logs for the execution of the most recent experiment (in the current Kubernetes context). medium priority

4. Deleting Experiments

To delete the Kubernetes objects related to an experiment, use:
kubectl iter8 manifest | kubectl delete -f -

Extensions

Where commands assume an experiment is defined in a (local) file experiment.yaml, we may want to add an option: -f <filename|URL> to allow an arbitrary file either local or accessible via HTTP.

Where an assumption of the latest experiment is made, we may want to allow the option to specify the experiment name.

A global option -v might be offered to control output verbosity.

[sriumcp]

@kalantar This is an excellent summary. I'm wondering if we can distill the above down to an MVP version (and maximize reuse of the base CLI commands) as follows.

---

1. To list all experiments... kubectl iter8 get (same as above)
2. To download/fetch the last experiment from cluster... kubectl iter8 get --last: download manifest.yaml, experiment.yaml, result.yaml, and logs.txt
3. To download/fetch an earlier experiment from cluster... kubectl iter8 get -e <experiment-name>: download manifest.yaml, experiment.yaml, result.yaml, and logs.txt

---

gen is a subcommand you get for free from the base CLI. The specific template for k8s might need to be supplied in the kubectl plugin layer
4. To run an experiment in cluster... kubectl iter8 gen -t k8s | kubectl apply -f - (same as above)

---

what you get for free from the base CLI
5. assert and report are the same as base CLI (reuse). They can be coupled with kubectl iter8 get --last and -e (see above)
6. no need for an explicit logs command, as logs are available in logs.txt simply through the above get command

---

you now get deletion for free using the above functionality as follows
kubectl iter8 get --last && kubectl delete -f manifest.yaml

---

To summarize, the only additional command we need to implement for kubectl iter8 to get all of the above functionality would be..

kubectl iter8 get (along with the --last optional flag, and -e optional flag)

---

@huang195 FYI.

[sriumcp]

Also... to minimize our efforts... we don't need ginkgo / env-tests for this type of functionality.

If we structure the plugin appropriately, we should be able to get away with unit testing with fake clients as in this example.

[huang195]

If I remember correctly, we said - we can defer most of the optional arguments till later - `--last` should be omitted for now in favor of `v0.7` style.

…

On Mon, Nov 8, 2021 at 3:57 PM Srinivasan Parthasarathy < ***@***.***> wrote: Also... just so we remember, we don't need ginkgo / env-tests for this type of functionality. If we structure the plugin appropriately, we should be able to get away with unit testing with fake clients as in this example <https://github.com/kubernetes/client-go/blob/master/examples/fake-client/main_test.go> . — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1046 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB5DHZIQJG7JAKEXBHOSX2DULA2SDANCNFSM5HTST4WA> .

[sriumcp]

I am good with leaving out --last, but I think we still want to provide a way to delete a specific experiment.

So, if we have kubectl iter8 get (for listing all experiments in a namespace), and kubectl iter8 get -e <experiment name> for getting a specific experiment, we should be good.

---

I believe we do need both. Without the former, the user will not have a simple way of knowing what experiments have been created. Without the latter, the user will not have a simple way of deleting a specific experiment.

[kalantar]

I’m seeing an apparent dichotomy in approaches. These are driven by contending demands: the desire to provide a simple, natural user experience and the desire to minimize the develop/test effort.

In concrete terms we see this tradeoff in how we think, for example, about the assert command. In the local case, iter8 assert basically returns true/false by evaluating an experiment. The user does not need to know anything about how an experiment is implemented (just how it is expressed).

The most natural extension to Kubernetes would be that kubectl iter8 assert do the same: return true or false without any requirement on the user. In practice, this would mean that it would have to determine whether the experiment was local or running in a cluster. I believe this is also the most natural expectation of a kubenetes user.

However, this approach requires additional work (distinguishing between local and remote) and therefore additional testing.

An alternative approach (described in the comments), is to get the experiment from the cloud and cache it locally. kubectl iter8 assert is exactly the same as iter8 assert). The only additional development/test work is the implementation and testing of get.

This alternative requires additional work/knowledge on the part of the user who must distinguish the difference between local/remote experiments and must have some notion of implementation: I should fetch it and query; if I want an update, I should fetch it again and so on. It is not the “natural” Kubernetes approach.

I’m not sure how to think long term.

Should we consider the first approach as the ideal approach BUT implement the second as a way to make faster progress. (Indeed, if we find that it doesn’t confuse users we might even stick with it). The full implementation can come over time. This would be similar to the way we are thinking about installing iter8: starting with go install; then manual selection and install of a binary; finally a native package installer.

Or should we really think about a model that fetches (and caches) and operates locally. How would we expect this to work in a non-kubernetes environment such as edge? If the first approach is the one that makes sense there maybe it makes sense here too.

[sriumcp]

This is a really practical question.

I agree we didn't fully think through this yet in our discussions... perhaps, we should focus on simplicity at the user level (i.e., have kubectl iter8 assert ic deliver the k8s equivalent of iter8 assert; ic = in cluster). But in order to keep dev effort practical, we narrow down the set of commands that are absolutely required and focus on those, and focus on obtaining end-user feedback in order to understand the high-value features we can build? In other words, embrace the principle of minimum viability (and agility)?

---

If we feel there are more commands that are really useful, perhaps we can mark them as help wanted or good first issues -- so that the community can contribute?

[sriumcp]

Adding to the above ... we will also have numerous opportunities to reach out to end-user communities (including other projects like KServe where Iter8 plays) and solicit feedback on what commands are really useful (and which ones are fluff).

[huang195]

A few comments from today's playback. The difference between local and remote version of kubectl plugin is clear. I see the local version being useful for a developer creating/debugging an experiment interactively, whereas the remote version would be useful for a longer running experiment. However I found the different versions of subcommands, e.g., run vs runr very distracting. Michael mentioned that the remote version is just a wrapper around the code that populates the templates. I wonder if there's a way that we can get away with just supporting the local version, and use unix pipes to achieve remote functionalities. E.g., to run a local experiment, we use

iter8 run

and if user wants to run the experiment remotely, it would be

iter8 run --dry-run | kubectl apply -f

The exact syntax might be wrong, but hopefully you get the point. I think it would be less confusing to users when we make the remote version more explicit. And our code could be a bit more crisp.

Moreover, the whole point of providing a remote version is that it's easier than letting users create their own K8s resource files. But in the CI/CD case, users would still need to create pipeline resources themselves to make use of Iter8. So, if we are supporting local and remote, should we also support tekton, github-action, and others? I feel we need to find the boundary between Iter8 and other tools in the ecosystem, so we have a clear vision of the responsibilities.

When I was looking at Tekton catalog a few weeks ago, tools are packaged in units of container images, and they have example yaml files that show users how to run these images as part of a Tekton task. I wonder if we're trying to roll too much "features" into iter8 command line.

[sriumcp]

During the previous playback on kubectl iter8 ... we went through exactly this thought process, and tried to rewrite the simple getting started tutorial in terms of remote experiments.

That's how we ended up with the commands we saw during the playback

---

In addition run and runr, there is also assert and assertr, gen and genr. Do you have thoughts on what the remote versions for these commands need to look like?

In particular, if you could provide an end-to-end set of commands to run a specific tutorial, it might help clarify further. For example, how should the user write the simplest possible load testing experiment inside Kubernetes, and assert SLOs, and generate a report out of it?

If they created a few experiments, how can day delete a specific one of them?

---

Conceptually, with the kubectl plugin user gets the ability to run and inspect experiments locally, as well as remotely. In terms of the commands, what should the end-to-end experiment interaction look like (not just for firing remote experiments, but also listing, debugging, reporting, asserting and deleting them)? This is design question for the plugin.

[sriumcp]

TBH, we don't really need to care about what the code needs to do under the covers at the moment (it is all code, which means we can make it do anything we want :) ). Given that the design is still under discussion -- a list of commands that demonstrates the user interaction for run, debug, assert, gen, list and delete is the thing that we need to nail first.

[huang195]

Let me make sure I understand these subcommands correctly first.

• run assumes there's an experiment.yaml in the current working directory, and uses it as input to execute each of the tasks described within. At the end of the experiment, it will generate an output.yaml in the current working directory.

• assert parses output.yaml and asserts if various user-provided conditions are met.

• gen parses output.yaml and outputs results in a human readable format.

• runr assumes there's an experiment.yaml in the current working directory, using which, it creates various K8s resources to start a pod that runs iter8 cmdline. experiment.yaml content is instantiated as a K8s secret, and experiment output is eventually written to the same secret.

• assertr parses the K8s secret and asserts if various user-provided conditions are met.

• genr parses the K8s secret and outputs results in a human readable format.

---

If this is somewhat along the line of what is being implemented, to not having to implement a separate assertr, is it possible to do something like

kubectl get secret [name of the secret] -o yaml > result.yaml && iter8 assert -c ...

The [name of the secret] could be obtained by some iter8 cmdline, like iter8 get experiment, so the above command will be something like:

kubectl get secret `iter8 get experiment --short` -o yaml > result.yaml && iter8 assert -c ...

so users don't have to find out what is the corresponding secret name of the last experiment.

Similarly, genr can be omitted the same way. It does expose users to the fact that we're storing experiment content in K8s secrets, but sometimes being more transparent could make the Iter8 blackbox more understandable.

[huang195]

So obviously

iter8 assertr -c

is much cleaner than

kubectl get secret `iter8 get experiment --short` -o yaml > result.yaml && iter8 assert -c ...

But the latter is more explicit what it is doing under the cover, and user doesn't have to think about should I use assert or assertr. I found there's merit in both, but leans a bit towards the latter. Your thoughts?

[sriumcp]

@huang195 If you remember, this is the substance of the discussion earlier in the thread: #1046 (comment)

---

We said we want to run a remote experiment because it might be long running. If we want to assert that SLOs are satisfied, but are willing to provide a timeout of 5m (5 minutes) for the assertion to return true... Is this a use-case you think we need to consider? Or do you think we should omit it? I ask because the local version of assert already supports a timeout now (for a long-running local experiment running in the background). How would a timeout work with the above?

[sriumcp]

Re: making the underlying mechanics of Iter8 more explicit to the enduser, I disagree that this is desirable. IMO, this makes the UX needlessly complex. End users should not have to understand these details.

---

Re: the simplification obtained by removing one of the assert commands, and keeping only assert instead of both assert and assertr. In theory, I agree this is desirable from the end-user perspective; but as described above, it is complicating the experience due to the need to couple it with a get command.

---

IMO, for the end-user, thinking in terms of experiments and operations on experiments is the right abstraction rather than thinking about the underlying experiment resources in kubernetes and operations on those resources. These are low level implementation details of the experiment run. We should avoid forcing this low level detail on the enduser (experiment being the central concept in Iter8).

---

A case in point is Helm. Helm stores release information within a K8s secret. But the Helm user is not forced to understand this low level implementation detail! Instead, Helm users can do helm get manifest, and also helm get values, all of which comes from the underlying secret, but without users ever having to make this connection explicitly.

This is because, releases are first class abstractions defined by Helm (and secrets and configmaps are the implementation details to realize the latter). Similarly, experiment is a first class abstraction defined by Iter8 (and secrets are an implementation detail when experiments run in K8s).

[huang195]

Exposing iter8 underlying mechanics is maybe not the right way to say it, but the helm example is convincing. Although my beef is with more with 2 similar versions of the same subcommand for multiple subcommands, which is not intuitive for users.

[sriumcp]

From the perspective of the generic kubectl flags, the following subcommand structure might be preferable.

### list all remote experiments (k stands for k8s)
kubectl iter8 k get
### run an experiment locally
kubectl iter8 run
### run an experiment remotely
kubectl iter8 k run
### assert a local experiment
kubectl iter8 assert -c ...
### assert a remote experiment
kubectl iter8 k assert  -c ...
### assert a remote experiment with timeout
kubectl iter8 k assert -c completed,nofailure,slos -t 5m
### generate a report from a local experiment
kubectl iter8 gen
### generate a report from a remote experiment
kubectl iter8 k gen
### delete a remote experiment
kubectl iter8 k delete

In other words, every remote command is now a subcommand of kubectl iter8 k, and every local command is now a subcommand of kubectl iter8. The main advantage of this approach is that we can surgically add the global flags recommended for all kubectl plugins (please see here: https://github.com/kubernetes/sample-cli-plugin) to only the remote commands kubectl iter8 k * and leave the local commands without those flags (the global kubectl flags do not make any sense for the local commands).

[sriumcp]

Re: Tekton / GHActions, I am not sure what the above discussion has to do with these two CD pipelines... Tekton task is the standard way to introduce new capabilities into Tekton. GH Action (available from the marketplace) is the standard way to introduce capabilities into GH Actions. We can think through the design of both of these in a separate discussion, as also how Iter8 will be used by Helm users (e.g., as part of a Helm test)? They all seem relatively simple with the new v0.8 architecture.

[huang195]

The following modification might be even more preferable

I like this better, but it still requires users to think about if he wants to do local or remote operation. Feels a bit clunky. Maybe this is inevitable. Will think more about it tomorrow.

[huang195]

I wonder if a config file (.iter8) would make this more intuitive for the user. E.g., in the config file, if it's specified remote: true, then kubectl iter8 [subcommand] does remote operations; otherwise, it does local operations. There's no separate assert and assertr. We should also allow override of this configuration in the command line, e.g., kubectl iter8 --local [subcommand] will do all operations locally even if remote: true is specified in the config file.

[sriumcp]

I wonder if we are missing cheaper things like aliasing?

alias ix="kubectl iter8 k"
# Now, you can do `ix assert`, `ix run`, etc, all of which are remote!

[sriumcp]

still requires users to think about if he wants to do local or remote operation. Feels a bit clunky. Maybe this is inevitable.

There is a difference between local and remote operations, not just in terms of the implementation details but also the UX. The remote operation (at the minimum) involves an experiment with a name, running in a specific namespace. Generic kubectl flags provide a number of ways to make things like default namespaces more robust across environments. We don't want to reinvent these mechanisms, but simply take advantage of them (the 10 lines code for generic flag inheritance that Michael demoed during the playback accomplishes this).

Local experiments do not have any of these generic flags (which means the commands dealing with local experiments will have a reduced flagset).

[huang195]

aliasing helps reduce the amount of typing, but I don't know if it will make iter8 k vs iter8 more clear. Mistyping an alias could cause some unexpected behavior for the user.

I feel we might need some fresh eyes on the cmdline design. Would it be possible to ask for 30 minutes from Nick Mitchell to do a quick demo and see if he has some recommendations? And see if someone with UI experience and some familiarity with Iter8 has an opinion one way or another?

The 3 options we have discussed so far are:

1. Use different subcommands for remote and local operations: e.g.,

kubectl iter8 run
kubectl iter8 runr

2. Use k8s subcommand structure:

kubectl iter8 run
kubectl iter8 k run

3. Use a config file (.iter8) w/ cmdline override:

kubectl iter8 run
kubectl iter8 run --remote

I feel options 1 and 2 are pretty much the same. User has to make a conscious decision if he wants to do a local or a remote operation, e.g., runr and k run aren't very different. If user forget to type k when he intended to run kubectl iter8 k run, it will be confusing.

In option 3, user pretty much always type kubectl iter8 run, and if he wants to switch from local to remote or vice versa, he could either change the global config in .iter8 or add --remote flag, and both are very explicit operations so the likelihood of making an inadvertent mistake is diminished (i.e., clarity through explicity) . However, this particular option assumes switching from local to remote or vice versa doesn't happen at a high frequency, or otherwise, this could be cumbersome. But I don't know if this assumption is true or not.

[sriumcp]

Could we please take 1) off the table? We all seem to be in agreement that we don't want 1).

[sriumcp]

Re: 3), the flagset for local and remote executions are different (for run, assert, etc.). This is the main reason why I prefer 2 over 3. It is easier both to implement and expose the flagsets cleanly with 2. With 3, you are going have a number of generic flags like --namespace; you have to explicitly mention in the documentation that these flags are irrelevant when used for local experiments. And you also need to explicitly make this distinction in the implementation level.

---

Also, 2 doesn't force the user to manipulate yet another artifact, namely the .iter8 config file. Seems like an overkill to me.

[kalantar]

A few thoughts.

In trying to determine the commands, I think the user should not be aware of underlying implementation. Today, we may implement assertr by reading the secret writing to a file and using the local assert to evaluate. But I would expect that in time we will rewrite assert to work on a yaml string. This would be available to assertr and to assert each of which will get the string differently. We should avoid forcing the user to think about implementation while giving us flexibility to make it be whatever makes sense.

It is hard to auto-magically determine if a command should be local or remote. In principle, one might first look to see if there is a k8s context, then figure out if there is a remote experiment otherwise assume local. However, this is heuristic and may not match what the user intends.

I think the user does think about local vs remote. I think it is ok to capture this in the commands. Remote execution at least has 2 additional requirements: list and delete.

I also don't like assert and assertr. I would prefer the same command. However, we need to be able to figure out exactly how to determine which is which.

We should not assume that since the current implementation adds all the k8s specific options to every command that this is a requirement. There is a lot I don't know here yet. For example, I thought i was going to get the commands displayed by group (local/remote) and I don't. Let's not make the assumption that this can't be fixed.

[huang195]

Yep, we can certainly discuss in parallel while the implementation is going forward, and fix things as we go.

So taking a step back, I remember hearing that the reasons we want to support local were: i) just simpler for developers to do some quick and dirty testing, ii) users can create and work with an experiment w/o an actual k8s cluster. Maybe there are other reasons that I missed, but I wonder if we can get out of this mess by not supporting local operations for users.

From the few things I've done in Iter8 in the past (GitOps, Iter8-trend, e2e testing), there's always a cluster that I'm working with, e.g., Minikube on my local laptop, Openshift on IBM cloud or Redhat Developer Sandbox, Minikube running on a VM in Github. From what I can tell from our playbacks, in local mode, we don't need to stand up an app for built-in metrics to work, which is great. However, standing up a Minikube with an dummy webapp takes a minute, and you can just let it run in the background. Is there something else that I'm overlooking?

[sriumcp]

Why are local operations useful?

Immediate benefits

1. Simpler to get started with Iter8 -- you don't need to put anything in your cluster to understand various Iter8 experiments
2. Simpler for users to do some quick and dirty testing
3. Simpler for users to get SLO validation insights without inserting anything into the cluster and subsequently cleaning up
4. Simpler for users to get A/B(/n) testing insights without inserting anything into the cluster and subsequently cleaning up
5. Simpler for users to debug experiments which are still in the process of being configured -- running locally produces logs locally
6. Simpler to run an experiment as a step in Tekton, instead of launching it into K8s and cleaning up later
7. Simpler to run an experiment in GH Action step, instead of launching it into K8s and cleaning up later

Long term benefits

1. Possible to use Iter8 for non-k8s applications like those on a VM
2. Possible to use Iter8 for non-k8s applications like those on an edge device

---

For me, the concept is really simple. There is something called an Iter8 experiment. You can run it locally, or in a K8s cluster.

---

To run it locally, do iter8 run.

To run it in a K8s cluster, do iter8 k run.

When running in a cluster, iter8 k will support all the generic flags that kubectl or other tools that interact with a cluster are expected to support.

---

I am not sure I understood where the mess/difficulty is in the above story :)

[huang195]

That's a great list of benefits. You sold me on the long term benefits as we do need to consider platforms other than K8s-based ones. Not so sure about the short term benefits, but seems to be a moot point now. OK, so we do need both local and remote.

[huang195]

So kubectl plugin is basically an executable file named in a certain way placed in the execution path. I wonder if we can make local and remote distinction by determining if iter8 is invoked as a subcommand of kubectl or by itself. So if user runs

iter8 assert

it's a local call. If user runs

kubectl iter8 assert

it's a remote call.

This means we need to be able to tell within the iter8 code how it was invoked. If we can differentiate who the invoker is, we can make different set of flags (e.g., --namespace) available depending on if iter8 was called or kubectl iter8 was called.

[sriumcp]

The user needs to create the soft link. I am still unclear as to why? If the user can use iter8 and kubectl iter8 after soft linking, why can't she use iter8 and iter8 k without the need to soft link?

[huang195]

The iter8ctl install looks like this today:

GOBIN=/usr/local/bin go install github.com/iter8-tools/etc3/iter8ctl@latest

It will still be a one-liner with soft link

GOBIN=/usr/local/bin go install github.com/iter8-tools/etc3/iter8ctl@latest && ln -s ...

I'm using v0.7 as an example, so translate this to v0.8, but you get the point. It's still one liner and user doesn't need to do extra stuff.

---

If the user can use iter8 and kubectl iter8 after soft linking, why can't she use iter8 and iter8 k without the need to soft link?

When you say iter8 and iter8 k, you mean kubectl iter8 and kubectl iter8 k, right? The point I'm trying to get across all along is we're trying to differentiate local and remote operations clearly to users, so i) the different flavors of the same subcommand are clear to them, and ii) they don't inadvertently make a mistake.

So, when a user runs iter8 directly, it means he's running local operations only. When user appends kubectl in front of iter8, it means he wants to interact with a cluster, so these are remote commands. The use of kubectl implies remote operations, whereas the lack of it implies local operations.

[huang195]

In other words, the difference between iter8 run and kubectl iter8 run is greater than kubectl iter8 run and kubectl iter8 k run, thus, the former is more explicit on intent.

[huang195]

It also allows us to retain the long-term benefits you pointed out earlier, so this would still be applicable in a non-k8s environment,

[sriumcp]

We actually won't have kubectl iter8 and kubectl iter8 k to begin with. We will simply have iter8 and iter8 k. Over time, we will also make it as a kubectl plugin, so iter8 and iter8 k can be replaced by kubectl iter8 and kubectl iter8 k by users who download Iter8 as a kubectl plugin. This is one of items we discussed in the (internal) Iter8 meeting yesterday.

Reason: The only real advantage of a kubectl plugin from an Iter8 user perspective is distribution (via krew manifests); we don't want to miss out on this popular distribution channel for making tools available; but we also don't need to start with this either (Iter8 v0.8 can also be installed using binaries, Go source, brew install, as a kubectl/krew plugin, and basically any other way in which Go-built executables can be installed in a cross-platform way).

---

Re: the points above ... at this point, it seems like we're willing to go to great lengths only to avoid iter8 k and I am still wondering why.

Anyway, let us take the points under discussion...

i) the different flavors of the same subcommand are clear to them: of course, iter8 assert and iter8 k assert are different flavors of assert and the documentation will make the difference obvious. The very fact that you have iter8 assert and iter8 k assert should indicate that the two asserts do somewhat different things (because of k) but also somewhat similar things (because of assert)?

ii) kubectl itself is an example of a tool that supports both remote and local operations. They do not ask users to symlink, etc. for local vs remote operations. If you need to kustomize a resource, you can do so using kubectl -k resource/folder. This is a local operation. They don't ask the users to do localctl -k resource/folder. Of course, they want to keep endusers' life simple, which is why they dispensed with symlinking and other needless complexities. Besides it is unclear to me if symlinking instructions can even be provided in a cross-platform way.

iii) they don't inadvertently make a mistake: if the user can type iter8 instead of iter8 k, of course, they can inadvertently type iter8 instead of kubectl iter8. Where is the guarantee that this will not happen?

iv) the difference between ... more explicit on intent:: we can replace the word k with the word k8s. I think shorter is better, but I am open to striking a balance between short and sweet on the one hand, and long and explicit on the other.

---

Knative provides a simple CLI called kn. They seem to have purposely kept everything short and sweet for end-user usability purposes. But again, we can strike a balance between brevity and explicitness.

---

Once again, iter8 k can be aliased as remoteiter8 if the user feels this is more explicit. But this is not something we need to force on every user. We can always point this as an option in the documentation / "Iter8 cheat sheet".

[kalantar]

I think .iter8 might be a good approach for some default templates to be used for the gen command. An environment like CE would then overwrite them.

[kalantar]

kubectl already suggests that it is for remote use. It is the kubernetes CLI. What if we use --local (-l) for local commands. It makes sense to explicitly indicate local use for a kubectl command.

[sriumcp]

Incidentally, @huang195 started this thread with --remote option which I must admit does look attractive to me, except for the above two issues.

[sriumcp]

I do wonder if iter8 and iter8 k can be starting point for us... we approach a few communities, try a few growth experiments, and based on usage we're observing (and the questions we get in Git/Slack) make a call on how the CLI must evolve.

---

As opposed to answering every possible question now with finality.

[huang195]

@kalantar I'm proposing to get rid of kubectl prefix for local commands. This removes the need to use a --local flag. So, instead of

kubectl iter8 run; <-- remote
kubectl iter8 run --local; <-- local

we do

kubectl iter8 run; <-- remote
iter8 run; <-- local

Isn't kubectl plugin just an executable? Can't we run it as a standalone binary in addition to being called by kubectl?

---

So, we are polluting a single command assert, with a mixture of flags...

@sriumcp This won't happen if iter8 can figure out if it was called as a kubectl subcommand or not. This logic can gate the flags you are talking about, i.e., iter8 --help will output different set of flags than kubectl iter8 --help.

[kalantar]

I'm proposing to get rid of kubectl prefix for local commands. This removes the need to use a --local flag.

I saw this using alias. I am not sure alias gives the ability to distinguish. I think two binaries would be necessary. I am not entirely opposed to this. In a sense, there is an argument that a kubectl plugin should be interaction with kubernetes. And that if you want local use a local command. And if we were interacting with something else, we should do something else.

[huang195]

I think two binaries would be necessary

I don't think this is necessary. Either alias or softlink should work. There is something at runtime that tells who is the parent process. If its name contains kubectl, we know it was invoked as a kubectl subcommand. Otherwise, we can assume it was invoked by a shell process.

[kalantar]

Is there any reason to believe running experiments inside the cluster is going to be the more prevalent mode of experimentation, compared to local? In other words, why not have --remote flag instead of --local flag?

No. I suggest this because we are writing a kubectl plugin. By its very nature, it implies it is for use in a cluster (ie, remote) environment. By the same token, if we offerted iter8ctl, I would suggest we lean it towards being always local. I think the natural assumption of the user, is that it is about the cluster. We offer an enhancement with the --local flag.

[sriumcp]

@kalantar One of the things we discussed in yesterday's meeting (which I think you agreed to) was that we will make this a kubectl plugin for distribution purposes.

But initially, it doesn't have to be kubectl plugin at all. We can distribute as via binary, brew, apt-get, etc.

---

Whether or not experiments will be run by users primarily locally vs. inside the cluster will be defined by user needs (which we are yet to assess). It will not be based on whether we distribute Iter8 as a kubectl plugin or a binary.

Even when it is available as a kubectl plugin, it is unclear why the experiment needs to predominantly run inside a cluster (instead of a short experiment running locally and interacting with things inside the cluster).

---

Besides, there are mixed mode operations. Tekton runs inside a cluster, and Iter8 can be invoked locally as a step in a Tekton task!

---

I don't think we should make a call on remote vs local without actually getting user data. At this point, we simply don't know enough about this.

[kalantar]

The k8s specific flags will be meaningless when used alongside --local. So, we are polluting a single command assert, with a mixture of flags that won't make sense together, and the documentation needs to call this out. This is going to make the documentation a bit tedious for almost every single Iter8 command.

While I agree they are meaningless, I also think that it is kubectl so the options should be there. It isn't hard to say that when --local is selected, cluster specific options are ignored. BTW, I have figured out how to not display them when using -h.

[sriumcp]

There is one reason to eventually make this a kubectl plugin, which is greater visibility and ease of download for users attuned to kubectl plugins.

---

Functionally, the binary is absolutely no different from the plugin! So, all of our collective wisdom on local/remote operations, inherited flagsets, etc. apply equally well to both.

---

kubectl plugin doesn't get us anything for free actually... except a new distribution channel and 8 extra characters prepended to every Iter8 command ;) If you don't write a krew manifest for the plugin, you lose the former and only get the 8 extra characters!

[huang195]

In that case, I think it makes sense to go w/ iter8 and iter8 k from the start. Then adding this as kubectl plugin and maybe even deprecating iter8 k in favor of kubectl plugin will depend on our own experience and user feedbacks.

[kalantar]

I guess I am ok with this; I can refactor what I have to do this.

When/if it comes time to consider it as a kubectl plugin, I think we will need to revisit. It may technically be possible to accomplish just by the name. However, I think that as a plugin it should "make sense".

[sriumcp]

@kalantar Perhaps you can do the refactoring after we discuss the base CLI (hopefully soon; a day or two) so as to avoid wasted effort. It might work better if you can ignore flags/subcommand structure for now and keep going with the remote ops which is the core business logic anyway... I am still working through the testing and documentation pieces and the code base for the base is still not there yet (in a presentable form).

[kalantar]

The remote ops are all implemented in some way -- mostly reliant on os.exec to call the base CLI. This is less than ideal; I will look at fixing this. But I am concerned that it will require duplicating code that would better be done just once (in the base).