Skip to content

Commit

Permalink
feat: add support for authenticated git clone
Browse files Browse the repository at this point in the history 
Added support for authenticated git cloning using `gitToken` and `gitTokenKey`
parameters. Updated documentation to reflect the new parameters and their
usage. Modified resolver functions to handle authentication when cloning
repositories. Added tests to verify the functionality of authenticated git
cloning.

The differences between the two modes are:

- The `git clone` method support anonymous cloning and authenticated cloning.
- Depending of the Git provider `git clone` has a lower rate limit (if none)
  than the authenticated API.
- The authenticated API supports private repositories and fetches only the file
  at the specified path rather than doing a full clone.

Signed-off-by: Chmouel Boudjnah <[email protected]>
  • Loading branch information
@chmouel
chmouel committed Jan 29, 2025
1 parent 0d698a7 commit a99385a
Show file tree
Hide file tree
Showing 7 changed files with 204 additions and 68 deletions.
82 changes: 57 additions & 25 deletions docs/git-resolver.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,17 +13,19 @@ This Resolver responds to type `git`.

## Parameters

| Param Name | Description | Example Value |
|--------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------|
| `url` | URL of the repo to fetch and clone anonymously. Either `url`, or `repo` (with `org`) must be specified, but not both. | `https://github.com/tektoncd/catalog.git` |
| `repo` | The repository to find the resource in. Either `url`, or `repo` (with `org`) must be specified, but not both. | `pipeline`, `test-infra` |
| `org` | The organization to find the repository in. Default can be set in [configuration](#configuration). | `tektoncd`, `kubernetes` |
| `token` | An optional secret name in the `PipelineRun` namespace to fetch the token from. Defaults to empty, meaning it will try to use the configuration from the global configmap. | `secret-name`, (empty) |
| `tokenKey` | An optional key in the token secret name in the `PipelineRun` namespace to fetch the token from. Defaults to `token`. | `token` |
| `revision` | Git revision to checkout a file from. This can be commit SHA, branch or tag. | `aeb957601cf41c012be462827053a21a420befca` `main` `v0.38.2` |
| `pathInRepo` | Where to find the file in the repo. | `task/golang-build/0.3/golang-build.yaml` |
| `serverURL` | An optional server URL (that includes the https:// prefix) to connect for API operations | `https:/github.mycompany.com` |
| `scmType` | An optional SCM type to use for API operations | `github`, `gitlab`, `gitea` |
| Param Name | Description | Example Value |
|---------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------|
| `url` | URL of the repo to fetch and clone anonymously. Either `url`, or `repo` (with `org`) must be specified, but not both. | `https://github.com/tektoncd/catalog.git` |
| `repo` | The repository to find the resource in. Either `url`, or `repo` (with `org`) must be specified, but not both. | `pipeline`, `test-infra` |
| `org` | The organization to find the repository in. Default can be set in [configuration](#configuration). | `tektoncd`, `kubernetes` |
| `token` | An optional secret name in the `PipelineRun` namespace to fetch the token from. Defaults to empty, meaning it will try to use the configuration from the global configmap. | `secret-name`, (empty) |
| `tokenKey` | An optional key in the token secret name in the `PipelineRun` namespace to fetch the token from. Defaults to `token`. | `token` |
| `gitToken` | An optional secret name in the `PipelineRun` namespace to fetch the token from when doing opration with the `git clone`. When empty it will use anonymous cloning. | `secret-gitauth-token` |
| `gitTokenKey` | An optional key in the token secret name in the `PipelineRun` namespace to fetch the token from when using the `git clone`. Defaults to `token`. | `token` |
| `revision` | Git revision to checkout a file from. This can be commit SHA, branch or tag. | `aeb957601cf41c012be462827053a21a420befca` `main` `v0.38.2` |
| `pathInRepo` | Where to find the file in the repo. | `task/golang-build/0.3/golang-build.yaml` |
| `serverURL` | An optional server URL (that includes the https:// prefix) to connect for API operations | `https:/github.mycompany.com` |
| `scmType` | An optional SCM type to use for API operations | `github`, `gitlab`, `gitea` |

## Requirements

Expand Down Expand Up @@ -55,11 +57,25 @@ for the name, namespace and defaults that the resolver ships with.

## Usage

The `git` resolver has two modes: cloning a repository anonymously, or fetching individual files via an SCM provider's API using an API token.
The `git` resolver has two modes: cloning a repository with `git clone` (with
the `go-git` library), or fetching individual files via an SCM provider's API
using an API token.

### Anonymous Cloning
The differences between the two modes are:

Anonymous cloning is supported only for public repositories. This mode clones the full git repo.
- The `git clone` method support anonymous cloning and authenticated cloning.
- When it comes to throughput, `git clone` is a more efficient option because
it is not subject to the same rate limits as API calls. This is a key
difference between the two modes: `git clone` supports both anonymous and
authenticated cloning, and depending on the Git provider, it often has a
higher rate limit compared to authenticated API calls.
- The `git clone` method is inefficient for larger repositories, as it clones the
entire repository in memory, whereas the authenticated API fetches only the
files at the specified path.

### Git Clone with git clone

Git clone with `git clone` is supported for anonymous and authenticated cloning.

#### Task Resolution

Expand All @@ -78,6 +94,11 @@ spec:
value: main
- name: pathInRepo
value: task/git-clone/0.6/git-clone.yaml
# Uncomment the following lines to use a secret with a token
# - name: gitToken
# value: "secret-with-token"
# - name: gitTokenKey (optional, defaults to "token")
# value: "token"
```

#### Pipeline resolution
Expand All @@ -97,6 +118,11 @@ spec:
value: main
- name: pathInRepo
value: pipeline/simple/0.1/simple.yaml
# Uncomment the following lines to use a secret with a token
# - name: gitToken
# value: "secret-with-token"
# - name: gitTokenKey (optional, defaults to "token")
# value: "token"
params:
- name: name
value: Ranni
Expand All @@ -108,11 +134,12 @@ The authenticated API supports private repositories, and fetches only the file a
When using the authenticated API, [providers with implementations in `go-scm`](https://github.com/jenkins-x/go-scm/tree/main/scm/driver) can be used.
Note that not all `go-scm` implementations have been tested with the `git` resolver, but it is known to work with:
* github.com and GitHub Enterprise
* gitlab.com and self-hosted Gitlab
* Gitea
* BitBucket Server
* BitBucket Cloud

- github.com and GitHub Enterprise
- gitlab.com and self-hosted Gitlab
- Gitea
- BitBucket Server
- BitBucket Cloud

#### Task Resolution

Expand Down Expand Up @@ -193,11 +220,11 @@ spec:

### Specifying Configuration for Multiple Git Providers

It is possible to specify configurations for multiple providers and even multiple configurations for same provider to use in
It is possible to specify configurations for multiple providers and even multiple configurations for same provider to use in
different tekton resources. Firstly, details need to be added in configmap with the unique identifier key prefix.
To use them in tekton resources, pass the unique key mentioned in configmap as an extra param to resolver with key
`configKey` and value will be the unique key. If no `configKey` param is passed, `default` will be used. Default
configuration to be used for git resolver can be specified in configmap by either mentioning no unique identifier or
To use them in tekton resources, pass the unique key mentioned in configmap as an extra param to resolver with key
`configKey` and value will be the unique key. If no `configKey` param is passed, `default` will be used. Default
configuration to be used for git resolver can be specified in configmap by either mentioning no unique identifier or
using identifier `default`

**Note**: `configKey` should not contain `.` while specifying configurations in configmap
Expand Down Expand Up @@ -253,7 +280,7 @@ data:

#### Task Resolution

A specific configurations from the configMap can be selected by passing the parameter `configKey` with the value
A specific configurations from the configMap can be selected by passing the parameter `configKey` with the value
matching one of the configuration keys used in the configMap.

```yaml
Expand Down Expand Up @@ -304,17 +331,21 @@ spec:
```

## `ResolutionRequest` Status

`ResolutionRequest.Status.RefSource` field captures the source where the remote resource came from. It includes the 3 subfields: `url`, `digest` and `entrypoint`.

- `url`
- If users choose to use anonymous cloning, the url is just user-provided value for the `url` param in the [SPDX download format](https://spdx.github.io/spdx-spec/package-information/#77-package-download-location-field).
- If scm api is used, it would be the clone URL of the repo fetched from scm repository service in the [SPDX download format](https://spdx.github.io/spdx-spec/package-information/#77-package-download-location-field).
- `digest`
- The algorithm name is fixed "sha1", but subject to be changed to "sha256" once Git eventually uses SHA256 at some point later. See https://git-scm.com/docs/hash-function-transition for more details.
- The algorithm name is fixed "sha1", but subject to be changed to "sha256" once Git eventually uses SHA256 at some point later. See <https://git-scm.com/docs/hash-function-transition> for more details.
- The value is the actual commit sha at the moment of resolving the resource even if a user provides a tag/branch name for the param `revision`.
- `entrypoint`: the user-provided value for the `path` param.

Example:

- Pipeline Resolution

```yaml
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
Expand All @@ -333,6 +364,7 @@ spec:
```

- `ResolutionRequest`

```yaml
apiVersion: resolution.tekton.dev/v1alpha1
kind: ResolutionRequest
Expand Down
12 changes: 10 additions & 2 deletions pkg/remoteresolution/resolver/git/resolver.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -119,11 +119,19 @@ func (r *Resolver) Resolve(ctx context.Context, req *v1beta1.ResolutionRequestSp
return nil, err
}

g := &git.GitResolver{
KubeClient: r.kubeClient,
Logger: r.logger,
Cache: r.cache,
TTL: r.ttl,
Params: params,
}

if params[git.UrlParam] != "" {
return git.ResolveAnonymousGit(ctx, params)
return g.ResolveGitClone(ctx)
}

return git.ResolveAPIGit(ctx, params, r.kubeClient, r.logger, r.cache, r.ttl, r.clientFunc)
return g.ResolveAPIGit(ctx, r.clientFunc)
}
// Remove this error once resolution of url has been implemented.
return nil, errors.New("the Resolve method has not been implemented.")
Expand Down
1 change: 1 addition & 0 deletions pkg/resolution/resolver/git/config.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ type ScmConfig struct {
Org string `json:"default-org"`
ServerURL string `json:"server-url"`
SCMType string `json:"scm-type"`
GitToken string `json:"git-token"`
APISecretName string `json:"api-token-secret-name"`
APISecretKey string `json:"api-token-secret-key"`
APISecretNamespace string `json:"api-token-secret-namespace"`
Expand Down
4 changes: 4 additions & 0 deletions pkg/resolution/resolver/git/params.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,10 @@ const (
TokenParam string = "token"
// TokenKeyParam is an optional reference to a key in the TokenParam secret for SCM API authentication
TokenKeyParam string = "tokenKey"
// GitTokenParam is an optional reference to a secret name when using go-git for git authentication
GitTokenParam string = "gitToken"
// GitTokenParam is an optional reference to a secret name when using go-git for git authentication
GitTokenKeyParam string = "gitTokenKey"
// DefaultTokenKeyParam is the default key in the TokenParam secret for SCM API authentication
DefaultTokenKeyParam string = "token"
// scmTypeParam is an optional string overriding the scm-type configuration (ie: github, gitea, gitlab etc..)
Expand Down
Loading

0 comments on commit a99385a

Please sign in to comment.