Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Provider implementation roadmap #3

Open
sqs opened this issue Dec 24, 2023 · 0 comments
Open

Provider implementation roadmap #3

sqs opened this issue Dec 24, 2023 · 0 comments

Comments

@sqs
Copy link
Member

sqs commented Dec 24, 2023 •
edited
Loading

Ways to make it easier to implement a provider:

Helpers to add to @opencodegraph/provider

  • createFilePositionCalculator?
  • excerpt/truncate?
thenamankumar pushed a commit that referenced this issue May 17, 2024
@keegancsmith
cli: refactor support more commands (#48)
b265fba 
The CLI is great for quickly testing of a provider. This setups up the
CLI to be able to do more than just `items` as a subcommand.

Note: This commit was extracted from a much larger change I have going
on for testing on top of #41. I'll send out the other changes once that
lands.

### Test Plan

Ran it with my sourcegraph-search provider. See the shell session below.

<details>

```shellsession
$ cd provider/sourcegraph-search/

$ pnpm bundle

> @openctx/[email protected] bundle /Users/keegan/src/github.com/sourcegraph/openctx/provider/sourcegraph-search
> esbuild --bundle --format=esm --outfile=dist/bundle.js index.ts


  dist/bundle.js  5.4kb

⚡ Done in 2ms

$ cd ../../bin

$ cat config.json 
{
  "enable": true,
  "debug": true,
  "providers": {
    "file:///Users/keegan/src/github.com/sourcegraph/openctx/provider/sourcegraph-search/dist/bundle.js": true
  }
}

$ OPENCTX_CONFIG=$(cat config.json) pnpm openctx items 'TODO(keegancsmith)'

> @openctx/cli@ openctx /Users/keegan/src/github.com/sourcegraph/openctx/bin
> esbuild --log-level=error --platform=node --bundle --outdir=dist --format=esm --out-extension:.js=.mjs cli.mts && node --no-warnings=ExperimentalWarning --es-module-specifier-resolution=node --loader ts-node/esm/transpile-only dist/cli.mjs "items" "TODO(keegancsmith)"

#1 github.com/sourcegraph/sourcegraph internal/httpcli/external.go:115-117
   - hover.text: From Sourcegraph query TODO(keegancsmith)
   - ai.content: (138 characters)
#2 github.com/sourcegraph/sourcegraph internal/ratelimit/monitor.go:88-90
   - hover.text: From Sourcegraph query TODO(keegancsmith)
   - ai.content: (105 characters)
#3 github.com/sourcegraph/cody vscode/src/services/GuardrailsProvider.ts:3-5
   - hover.text: From Sourcegraph query TODO(keegancsmith)
   - ai.content: (182 characters)
#4 github.com/sourcegraph/sourcegraph internal/updatecheck/client.go:461-463
   - hover.text: From Sourcegraph query TODO(keegancsmith)
   - ai.content: (92 characters)
#5 github.com/sourcegraph/sourcegraph internal/search/result/symbol.go:121-123
   - hover.text: From Sourcegraph query TODO(keegancsmith)
   - ai.content: (181 characters)
#6 github.com/sourcegraph/sourcegraph internal/search/symbol/symbol.go:45-47
   - hover.text: From Sourcegraph query TODO(keegancsmith)
   - ai.content: (335 characters)
#7 github.com/sourcegraph/sourcegraph cmd/frontend/internal/guardrails/dotcom/dotcom.go:31-33
   - hover.text: From Sourcegraph query TODO(keegancsmith)
   - ai.content: (200 characters)
#8 github.com/sourcegraph/sourcegraph internal/search/job/jobutil/filter_file_contains.go:224-226
   - hover.text: From Sourcegraph query TODO(keegancsmith)
   - ai.content: (181 characters)
#9 github.com/sourcegraph/sourcegraph cmd/frontend/internal/guardrails/attribution/attribution.go:137-139
   - hover.text: From Sourcegraph query TODO(keegancsmith)
   - ai.content: (149 characters)
#10 github.com/sourcegraph/zoekt api.go:814-816
   - hover.text: From Sourcegraph query TODO(keegancsmith)
   - ai.content: (153 characters)

$ OUTPUT_JSON=1 OPENCTX_CONFIG=$(cat config.json) pnpm openctx items 'TODO(keegancsmith)'

> @openctx/cli@ openctx /Users/keegan/src/github.com/sourcegraph/openctx/bin
> esbuild --log-level=error --platform=node --bundle --outdir=dist --format=esm --out-extension:.js=.mjs cli.mts && node --no-warnings=ExperimentalWarning --es-module-specifier-resolution=node --loader ts-node/esm/transpile-only dist/cli.mjs "items" "TODO(keegancsmith)"

[
  {
    "title": "github.com/sourcegraph/sourcegraph internal/httpcli/external.go:115-117",
    "uri": "https://sourcegraph.com/github.com/sourcegraph/sourcegraph/-/blob/internal/httpcli/external.go?L115-117",
    "ui": {
      "hover": {
        "text": "From Sourcegraph query TODO(keegancsmith)"
      }
    },
    "ai": {
      "content": "\t\t}\n\t\t// TODO(keegancsmith) ensure we validate these certs somewhere\n\t\teffective.TLSClientConfig.RootCAs.AppendCertsFromPEM([]byte(cert))\n"
    },
    "providerUri": "file:///Users/keegan/src/github.com/sourcegraph/openctx/provider/sourcegraph-search/dist/bundle.js"
  },
  {
    "title": "github.com/sourcegraph/sourcegraph internal/ratelimit/monitor.go:88-90",
    "uri": "https://sourcegraph.com/github.com/sourcegraph/sourcegraph/-/blob/internal/ratelimit/monitor.go?L88-90",
    "ui": {
      "hover": {
        "text": "From Sourcegraph query TODO(keegancsmith)"
      }
    },
    "ai": {
      "content": "\n// TODO(keegancsmith) Update RecommendedWaitForBackgroundOp to work with other\n// rate limits. Such as:\n"
    },
    "providerUri": "file:///Users/keegan/src/github.com/sourcegraph/openctx/provider/sourcegraph-search/dist/bundle.js"
  },
  {
    "title": "github.com/sourcegraph/cody vscode/src/services/GuardrailsProvider.ts:3-5",
    "uri": "https://sourcegraph.com/github.com/sourcegraph/cody/-/blob/vscode/src/services/GuardrailsProvider.ts?L3-5",
    "ui": {
      "hover": {
        "text": "From Sourcegraph query TODO(keegancsmith)"
      }
    },
    "ai": {
      "content": "export class GuardrailsProvider {\n    // TODO(keegancsmith) this provider should create the client since the guardrails client requires a dotcom graphql connection.\n    constructor(\n"
    },
    "providerUri": "file:///Users/keegan/src/github.com/sourcegraph/openctx/provider/sourcegraph-search/dist/bundle.js"
  },
  {
    "title": "github.com/sourcegraph/sourcegraph internal/updatecheck/client.go:461-463",
    "uri": "https://sourcegraph.com/github.com/sourcegraph/sourcegraph/-/blob/internal/updatecheck/client.go?L461-463",
    "ui": {
      "hover": {
        "text": "From Sourcegraph query TODO(keegancsmith)"
      }
    },
    "ai": {
      "content": "\n\t// TODO(keegancsmith) should be using pool.Get and closing conn?\n\tconn, err := dialFunc()\n"
    },
    "providerUri": "file:///Users/keegan/src/github.com/sourcegraph/openctx/provider/sourcegraph-search/dist/bundle.js"
  },
  {
    "title": "github.com/sourcegraph/sourcegraph internal/search/result/symbol.go:121-123",
    "uri": "https://sourcegraph.com/github.com/sourcegraph/sourcegraph/-/blob/internal/search/result/symbol.go?L121-123",
    "ui": {
      "hover": {
        "text": "From Sourcegraph query TODO(keegancsmith)"
      }
    },
    "ai": {
      "content": "func (s Symbol) Range() lsp.Range {\n\t// TODO(keegancsmith) For results from zoekt s.Character is not the start\n\t// of the symbol, but the start of the match. So doing s.Character +\n"
    },
    "providerUri": "file:///Users/keegan/src/github.com/sourcegraph/openctx/provider/sourcegraph-search/dist/bundle.js"
  },
  {
    "title": "github.com/sourcegraph/sourcegraph internal/search/symbol/symbol.go:45-47",
    "uri": "https://sourcegraph.com/github.com/sourcegraph/sourcegraph/-/blob/internal/search/symbol/symbol.go?L45-47",
    "ui": {
      "hover": {
        "text": "From Sourcegraph query TODO(keegancsmith)"
      }
    },
    "ai": {
      "content": "func (s *ZoektSymbolsClient) Compute(ctx context.Context, repoName types.MinimalRepo, commitID api.CommitID, inputRev *string, query *string, first *int32, includePatterns *[]string) (res []*result.SymbolMatch, err error) {\n\t// TODO(keegancsmith) we should be able to use indexedSearchRequest here\n\t// and remove indexedSymbolsBranch.\n"
    },
    "providerUri": "file:///Users/keegan/src/github.com/sourcegraph/openctx/provider/sourcegraph-search/dist/bundle.js"
  },
  {
    "title": "github.com/sourcegraph/sourcegraph cmd/frontend/internal/guardrails/dotcom/dotcom.go:31-33",
    "uri": "https://sourcegraph.com/github.com/sourcegraph/sourcegraph/-/blob/cmd/frontend/internal/guardrails/dotcom/dotcom.go?L31-33",
    "ui": {
      "hover": {
        "text": "From Sourcegraph query TODO(keegancsmith)"
      }
    },
    "ai": {
      "content": "func NewClient(externalHTTPClient httpcli.Doer, endpoint, token string) Client {\n\t// TODO(keegancsmith) we allow unauthed requests for now but should\n\t// require it when promoting guardrails for use.\n"
    },
    "providerUri": "file:///Users/keegan/src/github.com/sourcegraph/openctx/provider/sourcegraph-search/dist/bundle.js"
  },
  {
    "title": "github.com/sourcegraph/sourcegraph internal/search/job/jobutil/filter_file_contains.go:224-226",
    "uri": "https://sourcegraph.com/github.com/sourcegraph/sourcegraph/-/blob/internal/search/job/jobutil/filter_file_contains.go?L224-226",
    "ui": {
      "hover": {
        "text": "From Sourcegraph query TODO(keegancsmith)"
      }
    },
    "ai": {
      "content": "\tslices.SortFunc(cm.DiffPreview.MatchedRanges, func(a, b result.Range) int {\n\t\t// TODO(keegancsmith) I changed this from b.End to b.Start since that\n\t\t// matches the comment above.\n"
    },
    "providerUri": "file:///Users/keegan/src/github.com/sourcegraph/openctx/provider/sourcegraph-search/dist/bundle.js"
  },
  {
    "title": "github.com/sourcegraph/sourcegraph cmd/frontend/internal/guardrails/attribution/attribution.go:137-139",
    "uri": "https://sourcegraph.com/github.com/sourcegraph/sourcegraph/-/blob/cmd/frontend/internal/guardrails/attribution/attribution.go?L137-139",
    "ui": {
      "hover": {
        "text": "From Sourcegraph query TODO(keegancsmith)"
      }
    },
    "ai": {
      "content": "\n\t// TODO(keegancsmith) Reading the SearchClient code it seems to miss out\n\t// on some of the observability that we instead add in at a later stage.\n"
    },
    "providerUri": "file:///Users/keegan/src/github.com/sourcegraph/openctx/provider/sourcegraph-search/dist/bundle.js"
  },
  {
    "title": "github.com/sourcegraph/zoekt api.go:814-816",
    "uri": "https://sourcegraph.com/github.com/sourcegraph/zoekt/-/blob/api.go?L814-816",
    "ui": {
      "hover": {
        "text": "From Sourcegraph query TODO(keegancsmith)"
      }
    },
    "ai": {
      "content": "\t//\n\t// TODO(keegancsmith) audit updates to IndexTime and document how and when\n\t// it changes. Concerned about things like metadata updates or compound\n"
    },
    "providerUri": "file:///Users/keegan/src/github.com/sourcegraph/openctx/provider/sourcegraph-search/dist/bundle.js"
  }
]
```

</details>
keegancsmith added a commit that referenced this issue May 31, 2024
@keegancsmith
bin: introduce mention-items subcommand
73cc7ef 
It is quite clunky to copy paste the output mention JSON into the items
subcommand. So this introduces a new subcommand "mention-items" which
automates that process. It will take an item from the mentions results
and send it to items. By default it is the first mention, but can does
accept an argument for another mention.

Test Plan: Ran locally. See this example:

  $ OPENCTX_CONFIG=../provider/devdocs/dist/bundle.js pnpm openctx mentions 'abort controller'
  #1 AbortController — https://devdocs.io/dom/abortcontroller
  #2 AbortController.abort — https://devdocs.io/dom/abortcontroller/abort
  #3 AbortController.signal — https://devdocs.io/dom/abortcontroller/signal
  #4 AbortController.constructor — https://devdocs.io/dom/abortcontroller/abortcontroller
  #5 ReadableByteStreamController.byobRequest — https://devdocs.io/dom/readablebytestreamcontroller/byobrequest

  $ OPENCTX_CONFIG=../provider/devdocs/dist/bundle.js pnpm openctx mention-items 'abort controller'
  #1 AbortController — https://devdocs.io/dom/abortcontroller
     - ai.content: (13659 characters)
keegancsmith added a commit that referenced this issue May 31, 2024
@keegancsmith
bin: introduce mention-items subcommand
6bddb44 
It is quite clunky to copy paste the output mention JSON into the items
subcommand. So this introduces a new subcommand "mention-items" which
automates that process. It will take an item from the mentions results
and send it to items. By default it is the first mention, but can does
accept an argument for another mention.

Test Plan: Ran locally. See this example:

  $ OPENCTX_CONFIG=../provider/devdocs/dist/bundle.js pnpm openctx mentions 'abort controller'
  #1 AbortController — https://devdocs.io/dom/abortcontroller
  #2 AbortController.abort — https://devdocs.io/dom/abortcontroller/abort
  #3 AbortController.signal — https://devdocs.io/dom/abortcontroller/signal
  #4 AbortController.constructor — https://devdocs.io/dom/abortcontroller/abortcontroller
  #5 ReadableByteStreamController.byobRequest — https://devdocs.io/dom/readablebytestreamcontroller/byobrequest

  $ OPENCTX_CONFIG=../provider/devdocs/dist/bundle.js pnpm openctx mention-items 'abort controller'
  #1 AbortController — https://devdocs.io/dom/abortcontroller
     - ai.content: (13659 characters)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@sqs