Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: improve private network configuration guide tone and readability #910

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
230 changes: 188 additions & 42 deletions docs/admin/config/private-network.mdx
Original file line number Diff line number Diff line change
@@ -1,21 +1,33 @@
# Private network configuration

A **private network** refers to a secure network environment segregated from the public internet, designed to facilitate internal communications and operations within an organization. This network setup restricts external access, enhancing security and control over data flow by limiting exposure to external threats and unauthorized access.

When deploying self-hosted Sourcegraph instances in private networks with specific compliance and policy requirements, additional configuration may be required to ensure all networking features function correctly. The reasons for applying the following configuration options depend on the specific functionality of the Sourcegraph service and the unique network and infrastructure requirements of the organization.

The following is a list of Sourcegraph services and how and when each initiates outbound connections to external services:

- **executor**: Sourcegraph [Executor](../executors) batch change or precise indexing jobs may need to connect to services hosted within an organization's private network
- **frontend**: The frontend service communicates externally when connecting to external [auth providers](../auth), sending [telemetry data](../pings), testing code host connections, and connecting to [externally hosted](../external_services) Sourcegraph services
## Overview
A private network is your organization's secure, internal network space - separated from the public internet.
Think of it as your company's own protected environment where internal systems can communicate safely,
keeping your sensitive data and operations shielded from external access.

When deploying self-hosted Sourcegraph instances in private networks with specific compliance and policy requirements,
additional configuration may be required to ensure all networking features function correctly. The reasons for applying the following configuration options depend on the specific functionality of the Sourcegraph service and the unique network and infrastructure requirements of the organization.

The following is a list of Sourcegraph services that initiate outbound connections to external services. Sourcegraph services not included in this list can be assumed to only connect to services within the Sourcegraph deployment's network segment:
- **executor**: Sourcegraph [Executor](../executors) batch change or precise indexing jobs may need to connect to
services hosted within an organization's private network
- **frontend**: The frontend service communicates externally when connecting to:
* External [auth providers](../auth)
* Sending [telemetry data](../pings)
* Testing [code host connections](../code_hosts)
* Connecting to [externally hosted](../external_services) Sourcegraph services
* Connecting to external [LLM providers](../../cody/capabilities/supported-models) with Cody
- **gitserver**: Executes git commands against externally hosted [code hosts](../external_service)
- **migrator**: Connects to Postgres instances (which may be [externally hosted](../external_services/postgres)) to process database migrations
- **repo-updater**: Communicates with [code hosts](../external_service) APIs to coordinate repository synchronization
- **worker**: Sourcegraph [Worker](../workers) run various background jobs that may require establishing connections to services hosted within an organization's private network
- **worker**: Sourcegraph [Worker](../workers) run various background jobs that may require establishing connections to
services hosted within an organization's private network

## HTTP proxy configuration

All Sourcegraph services respect the conventional `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY` environment variables for routing Sourcegraph client application HTTP traffic through a proxy server. The steps for configuring proxy environment variables will depend on your Sourcegraph deployment method.
All Sourcegraph services respect the conventional `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY` environment variables for
routing Sourcegraph client application HTTP traffic through a proxy server. The steps for configuring proxy environment
variables will depend on your Sourcegraph deployment method.
trly marked this conversation as resolved.
Show resolved Hide resolved

### Kubernetes Helm

Expand All @@ -32,48 +44,182 @@ executor|frontend|gitserver|migrator|repo-updater|worker:
value: "blobstore,codeinsights-db,codeintel-db,sourcegraph-frontend-internal,sourcegraph-frontend,github-proxy,gitserver,grafana,indexed-search-indexer,indexed-search,jaeger-query,pgsql,precise-code-intel-worker,prometheus,redis-cache,redis-store,repo-updater,searcher,symbols,syntect-server,worker-executors,worker,cloud-sql-proxy,localhost,127.0.0.1,.svc,.svc.cluster.local,kubernetes.default.svc"
```

<Callout type="warning">Failure to configure `NO_PROXY` correctly can cause the proxy configuration to interfere with local networking between internal Sourcegraph services.</Callout>
### Docker Compose

## Using private CA root certificates
Some organizations maintain a private Certificate Authority (CA) for issuing certificates within their private network. When Sourcegraph connects to TLS encrypted service using a self-signed certificate that it does not trust, you will observe an `x509: certificate signed by unknown authority` error message in logs.
Add the proxy environment variables your docker compose override file.
```yaml
services:
<service-name>:
environment:
- HTTP_PROXY=http://proxy.example.com:8080
- HTTPS_PROXY=http://proxy.example.com:8080
- NO_PROXY='blobstore,caddy,cadvisor,codeintel-db,codeintel-db-exporter,codeinsights-db,codeinsights-db-exporter,sourcegraph-frontend-0,sourcegraph-frontend-internal,gitserver-0,grafana,migrator,node-exporter,otel-collector,pgsql,pgsql-exporter,precise-code-intel-worker,prometheus,redis-cache,redis-store,repo-updater,searcher-0,symbols-0,syntect-server,worker,zoekt-indexserver-0,zoekt-webserver-0,localhost,127.0.0.1'
```

In order for Sourcegraph to respect an organization's self-signed certificates, the private CA root certificate(s) will need to be appended to Sourcegraph's trusted CA root certificate list in `/etc/ssl/certs/ca-certificates.crt`.
<Callout type="warning">Failure to configure `NO_PROXY` correctly can cause the proxy configuration to interfere with
local networking between internal Sourcegraph services.</Callout>

### Configuring sourcegraph-frontend to recognize private CA root certificates
The following details the process for setting up the sourcegraph-frontend to acknowledge and trust a private CA root certificate for Sourcegraph instances deployed using [Helm](../deploy/kubernetes/helm). For any other Sourcegraph service that needs to trust an organization's private CA root certificate (including gitserver, repo-updater, or migrator), similar steps will need to be followed.
## Docker networking configuration
To avoid IP range collisions with the host network, it is recommended to explicitly configure a CIDR range for the
Docker network.

1. Copy out the existing `ca-certificates.crt` file from the sourcegraph-frontend container:
```sh
kubectl cp $(kubectl get pod -l app=sourcegraph-frontend -o jsonpath='{.items[0].metadata.name}'):/etc/ssl/certs/ca-certificates.crt sourcegraph-frontend-ca-certificates.crt
```yaml
networks:
default:
ipam:
driver: default
config:
- subnet: "172.20.2.0/27" # CIDR range for the Docker network that doesn't overlap with the host network.
```
2. Concatenate the private CA root certificate to the `sourcegraph-frontend-ca-certificates.crt` file:
```sh
cat sourcegraph-frontend-ca-certificates.crt {private-ca-certificate.crt file} > ca-certificates.crt

Additional information on docker networking can be found here:
* [Docker networking overview](https://docs.docker.com/network/)
* [Networking in Compose](https://docs.docker.com/compose/how-tos/networking/)

## Configuring TLS certificates for private networks

When deploying Sourcegraph in private networks, you'll often need to configure TLS certificates to establish trusted
connections with internal services like code hosts. The recommended approach is to configure root CA certificates
through Sourcegraph's site configuration using `tls.external` in the `experimentalFeatures` section.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to check if this will work for executors, or at least add a note that depending on your executor deployment method this might not work.


This method offers several advantages:
- Works consistently across both Cloud and self-hosted deployments
- Requires minimal configuration changes
- Can be managed entirely through the web UI
- Maintains certificates in a centralized location
- Aligns with enterprise PKI best practices

The configuration process involves identifying and adding the public key of your organization's root Certificate
Authority (CA) to Sourcegraph's site configuration. This approach is particularly efficient because:
* Root CA certificates typically have long expiration periods (often measured in years)
* A single root CA certificate usually covers multiple internal services
* The configuration can be managed without container modifications or filesystem changes

### Obtain the certificate chain
Use the OpenSSL command to extract the certificate chain from your code host.
Replace the domain and port with your internal code host's values:

```bash
openssl s_client -showcerts -connect example.com:8443 \
-nameopt lname < /dev/null > certs.log 2>&1
```
3. Create a new Kubernetes ConfigMap containing the concatenated `ca-certificates.crt` file:
```sh
kubectl create configmap sourcegraph-frontend-ca-certificates --from-file=ca-certificates.crt

### Identify the root certificate
In the generated `certs.log` file, locate the root CA certificate:

Certificate chains typically include 3 certificates:

* Root certificate authority (depth=2)
* Intermediate certificate authority (depth=1)
* Server (leaf) certificate (depth=0)

The last certificate in the chain will be the root CA certificate and will typically have:

* A long expiration period (years)
* A descriptive common name (e.g., "Enterprise Root CA 2023")

Example root CA certificate for github.com:

```text
Connecting to 140.82.114.3
depth=2 countryName=US, stateOrProvinceName=New Jersey, localityName=Jersey City, organizationName=The USERTRUST Network, commonName=USERTrust ECC Certification Authority
verify return:1
depth=1 countryName=GB, stateOrProvinceName=Greater Manchester, localityName=Salford, organizationName=Sectigo Limited, commonName=Sectigo ECC Domain Validation Secure Server CA
verify return:1
depth=0 commonName=github.com
verify return:1
CONNECTED(00000005)
---
...
2 s:countryName=US, stateOrProvinceName=New Jersey, localityName=Jersey City, organizationName=The USERTRUST Network, commonName=USERTrust ECC Certification Authority
i:countryName=GB, stateOrProvinceName=Greater Manchester, localityName=Salford, organizationName=Comodo CA Limited, commonName=AAA Certificate Services
a:PKEY: id-ecPublicKey, 384 (bit); sigalg: RSA-SHA384
v:NotBefore: Mar 12 00:00:00 2019 GMT; NotAfter: Dec 31 23:59:59 2028 GMT
-----BEGIN CERTIFICATE-----
MII...c=
-----END CERTIFICATE-----
```
4. Mount the `sourcegraph-frontend-ca-certificates` ConfigMap to the sourcegraph-frontend Deployment:
```yaml
frontend:
extraVolumes:
- name: ca-certificates
configMap:
name: sourcegraph-frontend-ca-certificates
extraVolumeMounts:
- name: ca-certificates
mountPath: /etc/ssl/certs/

### Format the certificate
Once you've identified the root CA certificate:
trly marked this conversation as resolved.
Show resolved Hide resolved

* Extract the certificate content including the BEGIN and END markers.
* Format the certificate for the site configuration:
* Replace newlines with \n characters
* Enclose the entire certificate in double quotes
* Add a trailing comma


The following command can be used to easily obtain, extract, and format the root certificate from a 3 certificate chain.
Be sure to adjust the hostname and port to match your internal code host. If your certificate chain is of a different
depth, adjust the awk command accordingly. `awk '/BEGIN CERTIFICATE/{i++} i==X'`
```bash
openssl s_client -showcerts -connect example.com:8443 \
-nameopt lname < /dev/null 2>&1 \
| awk '/BEGIN CERTIFICATE/,/END CERTIFICATE/' \
| awk '/BEGIN CERTIFICATE/{i++} i==2' \
| awk '{printf "%s\\n", $0}' | sed 's/\\n$//' \
| awk '{print "\"" $0 "\","}'
```

Once deployed, you should see the private CA root certificate in the sourcegraph-frontend container's `/etc/ssl/certs/ca-certificates.crt` file.
```sh
kubectl exec -it $(kubectl get pod -l app=sourcegraph-frontend -o jsonpath='{.items[0].metadata.name}') -- tail /etc/ssl/certs/ca-certificates.crt
### Add the certificate to the site configuration
Add the formatted certificate to your Sourcegraph site configuration.

```json
{
"experimentalFeatures": {
"tls.external": {
"certificates": [
"-----BEGIN CERTIFICATE-----\naZ...==\n-----END CERTIFICATE-----"
]
}
}
}
```

You can verify that the self-signed certificate is trusted using `curl`:
```sh
kubectl exec -it $(kubectl get pod -l app=sourcegraph-frontend -o jsonpath='{.items[0].metadata.name}') -- curl -v {https://internal.service.example.com} > /dev/null
For organizations with multiple root CAs (uncommon), additional certificates can be added to the array:
```json
{
"experimentalFeatures": {
"tls.external": {
"certificates": [
"-----BEGIN CERTIFICATE-----\naZ...==\n-----END CERTIFICATE-----",
"-----BEGIN CERTIFICATE-----\nMI...I7\n-----END CERTIFICATE-----"
]
}
}
}
```

<Callout type="note">It is recommended to repeat these steps on a regular cadence to ensure that Sourcegraph's CA root certificate list stays up to date.</Callout>
### Validation of certificate configuration
These steps confirms that configuring the root CA certificate through `tls.external` is sufficient for all standard
Sourcegraph operations that require secure connections to internal services.

1. **Code host connectivity**
- Verify using the UI "Test Connection" button
- Trigger validate completed sync jobs
<Callout type="info">Executed by: frontend service</Callout>

2. **Repository operations**
- Verify individual repository synchronization
- Verify cloning operations
<Callout type="info">Executed by: gitserver service</Callout>

3. **Permission synchronization**
- Verify user-centric permission sync jobs
<Callout type="info">Executed by: worker service</Callout>

<Callout type="note">
Repository-centric permission sync jobs are expected to behave identically, as they use the same underlying TLS configuration mechanisms.
</Callout>

### Recommended best practices
* Only include root CA certificates, not intermediate or server certificates.
* Avoid using `insecureSkipVerify: true` and add TLS certificates if needed, as it bypasses important security checks.
* Document certificate sources and expiration dates in your organization's runbooks.
* Plan for certificate rotation well before root CA expiration.
* Most enterprises use a single root CA, so adding one certificate often covers all internal services.
* Keep the certificate list minimal and well-maintained.



Loading