feat: Add self-managed debug tool #31282

SangJunBak · 2025-02-04T04:50:47Z

Introduces a new CLI tool for debugging self-managed Materialize deployments in Kubernetes. The tool can:

Collect pod logs
Collect info similar to kubectl get all -o wide
Up next will be getting events, describe output, rest of TODOs, and tests

I hope to merge all changes in a stack but decided to create a PR for early feedback.

Motivation

This PR adds a known-desirable feature.

https://github.com/MaterializeInc/database-issues/issues/8908

Tips for reviewer

Currently want to leave it all in one file and separate things out later
Curious if the amount of code for something like this is normal!
To quickly test, run cargo run --bin mz-self-managed-debug -- --kubernetes-context mzcloud-staging-us-east-1-0 --kubernetes-namespaces mz-balancer

Checklist

This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).
If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.

src/self-managed-debug/Cargo.toml

SangJunBak · 2025-02-04T04:53:47Z

src/self-managed-debug/src/main.rs

+    Ok(file_name)
+}
+
+fn format_duration(duration: Duration) -> String {


Wondering if there's a better function I can reuse here

You should be able to get a TimeDelta, which has a Display implementation.

Introduces a new CLI tool for debugging self-managed Materialize deployments in Kubernetes. The tool can: - Collect pod logs - Collect info similar to `kubectl get all -o wide` - Up next will be getting events, describe output, and rest of TODOs.

antiguru

Left some comments inline. Thanks for taking this on!

A high-level question I have is why we don't format the data as JSON (or another serialization framework). This would reduce complexity here as a lot of the types already have serde implementations, and we'd not accidentally lose information on the way.

We could then have a second binary that takes the serialized format and pretty-prints it for human consumption. (Or, send both files because it allows users to more easily verify what information we include.)

antiguru · 2025-02-04T10:08:26Z

src/self-managed-debug/Cargo.toml

+[package]
+name = "mz-self-managed-debug"
+description = "Debug tool for self-managed Materialize."
+version = "0.130.1"


You probably want the current version here and elsewhere (right now it's 0.133.0-dev.0, but you'll want to update it prior to merging.)

Also, to update it automatically, adjust line 41 and 61 in bin/bump-version so that the version gets updated automatically. The script takes care of calling bin/bazel gen to sync the bazel files.

Acc ik Marta mentioned we'd most likely want to decouple the tool from LTS versions/materialize https://materializeinc.slack.com/archives/C07PN7KSB0T/p1737567670003019?thread_ts=1737566344.547089&cid=C07PN7KSB0T, so we might want to make this just version 0.1.0. I think it's very doable that we can have the tool work for any version of materialize.

src/self-managed-debug/Cargo.toml

antiguru · 2025-02-04T10:10:28Z

src/self-managed-debug/src/main.rs

+use std::fmt::Debug;
+
+use std::process;
+
+use std::sync::LazyLock;
+
+use chrono::{Duration, Utc};
+use clap::Parser;
+use k8s_openapi::api::apps::v1::{Deployment, ReplicaSet, StatefulSet};
+use k8s_openapi::api::networking::v1::{NetworkPolicy, NetworkPolicyPeer, NetworkPolicyPort};
+use k8s_openapi::apimachinery::pkg::util::intstr::IntOrString;
+use kube::config::KubeConfigOptions;
+use kube::{Client, Config};
+use tabled::{Style, Table, Tabled};
+
+use mz_build_info::{build_info, BuildInfo};
+
+use mz_ore::cli::{self, CliConfig};
+
+use k8s_openapi::api::core::v1::{ContainerStatus, Pod, Service};
+
+use kube::api::{Api, ListParams, LogParams, ObjectMeta};
+use mz_ore::error::ErrorExt;
+
+use std::fs::File;
+use std::io::Write;


Nit: Structure uses as:

use std::...; use other-crate::...; use crate::...;

I.e., three sections: std, other crates, crate, separated by a newline.

antiguru · 2025-02-04T10:12:18Z

src/self-managed-debug/src/main.rs

+/**
+ * Creates a k8s client given a context. If no context is provided, the default context is used.
+ */


Suggested change

/**

* Creates a k8s client given a context. If no context is provided, the default context is used.

*/

/// Creates a k8s client given a context. If no context is provided, the default context is used.

antiguru · 2025-02-04T10:12:44Z

src/self-managed-debug/src/main.rs

+/**
+ * Write k8s pod logs to a file per pod as mz-pod-logs.<namespace>.<pod-name>.log.
+ * Returns a list of file names on success.
+ */


Suggested change

/**

* Write k8s pod logs to a file per pod as mz-pod-logs.<namespace>.<pod-name>.log.

* Returns a list of file names on success.

*/

/// Write k8s pod logs to a file per pod as `mz-pod-logs.<namespace>.<pod-name>.log`.

/// Returns a list of file names on success.

antiguru · 2025-02-04T10:16:02Z

src/self-managed-debug/src/main.rs

+        {
+            let pod_name = pod.metadata.name.clone().unwrap_or_default();
+            eprintln!("Failed to process pod {}: {}", pod_name, e);
+        }


This might not do what you want: the ? operator returns from the enclosing function, not the block.

Ah but if we're referring to the usage of ? in lines 132-168, wouldn't it return from the block given it's an async block?

src/self-managed-debug/src/main.rs

antiguru · 2025-02-04T10:24:57Z

src/self-managed-debug/src/main.rs

+    Ok(file_name)
+}
+
+fn format_duration(duration: Duration) -> String {


You should be able to get a TimeDelta, which has a Display implementation.

alex-hunt-materialize

+1 to everything Moritz said.

src/self-managed-debug/src/main.rs

alex-hunt-materialize · 2025-02-04T13:57:30Z

src/self-managed-debug/src/main.rs

+            {
+                Ok(logs) => logs,
+                Err(_) => {
+                    // If we get a bad request error, try without the previous flag.


I think we probably want to just always try to grab both the current and previous logs.

alex-hunt-materialize · 2025-02-04T13:59:03Z

src/self-managed-debug/src/main.rs

+
+            for line in logs.lines() {
+                writeln!(file, "{}", line)?;
+            }


Maybe just write the whole thing in one go? You aren't doing anything with the split lines.

alex-hunt-materialize · 2025-02-04T14:02:56Z

src/self-managed-debug/src/main.rs

+    let client = create_k8s_client(args.kubernetes_context.clone()).await?;
+    // TODO: Make namespaces mandatory
+    // TODO: Print a warning if namespace doesn't exist
+    for namespace in args.kubernetes_namespaces {


We probably want to store things in an organized directory structure. Maybe something like:
materialize-debug-{iso_datetime}/{namespace}/{resource_type}/{resource_name}.yaml

or, in the case of logs:
materialize-debug-{iso_datetime}/{namespace}/logs/{pod_name}.{current_or_previous}.log

That makes sense, even if we're zipping at the end!

src/self-managed-debug/src/main.rs

SangJunBak · 2025-02-04T16:50:59Z

After talking to the cloud team, decided to remove dump_k8s_get_all and can add it back later if we need to

Decided high level, human readable logs aren't useful for the cloud team.

- Rename Kubernetes-related CLI arguments to shortform - Make namespace argument required - Add error handling for non-existent namespaces

- Add timestamp-based directory structure for log files - Separate previous and current pod logs into distinct files - Introduce `Context` struct to track debug tool shared state

SangJunBak requested a review from a team February 4, 2025 04:50

SangJunBak requested a review from ParkMyCar as a code owner February 4, 2025 04:50

SangJunBak commented Feb 4, 2025

View reviewed changes

src/self-managed-debug/Cargo.toml Show resolved Hide resolved

SangJunBak commented Feb 4, 2025

View reviewed changes

SangJunBak force-pushed the jun/#8908/debug-tool branch from 63632d2 to a6465db Compare February 4, 2025 04:54

SangJunBak removed the request for review from ParkMyCar February 4, 2025 05:11

feat: Add self-managed debug tool

2460e24

Introduces a new CLI tool for debugging self-managed Materialize deployments in Kubernetes. The tool can: - Collect pod logs - Collect info similar to `kubectl get all -o wide` - Up next will be getting events, describe output, and rest of TODOs.

SangJunBak force-pushed the jun/#8908/debug-tool branch from a6465db to 2460e24 Compare February 4, 2025 05:44

SangJunBak requested review from a team, aljoscha and morsapaes as code owners February 4, 2025 05:44

SangJunBak changed the base branch from lts-v0.130 to main February 4, 2025 05:45

SangJunBak removed request for a team, aljoscha and morsapaes February 4, 2025 05:45

antiguru reviewed Feb 4, 2025

View reviewed changes

alex-hunt-materialize reviewed Feb 4, 2025

View reviewed changes

SangJunBak added 2 commits February 6, 2025 13:49

Clean up version and imports

29efe66

Remove get_all command

973cb84

Decided high level, human readable logs aren't useful for the cloud team.

SangJunBak added 2 commits February 6, 2025 14:38

Improve UI of tool

9449612

- Rename Kubernetes-related CLI arguments to shortform - Make namespace argument required - Add error handling for non-existent namespaces

Enhance log collection with improved file naming and organization

e9d47e9

- Add timestamp-based directory structure for log files - Separate previous and current pod logs into distinct files - Introduce `Context` struct to track debug tool shared state

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add self-managed debug tool #31282

feat: Add self-managed debug tool #31282

SangJunBak commented Feb 4, 2025 •

edited

Loading

SangJunBak Feb 4, 2025

antiguru Feb 4, 2025

antiguru left a comment

antiguru Feb 4, 2025

SangJunBak Feb 6, 2025

antiguru Feb 4, 2025

antiguru Feb 4, 2025

antiguru Feb 4, 2025

antiguru Feb 4, 2025

SangJunBak Feb 4, 2025

antiguru Feb 4, 2025

alex-hunt-materialize left a comment

alex-hunt-materialize Feb 4, 2025

alex-hunt-materialize Feb 4, 2025

alex-hunt-materialize Feb 4, 2025

SangJunBak Feb 4, 2025

SangJunBak commented Feb 4, 2025

feat: Add self-managed debug tool #31282

Are you sure you want to change the base?

feat: Add self-managed debug tool #31282

Conversation

SangJunBak commented Feb 4, 2025 • edited Loading

Motivation

Tips for reviewer

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

antiguru left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alex-hunt-materialize left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SangJunBak commented Feb 4, 2025

SangJunBak commented Feb 4, 2025 •

edited

Loading