Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added the default ignore config for service disable error #476

Merged
merged 14 commits into from
Feb 5, 2025
+30 −11
Merged

Added the default ignore config for service disable error #476

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
e1276f6
Added the default ignore config for service disable error
ParthaI Aug 7, 2023
c0187e0
Added support for ignoring errors by error message
ParthaI Aug 10, 2023
b3c186a
Merge branch 'main' of github.com:turbot/steampipe-plugin-gcp into is…
ParthaI Oct 3, 2023
ca35dd0
Made changes as per review comments
ParthaI Oct 4, 2023
fb92d9c
Updated the regex pattern
ParthaI Oct 4, 2023
ab5cc67
Updated the regex pattern for better understanding
ParthaI Oct 4, 2023
944ce9f
Updated the doc for ignoring the error by message
ParthaI Oct 4, 2023
f2eac5d
Update index.md
ParthaI Oct 4, 2023
48da7f7
Merge branch 'main' into issue-286
ParthaI Nov 28, 2024
3a7c37f
Fixed typo
ParthaI Nov 28, 2024
b6a1e4e
Update the config arg
ParthaI Nov 28, 2024
b775a5f
fix: ignore error messages from all error types (#688)
pdecat Dec 17, 2024
de603d6
Updated the error message for invalid regex pattern configured and ma…
ParthaI Jan 16, 2025
aecb618
Update the variable naming
ParthaI Feb 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions config/gcp.spc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,9 @@ connection "gcp" {
# If not set, no impersonation is done.
#impersonate_service_account = "YOUR_SERVICE_ACCOUNT"

# `ignore_error_messages` (optional) - List of additional GCP error message pattern to ignore for all queries.
cbruno10 marked this conversation as resolved.
Show resolved Hide resolved
# ignore_error_messages = ["API has not been used", "other regex"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ParthaI Is there a more exact regex (using ^, $) we should add as an example here, in particular for the "API has not been used" case? We can probably remove other regex as it's not helpful.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a really good addition to handle cases where, google uses the same 403 error code when - API are not enabled and when Quota exceeded. An exact regex mapping example to an error message would sure be helpful.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For reference, this is reported as this:

ERROR: googleapi: Error 403: Cloud DNS API has not been used in project XXX before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/dns.go…
Details:
[
  {
    "@type": "type.googleapis.com/google.rpc.Help",
    "links": [
      {
        "description": "Google developers console API activation",
        "url": "https://console.developers.google.com/apis/api/dns.googleapis.com/overview?project=XXX"
      }
    ]
  },
  {
    "@type": "type.googleapis.com/google.rpc.ErrorInfo",
    "domain": "googleapis.com",
    "metadata": {
      "consumer": "projects/XXX",
      "service": "dns.googleapis.com"
    },
    "reason": "SERVICE_DISABLED"
  }
]
, accessNotConfigured (SQLSTATE HV000)

So I think the current example with API has not been used would be enough.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cbruno10, I have added the example for the exact regex (using ^, $) i.e ignore_error_messages = ["^.*API\\s*has\\s*not\\s*been\\s*used.*$"]. Also we can pass a message just like a string format, in both of the case the code changes are working fine.

NOTE: In the .spc file, we should use \\ instead of \ for the regex pattern to avoid errors related to escape characters. Otherwise, parsing the .spc file might result in an error like "Invalid escape sequence: The symbol "d" is not a valid escape sequence selector."


# `ignore_error_codes` (optional) - List of additional GCP error codes to ignore for all queries.
# By default, common not found error codes are ignored and will still be ignored even if this argument is not set.
# Refer https://cloud.google.com/resource-manager/docs/core_errors#Global_Errors for more information on GCP error codes
Expand Down
5 changes: 5 additions & 0 deletions gcp/connection_config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ type gcpConfig struct {
Credentials *string `cty:"credentials"`
CredentialFile *string `cty:"credential_file"`
ImpersonateServiceAccount *string `cty:"impersonate_service_account"`
IgnoreErrorMessages []string `cty:"ignore_error_messages"`
IgnoreErrorCodes []string `cty:"ignore_error_codes"`
}

Expand All @@ -26,6 +27,10 @@ var ConfigSchema = map[string]*schema.Attribute{
"impersonate_service_account": {
Type: schema.TypeString,
},
"ignore_error_messages": {
Type: schema.TypeList,
Elem: &schema.Attribute{Type: schema.TypeString},
},
"ignore_error_codes": {
Type: schema.TypeList,
Elem: &schema.Attribute{Type: schema.TypeString},
Expand Down
18 changes: 14 additions & 4 deletions gcp/ignore_error_predicate.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ package gcp
import (
"context"
"path"
"regexp"

"github.com/turbot/go-kit/helpers"
"github.com/turbot/go-kit/types"
Expand All @@ -23,12 +24,21 @@ func isIgnorableError(notFoundErrors []string) plugin.ErrorPredicate {
// shouldIgnoreErrorPluginDefault:: Plugin level default function to ignore a set errors for hydrate functions based on "ignore_error_codes" config argument
func shouldIgnoreErrorPluginDefault() plugin.ErrorPredicateWithContext {
return func(ctx context.Context, d *plugin.QueryData, h *plugin.HydrateData, err error) bool {
if !hasIgnoredErrorCodes(d.Connection) {
return false
}

gcpConfig := GetConfig(d.Connection)

if gerr, ok := err.(*googleapi.Error); ok {
for _, pattern := range gcpConfig.IgnoreErrorMessages {
re := regexp.MustCompile(pattern)
result := re.FindString(types.ToString(gerr.Message))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does GCP ever send messages that don't have the Message property? If so, does this code still handle everything ok?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't encountered any errors without a message property. Additionally, we are converting errors of type googleapi.Error, which always includes a message property. I've tested it multiple times with various connection combinations, and it's working correctly, handling errors as expected.

If an error does not contain a message property, the code block above will not function as it is designed to handle errors based on their error message. In cases where the message property is not available, an alternative approach is to configure the ignore_error_codes in the .spc file. This will allow the following line of code to handle the error.

According to the GCP API design documentation, errors include a message property. However, it's important to note that Error messages are not part of the API surface. They are subject to changes without notice. Application code should not rely heavily on error messages as they may change.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this function support matching on full regular expressions? How is this different than the regex Match function?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this function support matching on full regular expressions.

  • *func (re Regexp) FindString(s string) string
    FindString returns a string holding the text of the leftmost match in s of the regular expression. If there is no match, the return value is an empty string, but it will also be empty if the regular expression successfully matches an empty string. Use FindStringIndex or FindStringSubmatch if it is necessary to distinguish these cases.

  • *func (re Regexp) Match(b []byte) bool
    Match reports whether the byte slice b contains any match of the regular expression re.

IMO, it would be better to use MatchString function here. I can see in most of the place we use MatchString function to validate patterns. https://github.com/search?q=repo%3Aturbot%2Fsteampipe%20MatchString&type=code

I have replace the function FindString to MatchString.

if result != "" {
return true
}
}

if !hasIgnoredErrorCodes(d.Connection) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this block still?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed this block of code as we do not need those.

return false
}

// Added to support regex in not found errors
for _, pattern := range gcpConfig.IgnoreErrorCodes {
if ok, _ := path.Match(pattern, types.ToString(gerr.Code)); ok {
Expand Down