NIFI-10831 enable JWT realm authentication with Elasticsearch #9605

ChrisSamo632 · 2024-12-30T21:47:09Z

Summary

NIFI-10831 enable JWT realm authentication with Elasticsearch.

Also allow for setting of runas_user in the Elasticsearch Controller Service, which may be of use to impersonate a user within Elasticsearch from NiFi, without having the user's credentials (but where the connecting NiFi user/token is granted permission to impersonate the alternate user).

Other changes include:

removing unused dependencies from Elasticsearch component POMs

Testing:

Attempted to verify changes using LogTo.io as an OIDC and JWT IdP, but it turns out that LogTo provides Access Token JWTs with a typ of at+jwt, which Elasticsearch cannot currently handle (Support at+jwt Types in JWT Realm elastic/elasticsearch#119370)
Verified using Keycloak instead (with a machine-to-machine/service-account only Client setup to provide Access Tokens)
Configure Elasticsearch to allow JWT realm authentication, an example might be like:

    jwt.keycloak_test_jwt:
      order: 3
      token_type: access_token
      client_authentication.type: shared_secret
      allowed_issuer: https://localhost:9443/realms/elasticsearch-test
      allowed_audiences: [account]
      allowed_subjects: [e61503d0-8fab-473e-9f70-95d91b085f52]
      allowed_signature_algorithms: [ES384]
      pkc_jwkset_path: https://keycloak:9443/realms/elasticsearch-test/protocol/openid-connect/certs
      claims.principal: preferred_username
      ssl.certificate_authorities:
        [/usr/share/elasticsearch/config/step/certs/root_ca.crt]

Note that NiFi requires the IdP to be running over HTTPS (for which SmallStep CA was used to generate certificates locally).

Tracking

Please complete the following tracking steps prior to pull request creation.

Issue Tracking

Apache NiFi Jira issue created

Pull Request Tracking

Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

Pull Request based on current revision of the main branch
Pull Request refers to a feature branch with one commit containing changes

Verification

Please indicate the verification steps performed prior to pull request creation.

Build

Build completed using mvn clean install -P contrib-check
- JDK 21

Licensing

~~[ ] New dependencies are compatible with the Apache License 2.0 according to the License Policy~~
~~[ ] New dependencies are documented in applicable LICENSE and NOTICE files~~

Documentation

~~[ ] Documentation formatting appears as expected in rendered files~~

exceptionfactory

Thanks for introducing JWT support and describing the details @ChrisSamo632.

Of particular note, since the new properties depend on the JWT scheme, they can be marked as required, which removes the need for custom validation and conditional checking.

On a related note, with the introduction of Run As User support here, does that remove one of the potential use cases for custom HTTP request headers in the related pull request?

exceptionfactory · 2025-01-06T15:35:15Z

...ient-service-api/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientService.java

+            .displayName("JWT Shared Secret")
+            .description("JWT realm Shared Secret.")
+            .dependsOn(AUTHORIZATION_SCHEME, AuthorizationScheme.JWT)
+            .required(false)


This can be changed to required(true) and the custom validation can be removed. Also recommend moving this directly under the OAuth2 Access Token Provider property.

Suggested change

.required(false)

.required(true)

exceptionfactory · 2025-01-06T15:35:35Z

nifi-extension-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/pom.xml

-        <dependency>
-            <groupId>commons-io</groupId>
-            <artifactId>commons-io</artifactId>
-        </dependency>
-        <dependency>
-            <groupId>org.apache.commons</groupId>
-            <artifactId>commons-compress</artifactId>
-        </dependency>


Are these dependencies not used?

IntelliJ reports that these dependencies are duplicated or not used here.

It looks like these are now inherited through the nifi-elasticsearch-bundle, which itself extends the nifi-standard-shared-bom, which provides these standard dependencies.

But this could be a mistake, I'm happy to add them back in if you think this might be an issue (I'm a little rusty on nested POMs) - things seem to be working OK when I run some tests though at least

exceptionfactory · 2025-01-06T15:35:57Z

...ient-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java

+    private static final List<PropertyDescriptor> properties = List.of(HTTP_HOSTS, PATH_PREFIX, AUTHORIZATION_SCHEME, USERNAME, PASSWORD,
+            API_KEY_ID, API_KEY, JWT_SHARED_SECRET, OAUTH2_ACCESS_TOKEN_PROVIDER, RUN_AS_USER, PROP_SSL_CONTEXT_SERVICE, PROXY_CONFIGURATION_SERVICE,
+            CONNECT_TIMEOUT, SOCKET_TIMEOUT, CHARSET, SUPPRESS_NULLS, COMPRESSION, SEND_META_HEADER, STRICT_DEPRECATION, NODE_SELECTOR,
+            SNIFF_CLUSTER_NODES, SNIFFER_INTERVAL, SNIFFER_REQUEST_TIMEOUT, SNIFF_ON_FAILURE, SNIFFER_FAILURE_DELAY);


This is a good opportunity to reformat these properties and list one per line for easier readability.

exceptionfactory · 2025-01-06T15:36:33Z

...ient-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java

+        if (authorizationScheme == AuthorizationScheme.JWT) {
+            if (oAuth2Provider == null) {
+                results.add(new ValidationResult.Builder().subject(OAUTH2_ACCESS_TOKEN_PROVIDER.getName()).valid(false)
+                        .explanation(String.format("if '%s' is '%s' then '%s' must be set.",
+                                AUTHORIZATION_SCHEME.getDisplayName(), authorizationScheme.getDisplayName(), OAUTH2_ACCESS_TOKEN_PROVIDER.getDisplayName())
+                        ).build()
+                );
+            }
+            if (!jwtSharedSecretSet) {
+                results.add(new ValidationResult.Builder().subject(JWT_SHARED_SECRET.getName()).valid(false)
+                        .explanation(String.format("if '%s' is '%s' then '%s' must be set.",
+                                AUTHORIZATION_SCHEME.getDisplayName(), authorizationScheme.getDisplayName(), JWT_SHARED_SECRET.getDisplayName())
+                        ).build()
+                );
+            }
+        }


This can be be removed given the use of dependent properties.

I took the opportunity to update the BASIC and API_KEY auth types too - I think we can get rid of those custom validations now too (with the associated dependsOn settings for those)

exceptionfactory · 2025-01-06T15:36:42Z

...ient-service-api/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientService.java

+            .displayName("OAuth2 Access Token Provider")
+            .description("The OAuth2 Access Token Provider used to provide JWTs for Bearer Token Authorization with Elasticsearch.")
+            .dependsOn(AUTHORIZATION_SCHEME, AuthorizationScheme.JWT)
+            .required(false)


Suggested change

.required(false)

.required(true)

exceptionfactory requested changes Jan 6, 2025

View reviewed changes

ChrisSamo632 mentioned this pull request Jan 6, 2025

NIFI-14104 allow setting of HTTP Request Headers for Elasticsearch requests #9606

Open

7 tasks

ChrisSamo632 force-pushed the NIFI-10831 branch from 6dc0dd2 to 78d2823 Compare January 23, 2025 20:57

ChrisSamo632 added 2 commits January 24, 2025 18:46

NIFI-10831 enable JWT realm authentication with Elasticsearch

90cada2

Update ElasticsearchClientService property descriptor dependencies

39b16c3

ChrisSamo632 force-pushed the NIFI-10831 branch from 78d2823 to 39b16c3 Compare January 24, 2025 18:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NIFI-10831 enable JWT realm authentication with Elasticsearch #9605

NIFI-10831 enable JWT realm authentication with Elasticsearch #9605

ChrisSamo632 commented Dec 30, 2024 •

edited

Loading

exceptionfactory left a comment

exceptionfactory Jan 6, 2025

exceptionfactory Jan 6, 2025

ChrisSamo632 Jan 23, 2025

exceptionfactory Jan 6, 2025

exceptionfactory Jan 6, 2025

ChrisSamo632 Jan 23, 2025

exceptionfactory Jan 6, 2025

NIFI-10831 enable JWT realm authentication with Elasticsearch #9605

Are you sure you want to change the base?

NIFI-10831 enable JWT realm authentication with Elasticsearch #9605

Conversation

ChrisSamo632 commented Dec 30, 2024 • edited Loading

Summary

Tracking

Issue Tracking

Pull Request Tracking

Pull Request Formatting

Verification

Build

Licensing

Documentation

exceptionfactory left a comment

Choose a reason for hiding this comment

exceptionfactory Jan 6, 2025

Choose a reason for hiding this comment

exceptionfactory Jan 6, 2025

Choose a reason for hiding this comment

ChrisSamo632 Jan 23, 2025

Choose a reason for hiding this comment

exceptionfactory Jan 6, 2025

Choose a reason for hiding this comment

exceptionfactory Jan 6, 2025

Choose a reason for hiding this comment

ChrisSamo632 Jan 23, 2025

Choose a reason for hiding this comment

exceptionfactory Jan 6, 2025

Choose a reason for hiding this comment

ChrisSamo632 commented Dec 30, 2024 •

edited

Loading