Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Csa 1.12.0 dh 7.2.18 #78

Open
wants to merge 28 commits into
base: develop
Choose a base branch
from
Open

Csa 1.12.0 dh 7.2.18 #78

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
d0f9745
Datahub 1.10.0 pom changes
Aug 2, 2023
09ea931
Merge branch 'develop' into cybersec-dh-1.10.0-2.3.2
Aug 7, 2023
054e548
[CYB-172] kafka topic created from tableapi kafka connector need ssb …
Aug 10, 2023
e8f4557
Merge branch 'develop' into cybersec-dh-1.10.0-2.3.2
Sep 20, 2023
eeb8f82
Datahub 1.10.0 pom changes
Oct 4, 2023
2d4dda8
Merge branch 'develop' into cybersec-dh-1.10.0-2.3.2
Jan 30, 2024
28f8f9a
[CYB-190] user can specify names of CDP cloud artifacts in example cr…
Jan 31, 2024
a41c542
[CYB-190] user can specify names of CDP cloud artifacts in example cr…
Feb 1, 2024
2a262dd
[CYB-191] schemas produced for topics can't be consumed by SMM becaus…
Feb 1, 2024
b061055
[CYB-206] dev log removed
stas-panasiuk Mar 21, 2024
bd26382
Merge remote-tracking branch 'origin/cybersec-dh-1.10.0-2.3.2' into c…
Mar 22, 2024
d82f591
[CYB-207] cybersec toolkit supports CSA 1.12.0 with CDP 7.1.9
Mar 22, 2024
63dcbfb
[CYB-207] fix indexing job serialization error
Mar 29, 2024
9f1033c
[CYB-207] add missing jupiter dependency. use jupiter for all tests
Mar 29, 2024
311d841
[CYB-207] remove deprecated forkMode for rest enrichments module
Mar 29, 2024
800bcf6
[CYB-207] add junit dependency
Apr 1, 2024
1fe5223
[CYB-207] use https for mock server for all tests.
Apr 2, 2024
ba0a0a9
[CYB-207] temporary change to run just problem module
Apr 3, 2024
f4fee5b
[CYB-207] update mockserver dependency
Apr 3, 2024
31cf004
Revert "[CYB-207] temporary change to run just problem module"
carolynduby Apr 4, 2024
9e7791d
Revert "[CYB-207] use https for mock server for all tests."
carolynduby Apr 4, 2024
056e7ea
[CYB-207] Update mockserver dependency to remove conflicts
carolynduby Apr 4, 2024
68095f8
Merge remote-tracking branch 'origin/csa-1.12.0-cdh-7.1.9' into csa-1…
carolynduby Apr 4, 2024
e8c7c24
Merge branch 'develop' into csa-1.12.0-cdh-7.1.9
stas-panasiuk Apr 29, 2024
deff285
Merge branch 'develop' into csa-1.12.0-cdh-7.1.9
stas-panasiuk Apr 29, 2024
629a2c1
typo fix
stas-panasiuk Apr 29, 2024
2c00a1d
merge in CSA 1.12.0 with 7.1.9 CDH
carolynduby May 16, 2024
d84c726
[CYB-211] cybersec runs on DH 7.2.18 with CSA 1.12.0
carolynduby May 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions flink-cyber/caracal-generator/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,13 @@
<artifactId>flink-connector-kafka</artifactId>
</dependency>

<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-clients</artifactId>
<scope>provided</scope>
<version>${flink.version}</version>
</dependency>

<dependency>
<groupId>com.hortonworks.smm</groupId>
<artifactId>monitoring-interceptors</artifactId>
Expand Down
13 changes: 11 additions & 2 deletions flink-cyber/caracal-parser/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,8 +126,12 @@

<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-cloudera-registry</artifactId>
<version>${flink.version}</version>
<artifactId>flink-connector-cloudera-registry</artifactId>
</dependency>

<dependency>
<groupId>com.hortonworks.registries</groupId>
<artifactId>schema-registry-serdes</artifactId>
</dependency>

<dependency>
Expand Down Expand Up @@ -164,6 +168,11 @@
<artifactId>flink-orc</artifactId>
</dependency>

<dependency>
<groupId>org.apache.orc</groupId>
<artifactId>orc-core</artifactId>
</dependency>

<dependency>
<groupId>org.apache.parquet</groupId>
<artifactId>parquet-avro</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion ...cyber/flink-alert-scoring-api/src/main/java/com/cloudera/cyber/scoring/ScoredMessage.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ public long getTs() {
return message.getTs();
}

public static final Schema SCHEMA$ = AvroSchemas.createRecordBuilder(ScoredMessage.class.getPackage().getName(), ScoredMessage.class.getName())
public static final Schema SCHEMA$ = AvroSchemas.createRecordBuilder(ScoredMessage.class.getPackage().getName(), ScoredMessage.class.getName(), null)
.fields()
.name("message").type(Message.SCHEMA$).noDefault()
.name("cyberScoresDetails").type(Schema.createArray(Scores.SCHEMA$)).noDefault()
Expand Down
5 changes: 5 additions & 0 deletions flink-cyber/flink-alert-scoring/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,11 @@
<groupId>org.apache.flink</groupId>
<artifactId>flink-statebackend-rocksdb</artifactId>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<scope>test</scope>
</dependency>

</dependencies>
<build>
Expand Down
12 changes: 5 additions & 7 deletions flink-cyber/flink-commands/json-commands/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,6 @@
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<scope>test</scope>
</dependency>

<dependency>
Expand Down Expand Up @@ -70,15 +69,14 @@
<version>${log4j.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>${guava.version}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>${jackson.version}</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<scope>test</scope>
</dependency>
</dependencies>

Expand Down
18 changes: 18 additions & 0 deletions flink-cyber/flink-commands/scoring-commands/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,14 @@
<version>${jackson.datatype.version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>com.hortonworks.registries</groupId>
<artifactId>schema-registry-serdes</artifactId>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</dependency>
<!-- Logging and Metrics -->
<dependency>
<groupId>org.slf4j</groupId>
Expand Down Expand Up @@ -90,6 +98,10 @@
<artifactId>avro</artifactId>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-java</artifactId>
</dependency>
</dependencies>
<build>
<plugins>
Expand Down Expand Up @@ -128,6 +140,12 @@
</excludes>
</filter>
</filters>
<relocations>
<relocation>
<pattern>com.google.</pattern>
<shadedPattern>com.cloudera.cyber.com.google.</shadedPattern>
</relocation>
</relocations>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>com.cloudera.cyber.scoring.ScoringJobKafka</mainClass>
Expand Down
15 changes: 14 additions & 1 deletion flink-cyber/flink-common/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,9 +43,22 @@
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-kafka</artifactId>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-cloudera-registry</artifactId>
</dependency>
<dependency>
<groupId>com.hortonworks.registries</groupId>
<artifactId>schema-registry-serdes</artifactId>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-cloudera-registry</artifactId>
<artifactId>flink-connector-base</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.avro</groupId>
Expand Down
20 changes: 19 additions & 1 deletion flink-cyber/flink-cyber-api/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,19 @@
</plugins>
</build>

<dependencyManagement>
<dependencies>
<dependency>
<!-- Import dependency management from Spring Boot -->
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-dependencies</artifactId>
<version>${spring-boot.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>

<dependencies>
<dependency>
<groupId>org.projectlombok</groupId>
Expand All @@ -55,10 +68,15 @@
<artifactId>flink-table-common</artifactId>
</dependency>


<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-core</artifactId>
</dependency>

<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
<version>${avro.version}</version>
</dependency>

<dependency>
Expand Down
3 changes: 2 additions & 1 deletion flink-cyber/flink-cyber-api/src/main/java/com/cloudera/cyber/Message.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
import lombok.Data;
import lombok.NoArgsConstructor;
import lombok.NonNull;
import org.apache.avro.LogicalTypes;
import org.apache.avro.Schema;
import org.apache.avro.SchemaBuilder;
import org.apache.avro.specific.SpecificRecord;
Expand Down Expand Up @@ -56,7 +57,7 @@ public class Message extends SpecificRecordBase implements SpecificRecord, Ident
public static final Schema SCHEMA$ = AvroSchemas.createRecordBuilder(Message.class.getPackage().getName(), Message.class.getName())
.fields()
.requiredString("id")
.requiredLong("ts")
.name("ts").type(LogicalTypes.timestampMillis().addToSchema(Schema.create(Schema.Type.LONG))).noDefault()
.name("originalSource").type(SignedSourceKey.SCHEMA$).noDefault()
.requiredString("message")
.name("threats").type().optional().type(SchemaBuilder.map().values(SchemaBuilder.array().items(ThreatIntelligence.SCHEMA$)))
Expand Down
14 changes: 11 additions & 3 deletions flink-cyber/flink-cyber-api/src/main/java/com/cloudera/cyber/avro/AvroSchemas.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,16 @@
public class AvroSchemas {

public static SchemaBuilder.RecordBuilder<org.apache.avro.Schema> createRecordBuilder(String namespace, String recordName) {
return SchemaBuilder.record(recordName).namespace(namespace)
.prop("ssb.rowtimeAttribute", "ts")
.prop("ssb.watermarkExpression", "`ts` - INTERVAL '30' SECOND");
return createRecordBuilder(namespace, recordName, "ts");
}

public static SchemaBuilder.RecordBuilder<org.apache.avro.Schema> createRecordBuilder(String namespace, String recordName, String tsFieldName) {
SchemaBuilder.RecordBuilder<org.apache.avro.Schema> recordBuilder = SchemaBuilder.record(recordName).namespace(namespace);
if (tsFieldName != null) {
recordBuilder
.prop("ssb.rowtimeAttribute", tsFieldName)
.prop("ssb.watermarkExpression", String.format("`%s` - INTERVAL '30' SECOND", tsFieldName));
}
return recordBuilder;
}
}
7 changes: 6 additions & 1 deletion flink-cyber/flink-dedupe/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,12 @@

<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-cloudera-registry</artifactId>
<artifactId>flink-connector-cloudera-registry</artifactId>
</dependency>

<dependency>
<groupId>com.hortonworks.registries</groupId>
<artifactId>schema-registry-serdes</artifactId>
</dependency>

<dependency>
Expand Down
5 changes: 5 additions & 0 deletions flink-cyber/flink-enrichment/flink-enrichment-cidr/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,11 @@
<classifier>tests</classifier>
</dependency>

<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</dependency>

<dependency>
<groupId>com.hortonworks.smm</groupId>
<artifactId>monitoring-interceptors</artifactId>
Expand Down
5 changes: 5 additions & 0 deletions flink-cyber/flink-enrichment/flink-enrichment-geocode/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -156,6 +156,11 @@
<version>${commons-validator.version}</version>
</dependency>

<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</dependency>

<dependency>
<groupId>org.assertj</groupId>
<artifactId>assertj-core</artifactId>
Expand Down
18 changes: 6 additions & 12 deletions flink-cyber/flink-enrichment/flink-enrichment-load/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,25 +72,19 @@
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-hbase-${flink.hbase.version}</artifactId>
<version>${flink.version}</version>
<exclusions>
<exclusion>
<groupId>org.glassfish</groupId>
<artifactId>javax.el</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-client</artifactId>
</exclusion>
</exclusions>
</dependency>

<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-cloudera-registry</artifactId>
<artifactId>flink-connector-cloudera-registry</artifactId>
<scope>compile</scope>
</dependency>

<dependency>
<groupId>com.hortonworks.registries</groupId>
<artifactId>schema-registry-serdes</artifactId>
</dependency>

<dependency>
<groupId>com.cloudera.cyber</groupId>
<artifactId>flink-cyber-api</artifactId>
Expand Down
8 changes: 8 additions & 0 deletions flink-cyber/flink-enrichment/flink-enrichment-lookup-common/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,14 @@
<artifactId>lombok</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
</dependency>

</dependencies>

Expand Down
14 changes: 6 additions & 8 deletions flink-cyber/flink-enrichment/flink-enrichment-lookup-hbase/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,21 +52,19 @@
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-hbase-${flink.hbase.version}</artifactId>
<version>${flink.version}</version>
<exclusions>
<exclusion>
<groupId>org.glassfish</groupId>
<artifactId>javax.el</artifactId>
</exclusion>
</exclusions>
</dependency>

<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-cloudera-registry</artifactId>
<artifactId>flink-connector-cloudera-registry</artifactId>
<scope>provided</scope>
</dependency>

<dependency>
<groupId>com.hortonworks.registries</groupId>
<artifactId>schema-registry-serdes</artifactId>
</dependency>

<dependency>
<groupId>com.cloudera.cyber</groupId>
<artifactId>flink-cyber-api</artifactId>
Expand Down
7 changes: 6 additions & 1 deletion flink-cyber/flink-enrichment/flink-enrichment-lookup-raw/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,10 +50,15 @@

<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-cloudera-registry</artifactId>
<artifactId>flink-connector-cloudera-registry</artifactId>
<scope>provided</scope>
</dependency>

<dependency>
<groupId>com.hortonworks.registries</groupId>
<artifactId>schema-registry-serdes</artifactId>
</dependency>

<dependency>
<groupId>com.cloudera.cyber</groupId>
<artifactId>flink-cyber-api</artifactId>
Expand Down
Loading
Loading