'Version 1.4.0 of the DynamoDB Streams Kinesis Adapter'

awslabs · Jun 13, 2018 · a904965 · a904965
1 parent 4e4e32f
commit a904965
Show file tree

Hide file tree

Showing 10 changed files with 1,760 additions and 29 deletions.
diff --git a/META-INF/MANIFEST.MF b/META-INF/MANIFEST.MF
@@ -2,9 +2,9 @@ Manifest-Version: 1.0
 Bundle-ManifestVersion: 2
 Bundle-Name: Amazon DynamoDB Streams Kinesis Adapter for Java
 Bundle-SymbolicName: com.amazonaws.dynamodb.streams.kinesis.adapter;singleton:=true
-Bundle-Version: 1.2.2
+Bundle-Version: 1.4.0
 Bundle-Vendor: Amazon Technologies, Inc
-Bundle-RequiredExecutionEnvironment: JavaSE-1.7
+Bundle-RequiredExecutionEnvironment: JavaSE-1.8
 Export-Package: com.amazonaws.services.dynamodbv2.streamsadapter,
     com.amazonaws.services.dynamodbv2.streamsadapter.exceptions,
     com.amazonaws.services.dynamodbv2.streamsadapter.model
diff --git a/README.md b/README.md
@@ -14,14 +14,29 @@
 * The KCL is designed to process streams from Amazon Kinesis, but by adding the DynamoDB Streams Kinesis Adapter, your application can process DynamoDB Streams instead, seamlessly and efficiently.
 
 ## Release Notes
-* Requires Kinesis Client Library version >= 1.7.5.
-* Requires AWS Java SDK version >= 1.11.115
+### Latest Release (v1.4.0)
+* This release fixes an issue of high propagation delay of streams records when processing streams on small tables. This issue occurs when KCL ShardSyncer is not discovering new shards due to server side delays in shard creation or in reporting new shard creation to internal services. The code is implemented in a new implementation of IKinesisProxy interface called DynamoDBStreamsProxy which is part of the latest release.
+* This release requires Kinesis Client Library version >= 1.8.10. It is however recommended to use Kinesis Client Library version 1.9.0. Versions >= 1.8.10 have changes to allow IKinesisProxy injection into the KCL Worker builder which is required by DynamoDB Streams Kinesis Adapter v1.4.0 for
+injection of DynamoDBStreamsProxy into the KCL worker during initialization. Please refer to [Kinesis Client Library release notes for 1.8.10](https://github.com/awslabs/amazon-kinesis-client/blob/master/CHANGELOG.md#release-1810) for more information.
+* Suggested AWS Java SDK version >= 1.11.218
 * It is highly recommended to [configure][kcl-configuration] Kinesis Client Library with `MaxRecords = 1000` and `IdleTimeInMillis = 500` to optimize DynamoDB Streams costs.
 
+### Guidance for injecting DynamoDBStreamsProxy into KCL worker when using DynamoDB Streams Kinesis Adapter v1.4.0.
+To fix high propagation delay problems, opt-into using DynamoDBStreamsProxy (instead of the default KinesisProxy) by using the StreamsWorkerFactory factory method (shown below). This injects an instance of DynamoDBStreamsProxy into the created KCL worker.
+```
+       final Worker worker = StreamsWorkerFactory
+           .createDynamoDbStreamsWorker(
+               recordProcessorFactory,
+               workerConfig,
+               adapterClient,
+               amazonDynamoDB,
+               amazonCloudWatchClient);
+```
+
 ## Getting Started
 
 1. **Sign up for AWS** - Before you begin, you need an AWS account. Please see the [AWS Account and Credentials][docs-signup] section of the developer guide for information about how to create an AWS account and retrieve your AWS credentials. You don’t need this if you’re using DynamoDB Local.
-1. **Minimum requirements** - To run the SDK you will need **Java 1.7+**. For more information about the requirements and optimum settings for the SDK, please see the [Java Development Environment][docs-signup] section of the developer guide.
+1. **Minimum requirements** - To run the SDK you will need **Java 1.8+**. For more information about the requirements and optimum settings for the SDK, please see the [Java Development Environment][docs-signup] section of the developer guide.
 1. **Install the DynamoDB Streams Kinesis Adapter** - Using ***Maven*** is the recommended way to install the DynamoDB Streams Kinesis Adapter and its dependencies, including the AWS SDK for Java.  To download the code from GitHub, simply clone the repository by typing: `git clone https://github.com/awslabs/dynamodb-streams-kinesis-adapter.git`, and run the Maven command described below in "Building From Source". You may also depend on the maven artifact [com.amazonaws:dynamodb-streams-kinesis-adapter][adapter-maven].
 1. **Build your first application** - There is a walkthrough to help you build first application using this adapter. Please see [Using the DynamoDB Streams Kinesis Adapter to Process Stream Records][docs-adapter].
 
@@ -32,7 +47,7 @@ Add the following to your Maven pom file:
 <dependency>
     <groupId>com.amazonaws</groupId>
     <artifactId>dynamodb-streams-kinesis-adapter</artifactId>
-    <version>1.2.2</version>
+    <version>1.4.0</version>
 </dependency>
 ```
 

diff --git a/pom.xml b/pom.xml
@@ -6,7 +6,7 @@
     <artifactId>dynamodb-streams-kinesis-adapter</artifactId>
     <packaging>jar</packaging>
     <name>DynamoDB Streams Adapter for Java</name>
-    <version>1.2.2</version>
+    <version>1.4.0</version>
     <description>The DynamoDB Streams Adapter implements the AmazonKinesis interface so that your application can use KCL to consume and process data from a DynamoDB stream.</description>
     <url>https://aws.amazon.com/dynamodb</url>
 
@@ -23,8 +23,8 @@
     </licenses>
 
     <properties>
-        <aws-java-sdk.version>[1.11.115, 2.0.0)</aws-java-sdk.version>
-        <amazon-kinesis-client.version>[1.7.5, 1.8.0)</amazon-kinesis-client.version>
+        <aws-java-sdk.version>1.11.218</aws-java-sdk.version>
+        <amazon-kinesis-client.version>1.9.0</amazon-kinesis-client.version>
         <powermock.version>1.6.2</powermock.version>
         <aws.dynamodblocal.version>1.11.86</aws.dynamodblocal.version>
         <maven.dependency.version>3.0.0</maven.dependency.version>
@@ -76,6 +76,13 @@
             <scope>test</scope>
         </dependency>
 
+        <dependency>
+            <groupId>org.hamcrest</groupId>
+            <artifactId>hamcrest-all</artifactId>
+            <version>1.3</version>
+            <scope>test</scope>
+        </dependency>
+
         <dependency>
             <groupId>org.powermock</groupId>
             <artifactId>powermock-module-junit4</artifactId>
@@ -136,8 +143,8 @@
                     <groupId>org.apache.maven.plugins</groupId>
                     <artifactId>maven-compiler-plugin</artifactId>
                     <configuration>
-                        <source>1.7</source>
-                        <target>1.7</target>
+                        <source>1.8</source>
+                        <target>1.8</target>
                         <encoding>UTF-8</encoding>
                     </configuration>
                 </plugin>