亚马逊DynamoDB本地 - 未知错误、异常或失败。

Question

我需要测试一个非常依赖于Amazon的DynamoDB的应用程序。为了使测试能够分开运行，我选择了DynamoDB Local .jar。我已经知道最近的更新，让我们能够在没有外部bash命令调用的情况下运行它。然而，当我尝试运行这里指定的示例（here）时，我遇到了以下堆栈跟踪：

Exception in thread "main" com.amazonaws.AmazonServiceException: The request processing has failed because of an unknown error, exception or failure. (Service: AmazonDynamoDBv2; Status Code: 500; Error Code: InternalFailure; Request ID: cab7a550-aaa6-4bfe-a591-0b255481cc14)
    at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1275)
    at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:873)
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:576)
    at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:362)
    at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:328)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:307)
    at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:1805)
    at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.listTables(AmazonDynamoDBClient.java:1223)
    at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.listTables(AmazonDynamoDBClient.java:1235)

这是我正在尝试运行的代码：

public static void main( String[] args ) throws Exception
    {
        AmazonDynamoDB dynamodb = null;

        DynamoDBProxyServer server = null;
        final String[] localArgs = { "-inMemory", "-port", "13005" };
        server = ServerRunner.createServerFromCommandLineArgs(localArgs);
        server.start();

        BasicAWSCredentials auth = new BasicAWSCredentials("key", "secret");
        dynamodb = new AmazonDynamoDBClient(auth);
        dynamodb.setEndpoint("http://127.0.0.1:13005");

        // use the DynamoDB API over HTTP
        System.out.println(dynamodb.listTables());

        // Stop the DynamoDB Local endpoint
        if(server != null) {
            server.stop();
        }
    }

我观察到，如果我尝试从Java程序本身完全运行它，就会抛出异常，并且指定的端口将不再可用（会抛出错误指出该端口已被占用）。但是，如果我从命令提示符中启动DynamoDB Local并仅使用Java程序作为访问客户端，则一切都正常运行。

有什么建议吗？

- user2403149

---

这是在 Mac OS X 上吗？如果是，那可能会有一个特定的问题。 - Jayson Minard

无论如何，我认为我现在在下面的答案中已经找到了通往成功的完整路径。 呜呜呜，那真是太痛苦了。 - Jayson Minard

你可能需要在根目录下或使用 sudo 命令启动本地 DynamoDB。 - Brilliantree

Answer 1

DynamoDBLocal存在至少2个问题。 解决这两个问题，您将运行嵌入式DynamoDB。

首先，-port参数不起作用。 因此，Jetty未在您期望的端口上设置。 相反，默认端口被设置为51205（或随机端口？）。

以下是我启动服务器的代码，避免使用内置的命令行解析，这种方式更好...以这种方式启动服务器后，http://localhost:19444/shell 可用，所以Jetty良好。 但是，然后您可能会遇到Sqlite4java的另一个问题（请参见代码块之后的内容）。

注意：代码是Kotlin，但Java非常相似。 此外，我已经配置了SLF4j，并且不让ServerRunner类破坏日志记录，因此这是启动服务器的更好方法，也是ServerRunner内部执行的方式。

class TestAccountManager {
    companion object {
        private val localDbPort = 19444

        private lateinit var localDb: DynamoDBProxyServer
        private lateinit var dbClient: AmazonDynamoDBClient

        @BeforeClass @JvmStatic fun setup() {
            System.setProperty("org.eclipse.jetty.util.log.class", "org.eclipse.jetty.util.log.Slf4jLog")
            localDb = DynamoDBProxyServer(localDbPort, LocalDynamoDBServerHandler(
                    LocalDynamoDBRequestHandler(0, true, null, true, true), null)
            )
            localDb.start()

            val auth = BasicAWSCredentials("fakeKey", "fakeSecret")
            dbClient = AmazonDynamoDBClient(auth)
            dbClient.signerRegionOverride = "us-east-1"
            dbClient.setEndpoint("http://localhost:$localDbPort")
        }

        @AfterClass @JvmStatic fun teardown() {
           localDb.stop()
        }
    }

    @Test fun testSomething() {
        dbClient.listTables().tableNames.forEach {
            println(it)
        }
    }
}

一旦您拥有了这段代码，您现在正在使用预期的端口运行。

现在您可能会遇到第二个错误，例如Sqlite4java未找到其正确平台的二进制文件。对于某些版本的Mac OSX，它将生成一个实际上不存在的二进制文件名。而DynamoDBLocal强制隐藏所有Sqlite4java日志（不可能覆盖它），因此您将无法看到它。

您可以通过下载分发包、解压缩并运行来测试sqlite库：

java -jar sqlite4java-1.0.392.jar -d

它将报告尝试加载的内容以及成功或失败的情况。您只需要从Gradle、Maven或您放置它的任何位置在系统上找到该JAR文件即可。我的输出结果带有错误信息：

sqlite4java 392
160212:002049.833 FINE [sqlite] Internal: loading library
160212:002049.834 FINE [sqlite] Internal: java.library.path=/Users/jminard/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
160212:002049.834 FINE [sqlite] Internal: sqlite4java.library.path=null
160212:002049.834 FINE [sqlite] Internal: cwd=/Users/jminard/DEV/Collokia/repos/collokia-web-back/.
160212:002049.834 FINE [sqlite] Internal: default path=/Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d
160212:002049.834 FINE [sqlite] Internal: forced path=null
160212:002049.834 FINE [sqlite] Internal: os.name=mac os x; os=osx
160212:002049.835 FINE [sqlite] Internal: os.arch=x86_64
160212:002049.835 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-x86_64-1.0.392.dylib
160212:002049.835 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-amd64-1.0.392.dylib
160212:002049.835 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-1.0.392.dylib
160212:002049.835 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-1.0.392.dylib
160212:002049.836 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-x86_64-d-1.0.392.dylib
160212:002049.836 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-amd64-d-1.0.392.dylib
160212:002049.836 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-d-1.0.392.dylib
160212:002049.836 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-d-1.0.392.dylib
160212:002049.836 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-x86_64.dylib
160212:002049.836 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-amd64.dylib
160212:002049.836 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx.dylib
160212:002049.837 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java.dylib
160212:002049.837 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-x86_64-d.dylib
160212:002049.837 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-amd64-d.dylib
160212:002049.837 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-d.dylib
160212:002049.837 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-d.dylib
160212:002049.837 FINE [sqlite] Internal: trying to load sqlite4java-osx-x86_64-1.0.392
160212:002049.838 FINE [sqlite] Internal: cannot load sqlite4java-osx-x86_64-1.0.392: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-x86_64-1.0.392 in java.library.path
160212:002049.838 FINE [sqlite] Internal: trying to load sqlite4java-osx-amd64-1.0.392
160212:002049.839 FINE [sqlite] Internal: cannot load sqlite4java-osx-amd64-1.0.392: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-amd64-1.0.392 in java.library.path
160212:002049.839 FINE [sqlite] Internal: trying to load sqlite4java-osx-1.0.392
160212:002049.840 FINE [sqlite] Internal: cannot load sqlite4java-osx-1.0.392: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-1.0.392 in java.library.path
160212:002049.840 FINE [sqlite] Internal: trying to load sqlite4java-1.0.392
160212:002049.841 FINE [sqlite] Internal: cannot load sqlite4java-1.0.392: java.lang.UnsatisfiedLinkError: no sqlite4java-1.0.392 in java.library.path
160212:002049.841 FINE [sqlite] Internal: trying to load sqlite4java-osx-x86_64-d-1.0.392
160212:002049.842 FINE [sqlite] Internal: cannot load sqlite4java-osx-x86_64-d-1.0.392: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-x86_64-d-1.0.392 in java.library.path
160212:002049.842 FINE [sqlite] Internal: trying to load sqlite4java-osx-amd64-d-1.0.392
160212:002049.842 FINE [sqlite] Internal: cannot load sqlite4java-osx-amd64-d-1.0.392: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-amd64-d-1.0.392 in java.library.path
160212:002049.843 FINE [sqlite] Internal: trying to load sqlite4java-osx-d-1.0.392
160212:002049.843 FINE [sqlite] Internal: cannot load sqlite4java-osx-d-1.0.392: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-d-1.0.392 in java.library.path
160212:002049.843 FINE [sqlite] Internal: trying to load sqlite4java-d-1.0.392
160212:002049.844 FINE [sqlite] Internal: cannot load sqlite4java-d-1.0.392: java.lang.UnsatisfiedLinkError: no sqlite4java-d-1.0.392 in java.library.path
160212:002049.844 FINE [sqlite] Internal: trying to load sqlite4java-osx-x86_64
160212:002049.845 FINE [sqlite] Internal: cannot load sqlite4java-osx-x86_64: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-x86_64 in java.library.path
160212:002049.845 FINE [sqlite] Internal: trying to load sqlite4java-osx-amd64
160212:002049.845 FINE [sqlite] Internal: cannot load sqlite4java-osx-amd64: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-amd64 in java.library.path
160212:002049.845 FINE [sqlite] Internal: trying to load sqlite4java-osx
160212:002049.846 FINE [sqlite] Internal: cannot load sqlite4java-osx: java.lang.UnsatisfiedLinkError: no sqlite4java-osx in java.library.path
160212:002049.846 FINE [sqlite] Internal: trying to load sqlite4java
160212:002049.847 FINE [sqlite] Internal: cannot load sqlite4java: java.lang.UnsatisfiedLinkError: no sqlite4java in java.library.path
160212:002049.847 FINE [sqlite] Internal: trying to load sqlite4java-osx-x86_64-d
160212:002049.848 FINE [sqlite] Internal: cannot load sqlite4java-osx-x86_64-d: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-x86_64-d in java.library.path
160212:002049.848 FINE [sqlite] Internal: trying to load sqlite4java-osx-amd64-d
160212:002049.849 FINE [sqlite] Internal: cannot load sqlite4java-osx-amd64-d: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-amd64-d in java.library.path
160212:002049.849 FINE [sqlite] Internal: trying to load sqlite4java-osx-d
160212:002049.849 FINE [sqlite] Internal: cannot load sqlite4java-osx-d: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-d in java.library.path
160212:002049.850 FINE [sqlite] Internal: trying to load sqlite4java-d
160212:002049.850 FINE [sqlite] Internal: cannot load sqlite4java-d: java.lang.UnsatisfiedLinkError: no sqlite4java-d in java.library.path
Error: cannot load SQLite
java.lang.UnsatisfiedLinkError: no sqlite4java-osx-x86_64-1.0.392 in java.library.path
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1864)
    at java.lang.Runtime.loadLibrary0(Runtime.java:870)
    at java.lang.System.loadLibrary(System.java:1122)
    at com.almworks.sqlite4java.Internal.tryLoadFromSystemPath(Internal.java:352)
    at com.almworks.sqlite4java.Internal.loadLibraryX(Internal.java:124)
    at com.almworks.sqlite4java.SQLite.main(SQLite.java:368)

如果你遇到动态库加载错误，请参考以下内容进行解决：

• UnsatisfiedLinkError with sqlite4java Jar on Mac OS X NetBeans
• UnsatisfiedLinkError with sqlite4java Jar on Mac OS X
• 并特别与此相关的DynamoDBLocal：在Mac OS X上使用java命令运行DynamoDB Local
• https://groups.google.com/forum/#!topic/sqlite4java/9J1lmCuoKLA
• https://groups.google.com/forum/#!topic/sqlite4java/jhwt44nYGvw

唯一可靠的解决方案是：

• 将从下载的sqlite4java分发版中获取的库添加到已知位置，并设置系统属性java.library.path
• 将从下载的sqlite4java分发版中获取的库添加到/Library/Java/Extensions（仅适用于macOS）

我个人更倾向于采用第一种方法，将库添加到项目中，并确保构建时添加了-Djava.library.path=./lib/sqlite4java，其中动态库被解压缩。为了进行更加可靠的测试，可以使用这个技巧在代码中以编程方式设置java.library.path（否则在代码中设置时会被忽略）：http://blog.cedarsoft.com/2010/11/setting-java-library-path-programmatically/

DynamoDBLocal隐藏所有Sqlite错误是非常不友好的，因此必须进行调试以查找悄无声息的故障。该库充满了各种调整日志级别的东西，使得调试变得困难，因为它们破坏了您查看错误的能力。例如，每次打开SqlLite文件时（类SQLiteDBAccess），都会发生这样的情况：

LocalDBUtils.setLog4jToUtilsLogging("com.almworks.sqlite4java");
LocalDBUtils.setLog4jToUtilsLogging("com.almworks.sqlite4java.Internal");
java.util.logging.Logger.getLogger("com.almworks.sqlite4java").setLevel(Level.OFF);
java.util.logging.Logger.getLogger("com.almworks.sqlite4java.Internal").setLevel(Level.OFF);

结束语：我正在寻找替代方案，这并不是地球上最好的东西，看DynamoDbLocal的代码和做出的决定让我对它没有信心。 Alternator或Jcabi-Dynamo Mock是我接下来要考虑的。

- Jayson Minard

---

我一段时间前尝试过 Alternator -- 我非常喜欢在同一JVM中运行DynamoDB模拟以加速我的测试的想法。但是我发现了许多边缘情况，这些情况在真正的DynamoDB中是被支持的，但在Alternator中失败了。再次强调，这是很久以前的事情，请分享您的好经验：）。现在，既然你正在寻找替代方案，我曾经使用 Docker 成功地运行 DynamoDB Local（即“官方”版本）进行一个小项目。我在Docker Hub上找到了可用的镜像，设置起来非常简单。 - Bruno Reis

@BrunoReis 很好，目前我正在使用上述的官方 DynamoDbLocal 嵌入式，并等待它失败后再尝试其他方法。 - Jayson Minard

在Mac上，将dylib保存到/Library/Java/Extensions中就可以了！ - Nic Cottrell

SQLite扩展已经包含在Gradle中的DynamoDB模拟器依赖项中。更多信息请参见我的答案这里。 - forresthopkinsa

Answer 2

2018年8月，亚马逊宣布推出新的Docker镜像，其中搭载了Amazon DynamoDB本地版本。它不需要下载和运行任何JAR文件，也不需要添加使用第三方特定于操作系统的二进制文件（我指的是sqlite4java）。

只需在测试之前启动Docker容器即可，非常简单：

docker run -p 8000:8000 amazon/dynamodb-local

您可以通过上面描述的方式手动进行本地开发。IDE通常提供在执行任务之前运行任意命令的方法，因此您可以使IDE为您启动容器。

或者您可以在CI流水线中使用它。许多CI服务提供在流水线期间启动附加容器以为您的测试提供依赖项的能力。以下是Gitlab CI/CD的示例：

test:
  stage: test
  image: openjdk:8-alpine
  services:
    - name: amazon/dynamodb-local
      alias: dynamodb-local
  script:
    - DYNAMODB_LOCAL_URL=http://dynamodb-local:8000 ./gradlew clean test

或者是 Bitbucket Pipelines：

definitions:
  services:
    dynamodb-local:
      image: amazon/dynamodb-local
…
step:
  name: test
  image:
    name: openjdk:8-alpine
  services:
    - dynamodb-local
  script:
    - DYNAMODB_LOCAL_URL=http://localhost:8000 ./gradlew clean test

- madhead