Clojure Cassandra 驱动性能

Question

Clojure Cassandra 驱动性能

3

我正在测试在一个6节点的Cassandra集群上，使用Clojure中Alia的性能。即使多线程，我最多只能获得约400个写操作/秒。但是使用Firebrand Cassandra驱动程序并手动处理Java线程，我们能够使用96个线程获得5000个写操作/秒。

我在使用agents时有什么错误吗？这台机器的CPU使用率只有约25％，这似乎非常低。

更新：根据Alia的作者建议，改用预编译语句而不是原始语句，以同步单线程方式实现了高达2500/秒的增益。我仍需要通过Clojure进行多线程测试，并分别利用Alia /底层Java驱动程序中内置的async功能来查看哪个更快。

更新2：通过额外利用驱动程序中内置的async功能，我现在看到类似于mpenet下面的结果。

(ns alia-perf-test.core
  (:gen-class)
  (:require [qbits.alia :as alia]
            [qbits.hayt :as hayt]))

(defn exec-query [session query]
  (alia/execute session (hayt/->raw query)))

(defmacro time-query
  [expr]
  `(let [start# (. System (nanoTime))
         ret# ~expr]
     (/ (double (- (. System (nanoTime)) start#)) 1000000.0)))

(defn write-entity
  [total-time session entity]
  (let [query (hayt/->raw (hayt/insert :entities (hayt/values entity) (hayt/using :timestamp 1234)))
        query-time (time-query (alia/execute session query))]
      (+ total-time query-time)))

(defn generate-entity []
  {:id (str (java.util.UUID/randomUUID)) :num 0})

(defn write-something
  [write-agent session]
  (send-off write-agent write-entity
        session 
        (generate-entity)))

(defn -main [& args]
  (let [cluster (alia/cluster ["server1"
                               "server2" 
                               "server3" 
                               "server4" 
                               "server5" 
                               "server6"]
                              :pooling-options {:core-connections-per-host [:local 16 :remote 16]
                                                :max-connections-per-host  [:local 1000 :remote 1000]
                                                :max-simultaneous-requests-per-connection [:local 32 :remote 32]
                                                :min-simultaneous-requests-per-connection [:local 16 :remote 16]})
        session (alia/connect cluster)]
    (alia/set-consistency! :any)
    (exec-query session (hayt/create-keyspace :aliaperftest
                                              (hayt/with {:replication
                                                          {:class "NetworkTopologyStrategy"
                                                           :dc1 3 :dc2 3}})))
    (exec-query session (hayt/use-keyspace :aliaperftest))
    (exec-query session (hayt/create-table :entities
                                           (hayt/column-definitions {:id :varchar
                                                                     :num :int
                                                                     :primary-key [:id]})))
    (let [num-entities 10000
          write-agent (agent 0)]
      (dotimes [n num-entities]
        (write-something write-agent session))
      (await write-agent)
      (println "Wrote" num-entities "entities in" @write-agent "ms -"
           (* (/ num-entities @write-agent) 1000.0) "ops/sec"))

    (exec-query session (hayt/drop-table :entities))
    (exec-query session (hayt/drop-keyspace :aliaperftest))
    (alia/shutdown session)
    (alia/shutdown cluster)
    (shutdown-agents)))

- Daniel Gibbons

1个回答

网页内容由stack overflow 提供, 点击上面的

可以查看英文原文，
原文链接

- mpenet · Answer 1

更新：我已经在异步模式下（使用alia的成功处理程序，使用单个节点）将其设为进行100,000个请求，使用原子接收结果并等待所有响应到达，大约需要8秒钟时间。

根据您的设置和批处理，使用自定义执行程序也可能会将其性能提升得更高，但我没有深入研究过。请参见：https://github.com/mpenet/alia/blob/master/docs/guide.md#executors。