LoginSignup
2
2

More than 5 years have passed since last update.

@akishin(Shin Akiyama)

Finagle から Solr 6 へアクセスしてみる

Last updated at Posted at 2016-05-08

Solr 6.0.0 のインストール で作成した検証環境に Finagle からアクセスしてみた。

Solrj を使う

取り敢えず Java 製の公式クライアントである Solrj を使ってみる。
Solrj にはまだ非同期 I/O の API が無いっぽい? のでそこで詰まるかも知れない。

build.sbt は以下のような内容。

build.sbt
name := "finagle-solr-ex"

version := "1.0"

scalaVersion := "2.11.8"

resolvers += "twitter-repo" at "https://maven.twttr.com"

libraryDependencies ++= Seq(
  "com.twitter" %% "finagle-http" % "6.35.+",
  "com.twitter" %% "twitter-server" % "1.20.+",
  "org.apache.solr" % "solr-solrj" % "6.0.0"
)

コードは以下。

HTTPServer.scala
import com.twitter.finagle.http.{Request, Response}
import com.twitter.finagle.{Service, Http}
import com.twitter.logging.Logger
import com.twitter.server.TwitterServer
import com.twitter.util.{FuturePool, Future, Await}
import org.apache.solr.client.solrj.SolrQuery
import org.apache.solr.client.solrj.impl.HttpSolrClient

class SolrService extends Service[Request, Response] {

  private[this] val log = Logger.get(getClass)
  private[this] val solrServer = "http://192.168.33.98:8983/solr/mycore"
  private[this] val solrClient = new HttpSolrClient(solrServer)
  private[this] val query = new SolrQuery("*:*")

  override def apply(request: Request): Future[Response] = FuturePool.unboundedPool {
    log.info("request received.")
    val queryResponse = solrClient.query(query)
    val response = Response()
    response.setContentString(queryResponse.getResults.toString)
    response
  }
}

object HTTPServer extends TwitterServer {

  def main:Unit = {
    val service = new SolrService

    val server = Http.serve(":8080", service)
    onExit {
      log.info("close http server")
      service.close()
      server.close()
    }
    Await.ready(server)
  }
}

sbt run して ab で適当にリクエストを発行。

% ab -n 10000 -c 10 http://localhost:8080/
This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        
Server Hostname:        localhost
Server Port:            8080

Document Path:          /
Document Length:        3212 bytes

Concurrency Level:      10
Time taken for tests:   15.684 seconds
Complete requests:      10000
Failed requests:        0
Total transferred:      32530000 bytes
HTML transferred:       32120000 bytes
Requests per second:    637.58 [#/sec] (mean)
Time per request:       15.684 [ms] (mean)
Time per request:       1.568 [ms] (mean, across all concurrent requests)
Transfer rate:          2025.45 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       1
Processing:     4   16  16.2     11     557
Waiting:        4   15  15.9     11     557
Total:          4   16  16.2     11     557

Percentage of the requests served within a certain time (ms)
  50%     11
  66%     15
  75%     18
  80%     21
  90%     30
  95%     39
  98%     54
  99%     67
 100%    557 (longest request)

VisualVM だと以下のような感じ。

WS000089.jpg

Finagle の Service を使う

Finagle の Service を使って非同期に Solr に HTTP アクセスしたらどうなるか見てみる。

ちなみに Scala 製で非同期API がある Solr クライアントとしては solr-scala-client や solrs があるが、どちらも使用している Async Http Client のバージョンが古く、その為 Finagle が使用している Netty とコンフリクトする模様。

Scala 力が低い事もありこういった場合の回避方法があるのかどうかもよく判らなかったので、直接 Solr に HTTP リクエストを送ってみる事にした。

build.sbt は以下のような内容。

build.sbt
name := "finagle-solr-ex2"

version := "1.0"

scalaVersion := "2.11.8"

resolvers += "twitter-repo" at "https://maven.twttr.com"

libraryDependencies ++= Seq(
  "com.twitter" %% "finagle-http" % "6.35.+",
  "com.twitter" %% "twitter-server" % "1.20.+"
)

コードは以下。

HTTPServer.scala
import com.twitter.finagle.http.{RequestBuilder, Request, Response}
import com.twitter.finagle.{Service, Http}
import com.twitter.logging.Logger
import com.twitter.server.TwitterServer
import com.twitter.util.{FuturePool, Future, Await}

class SolrService extends Service[Request, Response] {

  private[this] val log = Logger.get(getClass)
  private[this] val solrServer = "http://192.168.33.98:8983/solr/mycore"
  private[this] val client: Service[Request, Response] = Http.client.newService("192.168.33.98:8983")
  private[this] val solrRequest = RequestBuilder.safeBuildGet(RequestBuilder.create().url(s"${solrServer}/select?q=*:*&wt=json"))

  override def apply(request: Request): Future[Response] = FuturePool.unboundedPool {
    log.info("request received.")
    val f = client(solrRequest)
    Await.result(f)
  }
}

object HTTPServer extends TwitterServer {

  def main:Unit = {
    val service = new SolrService

    val server = Http.serve(":8080", service)
    onExit {
      log.info("close http server")
      service.close()
      server.close()
    }
    Await.ready(server)
  }
}

先ほどと同様に sbt run して ab で適当にリクエストを発行。

% ab -n 10000 -c 10 http://localhost:8080/
This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        
Server Hostname:        localhost
Server Port:            8080

Document Path:          /
Document Length:        3361 bytes

Concurrency Level:      10
Time taken for tests:   15.392 seconds
Complete requests:      10000
Failed requests:        19
   (Connect: 0, Receive: 0, Length: 19, Exceptions: 0)
Total transferred:      34420019 bytes
HTML transferred:       33610019 bytes
Requests per second:    649.70 [#/sec] (mean)
Time per request:       15.392 [ms] (mean)
Time per request:       1.539 [ms] (mean, across all concurrent requests)
Transfer rate:          2183.87 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       1
Processing:     3   15  14.6     12     254
Waiting:        3   15  14.5     12     253
Total:          3   15  14.6     12     254

Percentage of the requests served within a certain time (ms)
  50%     12
  66%     16
  75%     19
  80%     21
  90%     27
  95%     33
  98%     43
  99%     53
 100%    254 (longest request)

VisualVM だと以下のような感じ。

WS000090.jpg

結論

結果からみるとどちらもそこまで大きな差はないっぽい？
サンプルが単純だからなのか、ライブラリの使い方が間違っているのか、検証方法が間違ってるのか、それともこういうもんなのか。
unboundedPool 使ってるとそこまで差は出なかったりする、とかか？

さて、どうしたもんかなー・・・。

気になったので以下ちょっと追記。

Solrj(FuturePool.unboundedPool 無し ver.)

試す順番が逆のような気もするが FuturePool.unboundedPool 無しのバージョンを試してみる。
まずは Solrj。

HTTPServer.scala
import com.twitter.finagle.http.{Request, Response}
import com.twitter.finagle.{Http, Service}
import com.twitter.logging.Logger
import com.twitter.server.TwitterServer
import com.twitter.util.{Await, Future}
import org.apache.solr.client.solrj.SolrQuery
import org.apache.solr.client.solrj.impl.HttpSolrClient

class SolrService extends Service[Request, Response] {

  private[this] val log = Logger.get(getClass)
  private[this] val solrServer = "http://192.168.33.98:8983/solr/mycore"
  private[this] val solrClient = new HttpSolrClient(solrServer)
  private[this] val query = new SolrQuery("*:*")

  override def apply(request: Request): Future[Response] = {
    log.info("request received.")
    val queryResponse = solrClient.query(query)
    val response = Response()
    response.setContentString(queryResponse.getResults.toString)
    Future.value(response)
  }
}

object HTTPServer extends TwitterServer {

  def main:Unit = {
    val service = new SolrService

    val server = Http.serve(":8080", service)
    onExit {
      log.info("close http server")
      service.close()
      server.close()
    }
    Await.ready(server)
  }
}

実行してみる。

% ab -n 10000 -c 10 http://localhost:8080/
This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        
Server Hostname:        localhost
Server Port:            8080

Document Path:          /
Document Length:        3212 bytes

Concurrency Level:      10
Time taken for tests:   18.388 seconds
Complete requests:      10000
Failed requests:        0
Total transferred:      32530000 bytes
HTML transferred:       32120000 bytes
Requests per second:    543.82 [#/sec] (mean)
Time per request:       18.388 [ms] (mean)
Time per request:       1.839 [ms] (mean, across all concurrent requests)
Transfer rate:          1727.60 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       2
Processing:     3   18  20.1     13     472
Waiting:        3   18  20.0     13     472
Total:          3   18  20.1     14     472

Percentage of the requests served within a certain time (ms)
  50%     14
  66%     20
  75%     24
  80%     28
  90%     37
  95%     46
  98%     59
  99%     69
 100%    472 (longest request)

何度か試してみたが、こちらは FuturePool.unboundedPool を使った時よりパフォーマンスが悪くなった。
やはり Solrj がブロックするからか。

Finagle の Service(FuturePool.unboundedPool 無し ver.)

続いて Service 版。
FuturePool.unboundedPool を使うと Await.ready で待たないといけない（待たないで FuturePool 経由で Service を実行する方法が分からない・・・) ので、こちらは速くなるのではないかと予想。

HTTPServer.scala
import com.twitter.finagle.http.{Request, RequestBuilder, Response}
import com.twitter.finagle.{Http, Service}
import com.twitter.logging.Logger
import com.twitter.server.TwitterServer
import com.twitter.util.{Await, Future}

class SolrService extends Service[Request, Response] {

  private[this] val log = Logger.get(getClass)
  private[this] val solrServer = "http://192.168.33.98:8983/solr/mycore"
  private[this] val client: Service[Request, Response] = Http.client.newService("192.168.33.98:8983")
  private[this] val solrRequest = RequestBuilder.safeBuildGet(RequestBuilder.create().url(s"${solrServer}/select?q=*:*&wt=json"))

  override def apply(request: Request): Future[Response] = {
    log.info("request received.")
    client(solrRequest)
  }
}

object HTTPServer extends TwitterServer {

  def main:Unit = {
    val service = new SolrService

    val server = Http.serve(":8080", service)
    onExit {
      log.info("close http server")
      service.close()
      server.close()
    }
    Await.ready(server)
  }
}

実行。

% ab -n 10000 -c 10 http://localhost:8080/
This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        
Server Hostname:        localhost
Server Port:            8080

Document Path:          /
Document Length:        3361 bytes

Concurrency Level:      10
Time taken for tests:   14.566 seconds
Complete requests:      10000
Failed requests:        9
   (Connect: 0, Receive: 0, Length: 9, Exceptions: 0)
Total transferred:      34420009 bytes
HTML transferred:       33610009 bytes
Requests per second:    686.55 [#/sec] (mean)
Time per request:       14.566 [ms] (mean)
Time per request:       1.457 [ms] (mean, across all concurrent requests)
Transfer rate:          2307.71 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       1
Processing:     3   14  16.0     12     486
Waiting:        3   14  15.9     12     485
Total:          3   15  16.0     12     486

Percentage of the requests served within a certain time (ms)
  50%     12
  66%     16
  75%     18
  80%     20
  90%     25
  95%     30
  98%     37
  99%     44
 100%    486 (longest request)

そこまで変わらないけど何度か試すと毎回 FuturePool.unboundedPool 利用版よりはやや速い結果となった。
ちょっと速くなったと言えるのかも？
FuturePool.unboundedPool を使わない Solrj よりは確実に速い。

しかし この辺 見る限り Service も FuturePool 経由で実行されている訳でもないっぽい（？）し、Await せずに FuturePool.unboundedPool 経由で実行できればもっと速くなりそうな気もするんだけどそんな事ないのだろうか？
やはり何か根本的に使い方間違ってんのかなー。

2
2
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
2
2