Help us understand the problem. What is going on with this article?

cassandra-loaderでbulk load

More than 5 years have passed since last update.

install

git clone https://github.com/brianmhess/cassandra-loader.git
cd cassandra-load
gradle loader

load動作確認

make_test_csv.sh
#/bin/sh
for i in $(seq 1 100000) ; do
   echo "$i,$((i * 10))"
done
make_test_csv.sh > test.csv
cat test.csv
>1,10
>2,20
>3,30
>....

bulk insert

build/cassandra-loader -f test.csv -schema "test_from_spark.fun(k,v)" -host 127.0.0.1

cqlshで確認

cqlsh

cqlsh:test_from_spark> select * from fun limit 10;

 k    | v
------+-------
 4317 | 43170
 3372 | 33720
 1584 | 15840
 7034 | 70340
 9892 | 98920
 9640 | 96400
 9067 | 90670
 4830 | 48300
 2731 | 27310
 5056 | 50560

(10 rows)

spark-shell上で確認

scala> val t = sc.cassandraTable("test_from_spark", "fun");
t: com.datastax.spark.connector.rdd.CassandraTableScanRDD[com.datastax.spark.connector.CassandraRow] = CassandraTableScanRDD[3] at RDD at CassandraRDD.scala:15

scala> t.count
res6: Long = 10000


scala> t.collect.slice(1, 10).foreach(println)
CassandraRow{k: 9067, v: 90670}
CassandraRow{k: 4830, v: 48300}
CassandraRow{k: 2731, v: 27310}
CassandraRow{k: 5056, v: 50560}
CassandraRow{k: 6428, v: 64280}
CassandraRow{k: 2713, v: 27130}
CassandraRow{k: 769, v: 7690}
CassandraRow{k: 9973, v: 99730}
CassandraRow{k: 1863, v: 18630}


scala> t.take(10).foreach(println)
CassandraRow{k: 9640, v: 96400}
CassandraRow{k: 9067, v: 90670}
CassandraRow{k: 4830, v: 48300}
CassandraRow{k: 2731, v: 27310}
CassandraRow{k: 5056, v: 50560}
CassandraRow{k: 6428, v: 64280}
CassandraRow{k: 2713, v: 27130}
CassandraRow{k: 769, v: 7690}
CassandraRow{k: 9973, v: 99730}
CassandraRow{k: 1863, v: 18630}
Why not register and get more from Qiita?
  1. We will deliver articles that match you
    By following users and tags, you can catch up information on technical fields that you are interested in as a whole
  2. you can read useful information later efficiently
    By "stocking" the articles you like, you can search right away