GoogleのBigTableの概念を参考に作られたKeyValue型データストアであるLevelDBを、macのruby環境にインストールする方法について説明します。この記事では、leveldb-rubyではなく、leveldb-native(C++実装)について触れています。
いつもながら素人記事ですので、コメントありましたらぜひお寄せください！
実行環境
- 実行日: 2019-12-25
- OS version: MacOS Mojave 10.14.6
- Ruby version: 2.6.3
- leveldb version: 0.6
事象
上記実行環境でleveldb-nativeをgemでインストールしようとすると、次のようなコンパイルエラーが出て、インストールできません。
[ 19-12-24 19:34 ] ~/workspace/bitcoinrb
osada@mbp17e% gem install leveldb-native
Building native extensions. This could take a while...
ERROR: Error installing leveldb-native:
ERROR: Failed to build gem native extension.
current directory: /Users/osada/.rvm/gems/ruby-2.6.3@bitcoinrb/gems/leveldb-native-0.6/ext/leveldb-native
/Users/osada/.rvm/rubies/ruby-2.6.3/bin/ruby -I /Users/osada/.rvm/rubies/ruby-2.6.3/lib/ruby/site_ruby/2.6.0 -r ./siteconf20191224-34602-37ap6f.rb extconf.rb
checking for -lleveldb... yes
creating Makefile
current directory: /Users/osada/.rvm/gems/ruby-2.6.3@bitcoinrb/gems/leveldb-native-0.6/ext/leveldb-native
make "DESTDIR=" clean
current directory: /Users/osada/.rvm/gems/ruby-2.6.3@bitcoinrb/gems/leveldb-native-0.6/ext/leveldb-native
make "DESTDIR="
compiling leveldb-native.cc
In file included from leveldb-native.cc:4:
In file included from /usr/local/include/leveldb/db.h:12:
In file included from /usr/local/include/leveldb/iterator.h:19:
/usr/local/include/leveldb/slice.h:43:25: warning: defaulted function definitions are a C++11 extension [-Wc++11-extensions]
Slice(const Slice&) = default;
^
/usr/local/include/leveldb/slice.h:44:36: warning: defaulted function definitions are a C++11 extension [-Wc++11-extensions]
Slice& operator=(const Slice&) = default;
^
In file included from leveldb-native.cc:4:
In file included from /usr/local/include/leveldb/db.h:12:
In file included from /usr/local/include/leveldb/iterator.h:20:
/usr/local/include/leveldb/status.h:27:11: error: expected ';' at end of declaration list
Status() noexcept : state_(nullptr) {}
^
;
/usr/local/include/leveldb/status.h:33:16: warning: rvalue references are a C++11 extension [-Wc++11-extensions]
Status(Status&& rhs) noexcept : state_(rhs.state_) { rhs.state_ = nullptr; }
^
/usr/local/include/leveldb/status.h:33:23: error: expected ';' at end of declaration list
Status(Status&& rhs) noexcept : state_(rhs.state_) { rhs.state_ = nullptr; }
^
;
/usr/local/include/leveldb/status.h:103:16: error: 'Status' is missing exception specification 'throw()'
inline Status::Status(const Status& rhs) {
^
throw()
/usr/local/include/leveldb/status.h:24:22: note: previous declaration is here
class LEVELDB_EXPORT Status {
^
/usr/local/include/leveldb/status.h:115:40: warning: rvalue references are a C++11 extension [-Wc++11-extensions]
inline Status& Status::operator=(Status&& rhs) noexcept {
^
/usr/local/include/leveldb/status.h:115:48: error: expected function body after function declarator
inline Status& Status::operator=(Status&& rhs) noexcept {
^
In file included from leveldb-native.cc:4:
In file included from /usr/local/include/leveldb/db.h:12:
/usr/local/include/leveldb/iterator.h:28:31: warning: deleted function definitions are a C++11 extension [-Wc++11-extensions]
Iterator(const Iterator&) = delete;
^
/usr/local/include/leveldb/iterator.h:29:42: warning: deleted function definitions are a C++11 extension [-Wc++11-extensions]
Iterator& operator=(const Iterator&) = delete;
^
/usr/local/include/leveldb/iterator.h:80:27: warning: alias declarations are a C++11 extension [-Wc++11-extensions]
using CleanupFunction = void (*)(void* arg1, void* arg2);
^
In file included from leveldb-native.cc:4:
In file included from /usr/local/include/leveldb/db.h:13:
/usr/local/include/leveldb/options.h:49:26: warning: in-class initialization of non-static data member is a C++11 extension [-Wc++11-extensions]
bool create_if_missing = false;
^
/usr/local/include/leveldb/options.h:52:24: warning: in-class initialization of non-static data member is a C++11 extension [-Wc++11-extensions]
bool error_if_exists = false;
^
/usr/local/include/leveldb/options.h:59:24: warning: in-class initialization of non-static data member is a C++11 extension [-Wc++11-extensions]
bool paranoid_checks = false;
^
/usr/local/include/leveldb/options.h:69:20: warning: in-class initialization of non-static data member is a C++11 extension [-Wc++11-extensions]
Logger* info_log = nullptr;
^
/usr/local/include/leveldb/options.h:82:28: warning: in-class initialization of non-static data member is a C++11 extension [-Wc++11-extensions]
size_t write_buffer_size = 4 * 1024 * 1024;
^
/usr/local/include/leveldb/options.h:87:22: warning: in-class initialization of non-static data member is a C++11 extension [-Wc++11-extensions]
int max_open_files = 1000;
^
/usr/local/include/leveldb/options.h:94:22: warning: in-class initialization of non-static data member is a C++11 extension [-Wc++11-extensions]
Cache* block_cache = nullptr;
^
/usr/local/include/leveldb/options.h:100:21: warning: in-class initialization of non-static data member is a C++11 extension [-Wc++11-extensions]
size_t block_size = 4 * 1024;
^
/usr/local/include/leveldb/options.h:105:30: warning: in-class initialization of non-static data member is a C++11 extension [-Wc++11-extensions]
int block_restart_interval = 16;
^
/usr/local/include/leveldb/options.h:115:24: warning: in-class initialization of non-static data member is a C++11 extension [-Wc++11-extensions]
size_t max_file_size = 2 * 1024 * 1024;
^
/usr/local/include/leveldb/options.h:131:31: warning: in-class initialization of non-static data member is a C++11 extension [-Wc++11-extensions]
CompressionType compression = kSnappyCompression;
^
/usr/local/include/leveldb/options.h:137:19: warning: in-class initialization of non-static data member is a C++11 extension [-Wc++11-extensions]
bool reuse_logs = false;
^
/usr/local/include/leveldb/options.h:142:37: warning: in-class initialization of non-static data member is a C++11 extension [-Wc++11-extensions]
const FilterPolicy* filter_policy = nullptr;
^
/usr/local/include/leveldb/options.h:147:19: warning: defaulted function definitions are a C++11 extension [-Wc++11-extensions]
ReadOptions() = default;
^
/usr/local/include/leveldb/options.h:151:25: warning: in-class initialization of non-static data member is a C++11 extension [-Wc++11-extensions]
bool verify_checksums = false;
^
/usr/local/include/leveldb/options.h:155:19: warning: in-class initialization of non-static data member is a C++11 extension [-Wc++11-extensions]
bool fill_cache = true;
^
/usr/local/include/leveldb/options.h:161:28: warning: in-class initialization of non-static data member is a C++11 extension [-Wc++11-extensions]
const Snapshot* snapshot = nullptr;
^
/usr/local/include/leveldb/options.h:166:20: warning: defaulted function definitions are a C++11 extension [-Wc++11-extensions]
WriteOptions() = default;
^
/usr/local/include/leveldb/options.h:182:13: warning: in-class initialization of non-static data member is a C++11 extension [-Wc++11-extensions]
bool sync = false;
^
In file included from leveldb-native.cc:4:
/usr/local/include/leveldb/db.h:56:10: warning: defaulted function definitions are a C++11 extension [-Wc++11-extensions]
DB() = default;
^
/usr/local/include/leveldb/db.h:58:19: warning: deleted function definitions are a C++11 extension [-Wc++11-extensions]
DB(const DB&) = delete;
^
/usr/local/include/leveldb/db.h:59:30: warning: deleted function definitions are a C++11 extension [-Wc++11-extensions]
DB& operator=(const DB&) = delete;
^
In file included from leveldb-native.cc:5:
/usr/local/include/leveldb/cache.h:36:13: warning: defaulted function definitions are a C++11 extension [-Wc++11-extensions]
Cache() = default;
^
/usr/local/include/leveldb/cache.h:38:25: warning: deleted function definitions are a C++11 extension [-Wc++11-extensions]
Cache(const Cache&) = delete;
^
/usr/local/include/leveldb/cache.h:39:36: warning: deleted function definitions are a C++11 extension [-Wc++11-extensions]
Cache& operator=(const Cache&) = delete;
^
In file included from leveldb-native.cc:6:
/usr/local/include/leveldb/write_batch.h:45:35: warning: defaulted function definitions are a C++11 extension [-Wc++11-extensions]
WriteBatch(const WriteBatch&) = default;
^
/usr/local/include/leveldb/write_batch.h:46:46: warning: defaulted function definitions are a C++11 extension [-Wc++11-extensions]
WriteBatch& operator=(const WriteBatch&) = default;
^
34 warnings and 4 errors generated.
make: *** [leveldb-native.o] Error 1
make failed, exit code 2
Gem files will remain installed in /Users/osada/.rvm/gems/ruby-2.6.3@bitcoinrb/gems/leveldb-native-0.6 for inspection.
Results logged to /Users/osada/.rvm/gems/ruby-2.6.3@bitcoinrb/extensions/x86_64-darwin-18/2.6.0/leveldb-native-0.6/gem_make.out
当然、irbからleveldb-nativeを使おうとするとエラーになります。
[ 19-12-24 19:43 ] ~/workspace/bitcoinrb
osada@mbp17e% irb
2.6.3 :001 > require 'leveldb-native'
Traceback (most recent call last):
8: from /Users/osada/.rvm/gems/ruby-2.6.3@bitcoinrb/bin/ruby_executable_hooks:24:in `<main>'
7: from /Users/osada/.rvm/gems/ruby-2.6.3@bitcoinrb/bin/ruby_executable_hooks:24:in `eval'
6: from /Users/osada/.rvm/gems/ruby-2.6.3@bitcoinrb/bin/irb:23:in `<main>'
5: from /Users/osada/.rvm/gems/ruby-2.6.3@bitcoinrb/bin/irb:23:in `load'
4: from /Users/osada/.rvm/gems/ruby-2.6.3@bitcoinrb/gems/irb-1.2.1/exe/irb:11:in `<top (required)>'
3: from (irb):2
2: from /Users/osada/.rvm/rubies/ruby-2.6.3/lib/ruby/site_ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
1: from /Users/osada/.rvm/rubies/ruby-2.6.3/lib/ruby/site_ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
LoadError (cannot load such file -- leveldb-native)
解決策
事象の34個のwarningを見ると、「C++11 extension」で定義された関数が正しく使えていないようなことが見れます。また、4個のerrorを見ると、文法間違い扱いとなっているようで、コンパイルが通らない様になっているようです。
どうやら、私の実行環境はC++標準が古いようで、インストールするためには、コンパイル時のオプションでC++11以降を指定する必要があるようです。そこで、次のようにしてインストールするgemファイルを編集してからインストールすることにします。
手順1. ダウンロード
gem fetchで、インストールするgemファイルをダウンロードします。
[ 19-12-25 9:41 ] ~/workspace/bitcoinrb
osada@mbp17e% gem fetch leveldb-native
Fetching leveldb-native-0.6.gem
Downloaded leveldb-native-0.6
[ 19-12-25 9:41 ] ~/workspace/bitcoinrb
osada@mbp17e% ls
leveldb-native-0.6.gem
手順2. 展開
gem unpackで、gemファイルを展開します。
[ 19-12-25 9:41 ] ~/workspace/bitcoinrb
osada@mbp17e% gem unpack leveldb-native-0.6.gem
Unpacked gem: '/Users/osada/workspace/bitcoinrb/leveldb-native-0.6'
[ 19-12-25 9:42 ] ~/workspace/bitcoinrb
osada@mbp17e% ls
leveldb-native-0.6/
leveldb-native-0.6.gem
手順3. extconf.rbの編集
leveldb-native-0.6/ext/leveldb-native/extconf.rbに、コンパイル時に指定するオプションに情報を追記します。
leveldb-native-0.6/ext/leveldb-native/extconf.rb
require 'mkmf'
have_library "leveldb" or abort "Can't find leveldb library."
$CXXFLAGS += " -std=c++11 " # <= ココ!!!
create_makefile "leveldb-native/leveldb_native"
$CXXFLAGSに、-std=c++11を追加するわけです。-stdの前の半角スペースと、11の後ろの半角スペースを忘れずに。
ちなみに、このmkmfというパッケージはMakefileを生成するrubyのモジュールのようです。
手順4. gemspecの作成
gem fetch / gem unpackしたファイルには、gemspecがありません。次のように手動で作成します。
[ 19-12-25 14:19 ] ~/workspace/bitcoinrb
osada@mbp17e% gem spec --ruby leveldb-native-0.6.gem > leveldb-native-0.6.gemspec
[ 19-12-25 14:19 ] ~/workspace/bitcoinrb
osada@mbp17e% ls
leveldb-native-0.6/ leveldb-native-0.6.gem leveldb-native-0.6.gemspec
手順5. buildの実行
手順4.で作成したgemspecファイルを適切な場所に配置して、編集した手順3.を含んだleveldb-nativeをgem buildします。
[ 19-12-25 14:19 ] ~/workspace/bitcoinrb
osada@mbp17e% mv leveldb-native-0.6.gemspec leveldb-native-0.6
[ 19-12-25 14:19 ] ~/workspace/bitcoinrb
osada@mbp17e% cd leveldb-native-0.6
[ 19-12-25 14:20 ] ~/workspace/bitcoinrb/leveldb-native-0.6
osada@mbp17e% gem build leveldb-native-0.6.gemspec
WARNING: open-ended dependency on rake (>= 0.9, development) is not recommended
if rake is semantically versioned, use:
add_development_dependency 'rake', '~> 0.9'
WARNING: See http://guides.rubygems.org/specification-reference/ for help
Successfully built RubyGem
Name: leveldb-native
Version: 0.6
File: leveldb-native-0.6.gem
[ 19-12-25 14:20 ] ~/workspace/bitcoinrb/leveldb-native-0.6
osada@mbp17e% ls
LICENSE example/ leveldb-native-0.6.gemspec
README.md ext/ lib/
Rakefile leveldb-native-0.6.gem test/
まあ、なにかwarningが出ますが、無事新しいleveldb-native-0.6.gemが生成されました。
手順6. インストール
新しいgemファイルをinstallします。このとき、"./"を忘れずに。
[ 19-12-25 14:20 ] ~/workspace/bitcoinrb
osada@mbp17e% gem install ./leveldb-native-0.6.gem
Building native extensions. This could take a while...
Successfully installed leveldb-native-0.6
Parsing documentation for leveldb-native-0.6
Installing ri documentation for leveldb-native-0.6
Done installing documentation for leveldb-native after 0 seconds
1 gem installed
手順7. 使ってみる
[ 19-12-25 14:20 ] ~/workspace/bitcoinrb
osada@mbp17e% irb
2.6.3 :001 > require 'leveldb-native'
=> true
2.6.3 :002 >
できた！
原因調査
環境調査
C++のバージョンやgemの環境変数を確認します。
[ 19-12-24 19:47 ] ~/workspace/bitcoinrb
osada@mbp17e% which c++
/usr/bin/c++
[ 19-12-24 19:59 ] ~/workspace/bitcoinrb
osada@mbp17e% gem environment
RubyGems Environment:
- RUBYGEMS VERSION: 3.0.6
- RUBY VERSION: 2.6.3 (2019-04-16 patchlevel 62) [x86_64-darwin18]
- INSTALLATION DIRECTORY: /Users/osada/.rvm/gems/ruby-2.6.3@bitcoinrb
- USER INSTALLATION DIRECTORY: /Users/osada/.gem/ruby/2.6.0
- RUBY EXECUTABLE: /Users/osada/.rvm/rubies/ruby-2.6.3/bin/ruby
- GIT EXECUTABLE: /usr/bin/git
- EXECUTABLE DIRECTORY: /Users/osada/.rvm/gems/ruby-2.6.3@bitcoinrb/bin
- SPEC CACHE DIRECTORY: /Users/osada/.gem/specs
- SYSTEM CONFIGURATION DIRECTORY: /Users/osada/.rvm/rubies/ruby-2.6.3/etc
- RUBYGEMS PLATFORMS:
- ruby
- x86_64-darwin-18
- GEM PATHS:
- /Users/osada/.rvm/gems/ruby-2.6.3@bitcoinrb
- /Users/osada/.rvm/rubies/ruby-2.6.3/lib/ruby/gems/2.6.0
- GEM CONFIGURATION:
- :update_sources => true
- :verbose => true
- :backtrace => false
- :bulk_threshold => 1000
- REMOTE SOURCES:
- https://rubygems.org/
- SHELL PATH:
- /Users/osada/.rvm/gems/ruby-2.6.3@bitcoinrb/bin
- /Users/osada/.rvm/gems/ruby-2.6.3@global/bin
- /Users/osada/.rvm/rubies/ruby-2.6.3/bin
- /Users/osada/.rvm/bin
- /usr/local/opt/jpeg-turbo/bin
- /Users/osada/.pyenv/shims
- /Users/osada/.pyenv/bin
- /usr/local/bin
- /usr/bin
- /bin
- /usr/sbin
- /sbin
- /Applications/VMware Fusion.app/Contents/Public
- /Library/TeX/texbin
- /usr/local/share/dotnet
- /Applications/Wireshark.app/Contents/MacOS
- /sbin
- /usr/local/bin
- /Users/osada/bin
Makefile/Rakefile/extconf周辺
leveldb-naitiveは、nativeという言葉が意味する通り、C++でコンパイルしたバイナリで動作します。というわけで、Makefileを使う"make"でインストールすることになります。一方で、gemはrubyのパッケージマネージャであって、ここではRakefileを使う"rake"でインストールことになります。
Makefileの中に色々なコンパイルオプションが書かれていますが、ここではCXXFLAGSに注目して書いてみて、makeするとうまくコンパイルが通るようになります。
[ 19-12-25 13:12 ] ~/workspace/bitcoinrb/leveldb-native-0.6/ext/leveldb-native
osada@mbp17e% make
compiling leveldb-native.cc
In file included from leveldb-native.cc:1:
In file included from /Users/osada/.rvm/rubies/ruby-2.6.3/include/ruby-2.6.0/ruby.h:33:
In file included from /Users/osada/.rvm/rubies/ruby-2.6.3/include/ruby-2.6.0/ruby/ruby.h:2111:
/Users/osada/.rvm/rubies/ruby-2.6.3/include/ruby-2.6.0/ruby/intern.h:56:19: warning:
'register' storage class specifier is deprecated and incompatible with
C++17 [-Wdeprecated-register]
void rb_mem_clear(register VALUE*, register long);
^~~~~~~~~
/Users/osada/.rvm/rubies/ruby-2.6.3/include/ruby-2.6.0/ruby/intern.h:56:36: warning:
'register' storage class specifier is deprecated and incompatible with
C++17 [-Wdeprecated-register]
void rb_mem_clear(register VALUE*, register long);
^~~~~~~~~
leveldb-native.cc:205:3: warning: 'auto_ptr<bound_db>' is deprecated
[-Wdeprecated-declarations]
auto_ptr<bound_db> db(new bound_db);
^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/memory:2080:28: note:
'auto_ptr<bound_db>' has been explicitly marked deprecated here
class _LIBCPP_TEMPLATE_VIS _LIBCPP_DEPRECATED_IN_CXX11 auto_ptr
^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/__config:1101:39: note:
expanded from macro '_LIBCPP_DEPRECATED_IN_CXX11'
# define _LIBCPP_DEPRECATED_IN_CXX11 _LIBCPP_DEPRECATED
^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/__config:1090:48: note:
expanded from macro '_LIBCPP_DEPRECATED'
# define _LIBCPP_DEPRECATED __attribute__ ((deprecated))
^
3 warnings generated.
linking shared-object leveldb-native/leveldb_native.bundle
ただし、このままだとgemとして使うことができません。そこで、このMakefileを生成するrakeコマンドを見ていくわけです。rakeは、引数を指定して実行する必要があります。ここではextを選びます。
[ 19-12-25 13:11 ] ~/workspace/bitcoinrb/leveldb-native-0.6
osada@mbp17e% rake --task
rake clean # Clean compiled files
rake dist_clean # Clean compiled files and Makefile
rake ext # Build extension
rake release # Test, commit, tag, and push repo; build and push gem
rake release:diff # Diff to latest release
rake release:log # Log to latest release
rake test # Run tests for {:test=>:ext}
[ 19-12-25 13:11 ] ~/workspace/bitcoinrb/leveldb-native-0.6
osada@mbp17e% rake ext
cd ext/leveldb-native && ruby extconf.rb
checking for -lleveldb... yes
creating Makefile ## <= ココ!!!
cd ext/leveldb-native && make
compiling leveldb-native.cc
In file included from leveldb-native.cc:4:
In file included from /usr/local/include/leveldb/db.h:12:
In file included from /usr/local/include/leveldb/iterator.h:19:
/usr/local/include/leveldb/slice.h:43:25: warning: defaulted function definitions
are a C++11 extension [-Wc++11-extensions]
Slice(const Slice&) = default;
^
(略)
/usr/local/include/leveldb/write_batch.h:46:46: warning: defaulted function
definitions are a C++11 extension [-Wc++11-extensions]
WriteBatch& operator=(const WriteBatch&) = default;
^
34 warnings and 4 errors generated.
make: *** [leveldb-native.o] Error 1
rake aborted!
Command failed with status (2): [cd ext/leveldb-native && make...]
/Users/osada/workspace/bitcoinrb/leveldb-native-0.6/Rakefile:28:in `block in <top (required)>'
/Users/osada/.rvm/gems/ruby-2.6.3@bitcoinrb/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
/Users/osada/.rvm/gems/ruby-2.6.3@bitcoinrb/bin/ruby_executable_hooks:24:in `eval'
/Users/osada/.rvm/gems/ruby-2.6.3@bitcoinrb/bin/ruby_executable_hooks:24:in `<main>'
Tasks: TOP => ext => ext/leveldb-native/leveldb_native.so
(See full trace by running task with --trace)
Makefileを作っていることがわかりますね。rakeが指定するRakefileを見てみましょう。
Rakefile
desc "Build extension"
task :ext => File.join(ext_dir, so_name)
file File.join(ext_dir, so_name) => FileList[
File.join(ext_dir, "*.{c,cc,h}"),
File.join(ext_dir, "Makefile")] do
sh "cd #{ext_dir} && make"
end
file File.join(ext_dir, "Makefile") => File.join(ext_dir, "extconf.rb") do
sh "cd #{ext_dir} && ruby extconf.rb"
end
上記はextタスク部分を取り出したものです。Makefile作成にには、ext_dirにある"extconf.rb"をrubyで実行していることがわかります。
これを見に行くと、手順3に繋がります。
いや〜、C++の標準の違いは利用者も意識する必要があるので、骨が折れますね。c++11なのかc++14なのか、はたまたc++17なのか。コンパイルオプション"-std=xxx"で指定する部分は環境変数でなんとかならないものなんでしょうかね。
参考サイト
- LevelDBとは
- Unsupported leveldb 1.21 on MacOS
- How to add a C++ compiler flag to extconf.rb
- インストールできない gem を修正してインストールする