CDP私有云安装

概要

换工作休假中。趁着家里有网，搞一个私有云试试

一些参考资料

私有云下载： https://www.cloudera.com/downloads/cdp-private-cloud-trial/cdp-private-cloud-base-trial.html
官方文档： https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/installation/topics/cdpdc-installing-trial-cluster.html

注意：官方文档的版本是7.17，笔者自己玩的时候已经是7.4.4

服务器

zzeng01： manager+namenode
zzeng02： datanode
zzeng03： datanode

配置都是8cpu 32Gmem

安装和卸载

基本按着官网资料做就完事，卸载不要看官网，已经是旧的。

安装

参考： https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/installation/topics/cdp-quick-start-deployment-streams-install-runtime.html

$ wget https://archive.cloudera.com/cm7/7.4.4/cloudera-manager-installer.bin
$ chmod u+x cloudera-manager-installer.bin
$ sudo ./cloudera-manager-installer.bin

hint:为了方便可以设置/etc/hosts

[IP]  zzeng01.novalocal
[IP]  zzeng02.novalocal
[IP]  zzeng03.novalocal

卸载

sudo /opt/cloudera/installer/uninstall-cloudera-manager.sh

配置

访问网址进行修改 http://zzeng01.novalocal:7180/cmf/login

参考： https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/installation/topics/cdp-quick-start-deployment-streams-install-runtime.html

Failed to authenticate

Install Agents阶段出现Failed to authenticate
做了以下尝试后成功解决问题。

设置hosts和hostname

据说cdp内部是通过ip反找hostname，这里需要设置（但最后这里是不是问题的原因未知，总之我是一起修改了。。）

在三台服务器都要做

$ sudo cat /etc/hosts
10.113.205.187  ip-10.113.205.187.se.fuse.cloudera.com zzeng01.novalocal
10.113.206.35   ip-10.113.206.35.se.fuse.cloudera.com zzeng02.novalocal
10.113.204.129  ip-10.113.204.129.se.fuse.cloudera.com zzeng03.novalocal

在三台服务器都要做

$ sudo hostnamectl set-hostname ip-10.113.205.187.se.fuse.cloudera.com

unlock root

centos默认root用户是锁住的，因为我们用root，所以需要解锁

新建用户也可以

在每个server上准备一个新的用户名 参考： https://www.digitalocean.com/community/tutorials/how-to-add-and-delete-users-on-a-centos-7-server

$ sudo adduser cdp
$ sudo passwd cdp
$ sudo gpasswd -a cdp wheel
$ sudo lid -g wheel

这样的话会报错 no Privilege
然后查看/tmp/scm_prepare_node.XXXXX/scm_prepare_node.log

need root privileges but sudo requires password, exiting

然后修改/etc/sudoers.d下面的文件，添加

cdp ALL=(ALL) NOPASSWD:ALL

问题解决!

一些弯路：直接通过visudo来修改文件/etc/sudoers无法成功。原因未明

参考：

# set passwd
$ sudo passwd root

# unlock
$ sudo passwd -u root

sshd设置

sshd默认验证是的关着的，这里需要修改下

set_sshd

$ sudo vi /etc/ssh/sshd_config
PermitRootLogin yes
PasswordAuthentication yes 
$ systemctl restart sshd.service

swappiness

Inspect Cluster阶段出现警告，swap分区比重过大，要求下降（从60到10）
参考： https://qiita.com/ikuwow/items/f0b4d1f509a0b83b5d7e

具体log

Cloudera recommends setting /proc/sys/vm/swappiness to a maximum of 10. Current setting is 30. Use the sysctl command to change this setting at run time and edit /etc/sysctl.conf for this setting to be saved after a reboot. You can continue with installation, but Cloudera Manager might report that your hosts are unhealthy because they are swapping. The following hosts are affected:

一时修改

$ echo 10 > /proc/sys/vm/swappiness
$ cat /proc/sys/vm/swappiness
10

不怕重启的修改

$ sudo vi /etc/sysctl.conf
vm.swappiness = 10

transparent_hugepage

具体log

Transparent Huge Page Compaction is enabled and can cause significant performance problems. Run "echo never > /sys/kernel/mm/transparent_hugepage/defrag" and "echo never > /sys/kernel/mm/transparent_hugepage/enabled" to disable this, and then add the same command to an init script such as /etc/rc.local so it will be set on system reboot. 

照做

$ echo never > /sys/kernel/mm/transparent_hugepage/defrag
$ echo never > /sys/kernel/mm/transparent_hugepage/enabled
$ sudo vi /etc/rc.local

動作確認

beeline

安装好的状态：zzeng01装着hive；没有任何密码和验证
尝试：从zzeng03登陆hive

1. CDP UI确认开启hiveserver2
2. 登陆zzeng03
3. beeline

beeline -u jdbc:hive2://zzeng01.novalocal:10000

小插曲：用hive登陆的话，也会直接beeline登进去，CDP的优化？