本当の目的はDeploying Cloud Foundry on Google Compute Engineに従ってGCP上にCloud Foundryの環境を構築することなのですが、その前に、前提条件となるBOSHの動作環境が必要なので、Deploy BOSH on Google Cloud Platformに従って構築します。
関連ソフトウェア
この作業の中ではBOSH以外にも以下のソフトウェアが使用されています。
-
Terraform
- オーケストレーションツール
BOSH環境構築
以降の作業はブラウザからGoogle Cloud Shellを起動して実行します。
使用するGCPのアカウントやリージョンなどの情報を環境変数に設定します。
基本ドキュメントのままです。
export project_id=$(gcloud config list 2>/dev/null | grep project | sed -e 's/project = //g')
export region=us-east1
export zone=us-east1-d
export service_account_email=terraform@${project_id}.iam.gserviceaccount.com
gcloud config set compute/zone ${zone}
gcloud config set compute/region ${region}
gcloud iam service-accounts create terraform
gcloud iam service-accounts keys create ~/terraform.key.json \
--iam-account ${service_account_email}
gcloud projects add-iam-policy-binding ${project_id} \
--member serviceAccount:${service_account_email} \
--role roles/owner
export GOOGLE_CREDENTIALS=$(cat ~/terraform.key.json)
必要なソースをgitリポジトリからクローンします。
git clone https://github.com/cloudfoundry-incubator/bosh-google-cpi-release.git
cd bosh-google-cpi-release/docs/bosh
main.tfを覗いてみます。Terraformの設定ファイルです。
サブネット、ルート、ファイヤーウォール、GCEインスタンスなどの設定が記述してあります。
$ cat main.tf
variable "projectid" {
type = "string"
}
variable "region" {
type = "string"
default = "us-east1"
}
variable "zone" {
type = "string"
default = "us-east1-d"
}
variable "prefix" {
type = "string"
default = ""
}
variable "service_account_email" {
type = "string"
default = ""
}
provider "google" {
project = "${var.projectid}"
region = "${var.region}"
}
resource "google_compute_network" "bosh" {
name = "${var.prefix}bosh"
}
resource "google_compute_route" "nat-primary" {
name = "${var.prefix}nat-primary"
dest_range = "0.0.0.0/0"
network = "${google_compute_network.bosh.name}"
next_hop_instance = "${google_compute_instance.nat-instance-private-with-nat-primary.name}"
next_hop_instance_zone = "${var.zone}"
priority = 800
tags = ["no-ip"]
}
// Subnet for the BOSH director
resource "google_compute_subnetwork" "bosh-subnet-1" {
name = "${var.prefix}bosh-${var.region}"
ip_cidr_range = "10.0.0.0/24"
network = "${google_compute_network.bosh.self_link}"
}
// Allow SSH to BOSH bastion
resource "google_compute_firewall" "bosh-bastion" {
name = "${var.prefix}bosh-bastion"
network = "${google_compute_network.bosh.name}"
allow {
protocol = "icmp"
}
allow {
protocol = "tcp"
ports = ["22"]
}
target_tags = ["bosh-bastion"]
}
// Allow all traffic within subnet
resource "google_compute_firewall" "intra-subnet-open" {
name = "${var.prefix}intra-subnet-open"
network = "${google_compute_network.bosh.name}"
allow {
protocol = "icmp"
}
allow {
protocol = "tcp"
ports = ["1-65535"]
}
allow {
protocol = "udp"
ports = ["1-65535"]
}
source_tags = ["internal"]
}
// BOSH bastion host
resource "google_compute_instance" "bosh-bastion" {
name = "${var.prefix}bosh-bastion"
machine_type = "n1-standard-1"
zone = "${var.zone}"
tags = ["bosh-bastion", "internal"]
disk {
image = "ubuntu-1404-trusty-v20161020"
}
network_interface {
subnetwork = "${google_compute_subnetwork.bosh-subnet-1.name}"
access_config {
// Ephemeral IP
}
}
metadata_startup_script = <<EOT
#!/bin/bash
cat > /etc/motd <<EOF
# # ## ##### # # # # # ####
# # # # # # ## # # ## # # #
# # # # # # # # # # # # # #
# ## # ###### ##### # # # # # # # # ###
## ## # # # # # ## # # ## # #
# # # # # # # # # # # ####
Startup scripts have not finished installing, and the tools you need
are not ready yet. Please log out and log back in again in a few moments.
This warning will not appear when the system is ready.
EOF
apt-get update
apt-get install -y build-essential zlibc zlib1g-dev ruby ruby-dev openssl libxslt-dev libxml2-dev libssl-dev libreadline6 libreadline6-dev libyaml-dev libsqlite3-dev sqlite3 jq git unzip
gem install bosh_cli
curl -o /tmp/cf.tgz https://s3.amazonaws.com/go-cli/releases/v6.20.0/cf-cli_6.20.0_linux_x86-64.tgz
tar -zxvf /tmp/cf.tgz && mv cf /usr/bin/cf && chmod +x /usr/bin/cf
curl -o /usr/bin/bosh-init https://s3.amazonaws.com/bosh-init-artifacts/bosh-init-0.0.96-linux-amd64
chmod +x /usr/bin/bosh-init
cat > /etc/profile.d/bosh.sh <<'EOF'
#!/bin/bash
# Misc vars
export prefix=${var.prefix}
export ssh_key_path=$HOME/.ssh/bosh
# Vars from Terraform
export subnetwork=${google_compute_subnetwork.bosh-subnet-1.name}
export network=${google_compute_network.bosh.name}
# Vars from metadata service
export project_id=$$(curl -s -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/project/project-id)
export zone=$$(curl -s -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/zone)
export zone=$${zone##*/}
export region=$${zone%-*}
# Configure gcloud
gcloud config set compute/zone $${zone}
gcloud config set compute/region $${region}
EOF
# Clone repo
mkdir /share
git clone https://github.com/cloudfoundry-incubator/bosh-google-cpi-release.git /share
chmod -R 777 /share
# Install Terraform
wget https://releases.hashicorp.com/terraform/0.7.7/terraform_0.7.7_linux_amd64.zip
unzip terraform*.zip -d /usr/local/bin
rm /etc/motd
EOT
service_account {
email = "${var.service_account_email}"
scopes = ["cloud-platform"]
}
}
// NAT server (primary)
resource "google_compute_instance" "nat-instance-private-with-nat-primary" {
name = "${var.prefix}nat-instance-primary"
machine_type = "n1-standard-1"
zone = "${var.zone}"
tags = ["nat", "internal"]
disk {
image = "ubuntu-1404-trusty-v20161020"
}
network_interface {
subnetwork = "${google_compute_subnetwork.bosh-subnet-1.name}"
access_config {
// Ephemeral IP
}
}
can_ip_forward = true
metadata_startup_script = <<EOT
#!/bin/bash
sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
EOT
}
作成されるリソースを確認します。
docker run -i -t \
-e "GOOGLE_CREDENTIALS=${GOOGLE_CREDENTIALS}" \
-v `pwd`:/$(basename `pwd`) \
-w /$(basename `pwd`) \
hashicorp/terraform:light plan \
-var service_account_email=${service_account_email} \
-var projectid=${project_id} \
-var region=${region} \
-var zone=${zone}
作成するリソースがわかりやすく表示されています。
+ google_compute_firewall.bosh-bastion
allow.#: "2"
allow.1367131964.ports.#: "0"
allow.1367131964.protocol: "icmp"
allow.803338340.ports.#: "1"
allow.803338340.ports.0: "22"
allow.803338340.protocol: "tcp"
name: "bosh-bastion"
network: "bosh"
project: "<computed>"
self_link: "<computed>"
target_tags.#: "1"
target_tags.1860295641: "bosh-bastion"
+ google_compute_route.nat-primary
dest_range: "0.0.0.0/0"
name: "nat-primary"
network: "bosh"
next_hop_instance: "nat-instance-primary"
next_hop_instance_zone: "us-east1-d"
next_hop_network: "<computed>"
priority: "800"
self_link: "<computed>"
tags.#: "1"
tags.1472900357: "no-ip"
+ google_compute_subnetwork.bosh-subnet-1
gateway_address: "<computed>"
ip_cidr_range: "10.0.0.0/24"
name: "bosh-us-east1"
network: "<computed>"
self_link: "<computed>"
Plan: 7 to add, 0 to change, 0 to destroy.
リソースを作成します。
docker run -i -t \
-e "GOOGLE_CREDENTIALS=${GOOGLE_CREDENTIALS}" \
-v `pwd`:/$(basename `pwd`) \
-w /$(basename `pwd`) \
hashicorp/terraform:light apply \
-var service_account_email=${service_account_email} \
-var projectid=${project_id} \
-var region=${region} \
-var zone=${zone}
Cloud shellで確認してみると、リソースが作成されています。
$ gcloud compute networks list
NAME MODE IPV4_RANGE GATEWAY_IPV4
bosh custom
default auto
$ gcloud compute networks subnets list
NAME REGION NETWORK RANGE
default asia-northeast1 default 10.146.0.0/20
default us-west1 default 10.138.0.0/20
bosh-us-east1 us-east1 bosh 10.0.0.0/24
default us-east1 default 10.142.0.0/20
default europe-west1 default 10.132.0.0/20
default asia-east1 default 10.140.0.0/20
default us-central1 default 10.128.0.0/20
$ gcloud compute instances list
NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS
bosh-bastion us-east1-d n1-standard-1 10.0.0.2 XXX.XXX.XXX.XXX RUNNING
nat-instance-primary us-east1-d n1-standard-1 10.0.0.3 XXX.XXX.XXX.XXX RUNNING
bash-bastionにsshでログインします。
gcloud compute ssh bosh-bastion
VM上でGoogle Cloud SDKを最新版にアップデートします。
sudo gcloud components update
BOSHが使うサービスアカウントを作成します。
export service_account=bosh-user
export service_account_email=${service_account}@${project_id}.iam.gserviceaccount.com
gcloud iam service-accounts create ${service_account}
ここで以下のように権限エラーが出た場合は、現在ログインしているアカウントを確認し、iamの操作権限を持っているアカウントに変更します。
$ gcloud iam service-accounts create ${service_account}
ERROR: (gcloud.iam.service-accounts.create) You do not have permission to access project [cf-test] (or it may not exist
): Google Identity and Access Management (IAM) API has not been used in project cf-test before or it is disabled. Enabl
e it by visiting https://console.developers.google.com/apis/api/iam.googleapis.com/overview?project=cf-test then retry.
If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.
$ gcloud config list
Your active configuration is: [default]
[compute]
region = us-east1
zone = us-east1-d
[core]
account = terraform@cf-test.iam.gserviceaccount.com
project = cf-test
$ gcloud config set core/account tommarute@example.com
Updated property [core/account].
$ gcloud auth login
$ gcloud config list
Your active configuration is: [default]
[compute]
region = us-east1
zone = us-east1-d
[core]
account = tommarute@example.com
project = cf-test
アカウントに権限を設定します。
gcloud projects add-iam-policy-binding ${project_id} \
--member serviceAccount:${service_account_email} \
--role roles/compute.instanceAdmin
gcloud projects add-iam-policy-binding ${project_id} \
--member serviceAccount:${service_account_email} \
--role roles/compute.storageAdmin
gcloud projects add-iam-policy-binding ${project_id} \
--member serviceAccount:${service_account_email} \
--role roles/storage.admin
gcloud projects add-iam-policy-binding ${project_id} \
--member serviceAccount:${service_account_email} \
--role roles/compute.networkAdmin
gcloud projects add-iam-policy-binding ${project_id} \
--member serviceAccount:${service_account_email} \
--role roles/iam.serviceAccountActor
パスワードなしのsshキーを作成してアップロードします。
ssh-keygen -t rsa -f ~/.ssh/bosh -C bosh
gcloud compute project-info add-metadata --metadata-from-file \
sshKeys=<( gcloud compute project-info describe --format=json | jq -r '.commonInstanceMetadata.items[] | select(.key == "sshKeys") | .value' & echo "bosh:$(cat ~/.ssh/bosh.pub)" )
インストールされたBOSHのバージョンを確認します。
bosh-init -v
マニフェストテンプレートを作成するディレクトリを作ります。
mkdir google-bosh-director
cd google-bosh-director
テンプレートを作成します。
10.0.0.6がbosh-directorになるようです。
DirectorはVMの管理を行います。
CPIというのはBOSHがIaaSを操作するインタフェースです。
$ cat << EOF > manifest.yml.erb
> ---
> name: bosh
>
> releases:
> - name: bosh
> url: https://bosh.io/d/github.com/cloudfoundry/bosh?v=257.3
> sha1: e4442afcc64123e11f2b33cc2be799a0b59207d0
> - name: bosh-google-cpi
> url: https://bosh.io/d/github.com/cloudfoundry-incubator/bosh-google-cpi-release?v=25.6.0
> sha1: 3e01539a1228d62b8015feb388df2234978efaf6
>
> resource_pools:
> - name: vms
> network: private
> stemcell:
> url: https://bosh.io/d/stemcells/bosh-google-kvm-ubuntu-trusty-go_agent?v=3263.7
> sha1: a09ce8b4acfa9876f52ee7b4869b4b23f27d5ace
> cloud_properties:
> zone: <%= ENV['zone'] %>
> machine_type: n1-standard-1
> root_disk_size_gb: 40
> root_disk_type: pd-standard
> service_account: <%= ENV['service_account_email'] %>
>
> disk_pools:
> - name: disks
> disk_size: 32_768
> cloud_properties:
> type: pd-standard
>
> networks:
> - name: vip
> type: vip
> - name: private
> type: manual
> subnets:
> - range: 10.0.0.0/29
> gateway: 10.0.0.1
> static: [10.0.0.3-10.0.0.7]
> cloud_properties:
> network_name: <%= ENV['network'] %>
> subnetwork_name: <%= ENV['subnetwork'] %>
> ephemeral_external_ip: false
> tags:
> - internal
> - no-ip
>
> jobs:
> - name: bosh
> instances: 1
>
> templates:
> - name: nats
> release: bosh
> - name: postgres
> release: bosh
> - name: powerdns
> release: bosh
> - name: blobstore
> release: bosh
> - name: director
> release: bosh
> - name: health_monitor
> release: bosh
> - name: google_cpi
> release: bosh-google-cpi
>
> resource_pool: vms
> persistent_disk_pool: disks
>
> networks:
> - name: private
> static_ips: [10.0.0.6]
> default:
> - dns
> - gateway
>
> properties:
> nats:
> address: 127.0.0.1
> user: nats
> password: nats-password
>
> postgres: &db
> listen_address: 127.0.0.1
> host: 127.0.0.1
> user: postgres
> password: postgres-password
> database: bosh
> adapter: postgres
>
> dns:
> address: 10.0.0.6
> domain_name: microbosh
> db: *db
> recursor: 169.254.169.254
>
> blobstore:
> address: 10.0.0.6
> port: 25250
> provider: dav
> director:
> user: director
> password: director-password
> agent:
> user: agent
> password: agent-password
>
> director:
> address: 127.0.0.1
> name: micro-google
> db: *db
> cpi_job: google_cpi
> user_management:
> provider: local
> local:
> users:
> - name: admin
> password: admin
> - name: hm
> password: hm-password
> hm:
> director_account:
> user: hm
> password: hm-password
> resurrector_enabled: true
>
> google: &google_properties
> project: <%= ENV['project_id'] %>
>
> agent:
> mbus: nats://nats:nats-password@10.0.0.6:4222
> ntp: *ntp
> blobstore:
> options:
> endpoint: http://10.0.0.6:25250
> user: agent
> password: agent-password
>
> ntp: &ntp
> - 169.254.169.254
>
> cloud_provider:
> template:
> name: google_cpi
> release: bosh-google-cpi
>
> template:
> ssh_tunnel:
> host: 10.0.0.6
> port: 22
> user: bosh
> private_key: <%= ENV['ssh_key_path'] %>
>
> mbus: https://mbus:mbus-password@10.0.0.6:6868
>
> properties:
> google: *google_properties
> agent: {mbus: "https://mbus:mbus-password@0.0.0.0:6868"}
> blobstore: {provider: local, path: /var/vcap/micro_bosh/data/cache}
> ntp: *ntp
> EOF
テンプレートからYAMLファイルを作成します。
erb manifest.yml.erb > manifest.yml
BOSH Directorを作成するためのマニフェストを展開します。
bosh-init deploy manifest.yml
BOSH CLIにリモートの環境を指定します。
bosh target 10.0.0.6
# 認証はadmin/admin
次回はCloud Foundryの環境を構築します。