Introduction
IBM Cloud Object Storage has three types of endpoints.
https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-endpoints
- Public Endpoint
- Private Endpoint
- Direct Endpoint
The Private Endpoint and Direct Endpoint allows us to exchange data with ICOS over IBM Cloud's private network. Because it is secure and does not incur outbound transfer costs to the Internet, there have long been requests to access the private side of ICOS from on-premises via Direct Link.
Since ICOS could not be accessed directly from on-premises (because ICOS is located in the shared service segment, not the user's subnet, and routing information to ICOS is not advertised to on-premises), users had to manually set up a reverse proxy in IBM Cloud and access ICOS through it.
https://cloud.ibm.com/docs/direct-link?topic=direct-link-using-ibm-cloud-direct-link-to-connect-to-ibm-cloud-object-storage
There are also not a few requests to access ICOS from PowerVS, but as with the above, it was necessary to manually set up a reverse proxy on the x86 side because it could not go directly.
https://qiita.com/y_tama/items/bb9c553ba8f35f12f8a4 (Japanese)
This article is about a new feature of Virtual Private Endpoint (VPE), a service that allows users to create endpoints to ICOS and other IBM Cloud services on their VPC subnets.
https://cloud.ibm.com/docs/vpc?topic=vpc-about-vpe
Until now, VPEs could only communicate from the same VPC and could not be accessed over Direct Link or Transit Gateway.
The new feature allows access to VPE through Direct Link 2.0 and Transit Gateway, eliminating the need for users to configure their own reverse proxies.
Configuration Overview
An overview image is shown below.
In this article, PowerVS is used as the connection source. The same configuration can be used to connect from on-premises.
The outline of the procedure is as follows
- Connect PowerVS and VPC via Cloud Connections (Direct Link 2.0)
- Order a VPE linked to ICOS
- Create a Custom Resolver with DNS Services
- Set the DNS reference of PowerVS to the Custom Resolver created above.
- Setup ICOS client (s3cmd this time) on PowerVS (AIX).
- Check the connection.
Detailed Configuration
Connect PowerVS and VPC via Cloud Connections (Direct Link 2.0)
Connect PowerVS and VPC via Cloud Connections (Direct Link 2.0)
https://cloud.ibm.com/docs/power-iaas?topic=power-iaas-cloud-connections
When a public IP address is enabled on the PowerVS LPAR, the default gateway is set to the public side. Set a static route in AIX so that packets destined to the VPC are routed to the private side gateway.
# netstat -r
Routing tables
Destination Gateway Flags Refs Use If Exp Groups
Route tree for Protocol Family 2 (Internet):
default 192.168.167.65 UG 6 1625714 en0 - -
127/8 localhost U 4 409139 lo0 - -
172.16.10/24 172.16.52.1 UGS 2 99935 en1 - -
172.16.20/24 172.16.52.1 UGS 0 118 en1 - -
172.16.30/24 172.16.52.1 UGS 0 34689 en1 - -
172.16.52.0 172.16.52.253 UHSb 0 0 en1 - - =>
172.16.52/24 172.16.52.253 U 3 0 en1 - -
...
Order a VPE linked to ICOS
Order Virtual Private Endpoint.
https://cloud.ibm.com/vpc-ext/provision/endpointGateway
Select ICOS as the service to be bound to VPE.
In this article, select Osaka regional Direct Endpoint (s3.direct.jp-osa.cloud-object-storage.appdomain.cloud) in order to use Osaka regional ICOS.
A list of ICOS Endpoints can be found below.
https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-endpoints#endpoints-region
The IP addresses on the VPC to be bound to the VPE will be set later.
Click the create button.
Select one subnet from each zone and bind it to the created VPE.
This setup allows ICOS to be accessed using the VPE of the remaining zone in the event that one zone becomes unavailable for some reason.
Since communication to the VPE is protected by Security Groups, create a rule to allow the necessary communication.
In this case, we will allow TCP 443 from the PowerVS subnet range for REST API communication from PowerVS to ICOS.
Create a custom resolver with DNS Services
When using ICOS from PowerVS, we will specify the FQDN of the Direct Endpoint (s3.direct.jp-osa.cloud-object-storage.appdomain.cloud in this article).
In the default configuration of PowerVS with public IP enabled, name resolution for this FQDN uses a resolver on the Internet and it returns 161.26.x.x. This IP address is the IP address range that IBM Cloud uses for its private NW, but it is not routed directly from PowerVS, so it cannot be accessed.
# nslookup s3.direct.jp-osa.cloud-object-storage.appdomain.cloud
Server: 9.9.9.9
Address: 9.9.9.9#53
Non-authoritative answer:
Name: s3.direct.jp-osa.cloud-object-storage.appdomain.cloud
Address: 161.26.0.47
#
For LPARs that do not have public IP enabled, it is normal for them not to be able to resolve names by default.
# nslookup s3.direct.jp-osa.cloud-object-storage.appdomain.cloud
;; connection timed out; no servers could be reached
#
Therefore, we create a custom resolver, a function of IBM Cloud's DNS Services, and set the DNS for AIX to the custom resolver.
https://cloud.ibm.com/docs/dns-svcs?topic=dns-svcs-custom-resolver
Since the custom resolver reflects the VPE settings, it will return the VPE IP address in response to the name resolution request for s3.direct.jp-osa.cloud-object-storage.appdomain.cloud.
It would be possible to use the AIX's /etc/hosts to resolve the ICOS FQDN to the VPE IP address without using the custom resolver (and manually rewrite the /etc/hosts when switching VPEs).
In this article, I am using custom resolver from the perspective of automating the switchover when one VPE becomes unavailable.
First, create DNS Services from the catalog, then create a Custom Resolver within the created DNS Services.
Please see below for pricing for DNS Services and Custom Resolver.
https://cloud.ibm.com/docs/dns-svcs?topic=dns-svcs-pricing
A Custom Resolver is created on the user's VPC subnet. These are accessible from PowerVS.
Set the DNS reference of PowerVS to the Custom Resolver created above.
Specify the IP address of the Custom Resolver created above as the DNS lookup for AIX in PowerVS.
# cat /etc/resolv.conf
#nameserver 127.0.0.1
#nameserver 9.9.9.9
nameserver 172.16.10.8
nameserver 172.16.20.5
#
Custom Resolver performs name resolution that reflects the VPE configuration, so it returns the VPE's IP address in response to a name resolution request for the FQDN of the ICOS Direct Endpoint.
# nslookup s3.direct.jp-osa.cloud-object-storage.appdomain.cloud
Server: 172.16.10.8
Address: 172.16.10.8#53
Name: s3.direct.jp-osa.cloud-object-storage.appdomain.cloud
Address: 172.16.10.13
#
When I remove the IP address to be bound to the VPE as a test, the custom resolver return another IP address of the remaining zone after a certain amount of time (probably DNS TTL).
Delete 172.16.10.13 from VPE configuration, which was returned by custom resolver above.
After a certain period of time, the IP address of the remaining zone will be returned.
# nslookup s3.direct.jp-osa.cloud-object-storage.appdomain.cloud
Server: 172.16.10.8
Address: 172.16.10.8#53
Name: s3.direct.jp-osa.cloud-object-storage.appdomain.cloud
Address: 172.16.30.8
#
Setup ICOS client (s3cmd this time) on PowerVS (AIX).
As an ICOS client tool, we will use s3cmd, which can be installed as follows if yum is available in your environment.
# export PATH=$PATH:/opt/freeware/bin
# yum install python-pip -y
# pip install s3cmd
Configure the configuration file (.s3cfg). use ICOS HMAC credentials for access_key and secret_key.
# cat .s3cfg
[default]
access_key = xxxxxxxxxxxxxxxxxxxxxxxxx
secret_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
host_base = s3.direct.jp-osa.cloud-object-storage.appdomain.cloud
host_bucket = %(bucket)s.s3.direct.jp-osa.cloud-object-storage.appdomain.cloud
#
Check the connection.
Verify that ICOS is accessible using s3cmd.
# s3cmd ls
2021-11-08 15:04 s3://bucket-osa-20211109
# s3cmd ls s3://bucket-osa-20211109
2022-09-12 11:54 1073741824 s3://bucket-osa-20211109/tmpfile
2022-09-12 11:57 1073741824 s3://bucket-osa-20211109/tmpfile2
#
Since both VPE and Custom Resolver used in this configuration are managed services, it is now possible to access ICOS without having to manually configure reverse proxies as before, which is quite convenient.
That's it!