This is the step 2 of adding "Auto Scaling" to the Oracle Database Cloud Service.
If you want to check the other steps, please click the below links respectively.
- Step 1: Gather OCI DB System CPU Usage and write it into a custom log
- Step 2: Configure OCI Logging service to ingest the custom log
- Step 3: Configure OCI Service Connector Hub service to transfer the custom log to OCI Functions
- Step 4: Configure OCI Functions to act accordingly based on the CPU usage in the custom log
Configure OCI Logging service to ingest the custom log
In step 1: Gather OCI DB System CPU Usage and write it into a custom log, we have created and deployed a script to gather the realtime CPU usage and write it into the custom log. In the step 2, we are going to do following configuration to ingest and upload the custom logs to OCI Logging service.
- Create custom log in OCI Logging and setup Agent Configurations
- Install Oracle Unified Monitoring Agent in the target DB system node
Create Custom Log in OCI Logging and setup Agent Configurations
In this section, we will create Custom Log and Agent Configuration in OCI Logging service to retrieve the logs from client side.
OCI Logging service supports lots of pre-defined OCI services logs, and the list is still growing. We don't need to do many works and we can start to use the Logging service for them easily.
But this time, the log we are going to process is totally custom, so we need to use the Custom Logs to help us complete this task.
Create Custom Log
First, let's visit the Logs in the OCI Logging service.
[Figure: visit Logging service]
Then, click Create Custom Log button to start the creating.
[Figure: create Custom Logs]
Creating a Custom Log is very simple, just input a meaningful name and select COMPARTMENT and LOG GROUP. If you don't want to put your logs in the Default_Group you can create a new one.
[Figure: create Custom Log - input name]
In the additional options, we can select the expected Log Retention. We choose 1 month, that's the default option, but if you wish to store the logs longer, OCI Logging service provide you the option of 6 months maximumly.
If you wish even longer retention period, OCI also provide you the possibility by using Service Connector Hub and Object Storage. We can talk about this later.
[Figure: Log Retention]
Click the Create Custom Log button at bottom left corner, you will move to the next step Create Agent Config. We can create the configuration right now, or create it later and attach it to the Custom Log later.
The information you input here is quite important, because it decides the behavior the Oracle Unified Monitoring Agent in the future.
Input a meaningful name of this agent configuration, and select a compartment it belongs to.
Select User Group in the Choose Host Groups section, and select the user's group accordingly. You may ask the user mentioned here is which one? Well, it is the user who you added API keys to. This user owns the key and the agent will communicate with OCI via this user.
[Figure: Create Agent Configuration - 1]
In the Configure Log Inputs section, select Log Path and input a meaningful name as INPUT NAME. Then put the absolute path of your custom log on the target server into the FILE PATH field.
[Figure: Create Agent Configuration - 2]
You may notice that there is a Advanced Parser Options link on this page. We can configure the parser of this agent configuration. But for now, to keep the simplicity, we will configure the parser later.
Now, click the Create Custom Log button again, we will be back to the Logs page and you should see the newly created Custom Log and associated Agent Configuration in the list. If you see NONE for the Agent Configurations, just refresh the page should be fine.
[Figure: Custom Log created]
Basically, creating the Custom Log in OCI Logging service has been completed. In next section, we will install and do some configuration to the Oracle Unified Monitoring Agent. So that the agent can communicate with OCI and use the configuration to ingest and upload the custom logs onto OCI Logging service.
Install Oracle Unified Monitoring Agent in the target DB system node
Oracle Unified Monitoring Agent can help us to ingest, parse and upload the custom logs from client side to OCI Logging service.
Oracle Unified Monitoring Agent supports Linux and Windows platform, and in the new OCI compute instances, it has been pre-installed. But as we are deploying it to the DBCS node, we need to install it manually.
Download
For now, OCI DBCS instances are built on Oracle Linux 7.8, so we should download the OEL7 RPM package of Oracle Unified Monitoring Agent
Install
Once download of Oracle Unified Monitoring Agent is complete. Just perform following command to install it.
sudo yum install -y <rpm-name>
You should see something like this.
[Figure: install unified-monitoring-agent]
If you wish to, you can check the status of the Oracle Unified Monitoring Agent related services with following commands.
systemctl status unified-monitoring-agent
systemctl status unified-monitoring-agent_config_downloader.service
systemctl status unified-monitoring-agent_config_downloader.timer
systemctl status unified-monitoring-agent_restarter.path
You should see something like below.
[Figure: status of the agent services, right after installation]
Configure
To make the agent be able to communicate with OCI Logging service, we need to do some configuration.
Oracle Unified Monitoring Agent communicates with the OCI Logging service via OCI API (similar to OCI CLI), so we need to generate and configure the key pair firstly.
Generate key pair
For our setup, we need to perform following command to generate key pair at a specific location /etc/unified-monitoring-agent/.oci.
cd /etc/unified-monitoring-agent
sudo mkdir .oci
cd .oci
# Generate the private key with the following commands.
# To generate the key, encrypted with a passphrase you provide when prompted:
openssl genrsa -out oci_api_key.pem -aes128 2048
# pass phrase: WhatEverYouWish
# Change the file permission to ensure that only you can read the private key file:
sudo chmod go-rwx oci_api_key.pem
# Generate the public key from your new private key:
sudo openssl rsa -pubout -in oci_api_key.pem -out oci_api_key_public.pem
# Copy the contents of the public key, you'll need to paste the value into the Console later).
sudo cat oci_api_key_public.pem
[Figure: Generate the private key with passphrase]
[Figure: Copy the public key]
Once you copied the content of the public key, visit OCI Web console and add it as a new API key.
[Figure: add new API key]
OCI will generate and show a fingerprint automatically on the console. And you can generate locally to make a verification.
[Figure: new API key added]
# Geenrate fingerprint to verify the key is correct
sudo openssl rsa -pubout -outform DER -in oci_api_key.pem | openssl md5 -c
[Figure: Generate fingerprint for verification]
Now, let's restart the Oracle Unified Monitoring Agent service and check it's log to see how it is running.
# restart the Oracle Unified Monitoring Agent
sudo systemctl restart unified-monitoring-agent
# Check the log
tail -100f /var/log/unified-monitoring-agent/unified-monitoring-agent.log
[Figure: Check log, configuration is empty]
As we can see the using configuration file line in the log, the configuration is still empty. That's because the agent doesn't know where to grab the configuration yet.
For more information about generating OCI API key, please check How to Generate an API Signing Key.
Create profile for Oracle Unified Monitoring Agent
Now, the key pair is ready. We will create a Profile for the agent, so that it knows where to go.
The profile is saved in a config file which should be in the same directory of the key pair /etc/unified-monitoring-agent/.oci.
cd /etc/unified-monitoring-agent/.oci
sudo vim config
Input the configuration of the profile like below. You can get the OCID of user and tenancy from the OCI Web Console. If you have installed OCI CLI, this should be quite familiar for you.
[DEFAULT]
user=<OCID of OCI USER>
fingerprint=84:52:c9:35:9e:11:a6:dc:f3:7a:fb:dd:89:4b:7b:49
key_file=/etc/unified-monitoring-agent/.oci/oci_api_key.pem
tenancy=<OCID of OCI TENANCY>
region=ap-tokyo-1
pass_phrase="<WhatEverYouWish>"
[UNIFIED_MONITORING_AGENT]
user=ocid1.user.oc1..aaaaaaaaowxmwiw75v2ib3mcclcijhdib5ofcladbz2csr4tacg766wl4exa
fingerprint=84:52:c9:35:9e:11:a6:dc:f3:7a:fb:dd:89:4b:7b:49
key_file=/etc/unified-monitoring-agent/.oci/oci_api_key.pem
tenancy=ocid1.tenancy.oc1..aaaaaaaaro7aox2fclu4urtpgsbacnrmjv46e7n4fw3sc2wbq24l7dzf3kba
region=ap-tokyo-1
pass_phrase="<WhatEverYouWish>"
Save the config file and restart the Oracle Unified Monitoring Agent service.
sudo systemctl restart unified-monitoring-agent
Check the log to see what is going on.
tail -100f /var/log/unified-monitoring-agent/unified-monitoring-agent.log
We should see something like below. In the using configuration file line, there are some new content between the <ROOT>, that's the agent configuration just downloaded from OCI Logging service. Since I have configured two agent configurations under my user, so in the screenshot you can see two <source> and <match>, the latter one is the one we just created in OCI Logging service.
[Figure: downloaded agent configuration]
If you find that the agent configuration cannot be downloaded successfully. Highly recommend you to check the official document Agent Troubleshooting.
View the custom log uploaded to OCI Logging service
After a while (more than 3 minutes), we could check the agent log and should be able to see that our custom log (cpu usage) has been uploaded to OCI Logging service and the response code is 200.
[Figure: agent upload log]
Also, we should be able to see the uploaded custom log with cpu usage information in OCI Logging service.
[Figure: custom logs uploaded into OCI Logging]
If we want to check whether the log is correct, we can compare it with the original generated log on the DB System node.
[Figure: custom logs on DB System node]
Add parser to the agent configuration
Now, we can successfully ingest and upload the custom log from DB System node to the OCI Logging service. But you may notice that the log entry is just as it is in the log file. Because we have not add any parser yet. The agent doesn't know how to parse the log entry appropriately.
In this section we will add a parser to extract out the CPU Usage from each log entry.
Add parser
Visit the Agent Configuration we just created,
[Figure: Visit Agent Configuration]
Click Edit
[Figure: Edit the Agent Configuration]
Click Advanced Parser Options to start adding the new parser.
[Figure: Advanced Parser Options]
Select REGEXP option for the PARSER, as we are going to use regular expression to parse the log entry. Input ^(?<log-time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) (?<cpu-usage>.*)$ in the EXPRESSION field and %Y-%m-%d %H:%M:%S in the TIME FORMAT field. This expression will help us to extract the log entry time and corresponding CPU Usage.
[Figure: add REGEXP parser]
After adding the parser to the Agent Configuration, we need to wait a while because the synchronization of the new configuration takes some time. Or you could restart the Unified Monitoring Agent service to make it faster.
Check new configuration and parsed log entry
We can check the new agent configuration from the agent log
tail -100f /var/log/unified-monitoring-agent/unified-monitoring-agent.log
[Figure: Check the new agent configuration on DB System node]
In OCI Logging service, we should be able to see the parsed information log-time and cpu-usage.
[Figure: new parsed log entry in OCI Logging service]
Next
[Figure: Step 2 completed]
Until now, we have completed the step 2: Configure OCI Logging service to ingest the custom log. We can see the parsed log-time and cpu-usage information in the OCI Logging service.
In next step, we will continue to complete the step 3: Configure OCI Service Connector Hub service to transfer the custom log to OCI Functions