ネットレックスで開発しているあるサービスではApache Beamを使うことで大量データ処理を可能としています。Apache Beamの実行エンジンにはOCIのデータ・フローを使っています。
データ処理に長時間を要することもあり、処理中のCPU使用率をモニターしたいという要件が出てきたので、Oracle Cloud Infrastructure SDK for Javaを使って対応してみました。

SDK for Java

SDK for Javaを使用することでOCIの各リソースへJavaのプログラムからアクセスすることができます。

OciMonitoringService.java

import java.util.Date;

import com.oracle.bmc.Region;
import com.oracle.bmc.auth.AuthenticationDetailsProvider;
import com.oracle.bmc.auth.SimpleAuthenticationDetailsProvider;
import com.oracle.bmc.auth.StringPrivateKeySupplier;
import com.oracle.bmc.monitoring.MonitoringClient;
import com.oracle.bmc.monitoring.model.AggregatedDatapoint;
import com.oracle.bmc.monitoring.model.MetricData;
import com.oracle.bmc.monitoring.model.SummarizeMetricsDataDetails;
import com.oracle.bmc.monitoring.requests.SummarizeMetricsDataRequest;
import com.oracle.bmc.monitoring.responses.SummarizeMetricsDataResponse;

public class OciMonitoringService {

    private MonitoringClient monitoringClient;

    public OciMonitoringService(String tenantId, String userId, String fingerprint, String privateKey, String region) {
        AuthenticationDetailsProvider provider = SimpleAuthenticationDetailsProvider
            .builder()
            .tenantId(tenantId)
            .userId(userId)
            .fingerprint(fingerprint)
            .privateKeySupplier(new StringPrivateKeySupplier(privateKey))
            .region(Region.valueOf(region))
            .build();
        monitoringClient = MonitoringClient.builder().build(provider);
    }

    public void getSummarizeMetricsData(String namespace, String compartmentId, String query, Date start, Date end) {
    	final SummarizeMetricsDataDetails details = SummarizeMetricsDataDetails
            .builder()
            .namespace(namespace)
            .query(query)
            .startTime(start)
            .endTime(end)
            .build();
    	final SummarizeMetricsDataRequest request = SummarizeMetricsDataRequest
            .builder()
            .compartmentId(compartmentId)
            .summarizeMetricsDataDetails(details)
            .build();
        final SummarizeMetricsDataResponse response = monitoringClient.summarizeMetricsData(request);
        Map<String, List<BizzRepoMetricData>> map = new HashMap<>();
        for (MetricData md : response.getItems()) {
            String executorId = md.getDimensions().get("executorId");
            System.out.println("executorId: " + executorId);
            List<BizzRepoMetricData> list = new ArrayList<>();
            for (AggregatedDatapoint ad : md.getAggregatedDatapoints()) {
                System.out.println("timestamp: " + ad.getTimestamp() + ", value: " + ad.getValue());
            }
        }
    }
}

問い合わせ

SDKを使ったプログラミングの基本部分はパターン化されているので悩むことが少ないのですが問い合わせに何を与えれば良いか戸惑います。
今回はモニターしたい内容(CPU使用率)をOCIコンソールで見ることができるので、OCIコンソールのメトリック・エクスプローラを使い問い合わせを取得しました。

図中の赤枠内に以下のような問い合わせが表示されます。xxx部分はデータ・フロー実行時に払い出されるRun OCIDです。

CpuUtilization[1m]{resourceId = "xxx"}.groupBy(executorId).max()

実行

問い合わせとして設定する文字列がわかったので以下のように呼び出してみます。

Date end = new Date();
Date start = new Date(end.getTime() - TimeUnit.HOURS.toMillis(6));
OciMonitoringService service = new OciMonitoringService(tenantId, userId, fingerprint, privateKey, "ap-tokyo-1");
String query = String.format("CpuUtilization[1m]{resourceId = \"%s\"}.groupBy(executorId).max()", resourceId);
service.getSummarizeMetricsData("oci_dataflow", compartmentId, query, start, end);

OCIコンソールのチャートで表示されるデータを取得できました。

executorId: 2
timestamp: Tue Nov 26 10:14:00 JST 2024, value: 47.215111824728396
timestamp: Tue Nov 26 10:15:00 JST 2024, value: 33.66617551708633
timestamp: Tue Nov 26 10:16:00 JST 2024, value: 25.869712644927535
timestamp: Tue Nov 26 10:17:00 JST 2024, value: 16.644831202518994
timestamp: Tue Nov 26 10:18:00 JST 2024, value: 14.896989345511797
timestamp: Tue Nov 26 10:19:00 JST 2024, value: 16.073803849325337
timestamp: Tue Nov 26 10:20:00 JST 2024, value: 16.413830787684926
timestamp: Tue Nov 26 10:21:00 JST 2024, value: 16.308916143906828
timestamp: Tue Nov 26 10:22:00 JST 2024, value: 17.69114901414434
timestamp: Tue Nov 26 10:23:00 JST 2024, value: 14.950278708382877
timestamp: Tue Nov 26 10:24:00 JST 2024, value: 19.04996555179966
timestamp: Tue Nov 26 10:25:00 JST 2024, value: 20.202852900449102
timestamp: Tue Nov 26 10:26:00 JST 2024, value: 15.835965179461613
timestamp: Tue Nov 26 10:27:00 JST 2024, value: 21.024125493864112
timestamp: Tue Nov 26 10:28:00 JST 2024, value: 17.061395058982306
timestamp: Tue Nov 26 10:29:00 JST 2024, value: 17.44435005123463
timestamp: Tue Nov 26 10:30:00 JST 2024, value: 15.561718266443423
timestamp: Tue Nov 26 10:31:00 JST 2024, value: 16.27758532337065
timestamp: Tue Nov 26 10:32:00 JST 2024, value: 20.808900012496252
timestamp: Tue Nov 26 10:33:00 JST 2024, value: 22.08956719062375
timestamp: Tue Nov 26 10:34:00 JST 2024, value: 16.861339707779322
timestamp: Tue Nov 26 10:35:00 JST 2024, value: 14.667082148284313
timestamp: Tue Nov 26 10:36:00 JST 2024, value: 14.838265626864189
timestamp: Tue Nov 26 10:37:00 JST 2024, value: 15.538071675465684
timestamp: Tue Nov 26 10:38:00 JST 2024, value: 17.451260658877608
timestamp: Tue Nov 26 10:39:00 JST 2024, value: 15.521340474107767
timestamp: Tue Nov 26 10:40:00 JST 2024, value: 18.82509226654669
timestamp: Tue Nov 26 10:41:00 JST 2024, value: 17.192598966560034
timestamp: Tue Nov 26 10:42:00 JST 2024, value: 45.38113786988903
executorId: driver
timestamp: Tue Nov 26 10:14:00 JST 2024, value: 13.994138569078949
timestamp: Tue Nov 26 10:15:00 JST 2024, value: 3.009411539614347
timestamp: Tue Nov 26 10:16:00 JST 2024, value: 2.443987251424003
timestamp: Tue Nov 26 10:17:00 JST 2024, value: 3.2622595773381295
timestamp: Tue Nov 26 10:18:00 JST 2024, value: 2.725771807553957
timestamp: Tue Nov 26 10:19:00 JST 2024, value: 2.3127658679792122
timestamp: Tue Nov 26 10:20:00 JST 2024, value: 2.232527673018067
timestamp: Tue Nov 26 10:21:00 JST 2024, value: 2.978815507022612
timestamp: Tue Nov 26 10:22:00 JST 2024, value: 1.8027569095352085
timestamp: Tue Nov 26 10:23:00 JST 2024, value: 2.121465471444234
timestamp: Tue Nov 26 10:24:00 JST 2024, value: 1.5193901645064807
timestamp: Tue Nov 26 10:25:00 JST 2024, value: 1.907506912878788
timestamp: Tue Nov 26 10:26:00 JST 2024, value: 1.7157694627686157
timestamp: Tue Nov 26 10:27:00 JST 2024, value: 1.6959849550224888
timestamp: Tue Nov 26 10:28:00 JST 2024, value: 1.6008093387644942
timestamp: Tue Nov 26 10:29:00 JST 2024, value: 1.814838088455772
timestamp: Tue Nov 26 10:30:00 JST 2024, value: 3.0769256421789106
timestamp: Tue Nov 26 10:31:00 JST 2024, value: 1.9373515317341328
timestamp: Tue Nov 26 10:32:00 JST 2024, value: 1.5662248663895488
timestamp: Tue Nov 26 10:33:00 JST 2024, value: 1.8046221539071796
timestamp: Tue Nov 26 10:34:00 JST 2024, value: 1.6223892803315751
timestamp: Tue Nov 26 10:35:00 JST 2024, value: 1.4013509276303042
timestamp: Tue Nov 26 10:36:00 JST 2024, value: 1.3944580592734226
timestamp: Tue Nov 26 10:37:00 JST 2024, value: 1.805507703039362
timestamp: Tue Nov 26 10:38:00 JST 2024, value: 1.6221217216391803
timestamp: Tue Nov 26 10:39:00 JST 2024, value: 1.630967471264368
timestamp: Tue Nov 26 10:40:00 JST 2024, value: 1.3268796776611695
timestamp: Tue Nov 26 10:41:00 JST 2024, value: 4.30326849096175
timestamp: Tue Nov 26 10:42:00 JST 2024, value: 4.699013595483162
executorId: 1
timestamp: Tue Nov 26 10:14:00 JST 2024, value: 47.33160149186708
timestamp: Tue Nov 26 10:15:00 JST 2024, value: 35.736272612387616
timestamp: Tue Nov 26 10:16:00 JST 2024, value: 23.736135487055176
timestamp: Tue Nov 26 10:17:00 JST 2024, value: 14.732141665833668
timestamp: Tue Nov 26 10:18:00 JST 2024, value: 15.710203143428286
timestamp: Tue Nov 26 10:19:00 JST 2024, value: 15.299971605747354
timestamp: Tue Nov 26 10:20:00 JST 2024, value: 19.98200274070659
timestamp: Tue Nov 26 10:21:00 JST 2024, value: 17.25289678863614
timestamp: Tue Nov 26 10:22:00 JST 2024, value: 16.391403538841246
timestamp: Tue Nov 26 10:23:00 JST 2024, value: 14.841000216296804
timestamp: Tue Nov 26 10:24:00 JST 2024, value: 17.072250321153653
timestamp: Tue Nov 26 10:25:00 JST 2024, value: 18.81041086065574
timestamp: Tue Nov 26 10:26:00 JST 2024, value: 14.303889119514146
timestamp: Tue Nov 26 10:27:00 JST 2024, value: 14.963062573727882
timestamp: Tue Nov 26 10:28:00 JST 2024, value: 17.909522279566133
timestamp: Tue Nov 26 10:29:00 JST 2024, value: 18.78414842047386
timestamp: Tue Nov 26 10:30:00 JST 2024, value: 15.995661552878849
timestamp: Tue Nov 26 10:31:00 JST 2024, value: 19.590190742777168
timestamp: Tue Nov 26 10:32:00 JST 2024, value: 16.6897450039988
timestamp: Tue Nov 26 10:33:00 JST 2024, value: 17.70394955542383
timestamp: Tue Nov 26 10:34:00 JST 2024, value: 16.065008240223463
timestamp: Tue Nov 26 10:35:00 JST 2024, value: 13.519411181650371
timestamp: Tue Nov 26 10:36:00 JST 2024, value: 15.95819095563561
timestamp: Tue Nov 26 10:37:00 JST 2024, value: 18.233096428037847
timestamp: Tue Nov 26 10:38:00 JST 2024, value: 14.981370060957696
timestamp: Tue Nov 26 10:39:00 JST 2024, value: 13.619805594713657
timestamp: Tue Nov 26 10:40:00 JST 2024, value: 14.710250174947515
timestamp: Tue Nov 26 10:41:00 JST 2024, value: 14.670726207137857
timestamp: Tue Nov 26 10:42:00 JST 2024, value: 47.03027449390183