はじめに
OCI 監査ログを OCI AIデータ・プラットフォーム (AIDP) を使って集計・分析する連載の Part 2 です。
Qiita の記事は 3本に分けて掲載します。
Part 2 では、サービス・コネクタ・ハブ経由でオブジェクト・ストレージに格納された監査ログを、AIDP を使って分析用に整える作業を扱います。
尚、全編を通して、ociaidp と言うコンパートメント内で作業を行なっています。
監査ログを整える理由
監査ログをそのまま分析に使うのは、以下の理由であまり都合がよくありません。
- 分析に不要な項目が多く存在する
- データ形式を適切に変換した方が分析しやすい（String → Number, Timestamp 等）
- Jsonのようなネストした構造のままだと扱いにくい（分析の過程で Relational DB にエクスポートしたい場合など）
- 重複レコードが存在する - そのままだと不正確な分析結果になる
そこで、以降のステップで行う分析作業を簡単・効率的に行うために、AIDP を使って生の監査ログを整えていく訳です。
そしてここで作成された Parquet ファイルが、Part 3 で分析のための基本データとして更に分析用途毎に処理されていきます。
AIDP 環境の準備
まず最初に AIDP の準備を始めます。
色々と構築する要素が出てきますので、頭の整理をしておきましょう。
- まず、作業する場所として Workbench が必要です。
- Workbench の中は Workspace という複数の作業環境に分けられます。Workbench を作成する際に、デフォルトの Workspace も作成します。
- Workspace には、データを処理するランタイムが必要です。Workspace には
Default Master Catalog Computeと呼ばれるカタログを管理するランタイムが自動的に作られます。また、これとは別に Notebook がアタッチする Apache Spark のランタイム「Cluster」が必要です。これは自動で作成されません、用途に応じたスペックの Cluster を自分で作成します。 - Workbench には
Master Catalogと呼ばれるカタログがあり、必要に応じてここにデータベースやボリュームを登録します。
Workbench, Workspace の作成
OCIコンソールから アナリティクスとAI > データ・レイク AIデータ・プラットフォーム と進みます。
Create AI Data Platform Workbench ボタンを押すと作成画面が出てきます。
Workbench 名を workbench1、Workspace 名を workspace1として前に進みます。
必要なポリシーの追加を促す画面になりますが、ここでは
- コンパートメントレベルのポリシー追加
- オプションで
Enable object deletion
を追加します。
最後に create ボタンを押すと、Workbench が作成されます。
workbench1 のリンクをクリックすると、Workbench 環境にログインできます。
Cluster の作成
左側のメニューから Workspace / workspacw1 / Compute の画面に移ると、右上に Create ボタンがあるので、クリックします。
Cluster の名前を cluster1 にして、今回は Quickstart の構成で作成します。
Compute に cluster1 という Cluster が作成され、Active になっていることを確認して下さい。
※ cluster1 も Default Master Catalog Compute も課金の対象となりますのでご注意下さい。
Notebook で処理を実装する
では、いよいよ Notebook を使って、監査ログを整えます。
Notebook の作成と Cluster のアタッチ
workspace1 に AuditLog というフォルダを作成して、右上の Create ボタンから Notebook を選択して作成します。
Notebook を実行できるように、先ほど作成した cluster1 をアタッチします。
Notebook の名前も parse_audit_log.ipynb に変更しておきましょう。
Spark のお作法の確認
AIDP では、以下の流儀を守って下さい。
-
SparkSession はシンプルに取得する
from pyspark.sql import SparkSession spark = SparkSession.builder.getOrCreate() -
spark.stop() は呼ばない
監査ログデータの処理
監査ログのロード
まず、生の監査ログを全件ロードして、様子を見てみましょう。
$\small \textsf{script}$
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
spark.conf.set("spark.sql.caseSensitive", "true")
service_connector_id = "ocid1.serviceconnector.oc1....."
os_namespace = "xxxxxxxx"
audit_log_bucket = "audit_log"
df = spark.read.json(f"oci://{audit_log_bucket}@{os_namespace}/{service_connector_id}/*.log.gz")
print(f"Read {df.count():,} records")
df.printSchema()
監査ログは、1行1レコードの Json 形式で、複数行レコードが入った gzip 圧縮ファイルが、特定のフォルダに順次蓄積されていきます。AIDP (Spark)は、圧縮ファイルの読み込みに対応していて、ワイルドカード指定もできるので、これをまるっと読み込むことができます。
df = spark.read.json(f"oci://{audit_log_bucket}@{os_namespace}/{service_connector_id}/*.log.gz")
オブジェクト・ストレージへのアクセスは、OCI HDFS Connector 仕様なので、従来と同じように oci://<バケット>@<ネームスペース>/<フォルダ>/<ファイル> で行えます。AIDP では Volume というものを定義できて、Volume を経由してオブジェクト・ストレージにアクセスする方法もありますが、これは Part 3 で紹介します。
監査ログのデータには、同一レコード内に大文字/小文字の異なる同じカラム名が入っているケースがあり、Spark はデフォルトでカラム名（ここではJsonのキーの名前）の大文字/小文字を区別しないため、そのまま処理させようとするとエラーとなります。
具体例を挙げると、以下のようなJsonデータに遭遇して処理が中断したことがあります。
.data.request.headers.accept-language
.data.request.headers.Accept-Language
（case-insensitive だと同一になる二つの要素がレコード上に存在）
これを回避して処理を継続させるためには、spark.sql.caseSensitive を true にセットしておく必要があります。
spark.conf.set("spark.sql.caseSensitive", "true")
出力結果を見てみましょう
$\small \textsf{output}$
Read 12,059 records
root
|-- data: struct (nullable = true)
| |-- additionalDetails: struct (nullable = true)
| | |-- JobRunKey: string (nullable = true)
| | |-- X-Real-Port: long (nullable = true)
| | |-- bucketId: string (nullable = true)
| | |-- bucketName: string (nullable = true)
| | |-- connectorState: string (nullable = true)
| | |-- dataLakeId: string (nullable = true)
| | |-- description: string (nullable = true)
| | |-- eTag: string (nullable = true)
| | |-- isAccessable: boolean (nullable = true)
| | |-- jobKey: string (nullable = true)
| | |-- jobName: string (nullable = true)
| | |-- jobStatus: string (nullable = true)
| | |-- lifeCycleState: string (nullable = true)
| | |-- namespace: string (nullable = true)
| | |-- publicAccessType: string (nullable = true)
| | |-- timeUpdated: string (nullable = true)
| | |-- versioning: string (nullable = true)
| | |-- workspaceKey: string (nullable = true)
| |-- availabilityDomain: string (nullable = true)
| |-- compartmentId: string (nullable = true)
| |-- compartmentName: string (nullable = true)
| |-- definedTags: struct (nullable = true)
| | |-- Oracle-Tags: struct (nullable = true)
| | | |-- CreatedBy: string (nullable = true)
| | | |-- CreatedOn: string (nullable = true)
| |-- eventGroupingId: string (nullable = true)
| |-- eventName: string (nullable = true)
| |-- freeformTags: struct (nullable = true)
| | |-- WorkspaceKey: string (nullable = true)
| | |-- dataLakeId: string (nullable = true)
| |-- identity: struct (nullable = true)
| | |-- authType: string (nullable = true)
| | |-- callerId: string (nullable = true)
| | |-- callerName: string (nullable = true)
| | |-- consoleSessionId: string (nullable = true)
| | |-- credentials: string (nullable = true)
| | |-- ipAddress: string (nullable = true)
| | |-- principalId: string (nullable = true)
| | |-- principalName: string (nullable = true)
| | |-- tenantId: string (nullable = true)
| | |-- userAgent: string (nullable = true)
| |-- message: string (nullable = true)
| |-- request: struct (nullable = true)
| | |-- action: string (nullable = true)
| | |-- headers: struct (nullable = true)
| | | |-- Accept: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- Accept-Encoding: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- Accept-Language: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- Authorization: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- Connection: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- Content-Length: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- Content-Type: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- Date: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- If-None-Match: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- Oci-Original-Host: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- Oci-Splat-Service-Operation-Id: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- Opc-Request-Id: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- Origin: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- Referer: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- Sec-Ch-Ua: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- Sec-Ch-Ua-Mobile: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- Sec-Ch-Ua-Platform: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- Sec-Fetch-Dest: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- Sec-Fetch-Mode: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- Sec-Fetch-Site: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- User-Agent: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- X-Date: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- X-Forwarded-For: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- X-OCI-LB-NetworkMetadata: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- X-OCI-LB-PrivateAccessMetadata: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- X-Real-IP: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- X-Real-Port: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- accept-language: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- authorization: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- content-length: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- date: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- host: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- if-match: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- oci-original-host: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- oci-original-url: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- oci-skip-authorization-for-splat: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- oci-skip-nbac-for-splat: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- oci-splat-audited: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- oci-splat-authorization-request-hash: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- oci-splat-generated-ocids: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- oci-splat-internal-context: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- oci-splat-service-operation-id: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- opc-client-info: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- opc-client-request-id: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- opc-client-retries: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- opc-obo-token: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- opc-principal: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- opc-request-id: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- opc-retry-token: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- path: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- sec-ch-ua: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- sec-ch-ua-mobile: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- sec-ch-ua-platform: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- should-mark-all-notifications-read: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- should-update-recent: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- type: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- x-content-sha256: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- x-date: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- x-sdk-type: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- x-sdk-version: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | |-- id: string (nullable = true)
| | |-- parameters: struct (nullable = true)
| | | |-- 1767061942725: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767061945830: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767061946877: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767061978403: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767061979180: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767062011277: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767062078386: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767062079300: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767062111584: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767062113132: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767062115746: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767062278963: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767062279998: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767062349947: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767062351119: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767062364640: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767062372835: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767062381101: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767062383273: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767062388356: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767062398488: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767062409159: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767062471098: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767062471698: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767062473669: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767062477182: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767062497586: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767062498325: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767062545546: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767062578093: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767062580253: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767062580894: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767062606918: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767062609972: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767062635097: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767062635958: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767062663710: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767062673841: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767062674598: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767062680492: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767062755963: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767062756523: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767062788481: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767063119705: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767063120324: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767063122227: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767063258211: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767063258220: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767063414624: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767064017834: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767064019257: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767064021343: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767065047561: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767065048858: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767065168836: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767065169343: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767065201554: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767065310825: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767065311964: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767065311974: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767065312327: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767065357977: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767065358806: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767065360534: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767065383613: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767065384457: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767073361382: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767073362923: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767073482898: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767073484110: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767073486411: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767073607197: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767073607708: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767073609626: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767073850720: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767073851227: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767073852995: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767073970717: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767073971703: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767073973597: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767074211837: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767074213282: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767074333249: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767074333786: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767074335778: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767074504320: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767074505191: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767074548822: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767074550302: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767074648351: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767074649826: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767074690797: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767074691299: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767074693269: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767074708011: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767074710046: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767074748690: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767074749798: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767074750394: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767074818047: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767074819127: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767074939105: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767074939638: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767074941650: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767075062113: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767075062650: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767075064761: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767075185216: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767075186250: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767075188254: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767075308747: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767075309278: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767075311787: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767075432288: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767075433071: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767075435538: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767075556443: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767075556979: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767075559332: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767075679827: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767075680382: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767075682193: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767075802660: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767075803260: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767075804941: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767075924352: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767075924879: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767075927119: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767075927192: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767075952596: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767075953256: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076367130: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076368196: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076370672: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076371461: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076474730: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076477177: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076478270: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076578016: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076582058: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076582855: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076596617: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076597382: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076636040: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076636913: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076641908: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076665776: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076666256: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076667899: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076674235: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076675098: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076685899: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076685938: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076686542: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076699822: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076700523: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076726748: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076734228: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076737421: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076746613: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076757165: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076757454: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076758268: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076758999: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076764903: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076765731: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076788453: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076789143: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076800467: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076801348: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076815894: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076816496: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076820522: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076821001: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076822633: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076827547: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076828004: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076829667: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076830790: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076837066: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076837653: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076837784: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076838011: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076913174: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076913713: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076916018: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076916869: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076929896: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076930695: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076932077: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076933198: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076935596: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767076936374: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767077041004: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767077041775: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767077042677: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767077043673: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767077043686: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767077053970: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767077054994: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767077174993: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767077175650: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767077177730: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767077298250: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767077298773: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767077300649: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767077421093: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767077424606: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767077427104: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767077547591: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767077548137: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767077550615: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767077671070: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767077671687: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767077673739: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767077794180: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767077794731: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767077796885: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767077917324: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767077917857: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767077919956: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767078040742: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767078041284: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767078043326: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767078163987: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767078164503: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767078196849: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767078531013: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767078532348: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767078534958: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767078655773: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767078656324: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767078658327: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767078778784: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767078779330: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767078811659: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767079146463: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767079147017: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767079149059: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767079269531: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767079270809: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767079273025: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767079393489: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767079394591: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767079396527: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767079517114: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767079517656: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767079519590: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767079520298: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767079520738: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767079522431: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767079525947: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767079526408: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767079528184: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767079648642: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767079649194: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767079651192: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767079711201: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767079711765: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767079713706: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767079722043: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767079724451: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767079725375: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767079728412: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767079735755: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767079746108: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767079751229: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767079751415: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767079845386: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767079845911: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767079848645: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767079926303: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767079973205: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767079981678: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767079982186: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767079983795: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767094306541: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767094307793: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767094388542: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767094390076: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767094598353: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767094599142: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767094601118: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767094635446: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767094636435: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767094638425: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767094639441: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767094640874: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767094641974: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767094814755: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- 1767094815834: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- aiModelCapabilities: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- availabilityDomain: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- catalogKey: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- compartmentId: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- content: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- displayName: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- excludeLifecycleState: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- featureNames: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- fields: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- format: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- hash: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- includeLegacy: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- isLocalizedLicenseTypeReturned: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- isNewProductRequirement: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- isRead: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- isSent: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- jobKey: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- limit: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- logType: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- modelType: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- name: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- objectDescription: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- page: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- param0: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- param1: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- param2: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- path: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- recordCount: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- roleScope: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- schemaKey: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- serviceConnectorId: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- shouldFilterByCallingPrincipal: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- sortBy: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- sortOrder: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- sourceService: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- state: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- status: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- type: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | |-- path: string (nullable = true)
| |-- resourceId: string (nullable = true)
| |-- response: struct (nullable = true)
| | |-- headers: struct (nullable = true)
| | | |-- Access-Control-Allow-Credentials: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- Access-Control-Allow-Headers: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- Access-Control-Allow-Methods: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- Access-Control-Allow-Origin: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- Access-Control-Expose-Headers: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- Cache-Control: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- Connection: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- Content-Encoding: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- Content-Length: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- Content-Location: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- Content-Type: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- Date: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- ETag: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- Location: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- Opc-Request-Id: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- Pragma: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- Strict-Transport-Security: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- Timing-Allow-Origin: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- Transfer-Encoding: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- Vary: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- X-Content-Type-Options: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- X-Frame-Options: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- access-control-allow-credentials: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- access-control-allow-methods: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- access-control-allow-origin: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- access-control-expose-headers: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- cache-control: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- datalake-async-operation-key: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- date: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- etag: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- has-incomplete-policy-set: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- last-modified: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- location: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- oci-splat-authorization-verify-content: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- oidl-async-operation-key: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- opc-client-request-id: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- opc-limit: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- opc-next-page: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- opc-prev-page: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- opc-request-id: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- opc-total-items: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- opc-work-request-id: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- strict-transport-security: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- x-api-id: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- x-content-type-options: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | |-- message: string (nullable = true)
| | |-- payload: struct (nullable = true)
| | | |-- compartmentId: string (nullable = true)
| | | |-- id: string (nullable = true)
| | | |-- operationType: string (nullable = true)
| | | |-- percentComplete: double (nullable = true)
| | | |-- resourceName: string (nullable = true)
| | | |-- resources: array (nullable = true)
| | | | |-- element: struct (containsNull = true)
| | | | | |-- actionType: string (nullable = true)
| | | | | |-- entityType: string (nullable = true)
| | | | | |-- entityUri: string (nullable = true)
| | | | | |-- identifier: string (nullable = true)
| | | |-- status: string (nullable = true)
| | | |-- timeAccepted: string (nullable = true)
| | | |-- timeFinished: string (nullable = true)
| | | |-- timeStarted: string (nullable = true)
| | |-- responseTime: string (nullable = true)
| | |-- status: string (nullable = true)
| |-- stateChange: struct (nullable = true)
| | |-- current: struct (nullable = true)
| | | |-- JobRun: struct (nullable = true)
| | | | |-- createdBy: string (nullable = true)
| | | | |-- createdByName: string (nullable = true)
| | | | |-- delegationToken: string (nullable = true)
| | | | |-- endTime: long (nullable = true)
| | | | |-- executionDuration: long (nullable = true)
| | | | |-- isInternalOrchestration: boolean (nullable = true)
| | | | |-- jobKey: string (nullable = true)
| | | | |-- jobName: string (nullable = true)
| | | | |-- key: string (nullable = true)
| | | | |-- launched: string (nullable = true)
| | | | |-- lifecycleStates: array (nullable = true)
| | | | | |-- element: struct (containsNull = true)
| | | | | | |-- endTime: long (nullable = true)
| | | | | | |-- startTime: long (nullable = true)
| | | | | | |-- status: string (nullable = true)
| | | | |-- maxConcurrentRuns: long (nullable = true)
| | | | |-- name: string (nullable = true)
| | | | |-- rootJobRunKey: string (nullable = true)
| | | | |-- startTime: long (nullable = true)
| | | | |-- state: struct (nullable = true)
| | | | | |-- status: string (nullable = true)
| | | | |-- systemParameters: struct (nullable = true)
| | | | | |-- idl.id: string (nullable = true)
| | | | | |-- idl.region: string (nullable = true)
| | | | | |-- job.id: string (nullable = true)
| | | | | |-- job.name: string (nullable = true)
| | | | | |-- job.repair_count: string (nullable = true)
| | | | | |-- job.run_id: string (nullable = true)
| | | | | |-- job.start_time.day: string (nullable = true)
| | | | | |-- job.start_time.hour: string (nullable = true)
| | | | | |-- job.start_time.is_weekday: string (nullable = true)
| | | | | |-- job.start_time.iso_date: string (nullable = true)
| | | | | |-- job.start_time.iso_datetime: string (nullable = true)
| | | | | |-- job.start_time.iso_weekday: string (nullable = true)
| | | | | |-- job.start_time.minute: string (nullable = true)
| | | | | |-- job.start_time.month: string (nullable = true)
| | | | | |-- job.start_time.second: string (nullable = true)
| | | | | |-- job.start_time.timestamp_ms: string (nullable = true)
| | | | | |-- job.start_time.year: string (nullable = true)
| | | | | |-- job.trigger_type: string (nullable = true)
| | | | | |-- workspace.id: string (nullable = true)
| | | | | |-- workspace.url: string (nullable = true)
| | | | |-- taskToTaskRunMap: struct (nullable = true)
| | | | | |-- Notebook_Scheduler_Task: string (nullable = true)
| | | | |-- tasks: array (nullable = true)
| | | | | |-- element: struct (containsNull = true)
| | | | | | |-- cluster: struct (nullable = true)
| | | | | | | |-- clusterKey: string (nullable = true)
| | | | | | |-- dependsOn: array (nullable = true)
| | | | | | | |-- element: string (containsNull = true)
| | | | | | |-- maxRetries: long (nullable = true)
| | | | | | |-- notebookPath: string (nullable = true)
| | | | | | |-- runIf: string (nullable = true)
| | | | | | |-- taskKey: string (nullable = true)
| | | | | | |-- type: string (nullable = true)
| | | | |-- tenantId: string (nullable = true)
| | | | |-- unifiedJobAndJobRunParameters: array (nullable = true)
| | | | | |-- element: string (containsNull = true)
| | | | |-- version: long (nullable = true)
| | | |-- activeClusterResources: struct (nullable = true)
| | | | |-- activeCores: double (nullable = true)
| | | | |-- activeExecutorCount: double (nullable = true)
| | | | |-- activeGpuCores: double (nullable = true)
| | | | |-- activeGpuMemoryInGBs: double (nullable = true)
| | | | |-- activeMemoryInGBs: double (nullable = true)
| | | |-- aiDataPlatformType: string (nullable = true)
| | | |-- aiFeatureStatus: string (nullable = true)
| | | |-- aliasKey: string (nullable = true)
| | | |-- autoTerminationMinutes: long (nullable = true)
| | | |-- bucketName: string (nullable = true)
| | | |-- catalogName: string (nullable = true)
| | | |-- category: string (nullable = true)
| | | |-- clusterRuntimeConfig: struct (nullable = true)
| | | | |-- initScripts: array (nullable = true)
| | | | | |-- element: string (containsNull = true)
| | | | |-- sparkVersion: string (nullable = true)
| | | | |-- type: string (nullable = true)
| | | |-- cluster_id: string (nullable = true)
| | | |-- compartmentId: string (nullable = true)
| | | |-- configuration: string (nullable = true)
| | | |-- created: string (nullable = true)
| | | |-- createdBy: string (nullable = true)
| | | |-- createdByName: string (nullable = true)
| | | |-- data: array (nullable = true)
| | | | |-- element: struct (containsNull = true)
| | | | | |-- type: string (nullable = true)
| | | | | |-- value: string (nullable = true)
| | | |-- defaultCatalogKey: string (nullable = true)
| | | |-- definedTags: string (nullable = true)
| | | |-- description: string (nullable = true)
| | | |-- displayName: string (nullable = true)
| | | |-- driverConfig: struct (nullable = true)
| | | | |-- driverShape: string (nullable = true)
| | | | |-- driverShapeConfig: struct (nullable = true)
| | | | | |-- gpus: long (nullable = true)
| | | | | |-- memoryInGBs: long (nullable = true)
| | | | | |-- ocpus: long (nullable = true)
| | | |-- eTag: string (nullable = true)
| | | |-- freeformTags: string (nullable = true)
| | | |-- fullName: string (nullable = true)
| | | |-- id: string (nullable = true)
| | | |-- isEnabled: string (nullable = true)
| | | |-- isPrivateNetworkEnabled: boolean (nullable = true)
| | | |-- isTruncated: boolean (nullable = true)
| | | |-- items: array (nullable = true)
| | | | |-- element: struct (containsNull = true)
| | | | | |-- fields: array (nullable = true)
| | | | | | |-- element: string (containsNull = true)
| | | | | |-- results: array (nullable = true)
| | | | | | |-- element: struct (containsNull = true)
| | | | | | | |-- data: struct (nullable = true)
| | | | | | | | |-- datetime: long (nullable = true)
| | | | | | | | |-- logContent: struct (nullable = true)
| | | | | | | | | |-- data: struct (nullable = true)
| | | | | | | | | | |-- clusterKey: string (nullable = true)
| | | | | | | | | | |-- eventType: string (nullable = true)
| | | | | | | | | | |-- identifierKey: string (nullable = true)
| | | | | | | | | | |-- message: string (nullable = true)
| | | | | | | | | | |-- workspaceKey: string (nullable = true)
| | | | | | | | | |-- id: string (nullable = true)
| | | | | | | | | |-- oracle: struct (nullable = true)
| | | | | | | | | | |-- compartmentid: string (nullable = true)
| | | | | | | | | | |-- ingestedtime: string (nullable = true)
| | | | | | | | | | |-- loggroupid: string (nullable = true)
| | | | | | | | | | |-- logid: string (nullable = true)
| | | | | | | | | | |-- tenantid: string (nullable = true)
| | | | | | | | | |-- source: string (nullable = true)
| | | | | | | | | |-- specversion: string (nullable = true)
| | | | | | | | | |-- time: string (nullable = true)
| | | | | | | | | |-- type: string (nullable = true)
| | | | | |-- summary: struct (nullable = true)
| | | | | | |-- resultCount: long (nullable = true)
| | | |-- jdbcEndpointUrl: string (nullable = true)
| | | |-- jobClusters: array (nullable = true)
| | | | |-- element: struct (containsNull = true)
| | | | | |-- clusterKey: string (nullable = true)
| | | |-- jobKey: string (nullable = true)
| | | |-- jobName: string (nullable = true)
| | | |-- kernel: struct (nullable = true)
| | | | |-- connections: long (nullable = true)
| | | | |-- execution_state: string (nullable = true)
| | | | |-- id: string (nullable = true)
| | | | |-- last_activity: string (nullable = true)
| | | | |-- name: string (nullable = true)
| | | |-- key: string (nullable = true)
| | | |-- last_modified: string (nullable = true)
| | | |-- launched: string (nullable = true)
| | | |-- lifecycleState: string (nullable = true)
| | | |-- lifecycleStateDetails: string (nullable = true)
| | | |-- lifecycleStates: array (nullable = true)
| | | | |-- element: struct (containsNull = true)
| | | | | |-- startTime: long (nullable = true)
| | | | | |-- status: string (nullable = true)
| | | |-- logGroupId: string (nullable = true)
| | | |-- logId: string (nullable = true)
| | | |-- logType: string (nullable = true)
| | | |-- maxConcurrentRuns: long (nullable = true)
| | | |-- name: string (nullable = true)
| | | |-- objectLifecyclePolicyETag: string (nullable = true)
| | | |-- outputParameters: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- path: string (nullable = true)
| | | |-- resource: string (nullable = true)
| | | |-- results: array (nullable = true)
| | | | |-- element: struct (containsNull = true)
| | | | | |-- aggregatedDataPoints: array (nullable = true)
| | | | | | |-- element: struct (containsNull = true)
| | | | | | | |-- timestamp: string (nullable = true)
| | | | | | | |-- value: double (nullable = true)
| | | | | |-- compartmentId: string (nullable = true)
| | | | | |-- dimensions: struct (nullable = true)
| | | | | | |-- clusterKey: string (nullable = true)
| | | | | | |-- datalakeId: string (nullable = true)
| | | | | | |-- executorId: string (nullable = true)
| | | | | | |-- resourceName: string (nullable = true)
| | | | | | |-- tenantId: string (nullable = true)
| | | | | |-- metadata: struct (nullable = true)
| | | | | | |-- displayName: string (nullable = true)
| | | | | | |-- unit: string (nullable = true)
| | | | | |-- name: string (nullable = true)
| | | | | |-- namespace: string (nullable = true)
| | | |-- retentionDuration: string (nullable = true)
| | | |-- rootJobRunKey: string (nullable = true)
| | | |-- schemaName: string (nullable = true)
| | | |-- shouldAddMandatoryPolicies: boolean (nullable = true)
| | | |-- shouldAddNonAssessedPolicies: boolean (nullable = true)
| | | |-- shouldAddOptionalPolicies: boolean (nullable = true)
| | | |-- sourceApi: string (nullable = true)
| | | |-- startTime: long (nullable = true)
| | | |-- state: string (nullable = true)
| | | |-- stateDetails: string (nullable = true)
| | | |-- statements: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- stoppedBy: string (nullable = true)
| | | |-- stoppedByName: string (nullable = true)
| | | |-- storageLocation: string (nullable = true)
| | | |-- tags: struct (nullable = true)
| | | | |-- authorized: boolean (nullable = true)
| | | | |-- dataDropped: boolean (nullable = true)
| | | | |-- definedTags: struct (nullable = true)
| | | | | |-- empty: boolean (nullable = true)
| | | | | |-- present: boolean (nullable = true)
| | | | |-- droppedTagSlugSections: struct (nullable = true)
| | | | | |-- empty: boolean (nullable = true)
| | | | | |-- present: boolean (nullable = true)
| | | | |-- freeformTags: struct (nullable = true)
| | | | | |-- empty: boolean (nullable = true)
| | | | | |-- present: boolean (nullable = true)
| | | | |-- internalTags: struct (nullable = true)
| | | | | |-- empty: boolean (nullable = true)
| | | | | |-- present: boolean (nullable = true)
| | | | |-- securityAttributes: struct (nullable = true)
| | | | | |-- empty: boolean (nullable = true)
| | | | | |-- present: boolean (nullable = true)
| | | | |-- systemSecurityAttributes: struct (nullable = true)
| | | | | |-- empty: boolean (nullable = true)
| | | | | |-- present: boolean (nullable = true)
| | | | |-- systemTags: struct (nullable = true)
| | | | | |-- empty: boolean (nullable = true)
| | | | | |-- present: boolean (nullable = true)
| | | | |-- version: long (nullable = true)
| | | |-- taskType: string (nullable = true)
| | | |-- tasks: array (nullable = true)
| | | | |-- element: struct (containsNull = true)
| | | | | |-- cluster: struct (nullable = true)
| | | | | | |-- clusterKey: string (nullable = true)
| | | | | |-- dependsOn: array (nullable = true)
| | | | | | |-- element: string (containsNull = true)
| | | | | |-- maxRetries: long (nullable = true)
| | | | | |-- notebookPath: string (nullable = true)
| | | | | |-- runIf: string (nullable = true)
| | | | | |-- taskKey: string (nullable = true)
| | | | | |-- type: string (nullable = true)
| | | |-- tenancyId: string (nullable = true)
| | | |-- timeCreated: string (nullable = true)
| | | |-- timeLastModified: string (nullable = true)
| | | |-- timeUpdated: string (nullable = true)
| | | |-- type: string (nullable = true)
| | | |-- updatedBy: string (nullable = true)
| | | |-- version: long (nullable = true)
| | | |-- volumeType: string (nullable = true)
| | | |-- workRequestId: string (nullable = true)
| | | |-- workerConfig: struct (nullable = true)
| | | | |-- maxWorkerCount: long (nullable = true)
| | | | |-- minWorkerCount: long (nullable = true)
| | | | |-- workerShape: string (nullable = true)
| | | | |-- workerShapeConfig: struct (nullable = true)
| | | | | |-- gpus: long (nullable = true)
| | | | | |-- memoryInGBs: long (nullable = true)
| | | | | |-- ocpus: long (nullable = true)
| | | |-- writable: boolean (nullable = true)
| | |-- previous: struct (nullable = true)
| | | |-- bucketName: string (nullable = true)
| | | |-- compartmentId: string (nullable = true)
| | | |-- definedTags: struct (nullable = true)
| | | | |-- Oracle-Tags: struct (nullable = true)
| | | | | |-- CreatedBy: string (nullable = true)
| | | | | |-- CreatedOn: string (nullable = true)
| | | |-- displayName: string (nullable = true)
| | | |-- id: string (nullable = true)
| | | |-- lifecycleDetails: string (nullable = true)
| | | |-- lifecycleState: string (nullable = true)
| | | |-- lifecyleDetails: string (nullable = true)
| | | |-- objectLifecyclePolicyETag: string (nullable = true)
| | | |-- source: struct (nullable = true)
| | | | |-- kind: string (nullable = true)
| | | | |-- logSources: array (nullable = true)
| | | | | |-- element: struct (containsNull = true)
| | | | | | |-- compartmentId: string (nullable = true)
| | | | | | |-- logGroupId: string (nullable = true)
| | | |-- tags: struct (nullable = true)
| | | | |-- authorized: boolean (nullable = true)
| | | | |-- dataDropped: boolean (nullable = true)
| | | | |-- definedTags: struct (nullable = true)
| | | | | |-- empty: boolean (nullable = true)
| | | | | |-- present: boolean (nullable = true)
| | | | |-- droppedTagSlugSections: struct (nullable = true)
| | | | | |-- empty: boolean (nullable = true)
| | | | | |-- present: boolean (nullable = true)
| | | | |-- freeformTags: struct (nullable = true)
| | | | | |-- empty: boolean (nullable = true)
| | | | | |-- present: boolean (nullable = true)
| | | | |-- internalTags: struct (nullable = true)
| | | | | |-- empty: boolean (nullable = true)
| | | | | |-- present: boolean (nullable = true)
| | | | |-- securityAttributes: struct (nullable = true)
| | | | | |-- empty: boolean (nullable = true)
| | | | | |-- present: boolean (nullable = true)
| | | | |-- systemSecurityAttributes: struct (nullable = true)
| | | | | |-- empty: boolean (nullable = true)
| | | | | |-- present: boolean (nullable = true)
| | | | |-- systemTags: struct (nullable = true)
| | | | | |-- empty: boolean (nullable = true)
| | | | | |-- present: boolean (nullable = true)
| | | | |-- version: long (nullable = true)
| | | |-- target: struct (nullable = true)
| | | | |-- batchRolloverSizeInMBs: long (nullable = true)
| | | | |-- batchRolloverTimeInMs: long (nullable = true)
| | | | |-- bucketName: string (nullable = true)
| | | | |-- kind: string (nullable = true)
| | | |-- tasks: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- timeCreated: string (nullable = true)
| | | |-- timeUpdated: string (nullable = true)
| |-- systemTags: struct (nullable = true)
| | |-- orcl-aidp: struct (nullable = true)
| | | |-- governingAidpId: string (nullable = true)
|-- dataschema: string (nullable = true)
|-- id: string (nullable = true)
|-- oracle: struct (nullable = true)
| |-- compartmentid: string (nullable = true)
| |-- ingestedtime: string (nullable = true)
| |-- loggroupid: string (nullable = true)
| |-- tenantid: string (nullable = true)
|-- source: string (nullable = true)
|-- specversion: string (nullable = true)
|-- time: string (nullable = true)
|-- type: string (nullable = true)
ご覧の通り、スキーマが爆発していますw。今回の分析に必要十分な項目はもっと限られたものなので、これを整えていきます。
監査ログを整える
$\small \textsf{script}$
"""
上記画像より更新されています
userAgent は2箇所の値を評価しています → coalesce()
"""
from pyspark.sql.functions import col, to_timestamp, coalesce, udf
def int_status(status: str) -> int:
try: # 大抵の場合は、HTTPのステータスコードが入っている
return int(status)
except: # 'OK' とか 'Success' とかの場合は 0 にする
return 0
df_parsed = (
df
.withColumn("time", to_timestamp(col("time")))
.withColumn("message", col("data.message"))
.withColumn("ipAddress", col("data.identity.ipAddress"))
.withColumn("principalName", col("data.identity.principalName"))
.withColumn("eventName", col("data.eventName"))
.withColumn("compartmentName", col("data.compartmentName"))
.withColumn("resourceId", col("data.resourceId"))
.withColumn("path", col("data.request.path"))
.withColumn("action", col("data.request.action"))
.withColumn("userAgent", coalesce(col("data.identity.userAgent"), col("data.request.headers.User-Agent")[0]))
.withColumn("status", udf(int_status)("data.response.status").cast("int"))
.withColumn("opc-request-id", col("data.response.headers.opc-request-id")[0])
.select(
"id", "time", "type", "source", "message", "ipAddress", "principalName", "eventName",
"compartmentName", "resourceId", "path", "action", "userAgent", "status", "opc-request-id")
.dropDuplicates(["id"])
)
print(f"Parsed {df_parsed.count():,} records")
df_parsed.printSchema()
主な処理内容は以下の通り
- ネストされている必要項目を最上位の項目として再定義
-
time項目は String から Timestamp に変換 -
status項目は int型に変換- 普通に cast("int") したかったのですが、実はここには HTTP ステータス・コードの他にも "OK" や "Success" といった値が入っていて、そのまま数値型にキャストするとエラーで処理が止まってしまうことが判明。文字列のままとしても良かったのですが、後々の分析の効率を考えて、文字列は 数字の 0 に置換することとし、ユーザ定義関数 (udf) で変換をかけています。
- select() をかけて不要項目を削除
- 重複レコードを削除
- 監査ログに重複レコードがあることは盲点でした！ 重複データを削除するには
dropDuplicate([カラムの配列])の他に、distinct()（全カラムの値が一致するものを削除）も使えます。今回は、"id" カラムで一意性が保証されるので、これを使って重複データを削除しています。
- 監査ログに重複レコードがあることは盲点でした！ 重複データを削除するには
出力結果を見てましょう。
スキーマもスッキリして、レコード数も減っていますね。
整えられた監査ログを保存する
最後に、オブジェクト・ストレージに Parquet 形式で保存します。
$\small \textsf{script}$
parsed_file = f"oci://{audit_log_bucket}@{os_namespace}/analysis/parsed.parquet"
df_parsed.write.format("parquet").mode('overwrite').save(parsed_file)
Notebook をバッチ・ジョブにする
Notebook はバッチ・ジョブとして動かすことができます。
Notebook の右上に Action のメニューがあるので、そこから Schedule を選びます。
Job Name、Job Location、Cluster を入力し、Schedule を Manual run にしてジョブを作成します。
Workflow のページにジョブが作成されました。
右側の三角を押すとジョブが実行されます。
まとめ
そのままではスキーマも複雑で不要なデータが多く重複レコードも存在する生の監査ログを AIDPを使って分析用に整えました。Part 3 ではこの前処理済データを使って、本来の目的である分析作業を行います。
AIDP は構成要素が多くて全体感を掴むのが大変かもしれませんが、一度理解してしまうと非常にうまくデザインされた統合分析基盤だと感じていただけると思います。
参考