0
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

@torippy1024
in

【Splunk O11y Cloud】Splunk Add-On for OpenTelemetry Collectorのコンフィグフォーマットが変更になったので設定手順を更新する

0
Posted at

こんばんは。torippy1024です。
以前、Splunk Add-On for OpenTelemetry Collectorについての記事(https://qiita.com/torippy1024/items/5a48775b9f0252176cbf )を書いたのですが、バージョンアップに伴い、コンフィグフォーマットが変更になりました。

このため、実機で調査した結果をまとめます。
検証したバージョンはSplunk Add-On for OpenTelemetry Collector V0.160.1です。

バージョンアップによる変化

Splunk Add-On for OpenTelemetry Collectorは、2026年4月からコンフィグフォーマットが変更になりました。合わせて、バージョン番号についても、これまで本家OpenTelemetryとAdd-onバージョンが別々に管理されていたものを、同一バージョンで管理するように変更しました。この影響で、Add-on V1.12.0の次のバージョンはV0.148となりました。背景を知らずに見ると、バージョンが古くなったように勘違いする可能性があるので注意が必要です。
https://github.com/signalfx/splunk-otel-collector/releases/tag/Splunk_TA_otel/v1.12.0

Splunk Add-On for OpenTelemetry Collectorの構成とコンフィグ(V0.160.1)

次に、新しくなったOTel Add-onの構成とコンフィグを確認します。
執筆時点の最新バージョンであるV0.160.1でのフォルダ構成は以下の通りです。以前よりも構成がすっきりしました。

Splunk_TA_otelのフォルダ構造
Splunk_TA_otel
├─ configs
|  ├─ agent_config.yaml
|  └─ gateway_config.yaml
├─ default
|  ├─ app.conf
|  └─ inputs.conf
├─ linux_x86_64
│  └─ bin
│      └─ Splunk_TA_otel
├─ README
│  └─ inputs.conf.spec
├─ static
│  ├─ appIcon.png
│  └─ appIcon_2x.png
└─ windows_x86_64
   └─ bin
       └─ Splunk_TA_otel.exe

以前(V1.12.0)のフォルダ構成は以下を参照してください。
https://qiita.com/torippy1024/items/5a48775b9f0252176cbf#splunk-add-on-for-opentelemetry-collector%E3%81%A7%E4%BA%8B%E5%89%8D%E5%AE%9A%E7%BE%A9%E3%81%95%E3%82%8C%E3%81%A6%E3%81%84%E3%82%8B%E3%82%B3%E3%83%B3%E3%83%95%E3%82%A3%E3%82%B0

また以前までは、使用するときはlocalフォルダ配下にinputs.confとaccess_tokenファイルを作成しなければならなかったのですが、access_tokenはinputs.conf内部で指定する仕様に変更になりました。このため作成対象はinputs.confのみとなりました。
また、以前はconfigsフォルダ配下には、エージェントモードのyaml(agent_config.yaml)とは別に、ゲートウェイ経由でエージェントモードのデータを転送するためのyamlが存在していましたが、それはなくなっています。
agent_config.yamlの中にゲートウェイ経由の設定がコメントアウトされて入っているので、agent_config.yaml自体を修正して使用する形になったようです。(アップグレード時に誤って上書きされることを防ぐため、agent_config.yamlを別名保存して修正したほうがいいかもしれません)

また、default/inputs.confファイルの中身は以下です。

default/inputs.conf
# ATTENTION: If copying this file as base for your local/inputs.conf
# rename the stanza below to something in the form [Splunk_TA_otel://<data_input_name>]
[Splunk_TA_otel]

# Global settings, see https://help.splunk.com/en/splunk-enterprise/administer/admin-manual/9.4/configuration-file-reference/9.4.1-configuration-file-reference/inputs.conf#global-settings-0
disabled=false
start_by_shell=false
interval = 0
index = _internal
sourcetype = Splunk_TA_otel

# TA specific settings
splunk_access_token =
splunk_realm =
splunk_config = $SPLUNK_HOME/etc/apps/$SPLUNK_MODINPUT_BASE_DIR_NAME/configs/agent_config.yaml
splunk_collector_log_level = error
splunk_collector_env_vars =
splunk_collector_cmd_args =

重要なことがコメントに記載されています。local/inputs.confにコピーする際は、Stanzaを[Splunk_TA_otel]から、[Splunk_TA_otel://<data_input_name>]に変更して使うことが想定されています。また、disabled=falseがデフォルトで指定されています。

一般的(たぶん)な他のAdd-onでは、デフォルトの設定がStanza名込みでdefault/inputs.conf内で定義されdisabled=trueとなっており、有効にする場合はlocal/inputs.confにて同じStanza名でdisabled=falseを指定することが多いのですが、少しお作法が異なることに注意が必要です。

次に、OTel Collectorのパラメータを指定しているyamlファイルの中身を見てみます。agent_config.yamlの中身は以下です。
(まあgithub上に公開されているのでそちらを見たほうが正しいのですが。。https://github.com/signalfx/splunk-otel-collector/blob/v0.160.1/cmd/otelcol/config/collector/agent_config.yaml)

agent_config.yaml
# Default configuration file for the Linux (deb/rpm) and Windows MSI collector packages

# If the collector is installed without the Linux/Windows installer script, the following
# environment variables are required to be manually defined or configured below:
# - SPLUNK_ACCESS_TOKEN: The Splunk access token to authenticate requests
# - SPLUNK_API_URL: The Splunk API URL, e.g. https://api.us0.observability.splunkcloud.com
# - SPLUNK_HEC_TOKEN: The Splunk HEC authentication token
# - SPLUNK_HEC_URL: The Splunk HEC endpoint URL, e.g. https://http-inputs-acme.splunkcloud.com/services/collector
# - SPLUNK_INGEST_URL: The Splunk ingest URL, e.g. https://ingest.us0.observability.splunkcloud.com
# - SPLUNK_LISTEN_INTERFACE: The network interface the agent receivers listen on.

extensions:
  headers_setter:
    headers:
      - action: upsert
        key: X-SF-TOKEN
        from_context: X-SF-TOKEN
        default_value: "${SPLUNK_ACCESS_TOKEN}"
  health_check:
    endpoint: "${SPLUNK_LISTEN_INTERFACE}:13133"
  http_forwarder:
    ingress:
      endpoint: "${SPLUNK_LISTEN_INTERFACE}:6060"
    egress:
      endpoint: "${SPLUNK_API_URL}"
      # Use instead when sending to gateway
      #endpoint: "${SPLUNK_GATEWAY_URL}"
  http_forwarder/opamp_splunk_o11y:
    ingress:
      endpoint: "${SPLUNK_LISTEN_INTERFACE}:4320"
    egress:
      endpoint: "${SPLUNK_INGEST_URL}"
      # Use instead when sending to gateway
      #endpoint: "${SPLUNK_GATEWAY_URL}:4320"
      # Remove this header to use a token passed by the caller instead.
      headers:
        X-SF-Token: "${SPLUNK_ACCESS_TOKEN}"
  # opamp/splunk_o11y is included in the default config, but startup removes it
  # from service.extensions unless --feature-gates=+splunk.opamp.enabled is set.
  opamp/splunk_o11y:
    agent_description:
      # Include the Collector's resource attributes as OpAMP non-identifying attributes.
      include_resource_attributes: true
    server:
      http:
        endpoint: "${SPLUNK_INGEST_URL}/v1/opamp"
        # Use instead when sending to gateway (assuming the gateway is running http_forwarder/signalfx on port 4320)
        #endpoint: "${SPLUNK_GATEWAY_URL}:4320/v1/opamp"
        polling_interval: 30s
        headers:
          X-SF-Token: "${SPLUNK_ACCESS_TOKEN}"
  zpages:
    #endpoint: "${SPLUNK_LISTEN_INTERFACE}:55679"
    expvar:
      enabled: true

receivers:
  fluent_forward:
    endpoint: "${SPLUNK_LISTEN_INTERFACE}:8006"
  host_metrics:
    collection_interval: 10s
    scrapers:
      cpu:
      disk:
      filesystem:
      memory:
      network:
      # System load average metrics https://en.wikipedia.org/wiki/Load_(computing)
      load:
      # Paging/Swap space utilization and I/O metrics
      paging:
      # Aggregated system process count metrics
      processes:
      # System processes metrics, disabled by default
      # process:
  jaeger:
    protocols:
      grpc:
        endpoint: "${SPLUNK_LISTEN_INTERFACE}:14250"
      thrift_binary:
        endpoint: "${SPLUNK_LISTEN_INTERFACE}:6832"
      thrift_compact:
        endpoint: "${SPLUNK_LISTEN_INTERFACE}:6831"
      thrift_http:
        endpoint: "${SPLUNK_LISTEN_INTERFACE}:14268"
  otlp:
    protocols:
      grpc:
        endpoint: "${SPLUNK_LISTEN_INTERFACE}:4317"
        # Uncomment below config to preserve incoming access token and use it instead of the token value set in exporter config
        # include_metadata: true
      http:
        endpoint: "${SPLUNK_LISTEN_INTERFACE}:4318"
        # Uncomment below config to preserve incoming access token and use it instead of the token value set in exporter config
        # include_metadata: true
  # This section is used to collect the OpenTelemetry Collector metrics
  # Even if just a Splunk APM customer, these metrics are included
  prometheus/internal:
    config:
      scrape_configs:
      - job_name: 'otel-collector'
        scrape_interval: 10s
        static_configs:
        - targets: ["0.0.0.0:8888"]
        metric_relabel_configs:
          - source_labels: [ __name__ ]
            regex: 'promhttp_metric_handler_errors.*'
            action: drop
          - source_labels: [ __name__ ]
            regex: 'otelcol_processor_batch_.*'
            action: drop
  smartagent/processlist:
    type: processlist
  zipkin:
    endpoint: "${SPLUNK_LISTEN_INTERFACE}:9411"
  nop:

processors:
  batch:
    metadata_keys:
      - X-SF-Token
  # Enabling the memory_limiter is strongly recommended for every pipeline.
  # Configuration is based on the amount of memory allocated to the collector.
  # For more information about memory limiter, see
  # https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/memorylimiter/README.md
  memory_limiter:
    check_interval: 2s
    limit_mib: ${SPLUNK_MEMORY_LIMIT_MIB}

  # Detect if the collector is running on a cloud system, which is important for creating unique cloud provider dimensions.
  # Detector order is important: the `system` detector goes last so it can't preclude cloud detectors from setting host/os info.
  # Resource detection processor is configured to override all host and cloud attributes because instrumentation
  # libraries can send wrong values from container environments.
  # https://docs.splunk.com/Observability/gdi/opentelemetry/components/resourcedetection-processor.html#ordering-considerations
  resource_detection:
    detectors: [gcp, ecs, ec2, azure, system]
    override: true

  # Limit explicit histograms to at most 32 buckets before export through signalfx exporter.
  # With `send_otlp_histograms: true`, the backend drops histograms above this limit.
  # With `send_otlp_histograms: false`, this reduces the emitted `_bucket` series and `le` boundaries.
  transform/limit_histogram_buckets:
    metric_statements:
      - context: datapoint
        statements:
          - merge_histogram_buckets(32, method="limit_buckets")

  # Optional: The following processor can be used to add a default "deployment.environment.name" attribute to the logs and
  # traces when it's not populated by instrumentation libraries.
  # If enabled, make sure to enable this processor in a pipeline.
  # To use the deprecated attribute, change the key to "deployment.environment".
  # For more information, see https://docs.splunk.com/Observability/gdi/opentelemetry/components/resource-processor.html
  #resource/add_environment:
    #attributes:
      #- action: insert
        #value: staging/production/...
        #key: deployment.environment.name

exporters:
  # Traces
  otlp_http:
    traces_endpoint: "${SPLUNK_INGEST_URL}/v2/trace/otlp"
    headers:
      "X-SF-Token": "${SPLUNK_ACCESS_TOKEN}"
    auth:
      authenticator: headers_setter
  # Metrics + Events
  signalfx:
    access_token: "${SPLUNK_ACCESS_TOKEN}"
    api_url: "${SPLUNK_API_URL}"
    ingest_url: "${SPLUNK_INGEST_URL}"
    # Use instead when sending to gateway
    #api_url: http://${SPLUNK_GATEWAY_URL}:6060
    #ingest_url: http://${SPLUNK_GATEWAY_URL}:9943
    sync_host_metadata: true
    correlation:
  # Entities (applicable only if discovery mode is enabled)
  otlp_http/entities:
    logs_endpoint: "${SPLUNK_INGEST_URL}/v3/event"
    headers:
      "X-SF-Token": "${SPLUNK_ACCESS_TOKEN}"
    auth:
      authenticator: headers_setter
  # Logs
  splunk_hec:
    token: "${SPLUNK_HEC_TOKEN}"
    endpoint: "${SPLUNK_HEC_URL}"
    source: "otel"
    sourcetype: "otel"
    profiling_data_enabled: false
  # Profiling
  splunk_hec/profiling:
    token: "${SPLUNK_ACCESS_TOKEN}"
    endpoint: "${SPLUNK_INGEST_URL}/v1/log"
    log_data_enabled: false
  # Send to gateway
  otlp_grpc/gateway:
    endpoint: "${SPLUNK_GATEWAY_URL}:4317"
    tls:
      insecure: true
    auth:
      authenticator: headers_setter
  # Debug
  debug:
    verbosity: detailed

service:
  telemetry:
    logs:
      level: ${env:SPLUNK_COLLECTOR_LOG_LEVEL:-info}
    # Resource attributes attached to the Collector's internal telemetry. These
    # are also exposed via OpAMP when the opamp/splunk_o11y extension is enabled
    # with agent_description.include_resource_attributes set to true.
    resource:
      attributes:
        - name: otelcol.service.mode
          value: agent
        # Optional: Add a deployment environment to the Collector's internal telemetry.
        # To use the deprecated attribute, change the key to "deployment.environment".
        # - name: deployment.environment.name
        #   value: staging/production/...
    metrics:
      readers:
        - pull:
            exporter:
              prometheus:
                host: '127.0.0.1'
                port: 8888
  extensions: [headers_setter, health_check, http_forwarder, http_forwarder/opamp_splunk_o11y, opamp/splunk_o11y, zpages]
  pipelines:
    traces:
      receivers: [jaeger, otlp, zipkin]
      processors:
      - memory_limiter
      - batch
      - resource_detection
      #- resource/add_environment
      exporters: [otlp_http]
      # Use instead when sending to gateway
      #exporters: [otlp_grpc/gateway]
    metrics:
      receivers: [host_metrics, otlp]
      processors: [memory_limiter, transform/limit_histogram_buckets, batch, resource_detection]
      exporters: [signalfx]
      # Use instead when sending to gateway
      #exporters: [otlp_grpc/gateway]
    metrics/internal:
      receivers: [prometheus/internal]
      processors: [memory_limiter, batch, resource_detection]
      # When sending to gateway, at least one metrics pipeline needs
      # to use signalfx exporter so host metadata gets emitted
      exporters: [signalfx]
    logs/signalfx:
      receivers: [smartagent/processlist]
      processors: [memory_limiter, batch, resource_detection]
      exporters: [signalfx]
    logs/entities:
      # Receivers are dynamically added if discovery mode is enabled
      receivers: [nop]
      processors: [memory_limiter, batch, resource_detection]
      exporters: [otlp_http/entities]
      # Use instead when sending to gateway
      #exporters: [otlp_grpc/gateway]
    logs:
      receivers: [fluent_forward, otlp]
      processors:
      - memory_limiter
      - batch
      - resource_detection
      #- resource/add_environment
      exporters: [splunk_hec, splunk_hec/profiling]
      # Use instead when sending to gateway
      #exporters: [otlp_grpc/gateway]

ここでも、非常に重要な点があります。exporters - signalfxセクションでコメントアウトされている値が誤っています。

agent_config.yaml
exporters:
  signalfx:
    # Use instead when sending to gateway
    #api_url: http://${SPLUNK_GATEWAY_URL}:6060
    #ingest_url: http://${SPLUNK_GATEWAY_URL}:9943

上記のapi_urlとingest_urlは、Gateway経由でOTelデータを転送するときにコメントアウトを解除して利用するのですが、http:// という余計なURLスキーマがついてしまっています。正しくは${SPLUNK_GATEWAY_URL}:6060のように指定しなければ動作しないことに注意が必要です。
(私は動作確認中にこの影響でOTelが起動しなくなって困りました。。。。ログを見れば解決はするんですが)

設定手順

公式の設定手順は以下のドキュメントにあります。
https://help.splunk.com/en/splunk-observability-cloud/manage-data/splunk-distribution-of-the-opentelemetry-collector/get-started-with-the-splunk-distribution-of-the-opentelemetry-collector/splunk-add-on-for-opentelemetry-collector/install-the-add-on

inputs.confの設定

inputs.confについては、default/inputs.confをlocal/inputs.confにコピーして、以下を修正すれば動作しました。

  • [Splunk_TA_otel]を[Splunk_TA_otel://Splunk_TA_otel]に変更(以前までは、デフォルトのStanza名が設定されていたのですが、今バージョンからはStanza名ごとユーザーが定義する仕様になったようです)
  • splunk_access_tokenの値を指定
  • splunk_realmの値を指定
  • (ゲートウェイサーバー経由で転送する場合のエージェント側の設定)splunk_collector_env_varsでSPLUNK_GATEWAY_URL=xx.xx.xx.xx(GatewayサーバーのIPまたはホスト名)を指定
  • (ゲートウェイサーバー経由で転送する場合のゲートウェイ側の設定)splunk_collector_env_varsでSPLUNK_LISTEN_INTERFACE=0.0.0.0(全受信許可)を指定

また、OTel CollectorからログをSplunk Enterprise/Splunk Cloudに転送する場合は、HECトークンと転送先URLを指定する必要があります。この場合もsplunk_collector_env_varsで値を指定する必要があります。

具体例としては以下の通りです。(トークン値はサンプル)

local/inputs.conf(Agent)
# ATTENTION: If copying this file as base for your local/inputs.conf
# rename the stanza below to something in the form [Splunk_TA_otel://<data_input_name>]
[Splunk_TA_otel://Splunk_TA_otel]

# Global settings, see https://help.splunk.com/en/splunk-enterprise/administer/admin-manual/9.4/configuration-file-reference/9.4.1-configuration-file-reference/inputs.conf#global-settings-0
disabled=false
start_by_shell=false
interval = 0
index = _internal
sourcetype = Splunk_TA_otel

# TA specific settings
splunk_access_token = xxxxxxxxxxxxxxxx
splunk_realm = jp0
splunk_config = $SPLUNK_HOME/etc/apps/$SPLUNK_MODINPUT_BASE_DIR_NAME/configs/agent_config.yaml
splunk_collector_log_level = error
splunk_collector_env_vars = 
# Gateway経由で転送する場合
# splunk_collector_env_vars = SPLUNK_GATEWAY_URL=xx.xx.xx.xx
splunk_collector_cmd_args =
local/inputs.conf(Gateway)
# ATTENTION: If copying this file as base for your local/inputs.conf
# rename the stanza below to something in the form [Splunk_TA_otel://<data_input_name>]
[Splunk_TA_otel://Splunk_TA_otel]

# Global settings, see https://help.splunk.com/en/splunk-enterprise/administer/admin-manual/9.4/configuration-file-reference/9.4.1-configuration-file-reference/inputs.conf#global-settings-0
disabled=false
start_by_shell=false
interval = 0
index = _internal
sourcetype = Splunk_TA_otel

# TA specific settings
splunk_access_token = xxxxxxxxxxxxxxxx
splunk_realm = jp0
splunk_config = $SPLUNK_HOME/etc/apps/$SPLUNK_MODINPUT_BASE_DIR_NAME/configs/gateway_config.yaml
splunk_collector_log_level = error
splunk_collector_env_vars = SPLUNK_LISTEN_INTERFACE=0.0.0.0
splunk_collector_cmd_args =

Splunk Enterpriseにログを転送する場合のHECトークンと転送先URLの指定例は以下です。
(Splunk Cloudにログを転送する場合、URLやポート番号が異なるので注意してください)

splunk_collector_env_vars = SPLUNK_HEC_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx,SPLUNK_HEC_URL=https://\<Splunk Enterprise IP\>:8088/services/collector

yamlファイル(configs)の設定

次に、inputs.conf内で指定しているyamlファイルです。こちらは、AgentからSplunk Observability Cloudに直接通信して転送できる場合は特に変更不要です。

ただし、Gateway経由で転送する場合はコメントアウトされている箇所を修正する必要があります。また、前述した余計なURLスキーマの削除も必要です。

記事が長くなってしまうのですが、私が動作に成功した際のコンフィグを記載しておきます。

agent_config.yaml(Gatewayサーバー経由の場合)
# Default configuration file for the Linux (deb/rpm) and Windows MSI collector packages

# If the collector is installed without the Linux/Windows installer script, the following
# environment variables are required to be manually defined or configured below:
# - SPLUNK_ACCESS_TOKEN: The Splunk access token to authenticate requests
# - SPLUNK_API_URL: The Splunk API URL, e.g. https://api.us0.observability.splunkcloud.com
# - SPLUNK_HEC_TOKEN: The Splunk HEC authentication token
# - SPLUNK_HEC_URL: The Splunk HEC endpoint URL, e.g. https://http-inputs-acme.splunkcloud.com/services/collector
# - SPLUNK_INGEST_URL: The Splunk ingest URL, e.g. https://ingest.us0.observability.splunkcloud.com
# - SPLUNK_LISTEN_INTERFACE: The network interface the agent receivers listen on.

extensions:
  headers_setter:
    headers:
      - action: upsert
        key: X-SF-TOKEN
        from_context: X-SF-TOKEN
        default_value: "${SPLUNK_ACCESS_TOKEN}"
  health_check:
    endpoint: "${SPLUNK_LISTEN_INTERFACE}:13133"
  http_forwarder:
    ingress:
      endpoint: "${SPLUNK_LISTEN_INTERFACE}:6060"
    egress:
      #endpoint: "${SPLUNK_API_URL}"
      # Use instead when sending to gateway
      endpoint: "${SPLUNK_GATEWAY_URL}"
  http_forwarder/opamp_splunk_o11y:
    ingress:
      endpoint: "${SPLUNK_LISTEN_INTERFACE}:4320"
    egress:
      #endpoint: "${SPLUNK_INGEST_URL}"
      # Use instead when sending to gateway
      endpoint: "${SPLUNK_GATEWAY_URL}:4320"
      # Remove this header to use a token passed by the caller instead.
      headers:
        X-SF-Token: "${SPLUNK_ACCESS_TOKEN}"
  # opamp/splunk_o11y is included in the default config, but startup removes it
  # from service.extensions unless --feature-gates=+splunk.opamp.enabled is set.
  opamp/splunk_o11y:
    agent_description:
      # Include the Collector's resource attributes as OpAMP non-identifying attributes.
      include_resource_attributes: true
    server:
      http:
        endpoint: "${SPLUNK_INGEST_URL}/v1/opamp"
        # Use instead when sending to gateway (assuming the gateway is running http_forwarder/signalfx on port 4320)
        #endpoint: "${SPLUNK_GATEWAY_URL}:4320/v1/opamp"
        polling_interval: 30s
        headers:
          X-SF-Token: "${SPLUNK_ACCESS_TOKEN}"
  zpages:
    #endpoint: "${SPLUNK_LISTEN_INTERFACE}:55679"
    expvar:
      enabled: true

receivers:
  fluent_forward:
    endpoint: "${SPLUNK_LISTEN_INTERFACE}:8006"
  host_metrics:
    collection_interval: 10s
    scrapers:
      cpu:
      disk:
      filesystem:
      memory:
      network:
      # System load average metrics https://en.wikipedia.org/wiki/Load_(computing)
      load:
      # Paging/Swap space utilization and I/O metrics
      paging:
      # Aggregated system process count metrics
      processes:
      # System processes metrics, disabled by default
      # process:
  jaeger:
    protocols:
      grpc:
        endpoint: "${SPLUNK_LISTEN_INTERFACE}:14250"
      thrift_binary:
        endpoint: "${SPLUNK_LISTEN_INTERFACE}:6832"
      thrift_compact:
        endpoint: "${SPLUNK_LISTEN_INTERFACE}:6831"
      thrift_http:
        endpoint: "${SPLUNK_LISTEN_INTERFACE}:14268"
  otlp:
    protocols:
      grpc:
        endpoint: "${SPLUNK_LISTEN_INTERFACE}:4317"
        # Uncomment below config to preserve incoming access token and use it instead of the token value set in exporter config
        # include_metadata: true
      http:
        endpoint: "${SPLUNK_LISTEN_INTERFACE}:4318"
        # Uncomment below config to preserve incoming access token and use it instead of the token value set in exporter config
        # include_metadata: true
  # This section is used to collect the OpenTelemetry Collector metrics
  # Even if just a Splunk APM customer, these metrics are included
  prometheus/internal:
    config:
      scrape_configs:
      - job_name: 'otel-collector'
        scrape_interval: 10s
        static_configs:
        - targets: ["0.0.0.0:8888"]
        metric_relabel_configs:
          - source_labels: [ __name__ ]
            regex: 'promhttp_metric_handler_errors.*'
            action: drop
          - source_labels: [ __name__ ]
            regex: 'otelcol_processor_batch_.*'
            action: drop
  smartagent/processlist:
    type: processlist
  zipkin:
    endpoint: "${SPLUNK_LISTEN_INTERFACE}:9411"
  nop:

processors:
  batch:
    metadata_keys:
      - X-SF-Token
  # Enabling the memory_limiter is strongly recommended for every pipeline.
  # Configuration is based on the amount of memory allocated to the collector.
  # For more information about memory limiter, see
  # https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/memorylimiter/README.md
  memory_limiter:
    check_interval: 2s
    limit_mib: ${SPLUNK_MEMORY_LIMIT_MIB}

  # Detect if the collector is running on a cloud system, which is important for creating unique cloud provider dimensions.
  # Detector order is important: the `system` detector goes last so it can't preclude cloud detectors from setting host/os info.
  # Resource detection processor is configured to override all host and cloud attributes because instrumentation
  # libraries can send wrong values from container environments.
  # https://docs.splunk.com/Observability/gdi/opentelemetry/components/resourcedetection-processor.html#ordering-considerations
  resource_detection:
    detectors: [gcp, ecs, ec2, azure, system]
    override: true

  # Limit explicit histograms to at most 32 buckets before export through signalfx exporter.
  # With `send_otlp_histograms: true`, the backend drops histograms above this limit.
  # With `send_otlp_histograms: false`, this reduces the emitted `_bucket` series and `le` boundaries.
  transform/limit_histogram_buckets:
    metric_statements:
      - context: datapoint
        statements:
          - merge_histogram_buckets(32, method="limit_buckets")

  # Optional: The following processor can be used to add a default "deployment.environment.name" attribute to the logs and
  # traces when it's not populated by instrumentation libraries.
  # If enabled, make sure to enable this processor in a pipeline.
  # To use the deprecated attribute, change the key to "deployment.environment".
  # For more information, see https://docs.splunk.com/Observability/gdi/opentelemetry/components/resource-processor.html
  #resource/add_environment:
    #attributes:
      #- action: insert
        #value: staging/production/...
        #key: deployment.environment.name

exporters:
  # Traces
  otlp_http:
    traces_endpoint: "${SPLUNK_INGEST_URL}/v2/trace/otlp"
    headers:
      "X-SF-Token": "${SPLUNK_ACCESS_TOKEN}"
    auth:
      authenticator: headers_setter
  # Metrics + Events
  signalfx:
    access_token: "${SPLUNK_ACCESS_TOKEN}"
    #api_url: "${SPLUNK_API_URL}"
    #ingest_url: "${SPLUNK_INGEST_URL}"
    # Use instead when sending to gateway
    api_url: ${SPLUNK_GATEWAY_URL}:6060
    ingest_url: ${SPLUNK_GATEWAY_URL}:9943
    sync_host_metadata: true
    correlation:
  # Entities (applicable only if discovery mode is enabled)
  otlp_http/entities:
    logs_endpoint: "${SPLUNK_INGEST_URL}/v3/event"
    headers:
      "X-SF-Token": "${SPLUNK_ACCESS_TOKEN}"
    auth:
      authenticator: headers_setter
  # Logs
  splunk_hec:
    token: "${SPLUNK_HEC_TOKEN}"
    endpoint: "${SPLUNK_HEC_URL}"
    source: "otel"
    sourcetype: "otel"
    profiling_data_enabled: false
  # Profiling
  splunk_hec/profiling:
    token: "${SPLUNK_ACCESS_TOKEN}"
    endpoint: "${SPLUNK_INGEST_URL}/v1/log"
    log_data_enabled: false
  # Send to gateway
  otlp_grpc/gateway:
    endpoint: "${SPLUNK_GATEWAY_URL}:4317"
    tls:
      insecure: true
    auth:
      authenticator: headers_setter
  # Debug
  debug:
    verbosity: detailed

service:
  telemetry:
    logs:
      level: ${env:SPLUNK_COLLECTOR_LOG_LEVEL:-info}
    # Resource attributes attached to the Collector's internal telemetry. These
    # are also exposed via OpAMP when the opamp/splunk_o11y extension is enabled
    # with agent_description.include_resource_attributes set to true.
    resource:
      attributes:
        - name: otelcol.service.mode
          value: agent
        # Optional: Add a deployment environment to the Collector's internal telemetry.
        # To use the deprecated attribute, change the key to "deployment.environment".
        # - name: deployment.environment.name
        #   value: staging/production/...
    metrics:
      readers:
        - pull:
            exporter:
              prometheus:
                host: '127.0.0.1'
                port: 8888
  extensions: [headers_setter, health_check, http_forwarder, http_forwarder/opamp_splunk_o11y, opamp/splunk_o11y, zpages]
  pipelines:
    traces:
      receivers: [jaeger, otlp, zipkin]
      processors:
      - memory_limiter
      - batch
      - resource_detection
      #- resource/add_environment
      #exporters: [otlp_http]
      # Use instead when sending to gateway
      exporters: [otlp_grpc/gateway]
    metrics:
      receivers: [host_metrics, otlp]
      processors: [memory_limiter, transform/limit_histogram_buckets, batch, resource_detection]
      #exporters: [signalfx]
      # Use instead when sending to gateway
      exporters: [otlp_grpc/gateway]
    metrics/internal:
      receivers: [prometheus/internal]
      processors: [memory_limiter, batch, resource_detection]
      # When sending to gateway, at least one metrics pipeline needs
      # to use signalfx exporter so host metadata gets emitted
      exporters: [signalfx]
    logs/signalfx:
      receivers: [smartagent/processlist]
      processors: [memory_limiter, batch, resource_detection]
      exporters: [signalfx]
    logs/entities:
      # Receivers are dynamically added if discovery mode is enabled
      receivers: [nop]
      processors: [memory_limiter, batch, resource_detection]
      #exporters: [otlp_http/entities]
      # Use instead when sending to gateway
      exporters: [otlp_grpc/gateway]
    logs:
      receivers: [fluent_forward, otlp]
      processors:
      - memory_limiter
      - batch
      - resource_detection
      #- resource/add_environment
      #exporters: [splunk_hec, splunk_hec/profiling]
      # Use instead when sending to gateway
      exporters: [otlp_grpc/gateway]

OTel Add-onをForwarderにインストールし、inputs.confとconfigsを適切に修正してForwarderを再起動すれば、Splunk Observability Cloudにメトリクスが転送されるようになります。

image.png

以上です。

参考資料

Splunkbase(Splunk Add-On for OpenTelemetry Collector)
https://splunkbase.splunk.com/app/7125

Github(splunk-otel-collector)
https://github.com/signalfx/splunk-otel-collector/

Install the Splunk Add-on for OpenTelemetry Collector
https://help.splunk.com/en/splunk-observability-cloud/manage-data/splunk-distribution-of-the-opentelemetry-collector/get-started-with-the-splunk-distribution-of-the-opentelemetry-collector/splunk-add-on-for-opentelemetry-collector/install-the-add-on

0
0
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
0
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?