LoginSignup
16
3

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

Cisco Systems JapanAdvent Calendar 2024

Day 23
@rytakao

IOS-XE の EEM と OpenAI API を使用したトラブルシューティング

Last updated at Posted at 2024-12-22

IOS-XE の EEM と OpenAI API を使用したトラブルシューティング

はじめに

この記事では Cisco の IOS-XE Router のトラブルシューティングを IP SLA, Object Tracking, EEM, Python, ChatGPT API を使用して行います。


IOS-XE Router で Cisco の Japan TAC(Technical Assistance Center) によく来る障害のお問い合わせは大きくわけると以下の 4 つです。

  • パケットドロップ
  • High CPU 関連
  • Memory 関連
  • 再起動

再起動については crashinfo、core file の解析が必要なため Cisco 社内のリソースを使わないと解析が出来ません。そのためここではそれ以外の 3 つのトラブルシューティングの自動化を目指します。
大きくわけて以下の２つのステップでこの実現を試みます。

  • Step 1: EEM(Embedded Event Manager) によるログの自動取得
  • Step 2: guestshell, Python, ChatGPT API を使用したログの解析

ネットワーク構成等

以下のように IOS-XE Router の GigabitEthernet1 が Internet に接続されているような環境を前提としています。

+----------+
|  IOS-XE  |
|  Router  |GigabitEthernet1-----------Internet
|          |
+----------+

今回の検証では IOS-XE Router には Catalyst8000v version 17.12.04 を使用しています。

Step 1: EEM(Embedded Event Manager) によるログの自動取得

EEM(Embedded Event Manager) を使用して障害が起きた際にログを自動で取得するようにあらかじめ設定をしておきます。
それぞれの障害に応じてログを取得するように３つの EEM を設定しておきます。

  • パケットドロップ時にログを取得する EEM
  • High CPU 時にログを取得する EEM
  • Memory が不足した時にログを取得する EEM
  • Linux のリソースが不足した時にログを取得する EEM

パケットドロップ時にログを取得する EEM

以下の設定を行い、IP SLA/Object Tracking で 8.8.8.8(Google の DNS Server) への poling をするようにし、poling に失敗したら、EEM を動作させログを取得します。

!
ip sla 1
 icmp-echo 8.8.8.8 source-interface GigabitEthernet1
 frequency 10
ip sla schedule 1 life forever start-time now
!
track 1 ip sla 1 reachability
 delay down 30
!
event manager applet PACKET_DROP
 event track 1 state down
 action 011 cli command "enable"
 action 012 cli command "terminal length 0"
 action 013 cli command "terminal exec prompt timestamp"
 action 100 syslog msg "'event manager applet PACKET_DROP' takes show commands: 1st"
 action 101 cli command "show arp | redirect bootflash:/guest-share/PACKET_DROP.txt"
 action 102 cli command "show ip traffic | append bootflash:/guest-share/PACKET_DROP.txt"
 action 103 cli command "show buffer | append bootflash:/guest-share/PACKET_DROP.txt"
 action 104 cli command "show interfaces | append bootflash:/guest-share/PACKET_DROP.txt"
 action 105 cli command "show policy-map interface | append bootflash:/guest-share/PACKET_DROP.txt"
 action 106 cli command "show platform hardware qfp active datapath utilization | append bootflash:/guest-share/PACKET_DROP.txt"
 action 107 cli command "show platform hardware qfp active statistics drop detail | append bootflash:/guest-share/PACKET_DROP.txt"
 action 108 cli command "show platform software punt-policer | append bootflash:/guest-share/PACKET_DROP.txt"
 action 109 wait 10
 action 200 syslog msg "'event manager applet PACKET_DROP' takes show commands: 2nd"
 action 201 cli command "show arp | append bootflash:/guest-share/PACKET_DROP.txt"
 action 202 cli command "show ip traffic | append bootflash:/guest-share/PACKET_DROP.txt"
 action 203 cli command "show buffer | append bootflash:/guest-share/PACKET_DROP.txt"
 action 204 cli command "show interfaces | append bootflash:/guest-share/PACKET_DROP.txt"
 action 205 cli command "show policy-map interface | append bootflash:/guest-share/PACKET_DROP.txt"
 action 206 cli command "show platform hardware qfp active datapath utilization | append bootflash:/guest-share/PACKET_DROP.txt"
 action 207 cli command "show platform hardware qfp active statistics drop detail | append bootflash:/guest-share/PACKET_DROP.txt"
 action 208 cli command "show platform software punt-policer | append bootflash:/guest-share/PACKET_DROP.txt"
 action 300 syslog msg "'event manager applet PACKET_DROP' takes basic show commands"
 action 301 cli command "show version | append bootflash:/guest-share/PACKET_DROP.txt"
 action 302 cli command "show run | append bootflash:/guest-share/PACKET_DROP.txt"
 action 303 cli command "show platform | append bootflash:/guest-share/PACKET_DROP.txt"
 action 304 cli command "show inventory | append bootflash:/guest-share/PACKET_DROP.txt"
 action 300 cli command "show logging | append bootflash:/guest-share/PACKET_DROP.txt"

設定を簡単に説明します。

  • ip sla 1: IP SLA 設定を定義しています。識別子として 1 を割り当てています。
  • icmp-echo 8.8.8.8 source-interface GigabitEthernet1: プローブパケットとして ICMP echo request を 8.8.8.8 宛に GigabitEthernet1 の IP をソースアドレスとして送信します。source-interface は環境に合わせて変更してください。
  • frequency 30: プローブパケットの送信間隔です。10秒毎に送信します。
  • ip sla schedule 1 life forever start-time now: IP SLA の識別子 1 のオペレーションを開始します。
  • track 1 ip sla 1 reachability: IP SLA の識別子 1 を利用した object tracking の設定です。1 を識別子として割り当てています。object tracking は UP, DOWN の2つの状態を取り、IP SLA のプローブパケットに返答があれば UP、timeout になると DOWN, となります。
  • delay down 30: IP SLA が timeout してから object tracking が DOWN 状態に遷移するまでに 30 秒の遅延を追加しています。delay を追加しないとプローブパケットを1パケット落としただけで DOWN してしまいます。
  • action 101 cli command "show arp | redirect bootflash:/guest-share/PACKET_DROP.txt": コマンドの出力を bootflash:/guest-share/PACKET_DROP.txt に保存します。既にファイルがある場合、古いものは削除されます。
  • action xxx cli command "show xxx | append bootflash:/guest-share/PACKET_DROP.txt": コマンドの出力を bootflash:/guest-share/PACKET_DROP.txt に追記します。

今回の例では 8.8.8.8 に poling していますが、環境に合わせて対抗機器に設定するなどしてください。
EEM では track 1 が DOWN した場合に show command を取得するように設定しています。
ログは bootflash:/guest-share/ 配下に保存していますが、これは後述する guestshell がここの配下のファイルでないと取り扱えないためです。

High CPU 時にログを取得する EEM

以下の設定を行い、CPU 使用率が 80 % を 5 秒間超えた場合に EEM を動作させログを取得します。

process cpu threshold type total rising 80 interval 5
event manager environment pid1 0
event manager environment pid2 0
event manager environment pid3 0
event manager applet HIGH_CPU
 event syslog pattern "%SYS-1-CPURISINGTHRESHOLD:" maxrun 600
 action 011 cli command "enable"
 action 012 cli command "terminal length 0"
 action 013 cli command "terminal exec prompt timestamp"
 action 014 regexp "Top 3 processes¥(Pid/Util¥): +([0-9]+)/[0-9]+%, +([0-9]+)/[0-9]+%, +([0-9]+)/[0-9]+%" "$_syslog_msg" line pid1 pid2 pid3
 action 100 syslog msg "'event manager applet HIGH_CPU' takes show commands: 1st"
 action 101 cli command "show process $pid1 | redirect bootflash:/guest-share/HIGH_CPU.txt"
 action 102 cli command "show stack $pid1 | append bootflash:/guest-share/HIGH_CPU.txt"
 action 103 cli command "show process $pid2 | append bootflash:/guest-share/HIGH_CPU.txt"
 action 104 cli command "show stack $pid2 | append bootflash:/guest-share/HIGH_CPU.txt"
 action 105 cli command "show process $pid3 | append bootflash:/guest-share/HIGH_CPU.txt"
 action 106 cli command "show stack $pid3 | append bootflash:/guest-share/HIGH_CPU.txt"
 action 107 cli command "show interfaces | append bootflash:/guest-share/HIGH_CPU.txt"
 action 108 cli command "show interfaces switching | append bootflash:/guest-share/HIGH_CPU.txt"
 action 109 cli command "show controllers | append bootflash:/guest-share/HIGH_CPU.txt"
 action 110 cli command "show ip traffic | append bootflash:/guest-share/HIGH_CPU.txt"
 action 111 wait 10
 action 200 syslog msg "'event manager applet HIGH_CPU' takes show commands: 2nd"
 action 201 cli command "show process $pid1 | append bootflash:/guest-share/HIGH_CPU.txt"
 action 202 cli command "show stack $pid1 | append bootflash:/guest-share/HIGH_CPU.txt"
 action 203 cli command "show process $pid2 | append bootflash:/guest-share/HIGH_CPU.txt"
 action 204 cli command "show stack $pid2 | append bootflash:/guest-share/HIGH_CPU.txt"
 action 205 cli command "show process $pid3 | append bootflash:/guest-share/HIGH_CPU.txt"
 action 206 cli command "show stack $pid3 | append bootflash:/guest-share/HIGH_CPU.txt"
 action 207 cli command "show interfaces | append bootflash:/guest-share/HIGH_CPU.txt"
 action 208 cli command "show interfaces switching | append bootflash:/guest-share/HIGH_CPU.txt"
 action 209 cli command "show controllers | append bootflash:/guest-share/HIGH_CPU.txt"
 action 210 cli command "show ip traffic | append bootflash:/guest-share/HIGH_CPU.txt"
 action 300 syslog msg "'event manager applet HIGH_CPU' takes basic show commands"
 action 301 cli command "show version | append bootflash:/guest-share/HIGH_CPU.txt"
 action 302 cli command "show run | append bootflash:/guest-share/HIGH_CPU.txt"
 action 303 cli command "show platform | append bootflash:/guest-share/HIGH_CPU.txt"
 action 304 cli command "show inventory | append bootflash:/guest-share/HIGH_CPU.txt"
 action 300 cli command "show logging | append bootflash:/guest-share/HIGH_CPU.txt"

簡単に設定について説明します。

  • process cpu threshold type total rising 80 interval 5: Total の CPU 使用率が5秒以上 80% を超えたら以下のようなログを出力する。Total/Interrupt のCPU使用率と TOP 3 の PID と CPU 使用率を出力する。
%SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU
Utilization(Total/Intr): 84%/44%, Top 3 processes(Pid/Util):
116/48%, 111/0%, 125/0%
  • event manager environment pid1 0: 変数 pid1 を定義し初期値を 0 に設定
  • event manager applet HIGH_CPU: HIGH_CPU という名前で EEM を定義
  • event syslog pattern "%SYS-1-CPURISINGTHRESHOLD:" maxrun 600: "%SYS-1-CPURISINGTHRESHOLD:" というパターンの syslog があったら EEM を動作させる。
  • action 101 cli command "show process $pid1 | redirect bootflash:/guest-share/HIGH_CPU.txt": コマンドの出力を bootflash:/guest-share/HIGH_CPU_MEMORY.txt に保存します。既にファイルがある場合、古いものは削除されます。
    必要に応じて redirect の部分を append に変更してください。ただし append にした場合は EEM が動作するたびに追記されるので、ファイルサイズについて考慮する必要があります。
  • action xxx cli command "show xxx | append bootflash:/guest-share/HIGH_CPU_MEMORY.txt": コマンドの出力を bootflash:/guest-share/HIGH_CPU_MEMORY.txt に追記します。

ログは bootflash:/guest-share/ 配下に保存していますが、これは後述する guestshell がここの配下のファイルでないと取り扱えないためです。

Memory が不足した時にログを取得する EEM

以下の設定を行い、Processor Memory の残りが 20000KB 以下場合に EEM を動作させログを取得します。

memory free low-watermark processor 20000
event manager applet LOW_FREE_MEMORY
 event syslog pattern "%SYS-4-FREEMEMLOW:" maxrun 600
 action 011 cli command "enable"
 action 012 cli command "terminal length 0"
 action 013 cli command "terminal exec prompt timestamp"
 action 100 syslog msg "'event manager applet LOW_FREE_MEMORY' takes show commands: 1st"
 action 101 cli command "show region | redirect bootflash:/guest-share/LOW_FREE_MEMORY.txt"
 action 102 cli command "show processes memory sorted | append bootflash:/guest-share/LOW_FREE_MEMORY.txt"
 action 103 cli command "show memory allocating-process totals | append bootflash:/guest-share/LOW_FREE_MEMORY.txt"
 action 104 cli command "show memory statistics history | append bootflash:/guest-share/LOW_FREE_MEMORY.txt"
 action 105 cli command "show memory statistics history table | append bootflash:/guest-share/LOW_FREE_MEMORY.txt"
 action 106 cli command "show buffers | append bootflash:/guest-share/LOW_FREE_MEMORY.txt"
 action 107 cli command "show buffers leak | append bootflash:/guest-share/LOW_FREE_MEMORY.txt"
 action 108 cli command "show ip traffic | append bootflash:/guest-share/LOW_FREE_MEMORY.txt"
 action 109 cli command "show ip route summary | append bootflash:/guest-share/LOW_FREE_MEMORY.txt"
 action 110 wait 10
 action 200 syslog msg "'event manager applet LOW_FREE_MEMORY' takes show commands: 2nd"
 action 201 cli command "show region | append bootflash:/guest-share/LOW_FREE_MEMORY.txt"
 action 202 cli command "show processes memory sorted | append bootflash:/guest-share/LOW_FREE_MEMORY.txt"
 action 203 cli command "show memory allocating-process totals | append bootflash:/guest-share/LOW_FREE_MEMORY.txt"
 action 204 cli command "show memory statistics history | append bootflash:/guest-share/LOW_FREE_MEMORY.txt"
 action 205 cli command "show memory statistics history table | append bootflash:/guest-share/LOW_FREE_MEMORY.txt"
 action 206 cli command "show buffers | append bootflash:/guest-share/LOW_FREE_MEMORY.txt"
 action 207 cli command "show buffers leak | append bootflash:/guest-share/LOW_FREE_MEMORY.txt"
 action 208 cli command "show ip traffic | append bootflash:/guest-share/LOW_FREE_MEMORY.txt"
 action 209 cli command "show ip route summary | append bootflash:/guest-share/LOW_FREE_MEMORY.txt"
 action 300 syslog msg "'event manager applet LOW_FREE_MEMORY' takes basic show commands"
 action 301 cli command "show version | append bootflash:/guest-share/LOW_FREE_MEMORY.txt"
 action 302 cli command "show run | append bootflash:/guest-share/LOW_FREE_MEMORY.txt"
 action 303 cli command "show platform | append bootflash:/guest-share/LOW_FREE_MEMORY.txt"
 action 304 cli command "show inventory | append bootflash:/guest-share/LOW_FREE_MEMORY.txt"
 action 300 cli command "show logging | append bootflash:/guest-share/LOW_FREE_MEMORY.txt"

簡単に設定について説明します。

  • memory free low-watermark processor 10000: Processor Memory の残りが 20000KB 以下になると以下のようなログを出力します。
%SYS-4-FREEMEMLOW: Free Memory has dropped below 20000k
Pool: Processor Free: 66814056 freemem_lwm: 204800000
  • action 101 cli command "show xxx | redirect bootflash:/guest-share/LOW_FREE_MEMORY.txt": コマンドの出力を bootflash:/guest-share/LOW_FREE_MEMORY.txt に保存します。既にファイルがある場合、古いものは削除されます。
    必要に応じて redirect の部分を append に変更してください。ただし append にした場合は EEM が動作するたびに追記されるので、ファイルサイズについて考慮する必要があります。
  • action xxx cli command "show xxx | append bootflash:/guest-share/LOW_FREE_MEMORY.txt": コマンドの出力を bootflash:/guest-share/LOW_FREE_MEMORY.txt に追記します。

ログは bootflash:/guest-share/ 配下に保存していますが、これは後述する guestshell がここの配下のファイルでないと取り扱えないためです。

Linux 側のリソースが不足した時にログを取得する EEM

前述の２つの EEM(High CPU 時にログを取得する EEM, Memory が不足した時にログを取得する EEM) は主に IOSd 側のトラブルシューティングを行うためのものです。
IOS-XE では Linux の上に IOSd が Linux のプロセスの 1 つとして動いています。
そのため IOSd 上だけでなく Linux 全体のトラブルシューティングも行う必要があります。
Linux 側がリソース不足に陥った場合にログを取得するために以下の EEM を設定しておきます。
ここで言うリソース不足とは主に CPU のリソースと、Memory のリソースの不足になります。

以下のような EEM を設定します。

event manager applet LINUX＿TSHOOT
 event syslog pattern "%PLATFORM-4-ELEMENT_WARNING:" maxrun 600
 action 011 cli command "enable"
 action 012 cli command "terminal length 0"
 action 013 cli command "terminal exec prompt timestamp"
 action 100 syslog msg "'event manager applet LINUX＿TSHOOT' takes show commands: 1st"
 action 101 cli command "show processes cpu platform sorted | redirect bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 102 cli command "show platform software status control-processor brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 103 cli command "show platform software process environment ios RP active | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 104 cli command "show platform software process slot RP active monitor cycles 1 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 105 cli command "show platform software process list RP active sort memory | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 106 cli command "show platform software memory chassis-manager RP active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 107 cli command "show platform software memory cli-agent RP active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 108 cli command "show platform software memory dbm RP active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 109 cli command "show platform software memory emd RP active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 110 cli command "show platform software memory forwarding-manager RP active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 111 cli command "show platform software memory host-manager RP active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 112 cli command "show platform software memory ios RP active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 113 cli command "show platform software memory license-manager RP active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 114 cli command "show platform software memory logger RP active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 115 cli command "show platform software memory pluggable-services RP active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 116 cli command "show platform software memory shell-manager RP active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 117 cli command "show platform software memory smd RP active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 118 cli command "show platform software memory virt-manager RP active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 119 cli command "show platform software memory chunk host-manager RP active | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 120 cli command "show platform software memory database cli-agent RP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 121 cli command "show platform software memory database dbm RP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 122 cli command "show platform software memory database forwarding-manager RP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 123 cli command "show platform software memory database ios RP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 124 cli command "show platform software memory database smd RP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 125 cli command "show platform software memory database virt-manager RP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 126 cli command "show platform software memory messaging chassis-manager RP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 127 cli command "show platform software memory messaging cli-agent RP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 128 cli command "show platform software memory messaging dbm RP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 129 cli command "show platform software memory messaging emd RP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 130 cli command "show platform software memory messaging forwarding-manager RP active brief | exclude diff:0  | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 131 cli command "show platform software memory messaging host-manager RP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 132 cli command "show platform software memory messaging ios RP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 133 cli command "show platform software memory messaging license-manager RP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 134 cli command "show platform software memory messaging logger RP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 135 cli command "show platform software memory messaging pluggable-services RP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 136 cli command "show platform software memory messaging shell-manager RP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 137 cli command "show platform software memory messaging smd RP active brief | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 138 cli command "show platform software memory messaging virt-manager RP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 139 cli command "show platform software memory chassis-manager FP active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 140 cli command "show platform software memory forwarding-manager FP active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 141 cli command "show platform software memory host-manager FP active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 142 cli command "show platform software memory logger FP active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 143 cli command "show platform software memory chunk chassis-manager FP active | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 144 cli command "show platform software memory chunk forwarding-manager FP active | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 145 cli command "show platform software memory chunk host-manager FP active | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 146 cli command "show platform software memory messaging chassis-manager FP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 147 cli command "show platform software memory messaging forwarding-manager FP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 148 cli command "show platform software memory messaging host-manager FP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 149 cli command "show platform software memory messaging logger FP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 150 cli command "show platform hardware qfp active infrastructure exmem statistics | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 151 cli command "show platform software memory qfp-control-process qfp active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 152 cli command "show platform software memory qfp-driver qfp active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 153 cli command "show platform software memory qfp-ha-server qfp active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 154 cli command "show platform software memory qfp-service-process qfp active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 155 cli command "show platform software memory chunk qfp-control-process qfp active | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 156 cli command "show platform software memory chunk qfp-driver qfp active | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 157 cli command "show platform software memory chunk qfp-ha-server qfp active | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 158 cli command "show platform software memory chunk qfp-service-process qfp active | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 159 cli command "show platform software memory messaging qfp-control-process qfp active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 160 cli command "show platform software memory messaging qfp-driver qfp active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 161 cli command "show platform software memory messaging qfp-ha-server qfp active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 162 cli command "show platform software memory messaging qfp-service-process qfp active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 200 syslog msg "'event manager applet LINUX＿TSHOOT' takes show commands: 1st"
 action 201 cli command "show processes cpu platform sorted | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 202 cli command "show platform software status control-processor brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 203 cli command "show platform software process environment ios RP active | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 204 cli command "show platform software process slot RP active monitor cycles 1 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 205 cli command "show platform software process list RP active sort memory | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 206 cli command "show platform software memory chassis-manager RP active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 207 cli command "show platform software memory cli-agent RP active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 208 cli command "show platform software memory dbm RP active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 209 cli command "show platform software memory emd RP active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 210 cli command "show platform software memory forwarding-manager RP active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 211 cli command "show platform software memory host-manager RP active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 212 cli command "show platform software memory ios RP active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 213 cli command "show platform software memory license-manager RP active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 214 cli command "show platform software memory logger RP active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 215 cli command "show platform software memory pluggable-services RP active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 216 cli command "show platform software memory shell-manager RP active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 217 cli command "show platform software memory smd RP active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 218 cli command "show platform software memory virt-manager RP active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 219 cli command "show platform software memory chunk host-manager RP active | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 220 cli command "show platform software memory database cli-agent RP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 221 cli command "show platform software memory database dbm RP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 222 cli command "show platform software memory database forwarding-manager RP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 223 cli command "show platform software memory database ios RP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 224 cli command "show platform software memory database smd RP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 225 cli command "show platform software memory database virt-manager RP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 226 cli command "show platform software memory messaging chassis-manager RP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 227 cli command "show platform software memory messaging cli-agent RP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 228 cli command "show platform software memory messaging dbm RP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 229 cli command "show platform software memory messaging emd RP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 230 cli command "show platform software memory messaging forwarding-manager RP active brief | exclude diff:0  | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 231 cli command "show platform software memory messaging host-manager RP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 232 cli command "show platform software memory messaging ios RP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 233 cli command "show platform software memory messaging license-manager RP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 234 cli command "show platform software memory messaging logger RP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 235 cli command "show platform software memory messaging pluggable-services RP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 236 cli command "show platform software memory messaging shell-manager RP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 237 cli command "show platform software memory messaging smd RP active brief | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 238 cli command "show platform software memory messaging virt-manager RP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 239 cli command "show platform software memory chassis-manager FP active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 240 cli command "show platform software memory forwarding-manager FP active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 241 cli command "show platform software memory host-manager FP active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 242 cli command "show platform software memory logger FP active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 243 cli command "show platform software memory chunk chassis-manager FP active | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 244 cli command "show platform software memory chunk forwarding-manager FP active | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 245 cli command "show platform software memory chunk host-manager FP active | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 246 cli command "show platform software memory messaging chassis-manager FP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 247 cli command "show platform software memory messaging forwarding-manager FP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 248 cli command "show platform software memory messaging host-manager FP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 249 cli command "show platform software memory messaging logger FP active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 250 cli command "show platform hardware qfp active infrastructure exmem statistics | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 251 cli command "show platform software memory qfp-control-process qfp active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 252 cli command "show platform software memory qfp-driver qfp active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 253 cli command "show platform software memory qfp-ha-server qfp active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 254 cli command "show platform software memory qfp-service-process qfp active brief | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 255 cli command "show platform software memory chunk qfp-control-process qfp active | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 256 cli command "show platform software memory chunk qfp-driver qfp active | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 257 cli command "show platform software memory chunk qfp-ha-server qfp active | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 258 cli command "show platform software memory chunk qfp-service-process qfp active | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 259 cli command "show platform software memory messaging qfp-control-process qfp active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 260 cli command "show platform software memory messaging qfp-driver qfp active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 261 cli command "show platform software memory messaging qfp-ha-server qfp active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 262 cli command "show platform software memory messaging qfp-service-process qfp active | exclude diff:0 | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 300 syslog msg "'event manager applet LINUX＿TSHOOT' takes basic show commands"
 action 301 cli command "show version | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 302 cli command "show run | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 303 cli command "show platform | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 304 cli command "show inventory | append bootflash:/guest-share/LINUX＿TSHOOT.txt"
 action 300 cli command "show logging | append bootflash:/guest-share/LINUX＿TSHOOT.txt"

この EEM について簡単に解説します。
Linux 側でリソース不足等を検知すると以下のような WARNING メッセージが出力されます。

%PLATFORM-4-ELEMENT_WARNING: R0/0: smand: RP/0: Used Memory value 92% exceeds warning level 88%. Top memory allocators are: Process: xxxxxxxx. Tracekey: x#xxxxxxxxxxxxxxx Callsite ID: xxxxxxxxx (diff_call: xxxxxx). 
%PLATFORM-4-ELEMENT_WARNING: Chassis 1 R0/0: smand: 1/RP/0: 5-Minute Load Average value x.xx exceeds warning level 5.00.

この EEM ではこのような "%PLATFORM-4-ELEMENT_WARNING:" を含むメッセージをトリガーにログの取得を行っています。
なお、Memory Leak のような問題の場合は、Memory の Leak が少しづつ時間をかけて発生することが多く、ログを取得してから時間経過後(数週間から数ヶ月後)に再度ログを取得しないと Memory Leak が発生しているか判断できない場合もあります。

  • action 101 cli command "show xxx | redirect bootflash:/guest-share/LINUX＿TSHOOT.txt": コマンドの出力を bootflash:/guest-share/LINUX＿TSHOOT.txt に保存します。既にファイルがある場合、古いものは削除されます。
    必要に応じて redirect の部分を append に変更してください。ただし append にした場合は EEM が動作するたびに追記されるので、ファイルサイズについて考慮する必要があります。
  • action xxx cli command "show xxx | append bootflash:/guest-share/LINUX＿TSHOOT.txt": コマンドの出力を bootflash:/guest-share/LINUX＿TSHOOT.txt に追記します。

ログは bootflash:/guest-share/ 配下に保存していますが、これは後述する guestshell がここの配下のファイルでないと取り扱えないためです。

Step 2: guestshell, Python, ChatGPT API を使用した ChatGPT によるログの解析

guestshell の設定と有効化

guestshell を使用し、Internet と通信するために Catalyst8000v に以下の設定を行います。
<GigabitEthernet1 IP>, <Gateway IP>, <DNS Server IP> のところは環境に合わせて設定してください。

iox
!
vrf definition GuestShell
 rd 1:1
 !
 address-family ipv4
 exit-address-family
!
interface GigabitEthernet1
 ip address <GigabitEthernet1 IP> 255.255.255.0
 ip nat outside
!
interface VirtualPortGroup0
 vrf forwarding GuestShell
 ip address 10.0.0.254 255.255.255.0
 ip nat inside
!
ip access-list standard GuestShell
 permit 10.0.0.1
!
ip nat inside source list GuestShell interface GigabitEthernet1 vrf GuestShell overload
ip route 0.0.0.0 0.0.0.0 <Gateway IP>
ip route vrf GuestShell 0.0.0.0 0.0.0.0 GigabitEthernet1 <Gateway IP> global
!
app-hosting appid guestshell
 app-vnic gateway1 virtualportgroup 0 guest-interface 0
  guest-ipaddress 10.0.0.1 netmask 255.255.255.0
 app-default-gateway 10.0.0.254 guest-interface 0
 name-server0 <DNS Server IP>
!

設定が完了したら guestshell enable で guestshell を有効化します。

Router#guestshell enable
Interface will be selected if configured in app-hosting
Please wait for completion
guestshell installed successfully
Current state is: DEPLOYED
guestshell activated successfully
Current state is: ACTIVATED
guestshell started successfully
Current state is: RUNNING
Guestshell enabled successfully

Router#

guestshell コマンドで guestshell にアクセスします。

Router#guestshell
[guestshell@guestshell ~]$

ping -c 1 8.8.8.8 コマンドで guestshell から Google の DNS Server への疎通を確認します。

[guestshell@guestshell ~]$ ping -c 1 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=111 time=3.39 ms

--- 8.8.8.8 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 3.392/3.392/3.392/0.000 ms
[guestshell@guestshell ~]$

python3.8+ の install

OpenAI Python API library を使うためには python 3.8+ が必要なので、
python 3.8+ を install します。やり方は以下を参照してください。
https://pypi.org/project/network3-perseus/

OpenAI Python API library の install

python3.8 -m pip install openai

Python File を作成して実行

以下のような python file を作成して実行してください。

from openai import OpenAI

path = input("Please input file path to analyze.: ")
prompt = "Please analyze the logs for any problems. Also, if there are logs that appear to be problematic, please excerpt those sections.\n\n\n"

client = OpenAI(
  api_key="xxxxx"
)

with open(path) as f:
    s = f.read()
    s = prompt + s

completion = client.chat.completions.create(
  model="gpt-4o-mini",
  store=True,
  messages=[
    {"role": "user", "content": s}
  ]
)

print(completion.choices[0].message);

コードの api_key には自身の アカウントの ChatGPT の API Key を入力してください。
このファイルを実行すると "Please input file path to analyze.:" と表示されるので EEM で作成されたファイルのパスを入力してください。例) /bootflash/guest-share/HIGH_CPU.txt
コードの prompt の部分とファイルパスを指定したファイルが結合された ChatGPT に渡され、解析結果が返されます。正直 prompt の部分はもう少し良い感じに出力されるように検討する必要があると思います。
少し変えて EEM によりログファイルが生成されたら自動で OpenAI API を叩くようにしても良いと思いますが、従量課金なので一旦は python を実行して API 叩くような形にしてあります。

最後に

前半の EEM の部分は多くの IOS-XE で役立つかなと思います。show platform 系のコマンドを削るか platform に合わせたものに変えれば IOS Router/Switch 等でも活用出来ると思います。
後半の OpenAI API の部分は show logging の出力に対しては色々反応してくれますが、それ以外のログの部分に対しては思ったような解析結果を返してくれなかったのでこのままだと実用性は低いです。prompt を工夫するなどもう少し前提知識を与えてあげる必要がありそうです。一応こういうことも出来るよということで。
また、EEM で取得しているコマンドは version や platform によっては取得できないものもあるかもしれないので検証の上設定してください。

16
3
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
16
3

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?