Whisperで文字起こししてみた&話者分離をしてみた&ChatGPTと合わせて英検１級のリスニング問題を解かせてみた。

Whisperとは

WhisperはOpenAI社が出している文字起こしサービスです。
WhisperのAPIを利用することで無料で文字起こしすることができます。
便利な世の中になったものです。

この記事ではPythonを用いて文字起こしする方法とその他応用事例を紹介します。

環境

私の環境は下記の表の通りです。

環境	スペック
OS	Windows 11
CPU	Intel Core i5-11400H
GPU	NVIDIA GeForce RTX 3050 Ti Laptop
メモリ	16GB

このスペックがあれば最低限Whisperを動かせると思います。

[基本編]Whisperの導入

実際にwhisperを利用するには事前にwhisperの環境構築が必要になります。
導入方法を2つご紹介します。

導入①

ローカルの実機に構築する方法になります。
インストールが必要なもの

1. Python 3.7以降かつ3.11より前
2. PyTorch 最新版
3. Git
4. whisper

こちらの記事を参考にしました。

導入②

実機の仮想コンテナDocker上に構築する方法になります。
インストールが必要なもの

1. Docker

こちらのほうが手間なくできるので、簡単に構築可能です。
Dockerfileを載せております。https://github.com/minobun/test_whisper

[基本編]文字起こし方法

まず下記のようなファイルを作成します。

sample.py

# whisperの読み込み
import whisper

# 音声ファイルの読み込み
audio_file = "文字起こししたいファイル"

# モデルの読み込み
model = whisper.load_model("medium")

# 文字起こし
result = model.transcribe(audio_file, fp16 = False)

# 出力
print(result)

先ほど作成したファイルと同じディレクトリ上でsample.pyを下記のように実行すると、文字起こしした結果が出力されます。

> python sample.py

[基本編]実際の例

実際に上記のやり方でやってみます。
英検の過去問の音声スクリプトを文字起こししてみました。

まずは英検の音声ファイルをダウンロードします。

私は2022年度第3回のリスニング音源（Part1）をダウンロードしました。

任意のフォルダの中に上記のリスニング音源を配置します。
私はtest_whisperというフォルダの中に音源(1Q-part1.mp3)を配置しました。

C:\Users\user\デスクトップ\app\test_whisper> ls
Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a---l        2023/03/29     22:22       15577585 1Q-part1.mp3

sample.pyを作成します。

sample.py

# whisperの読み込み
import whisper

# 音声ファイルの読み込み
audio_file = "1Q-part1.mp3"

# モデルの読み込み
model = whisper.load_model("medium")

# 文字起こし
result = model.transcribe(audio_file, fp16 = False)

# 出力
print(result)

そして、このsample.pyファイルを実行します。
ちゃんと出力されていることが確認できました。

C:\Users\user\デスクトップ\app\test_whisper> python sample.py
Please check the volume and sound of the room. The third test of the year, the first-class listening test. We will now begin the first-class listening test. Please be careful as you will not be asked questions about what you will be talking about. If there is a noise or noise during the listening test broadcast, we will resume the problem at the beginning of the track and resume. However, if the teacher does not notice, please raise your hand and let us know. The listening test for the grade 1 examination is about to begin. Listen carefully to the directions. You will not be permitted to ask questions during the test. This test has four parts. All of the questions in these four parts are multiple-choice questions. For each question, choose the best answer from among the four choices written in your test booklet. On your answer sheet, find the number of the question and mark your answer. You are permitted to take notes for every part of this listening test. Now, here are the directions for part 1. In this part, you will hear 10 dialogues, number 1 through number 10. Each dialogue will be followed by one question. For each question, you will have 10 seconds to choose the best answer and mark your answer on your answer sheet. The dialogue and the question will be given only once. Now we will begin the grade 1 listening test. Number 1 What a day! And I'm still way behind schedule. I remember you were complaining a couple of months ago about being understaffed. But didn't two new people start in your section last week? That's exactly the problem. I'm spending all my time training them and hardly any time on my own projects. Ah, that makes sense. Well, it'll pay off eventually. I hope so. Right now though, I'd be much better off if I just did everything myself. Hang in there. Once the new hires are up to speed, your load should become much lighter. Question What is one problem the man has? Number 2 It looks like the pilot strike isn't going to end by Friday. I guess we're back to the drawing board with our weekend getaway plans then. I just can't believe they didn't accept the deal the airlines offered them. Well, I'm sympathetic to the pilots. Their salaries aren't nearly high enough, considering how stressful the work is. Anyway, staying home isn't so bad. We can still have fun driving around town. I suppose. What do people call it nowadays? A staycation? Exactly. There's plenty we haven't done right at our own doorstep. And we'll save money too. Question What does the woman imply? Number 3 What's up, Marie? Oh, not much. I'm on a new diet. Again? What is it this time? I'm avoiding anything that contains gluten, like bread or pasta. That sounds a bit extreme. Maybe, but this time it's on my doctor's advice. I was getting terrible indigestion and constipation. I don't think I could live without pasta. Is it helping? Seems to be. Making all my meals without bread or pasta gets kind of expensive though, and so many things contain gluten. I need to be very diligent about checking labels. Question Question Number 4 How was your internship, Thomas? Awful. What? You were ecstatic when you got accepted. You said the company's quite a famous publisher. Well, when I was interviewed for the position, I got the impression I would mostly be doing research, and possibly even contributing directly to articles. I take it that was not the case? No. Pretty much all I did was run errands and do some occasional proofreading for them. I believe they misled me intentionally. That's a shame. Regardless of your actual experience though, it will still look good on your resume. Question Number 5 My textbooks are costing more than I thought. My chemistry textbook alone is going to be two hundred and fifty dollars. Ouch! Can't you find a used copy? Unfortunately, I need this year's edition. That seems a bit unreasonable on the part of your professor. I think the point is that science is always advancing, so he wants to be sure we're using the latest information. Still, allowing the previous year's edition would save students a lot. You would think the professor could supply any new information with handouts. I guess it's just easier to have everyone get the newest copy. Question Number 6 Honey, did you hear what the neighbors decided to do about their raccoon problem? Are they finally going to start storing their garbage properly? Apparently that's too much work for them. Instead, it seems they are setting traps to catch and kill the raccoons. That's a rather cruel solution. I agree. It's not as though there have been reports of attacks or rabies, and there are much more humane ways of dealing with the problem. Unfortunately, they're within their rights as far as the law is concerned. Question Number 7 Are you going to order some wine? Yes, but I'm having a hard time deciding which one to choose. I love red wine, but I should probably go with white because I'm having fish, right? That may be the traditional pairing, but I think of it as a suggestion. The server isn't going to kick you out of the restaurant if you choose red. But this is such a fancy French restaurant. I don't want to do the wrong thing. In the end, it's your meal, so you should do what you want. I'm having a beer. You're right. Red it is. Question What is the man's opinion? Number 8 Pam, I'm looking for someone to do my taxes. Do you know anyone? Well, the accountant I use is good, but his rates are a little high. Is he worth it? I'd say so. He knows his stuff, including all the loopholes, and really takes his time. Despite how much he charges, he's saved me a bundle over the years. He sounds perfect. Could I have his number? Sure. He's pretty popular, so his schedule is packed. If you say I introduced you, though, I bet he'll find a way to fit you in. I see. Thanks. Question Why does the woman recommend her accountant? Number 9 Thank you for coming to see us here at Andex Computers. The recruiter was really impressed with your qualifications. Thank you. I hope I have the skills to contribute to the team here. Before we go any further, there are a couple of points in your application materials that I'd like to clear up. Of course. You've been in your present position as a sales representative for half a year, is that correct? Yes, and in that time I've always made or beaten my monthly sales targets. I'm sorry I can't supply a letter of recommendation, but my supervisor doesn't know I'm looking for another position. Understood. Before your present job, however, you worked at Logan TC, Inc. for less than a year. That's right. Your long-term commitment to the company is a high priority for us. To be honest, your record is a bit sparse on that point. Could you talk about that issue a bit? Well, if I can speak frankly, Logan TC was a dead-end position with almost no opportunity for advancement, and my current company, though I enjoy the work, is small. I think my organisational and sales skills are more suited to a larger organisation like Andex. I see. Thank you. Question. Why is the interviewer concerned about the applicant? I'm not sure. I'm not sure. I'm not sure. I'm not sure. Number 10. Okay, Mr. and Mrs. Fernandez, before you take me on as your financial consultant, I want to make sure we're on the same page. This service will cost you $500. You'll get a report and three follow-up meetings. That's fine with me. How about you, Mrs. Fernandez? Do you have any concerns? Not about your services. It's just that we're pretty deep in debt at this point. And your debts will only get worse unless you make major changes. There's no easy cure here. However, I can help you consolidate your debts and reduce the interest you're paying, especially on your credit cards. We're barely keeping up with the interest right now. I understand that. But you also have several hundred thousand dollars of equity tied up in your rental property. If you sold it and reinvested your equity in stocks, it would change your situation significantly. Isn't it a bad time to put a property on the market? True. It is a buyer's market. But since a lot of stocks are undervalued, you'll likely get very high returns on your investments. Anyway, I think things will look a whole lot better to you when I show you the comprehensive report. Question. What does the consultant recommend the couple do?

文字起こしの精度は見たところかなり高そうです。

応用編①

次は応用として話者分離を行います。

話者分離とは、誰がどの部分を話しているのか区別することです。

先ほどの文字起こしした結果はどの部分を誰が話したかわかりません。
そのため、話者分離をすることで初めてどの部分を誰が話したかわかるので、さらに文字起こしとして有意義な結果が得られます。

今回はpyannoteを利用して話者分離を行う方法と例を紹介します。

[応用編①]事前準備

pyannoteのインストールを行います。
また、pyannoteは（おそらく）mp3ファイルは対応していないため、mp3をwavに変換する用のライブラリをインストールします。
下記のコマンドを実行してインストールします。

C:\Users\user\デスクトップ\app\test_whisper> pip install pyannote.audio pydub

次にHugging Faceからpyannoteを利用するためのアクセストークンを作成します。

下記URLにアクセスしてください。
https://huggingface.co/pyannote

1. アカウント作成
   まず右上の[Sign Up]をクリックしてアカウント作成を行います。
   
2. アクセストークンの取得
   右上のプロフィール写真にカーソルを合わせて、[Settings]をクリックしてください。
   
   左に[Access Tokens]をクリックしてください。
   
   User Access Tokensにある[New token]をクリックしてください。
   
   Nameには任意の名前、RoleにはReadを選択し、[Generate Token]をクリックしてください。
   
   生成されたアクセストークンをコピーしてください。
3. アクセストークンの環境変数への登録
   [Windows + R]を押し、sysdm.cplと入力してください。
   
   そうすると、システムのプロパティが表示されるので[環境変数]をクリックします。
   
   ユーザー環境変数にて[新規]をクリックします。
   
   変数名に「PYANNOTE_KEY」、変数値に先ほどコピーしたアクセストークンを入力し、[OK]をクリックします。
   
   その後[OK]をクリックして、システムのプロパティ設定を閉じます。
4. pyannoteの利用にあたって条件の承諾
   Hugging Faceへログインしたまま下記の2つのサイトへアクセスし、それぞれのUser ConditionsについてAcceptしてください。
   https://huggingface.co/pyannote/speaker-diarization
   https://huggingface.co/pyannote/segmentation
   これでpyannoteを利用できる準備が整いました。

[応用編①]作成方法とその実行例

1. pythonファイルの作成
   pythonのファイルを作成します。
   「# ファイルの準備」下にあるファイル名はお使いの音源名に合わせてご利用ください。

test.py

# ライブラリの読み込み
import whisper
import os

from pyannote.audio import Pipeline
from pyannote.audio import Audio
from pydub import AudioSegment

# トークンの準備
auth_token = os.environ['PYANNOTE_KEY']

# ファイルの準備
audio_file = "1Q-part1.mp3"
file = "1Q-part1.wav"

# mp3ファイルを読み込む
audio = AudioSegment.from_file(audio_file, format="mp3")
# wavファイルに変換して保存する
audio.export(file, format="wav")

# パイプライン
pipeline = Pipeline.from_pretrained("pyannote/speaker-diarization@2.1", use_auth_token=auth_token)
diarization = pipeline(file)
audio = Audio(sample_rate=16000, mono=True)

# モデルの読み込み
model = whisper.load_model("medium")

# 話者分離して出力する
for segment, _, speaker in diarization.itertracks(yield_label = True):
    waveform, sample_rate = audio.crop(file,segment)
    text = model.transcribe(waveform.squeeze().numpy(), fp16 = False, language="en")
    # 認識したテキストの出力
    result = f"[{segment.start:06.1f}s - {segment.end:06.1f}s] {speaker}:{text}"
    print(result)

1. pythonファイルの実行
   pythonファイルを実行します。（なお、先ほどとは異なり、管理者権限でターミナルに入ってください。）

C:\Users\user\デスクトップ\app\test_whisper> python test.py
The torchaudio backend is switched to 'soundfile'. Note that 'sox_io' is not supported on Windows.
torchvision is not available - cannot save figures
The torchaudio backend is switched to 'soundfile'. Note that 'sox_io' is not supported on Windows.
[0000.5s - 0006.5s] SPEAKER_01: Please check the volume and sound of the room.
[0011.3s - 0013.2s] SPEAKER_01: 2022
[0013.8s - 0014.9s] SPEAKER_01: 第3回
[0015.8s - 0020.9s] SPEAKER_01: 実用英語技能検定 1級リスニングテスト
[0024.0s - 0028.0s] SPEAKER_01: We will now conduct a first-class listening test.
[0029.3s - 0035.7s] SPEAKER_01: We will not be answering any questions about what we are going to talk about, so please be careful.
[0036.8s - 0047.0s] SPEAKER_01: If there is a sound or a noise during the listening test broadcast, we will restart the problem by returning it to the beginning of the track.
[0047.8s - 0056.0s] SPEAKER_01: However, if the teacher does not notice, please raise your hand and let him know.
[0057.2s - 0063.7s] SPEAKER_01: This test will show you how and when to take the test in English.
[0065.8s - 0069.7s] SPEAKER_00: The listening test for the grade one examination is about to begin.
[0070.4s - 0072.4s] SPEAKER_00: Listen carefully to the directions.
[0073.1s - 0083.4s] SPEAKER_00: You will not be permitted to ask questions during the test. This test has four parts. All of the questions in these four parts are multiple choice questions.
[0084.4s - 0090.4s] SPEAKER_00: For each question, choose the best answer from among the four choices written in your test booklet.
[0091.4s - 0095.8s] SPEAKER_00: On your answer sheet, find the number of the question and mark your answer.
[0096.7s - 0100.6s] SPEAKER_00: You are permitted to take notes for every part of this listening test.
[0102.0s - 0109.7s] SPEAKER_00: Now here are the directions for part 1. In this part, you will hear 10 dialogues, number 1 through number 10.
[0110.6s - 0113.3s] SPEAKER_00: Each dialogue will be followed by one question.
[0114.6s - 0121.2s] SPEAKER_00: For each question, you will have 10 seconds to choose the best answer and mark your answer on your answer sheet.
[0122.2s - 0125.4s] SPEAKER_00: The dialogue and the question will be given only once.
[0126.9s - 0126.9s] SPEAKER_00: you
[0126.9s - 0130.2s] SPEAKER_02: Now we will begin the Grade 1 Listening Test.
[0133.3s - 0134.5s] SPEAKER_02: Number 1
[0135.2s - 0139.0s] SPEAKER_04: What a day! And I'm still way behind schedule.
[0139.3s - 0148.8s] SPEAKER_03: I remember you were complaining a couple of months ago about being understaffed, but didn't two new people start in your section last week?
[0149.1s - 0156.5s] SPEAKER_04: That's exactly the problem. I'm spending all my time training them, and hardly any time on my own projects.
[0156.9s - 0161.2s] SPEAKER_03: Ah, that makes sense. Well, it'll pay off eventually.
[0161.5s - 0167.7s] SPEAKER_04: I hope so. Right now though, I'd be much better off if I just did everything myself.
[0167.9s - 0174.1s] SPEAKER_03: Hang in there. Once the new hires are up to speed, your load should become much lighter.
[0174.9s - 0178.8s] SPEAKER_02: Question. What is one problem the man has?
[0188.9s - 0189.5s] SPEAKER_04: Number 2
[0189.5s - 0189.7s] SPEAKER_05: you
[0190.7s - 0194.4s] SPEAKER_05: It looks like the pilot strike isn't going to end by Friday.
[0194.7s - 0198.8s] SPEAKER_02: I guess we're back to the drawing board with our weekend getaway plans then.
[0199.3s - 0203.8s] SPEAKER_05: I just can't believe they didn't accept the deal the airlines offered them.
[0204.1s - 0219.0s] SPEAKER_02: Well, I'm sympathetic to the pilots. Their salaries aren't nearly high enough, considering how stressful the work is. Anyway, staying home isn't so bad. We can still have fun driving around town. I suppose.
[0219.0s - 0222.4s] SPEAKER_05: What do people call it nowadays? A staycation?
[0222.8s - 0229.8s] SPEAKER_02: Exactly! There's plenty we haven't done right at our own doorstep, and we'll save money too!
[0230.9s - 0234.1s] SPEAKER_04: Question. What does the woman imply?
[0244.1s - 0244.3s] SPEAKER_05: Number 2.
[0244.3s - 0245.0s] SPEAKER_04: Number 3
[0246.1s - 0247.1s] SPEAKER_04: What's up, Marie?
[0247.2s - 0250.7s] SPEAKER_03: Oh, not much. I'm on a new diet.
[0250.8s - 0253.2s] SPEAKER_04: Again? What is it this time?
[0253.2s - 0258.6s] SPEAKER_03: I'm avoiding anything that contains gluten, like bread or pasta.
[0258.6s - 0261.9s] SPEAKER_04: That sounds a bit extreme. Maybe.
[0261.9s - 0268.0s] SPEAKER_03: but this time it's on my doctor's advice. I was getting terrible indigestion and constipation.
[0268.0s - 0273.9s] SPEAKER_04: I don't think I could live without pasta. Is it helping? Seems to be.
[0273.9s - 0285.8s] SPEAKER_03: Making all my meals without bread or pasta gets kind of expensive though, and so many things contain gluten. I need to be very diligent about checking labels.
[0286.8s - 0291.0s] SPEAKER_05: Question Why did the woman change what she eats?
[0301.0s - 0301.4s] SPEAKER_03: Number 4
[0301.4s - 0302.0s] SPEAKER_02: for.
[0302.8s - 0305.0s] SPEAKER_02: How was your internship, Thomas?
[0305.4s - 0306.3s] SPEAKER_05: Awful.
[0306.5s - 0313.0s] SPEAKER_02: What? You were ecstatic when you got accepted. You said the company's quite a famous publisher.
[0313.3s - 0323.1s] SPEAKER_05: Well, when I was interviewed for the position, I got the impression I would mostly be doing research, and possibly even contributing directly to articles.
[0323.6s - 0325.7s] SPEAKER_02: I take it that was not the case?
[0326.2s - 0336.3s] SPEAKER_05: No. Pretty much all I did was run errands and do some occasional proofreading for them. I believe they misled me intentionally.
[0336.7s - 0343.0s] SPEAKER_02: That's a shame. Regardless of your actual experience though, it will still look good on your resume.
[0343.9s - 0348.0s] SPEAKER_03: Question. What does the man say about the internship?
[0358.1s - 0359.0s] SPEAKER_04: Number 5
[0362.1s - 0370.0s] SPEAKER_05: My textbooks are costing more than I thought. My chemistry textbook alone is going to be $250.
[0370.0s - 0373.5s] SPEAKER_03: Ouch! Can't you find a used copy?
[0374.0s - 0376.9s] SPEAKER_05: Unfortunately, I need this year's edition.
[0377.5s - 0381.1s] SPEAKER_03: That seems a bit unreasonable on the part of your professor.
[0381.5s - 0388.3s] SPEAKER_05: I think the point is that science is always advancing, so he wants to be sure we're using the latest information.
[0388.7s - 0398.2s] SPEAKER_03: Still, allowing the previous year's edition would save students a lot. You would think the professor could supply any new information with handouts.
[0398.5s - 0402.4s] SPEAKER_05: I guess it's just easier to have everyone get the newest copy.
[0403.3s - 0403.4s] SPEAKER_05:
[0403.4s - 0407.6s] SPEAKER_04: Question. What does the woman imply about the professor?
[0417.6s - 0418.7s] SPEAKER_02: Number 6
[0418.7s - 0418.7s] SPEAKER_03: you
[0419.7s - 0424.8s] SPEAKER_03: Honey, did you hear what the neighbours decided to do about their raccoon problem?
[0424.9s - 0429.4s] SPEAKER_04: Are they finally going to start storing their garbage properly? Apparently.
[0429.4s - 0435.9s] SPEAKER_03: that's too much work for them. Instead it seems they are setting traps to catch and kill the raccoon.
[0435.9s - 0439.6s] SPEAKER_04: That's a rather cruel solution.
[0439.6s - 0448.4s] SPEAKER_03: agree. It's not as though there have been reports of attacks or rabies and there are much more humane ways of dealing with the problem.
[0448.1s - 0453.4s] SPEAKER_04: Unfortunately, they're within their rights as far as the law is concerned.
[0454.4s - 0459.1s] SPEAKER_02: Question. What does the couple imply about the situation?
[0469.2s - 0469.2s] SPEAKER_05: you
[0469.2s - 0470.1s] SPEAKER_04: Number 7
[0471.1s - 0473.0s] SPEAKER_04: Are you going to order some wine?
[0473.3s - 0484.2s] SPEAKER_02: Yes, but I'm having a hard time deciding which one to choose. I love red wine, but I should probably go with white because I'm having fish, right?
[0484.5s - 0493.6s] SPEAKER_04: That may be the traditional pairing, but I think of it as a suggestion. The server isn't going to kick you out of the restaurant if you choose red.
[0494.1s - 0499.3s] SPEAKER_02: But this is such a fancy French restaurant. I don't want to do the wrong thing.
[0499.5s - 0505.3s] SPEAKER_04: In the end, it's your meal, so you should do what you want. I'm having a beer.
[0505.3s - 0507.4s] SPEAKER_02: You're right. Red it is!
[0508.4s - 0511.4s] SPEAKER_05: Question. What is the man's opinion?
[0521.6s - 0522.3s] SPEAKER_03: Number 8
[0522.2s - 0522.3s] SPEAKER_05:
[0523.5s - 0528.3s] SPEAKER_05: Pam, I'm looking for someone to do my taxes. Do you know anyone?
[0528.6s - 0533.4s] SPEAKER_02: Well, the accountant I use is good, but his rates are a little high.
[0533.8s - 0534.8s] SPEAKER_05: Is he worth it?
[0535.2s - 0545.9s] SPEAKER_02: I'd say so. He knows his stuff, including all the loopholes, and really takes his time. Despite how much he charges, he's saved me a bundle over the years.
[0546.4s - 0549.6s] SPEAKER_05: He sounds perfect. Could I have his number?
[0550.0s - 0558.1s] SPEAKER_02: Sure, he's pretty popular, so his schedule is packed. If you say I introduced you though, I bet he'll find a way to fit you in.
[0558.6s - 0561.0s] SPEAKER_05: I see. Thanks.
[0561.0s - 0565.3s] SPEAKER_03: Question. Why does the woman recommend her accountant?
[0575.4s - 0576.4s] SPEAKER_02: Number nine.
[0576.4s - 0576.5s] SPEAKER_01: you
[0576.5s - 0576.5s] SPEAKER_05: you
[0577.5s - 0585.5s] SPEAKER_05: Thank you for coming to see us here at Andex Computers. The recruiter was really impressed with your qualifications.
[0585.5s - 0589.9s] SPEAKER_03: Thank you. I hope I have the skills to contribute to the team here.
[0589.9s - 0604.3s] SPEAKER_05: Before we go any further, there are a couple of points in your application materials that I'd like to clear up. Of course. You've been in your present position as a sales representative for half a year, is that correct?
[0604.9s - 0610.5s] SPEAKER_03: Yes, and in that time I've always made or beaten my monthly sales targets.
[0611.3s - 0618.3s] SPEAKER_03: I'm sorry I can't supply a letter of recommendation, but my supervisor doesn't know I'm looking for another position.
[0618.8s - 0627.8s] SPEAKER_05: Understood. Before your present job, however, you worked at Logan TC, Incorporated for less than a year. That's right.
[0628.7s - 0632.9s] SPEAKER_05: Long-term commitment to the company is a high priority for us.
[0633.7s - 0640.3s] SPEAKER_05: To be honest, your record is a bit sparse on that point. Could you talk about that issue a bit?
[0640.8s - 0663.1s] SPEAKER_03: Well, if I can speak frankly, Logan TC was a dead-end position with almost no opportunity for advancement. And my current company, though I enjoy the work, is small. I think my organisational and sales skills are more suited to a larger organisation like Andex. I see. Thank you.
[0662.5s - 0663.2s] SPEAKER_05: Thank you.
[0664.3s - 0664.3s] SPEAKER_05:
[0664.3s - 0669.3s] SPEAKER_01: Question, why is the interviewer concerned about the applicant?
[0679.3s - 0680.3s] SPEAKER_02: Number 10
[0681.4s - 0702.2s] SPEAKER_04: Okay, Mr. and Mrs. Fernandez, before you take me on as your financial consultant, I want to make sure we're on the same page. This service will cost you $500. You'll get a report and three follow-up meetings. That's fine with me. How about you, Mrs. Fernandez? Do you have any concerns?
[0702.6s - 0708.4s] SPEAKER_03: Not about your services, it's just that we're pretty deep in debt.
[0708.4s - 0714.3s] SPEAKER_04: at this point. And your debts will only get worse unless you make major changes.
[0714.9s - 0725.3s] SPEAKER_04: There's no easy cure here. However, I can help you consolidate your debts and reduce the interest you're paying, especially on your credit cards.
[0724.7s - 0728.5s] SPEAKER_05: We're barely keeping up with the interest right now.
[0727.6s - 0735.2s] SPEAKER_04: I understand that. But you also have several hundred thousand dollars of equity tied up in your rental property.
[0735.8s - 0742.7s] SPEAKER_04: If you sold it and reinvested your equity in stocks, it would change your situation significantly.
[0743.1s - 0746.4s] SPEAKER_03: Isn't it a bad time to put a property on the market?
[0746.7s - 0763.5s] SPEAKER_04: True, it is a buyer's market, but since a lot of stocks are undervalued, you'll likely get very high returns on your investments. Anyway, I think things will look a whole lot better to you when I show you the comprehensive report.
[0764.5s - 0769.1s] SPEAKER_02: Question. What does the consultant recommend the couple do?

このようにきれいに話者分離することができました。

[応用編②]ChatGPTとWhisperを併用してみる。

また、別途応用としてWhisperとChatGPTを利用して英検のリスニング問題を解かせることにトライしてみました。

対象となる問題は2022年度第3回のPart1です。

なお、事前にPythonのopenaiライブラリとPyPDF2のライブラリを読み込んでいます。

> pip install openai PyPDF2

1. 問題ファイルとリスニング音源をそれぞれテキスト形式にします。
   問題ファイルはtext.txtへ文字起こし、リスニング音源は1Qonsei.txtに文字起こししています。
   text.txtにはPDFから文字起こししたファイルを私がちょっと手を加えています。（よろしくない。）
   1Qonsei.txtには先に起こしていた音源の文字起こしデータから問題部分を切り取ったものです。
   この2ファイルについては恣意的に操作している関係、気を付けて参照してください。

問題のPDFをPyPDF2を用いて文字起こしします。

text.py

import PyPDF2
# PDFの読み込み
with open("2022-3-1ji-1kyu.pdf", "rb") as f:
    reader = PyPDF2.PdfReader(f)
    page = reader.pages[15].extract_text() + "\n" +  reader.pages[16].extract_text()
# テキストファイルへ出力
with open("text.txt",'w', encoding='UTF-8') as f:
    f.write(page)

その後、文字化けしている部分を削除し、下記のように手動で修正しました。

text.txt

No. 1 1He did not receive training for his current project.
2He is not keeping up with his workload.
3His deadlines were suddenly shortened.
4Many of his colleagues suddenly quit.
No. 2 1The pilots are justified in striking.
2The couple cannot afford an expensive vacation.
3The couple should arrange a different flight.
4The strike may not last much longer.
No. 3 1Her friend recommended a new diet.
2She wanted to lose weight.
3She is trying to save money.
4She was having health problems.
No. 4 1He was not praised for his work.
2The company was not truthful about his duties.
3The company asked him to do too much.
4It was more exhausting than he expected.
No. 5 1He should stop using so much supplemental material.
2He is using outdated information.
3He should be more considerate to his students.
4He has published important research recently.
No. 6 1The raccoons are a health threat.
2The neighbors should consult a lawyer.
3The garbage problem is going to get worse.
4The neighbors’ solution is a bad one.
No. 7 1Wine prices are too high at the restaurant.
2The server made a good recommendation.
3The woman should worry less about what people say.
4Beer would be best with the woman’s meal.
No. 8 1He charges less than most people.
2His tax knowledge is extensive.
3He can do the man’s taxes quickly.
4He does not have very many clients.
No. 9 1She lacks some necessary academic qualifications.
2She was asked to leave a previous job.
3She has not included a letter of recommendation.
4She has not stayed at any one job for long.
No. 10 1Sell some property and invest the money.
2Sell off their stocks.
3Get a loan to pay their credit card debt.
4Increase the rent on their property.

音声の文字起こしは問題となる部分だけを手動で持ってきています。

1Qonsei.txt

Number 1 What a day! And I'm still way behind schedule. I remember you were complaining a couple of months ago about being understaffed. But didn't two new people start in your section last week? That's exactly the problem. I'm spending all my time training them and hardly any time on my own projects. Ah, that makes sense. Well, it'll pay off eventually. I hope so. Right now though, I'd be much better off if I just did everything myself. Hang in there. Once the new hires are up to speed, your load should become much lighter. Question What is one problem the man has? Number 2 It looks like the pilot strike isn't going to end by Friday. I guess we're back to the drawing board with our weekend getaway plans then. I just can't believe they didn't accept the deal the airlines offered them. Well, I'm sympathetic to the pilots. Their salaries aren't nearly high enough, considering how stressful the work is. Anyway, staying home isn't so bad. We can still have fun driving around town. I suppose. What do people call it nowadays? A staycation? Exactly. There's plenty we haven't done right at our own doorstep. And we'll save money too. Question What does the woman imply? Number 3 What's up, Marie? Oh, not much. I'm on a new diet. Again? What is it this time? I'm avoiding anything that contains gluten, like bread or pasta. That sounds a bit extreme. Maybe, but this time it's on my doctor's advice. I was getting terrible indigestion and constipation. I don't think I could live without pasta. Is it helping? Seems to be. Making all my meals without bread or pasta gets kind of expensive though, and so many things contain gluten. I need to be very diligent about checking labels. Question Question Number 4 How was your internship, Thomas? Awful. What? You were ecstatic when you got accepted. You said the company's quite a famous publisher. Well, when I was interviewed for the position, I got the impression I would mostly be doing research, and possibly even contributing directly to articles. I take it that was not the case? No. Pretty much all I did was run errands and do some occasional proofreading for them. I believe they misled me intentionally. That's a shame. Regardless of your actual experience though, it will still look good on your resume. Question Number 5 My textbooks are costing more than I thought. My chemistry textbook alone is going to be two hundred and fifty dollars. Ouch! Can't you find a used copy? Unfortunately, I need this year's edition. That seems a bit unreasonable on the part of your professor. I think the point is that science is always advancing, so he wants to be sure we're using the latest information. Still, allowing the previous year's edition would save students a lot. You would think the professor could supply any new information with handouts. I guess it's just easier to have everyone get the newest copy. Question Number 6 Honey, did you hear what the neighbors decided to do about their raccoon problem? Are they finally going to start storing their garbage properly? Apparently that's too much work for them. Instead, it seems they are setting traps to catch and kill the raccoons. That's a rather cruel solution. I agree. It's not as though there have been reports of attacks or rabies, and there are much more humane ways of dealing with the problem. Unfortunately, they're within their rights as far as the law is concerned. Question Number 7 Are you going to order some wine? Yes, but I'm having a hard time deciding which one to choose. I love red wine, but I should probably go with white because I'm having fish, right? That may be the traditional pairing, but I think of it as a suggestion. The server isn't going to kick you out of the restaurant if you choose red. But this is such a fancy French restaurant. I don't want to do the wrong thing. In the end, it's your meal, so you should do what you want. I'm having a beer. You're right. Red it is. Question What is the man's opinion? Number 8 Pam, I'm looking for someone to do my taxes. Do you know anyone? Well, the accountant I use is good, but his rates are a little high. Is he worth it? I'd say so. He knows his stuff, including all the loopholes, and really takes his time. Despite how much he charges, he's saved me a bundle over the years. He sounds perfect. Could I have his number? Sure. He's pretty popular, so his schedule is packed. If you say I introduced you, though, I bet he'll find a way to fit you in. I see. Thanks. Question Why does the woman recommend her accountant? Number 9 Thank you for coming to see us here at Andex Computers. The recruiter was really impressed with your qualifications. Thank you. I hope I have the skills to contribute to the team here. Before we go any further, there are a couple of points in your application materials that I'd like to clear up. Of course. You've been in your present position as a sales representative for half a year, is that correct? Yes, and in that time I've always made or beaten my monthly sales targets. I'm sorry I can't supply a letter of recommendation, but my supervisor doesn't know I'm looking for another position. Understood. Before your present job, however, you worked at Logan TC, Inc. for less than a year. That's right. Your long-term commitment to the company is a high priority for us. To be honest, your record is a bit sparse on that point. Could you talk about that issue a bit? Well, if I can speak frankly, Logan TC was a dead-end position with almost no opportunity for advancement, and my current company, though I enjoy the work, is small. I think my organisational and sales skills are more suited to a larger organisation like Andex. I see. Thank you. Question. Why is the interviewer concerned about the applicant? I'm not sure. I'm not sure. I'm not sure. I'm not sure. Number 10. Okay, Mr. and Mrs. Fernandez, before you take me on as your financial consultant, I want to make sure we're on the same page. This service will cost you $500. You'll get a report and three follow-up meetings. That's fine with me. How about you, Mrs. Fernandez? Do you have any concerns? Not about your services. It's just that we're pretty deep in debt at this point. And your debts will only get worse unless you make major changes. There's no easy cure here. However, I can help you consolidate your debts and reduce the interest you're paying, especially on your credit cards. We're barely keeping up with the interest right now. I understand that. But you also have several hundred thousand dollars of equity tied up in your rental property. If you sold it and reinvested your equity in stocks, it would change your situation significantly. Isn't it a bad time to put a property on the market? True. It is a buyer's market. But since a lot of stocks are undervalued, you'll likely get very high returns on your investments. Anyway, I think things will look a whole lot better to you when I show you the comprehensive report. Question. What does the consultant recommend the couple do?

1. ChatGPTに問題文とリスニング問題を読み込ませて答えを出力させます。

answer.py

import os
import openai

# openai.api_keyにOpenAIのAPIキーを入れる
openai.api_key = os.environ['OPEN_API_KEY']

# ファイルの読み込み

text = open('text.txt', 'r', encoding='UTF-8')
onsei = open('1Qonsei.txt', 'r', encoding='UTF-8')


# ChatGPTを使用して、応答を生成する

prompt = "会話文と質問文を含む英語の文章と選択肢を与えます。英語の文章にはNo.1~10が10個の英文とそれに対応する質問があります。No.1~10の質問に対して後に与える選択肢から解答を出力してください。 \n"

prompt += "\n英語の文章です。\n"

prompt += onsei.read()

prompt += "\n選択肢です。\n"

prompt += text.read()

# 問題を出力する。
print(prompt)

response = openai.Completion.create(
    engine="text-davinci-002",
    prompt=prompt,
    temperature=0.5,
    max_tokens=100,
    n=1,
    stop=None,
    frequency_penalty=0,
    presence_penalty=0
)

# ChatGPTからの応答を出力する
print("=========================================================")
print(response.choices[0].text.strip())
print("=========================================================")

[応用編②]結果

ChatGPTの結果...

1. 2 ✅
2. 1 ✅
3. 4 ✅
4. 2 ✅
5. 3 ✅
6. 4 ✅
7. 2 ❌
8. 2 ✅
9. 4 ✅
10. 1 ✅

10問中9問正解でした。なかなかやりますね。
ちなみに私は10問中7問正解でした。勝てません。

下記出力の結果です。

会話文と質問文を含む英語の文章と選択肢を与えます。英語の文章にはNo.1~10が10個の英文とそれに対応する質問があります。No.1~10の質問に対して後に与える選択肢から解答を出力してください。

英語の文章です。
Number 1 What a day! And I'm still way behind schedule. I remember you were complaining a couple of months ago about being understaffed. But didn't two new people start in your section last week? That's exactly the problem. I'm spending all my time training them and hardly any time on my own projects. Ah, that makes sense. Well, it'll pay off eventually. I hope so. Right now though, I'd be much better off if I just did everything myself. Hang in there. Once the new hires are up to speed, your load should become much lighter. Question What is one problem the man has? Number 2 It looks like the pilot strike isn't going to end by Friday. I guess we're back to the drawing board with our weekend getaway plans then. I just can't believe they didn't accept the deal the airlines offered them. Well, I'm sympathetic to the pilots. Their salaries aren't nearly high enough, considering how stressful the work is. Anyway, staying home isn't so bad. We can still have fun driving around town. I suppose. What do people call it nowadays? A staycation? Exactly. There's plenty we haven't done right at our own doorstep. And we'll save money too. Question What does the woman imply? Number 3 What's up, Marie? Oh, not much. I'm on a new diet. Again? What is it this time? I'm avoiding anything that contains gluten, like bread or pasta. That sounds a bit extreme. Maybe, but this time it's on my doctor's advice. I was getting terrible indigestion and constipation. I don't think I could live without pasta. Is it helping? Seems to be. Making all my meals without bread or pasta gets kind of expensive though, and so many things contain gluten. I need to be very diligent about checking labels. Question Question Number 4 How was your internship, Thomas? Awful. What? You were ecstatic when you got accepted. You said the company's quite a famous publisher. Well, when I was interviewed for the position, I got the impression I would mostly be doing research, and possibly even contributing directly to articles. I take it that was not the case? No. Pretty much all I did was run errands and do some occasional proofreading for them. I believe they misled me intentionally. That's a shame. Regardless of your actual experience though, it will still look good on your resume. Question Number 5 My textbooks are costing more than I thought. My chemistry textbook alone is going to be two hundred and fifty dollars. Ouch! Can't you find a used copy? Unfortunately, I need this year's edition. That seems a bit unreasonable on the part of your professor. I think the point is that science is always advancing, so he wants to be sure we're using the latest information. Still, allowing the previous year's edition would save students a lot. You would think the professor could supply any new information with handouts. I guess it's just easier to have everyone get the newest copy. Question Number 6 Honey, did you hear what the neighbors decided to do about their raccoon problem? Are they finally going to start storing their garbage properly? Apparently that's too much work for them. Instead, it seems they are setting traps to catch and kill the raccoons. That's a rather cruel solution. I agree. It's not as though there have been reports of attacks or rabies, and there are much more humane ways of dealing with the problem. Unfortunately, they're within their rights as far as the law is concerned. Question Number 7 Are you going to order some wine? Yes, but I'm having a hard time deciding which one to choose. I love red wine, but I should probably go with white because I'm having fish, right? That may be the traditional pairing, but I think of it as a suggestion. The server isn't going to kick you out of the restaurant if you choose red. But this is such a fancy French restaurant. I don't want to do the wrong thing. In the end, it's your meal, so you should do what you want. I'm having a beer. You're right. Red it is. Question What is the man's opinion? Number 8 Pam, I'm looking for someone to do my taxes. Do you know anyone? Well, the accountant I use is good, but his rates are a little high. Is he worth it? I'd say so. He knows his stuff, including all the loopholes, and really takes his time. Despite how much he charges, he's saved me a bundle over the years. He sounds perfect. Could I have his number? Sure. He's pretty popular, so his schedule is packed. If you say I introduced you, though, I bet he'll find a way to fit you in. I see. Thanks. Question Why does the woman recommend her accountant? Number 9 Thank you for coming to see us here at Andex Computers. The recruiter was really impressed with your qualifications. Thank you. I hope I have the skills to contribute to the team here. Before we go any further, there are a couple of points in your application materials that I'd like to clear up. Of course. You've been in your present position as a sales representative for half a year, is that correct? Yes, and in that time I've always made or beaten my monthly sales targets. I'm sorry I can't supply a letter of recommendation, but my supervisor doesn't know I'm looking for another position. Understood. Before your present job, however, you worked at Logan TC, Inc. for less than a year. That's right. Your long-term commitment to the company is a high priority for us. To be honest, your record is a bit sparse on that point. Could you talk about that issue a bit? Well, if I can speak frankly, Logan TC was a dead-end position with almost no opportunity for advancement, and my current company, though I enjoy the work, is small. I think my organisational and sales skills are more suited to a larger organisation like Andex. I see. Thank you. Question. Why is the interviewer concerned about the applicant? I'm not sure. I'm not sure. I'm not sure. I'm not sure. Number 10. Okay, Mr. and Mrs. Fernandez, before you take me on as your financial consultant, I want to make sure we're on the same page. This service will cost you $500. You'll get a report and three follow-up meetings. That's fine with me. How about you, Mrs. Fernandez? Do you have any concerns? Not about your services. It's just that we're pretty deep in debt at this point. And your debts will only get worse unless you make major changes. There's no easy cure here. However, I can help you consolidate your debts and reduce the interest you're paying, especially on your credit cards. We're barely keeping up with the interest right now. I understand that. But you also have several hundred thousand dollars of equity tied up in your rental property. If you sold it and reinvested your equity in stocks, it would change your situation significantly. Isn't it a bad time to put a property on the market? True. It is a buyer's market. But since a lot of stocks are undervalued, you'll likely get very high returns on your investments. Anyway, I think things will look a whole lot better to you when I show you the comprehensive report. Question. What does the consultant recommend the couple do?
選択肢です。
No. 1 1He did not receive training for his current project.
2He is not keeping up with his workload.
3His deadlines were suddenly shortened.
4Many of his colleagues suddenly quit.
No. 2 1The pilots are justified in striking.
2The couple cannot afford an expensive vacation.
3The couple should arrange a different flight.
4The strike may not last much longer.
No. 3 1Her friend recommended a new diet.
2She wanted to lose weight.
3She is trying to save money.
4She was having health problems.
No. 4 1He was not praised for his work.
2The company was not truthful about his duties.
3The company asked him to do too much.
4It was more exhausting than he expected.
No. 5 1He should stop using so much supplemental material.
2He is using outdated information.
3He should be more considerate to his students.
4He has published important research recently.
No. 6 1The raccoons are a health threat.
2The neighbors should consult a lawyer.
3The garbage problem is going to get worse.
4The neighbors’ solution is a bad one.
No. 7 1Wine prices are too high at the restaurant.
2The server made a good recommendation.
3The woman should worry less about what people say.
4Beer would be best with the woman’s meal.
No. 8 1He charges less than most people.
2His tax knowledge is extensive.
3He can do the man’s taxes quickly.
4He does not have very many clients.
No. 9 1She lacks some necessary academic qualifications.
2She was asked to leave a previous job.
3She has not included a letter of recommendation.
4She has not stayed at any one job for long.
No. 10 1Sell some property and invest the money.
2Sell off their stocks.
3Get a loan to pay their credit card debt.
4Increase the rent on their property.
=========================================================
1. 2
2. 1
3. 4
4. 2
5. 3
6. 4
7. 2
8. 2
9. 4
10. 1
=========================================================

解答は下記のページにあります。ご参照ください。
https://www.eiken.or.jp/eiken/exam/grade_1/pdf/202203/1kyu.pdf

感想

ちゃんと問題の形式を理解して出力させられたのがすごい。
次回は対話形式のリスニング音声文字起こしスクリプトを読ませてChatGPTに問題を解かせてみたい。

何か間違いや気になることがあればコメントお願いします。

参考

https://github.com/openai/whisper
https://happy-shibusawake.com/openai_whisper/696/
https://qiita.com/sayo0127/items/e22fdc229d2dfd879f75
https://huggingface.co/pyannote