ナレッジベースのチャンキング戦略がいろいろ増えたので動きを確認していきます。
日本語文書では微妙だったので、英語のエルマーの冒険をS3に格納してチャンキングしていきます。
デフォルトチャンキング
ナレッジベースを同期後、OpenSearchのダッシュボードのDev Toolsで内容を確認してみます。
"_source": {
"x-amz-bedrock-kb-source-uri": "s3://***/pg30017.txt",
"AMAZON_BEDROCK_METADATA": """{"source":"s3://***/pg30017.txt"}""",
"AMAZON_BEDROCK_METADATA"にはソースドキュメントの場所が格納されているようです。
冒頭幾つかのチャンクの内容を確認してみます（長いので畳みます）。
"_source": {
"AMAZON_BEDROCK_TEXT_CHUNK": """
The Project Gutenberg eBook of My Father's Dragon
This ebook is for the use of anyone anywhere in the United States and
most other parts of the world at no cost and with almost no restrictions
whatsoever. You may copy it, give it away or re-use it under the terms
of the Project Gutenberg License included with this ebook or online
at www.gutenberg.org. If you are not located in the United States,
you will have to check the laws of the country where you are located
before using this eBook."""
}
"_source": {
"AMAZON_BEDROCK_TEXT_CHUNK": """
Title: My Father's Dragon
Author: Ruth Stiles Gannett
Illustrator: Ruth Chrisman Gannett
Release date: September 18, 2009 [eBook #30017]
Language: English
Credits: Produced by Sankar Viswanathan, Greg Weeks, and the Online
Distributed Proofreading Team at http://www.pgdp.net
*** START OF THE PROJECT GUTENBERG EBOOK MY FATHER'S DRAGON ***
Produced by Sankar Viswanathan, Greg Weeks, and the Online
Distributed Proofreading Team at http://www.pgdp.net
Transcriber's Note:
Extensive research did not uncover any evidence that the
U.S. copyright on this publication was renewed.
MY FATHER'S DRAGON
STORY BY
RUTH STILES GANNETT
ILLUSTRATIONS BY
RUTH CHRISMAN GANNETT
RANDOM HOUSE . NEW YORK
COPYRIGHT 1948 BY RANDOM HOUSE, INC."""
}
"_source": {
"AMAZON_BEDROCK_TEXT_CHUNK": """
* * * * *
For My
FATHER
* * * * *
CONTENTS
1. My Father Meets the Cat 9
2. My Father Runs Away 15
3. My Father Finds the Island 22
4. My Father Finds the River 31
5. My Father Meets Some Tigers 39
6. My Father Meets A Rhinoceros 48
7. My Father Meets A Lion 56
8. My Father Meets A Gorilla 63
9. My Father Makes A Bridge 73
10. My Father Finds the Dragon 79
* * * * *
_Chapter One_
MY FATHER MEETS THE CAT
One cold rainy day when my father was a little boy, he met an old
alley cat on his street."""
}
"_source": {
"AMAZON_BEDROCK_TEXT_CHUNK": """
My Father Makes A Bridge 73
10. My Father Finds the Dragon 79
* * * * *
_Chapter One_
MY FATHER MEETS THE CAT
One cold rainy day when my father was a little boy, he met an old
alley cat on his street. The cat was very drippy and uncomfortable so
my father said, "Wouldn't you like to come home with me?"
This surprised the cat--she had never before met anyone who cared
about old alley cats--but she said, "I'd be very much obliged if I
could sit by a warm furnace, and perhaps have a saucer of milk."
"We have a very nice furnace to sit by," said my father, "and I'm sure
my mother has an extra saucer of milk."
[Illustration]
My father and the cat became good friends but my father's mother was
very upset about the cat. She hated cats, particularly ugly old alley
cats. "Elmer Elevator," she said to my father, "if you think I'm going
to give that cat a saucer of milk, you're very wrong."""
}
Semanticチャンキング
こちらもOpenSearchのダッシュボードのDev Toolsで内容を確認してみます。
"_source": {
"x-amz-bedrock-kb-source-uri": "s3://***/pg30017.txt",
"AMAZON_BEDROCK_METADATA": """{"source":"s3://***/pg30017.txt"}""",
"AMAZON_BEDROCK_METADATA"には同様にソースドキュメントの場所が格納されているようです。
こちらも冒頭幾つかのチャンクの内容を確認してみます（長いので畳みます）。
"_source": {
"AMAZON_BEDROCK_TEXT_CHUNK": """
The Project Gutenberg eBook of My Father's Dragon
This ebook is for the use of anyone anywhere in the United States and
most other parts of the world at no cost and with almost no restrictions
whatsoever. You may copy it, give it away or re-use it under the terms
of the Project Gutenberg License included with this ebook or online
at www.gutenberg.org. If you are not located in the United States,
you will have to check the laws of the country where you are located
before using this eBook.
Title: My Father's Dragon
Author: Ruth Stiles Gannett
Illustrator: Ruth Chrisman Gannett
Release date: September 18, 2009 [eBook #30017]
Language: English
Credits: Produced by Sankar Viswanathan, Greg Weeks, and the Online
Distributed Proofreading Team at http://www.pgdp.net
*** START OF THE PROJECT GUTENBERG EBOOK MY FATHER'S DRAGON ***
Produced by Sankar Viswanathan, Greg Weeks, and the Online
Distributed Proofreading Team at http://www.pgdp.net
Transcriber's Note:
Extensive research did not uncover any evidence that the
U.S. copyright on this publication was renewed."""
}
"_source": {
"AMAZON_BEDROCK_TEXT_CHUNK": "
MY FATHER'S DRAGON
STORY BY
RUTH STILES GANNETT
ILLUSTRATIONS BY
RUTH CHRISMAN GANNETT
RANDOM HOUSE . NEW YORK
COPYRIGHT 1948 BY RANDOM HOUSE, INC.
* * * * *
For My
FATHER
* * * * *
CONTENTS
1. My Father Meets the Cat 9
2. My Father Runs Away 15
3. My Father Finds the Island 22
4. My Father Finds the River 31
5. My Father Meets Some Tigers 39
6. My Father Meets A Rhinoceros 48
7. My Father Meets A Lion 56
8. My Father Meets A Gorilla 63
9. My Father Makes A Bridge 73
10. My Father Finds the Dragon 79
* * * * *
_Chapter One_
MY FATHER MEETS THE CAT
One cold rainy day when my father was a little boy, he met an old
alley cat on his street. The cat was very drippy and uncomfortable so
my father said, \"Wouldn't you like to come home with me?\""
}
"_source": {
"AMAZON_BEDROCK_TEXT_CHUNK": """
This surprised the cat--she had never before met anyone who cared
about old alley cats--but she said, "I'd be very much obliged if I
could sit by a warm furnace, and perhaps have a saucer of milk."
"We have a very nice furnace to sit by," said my father, "and I'm sure
my mother has an extra saucer of milk."
[Illustration]
My father and the cat became good friends but my father's mother was
very upset about the cat. She hated cats, particularly ugly old alley
cats. "Elmer Elevator," she said to my father, "if you think I'm going
to give that cat a saucer of milk, you're very wrong. Once you start
feeding stray alley cats you might as well expect to feed every stray
in town, and I am _not_ going to do it!"
This made my father very sad, and he apologized to the cat because his
mother had been so rude. He told the cat to stay anyway, and that
somehow he would bring her a saucer of milk each day. My father fed
the cat for three weeks, but one day his mother found the cat's saucer
in the cellar and she was extremely angry. She whipped my father and
threw the cat out the door, but later on my father sneaked out and
found the cat."""
}
これが良い感じでチャンクしてくれたのかどうかは何とも言えないですが、冒頭部分についてはデフォルトチャンキングよりも大きく切られていました。但し全体でみると、チャンク数はデフォルト：67チャンク、Semantic：64チャンクとさほど違いはありませんでした。
Hierarchicalチャンキング
一番確認したかったHierarchicalチャンキングを確認します。
同様にOpenSearchのダッシュボードのDev Toolsで内容を確認したところ、デフォルトチャンキングやSemanticチャンキングと異なり、
"AMAZON_BEDROCK_METADATA"にはソースドキュメントの場所だけでなく、"parentText"にドキュメント自体が格納されています。
抜粋
"_source": {
"x-amz-bedrock-kb-source-uri": "s3://***/pg30017.txt",
"AMAZON_BEDROCK_METADATA": """{"source":"s3://***/pg30017.txt",
"parentText":"
The Project Gutenberg eBook of My Father's Dragon
...
全体は長いので畳みます
"_source": {
"x-amz-bedrock-kb-source-uri": "s3://***/pg30017.txt",
"AMAZON_BEDROCK_METADATA": """{"source":"s3://***/pg30017.txt",
"parentText":"
The Project Gutenberg eBook of My Father's Dragon
This ebook is for the use of anyone anywhere in the United States and
most other parts of the world at no cost and with almost no restrictions
whatsoever. You may copy it, give it away or re-use it under the terms
of the Project Gutenberg License included with this ebook or online
at www.gutenberg.org. If you are not located in the United States,
you will have to check the laws of the country where you are located
before using this eBook.
Title: My Father's Dragon
Author: Ruth Stiles Gannett
Illustrator: Ruth Chrisman Gannett
Release date: September 18, 2009 [eBook #30017]
Language: English
Credits: Produced by Sankar Viswanathan, Greg Weeks, and the Online
Distributed Proofreading Team at http://www.pgdp.net
*** START OF THE PROJECT GUTENBERG EBOOK MY FATHER'S DRAGON ***
Produced by Sankar Viswanathan, Greg Weeks, and the Online
Distributed Proofreading Team at http://www.pgdp.net
Transcriber's Note:
Extensive research did not uncover any evidence that the
U.S. copyright on this publication was renewed.
MY FATHER'S DRAGON
STORY BY
RUTH STILES GANNETT
ILLUSTRATIONS BY
RUTH CHRISMAN GANNETT
RANDOM HOUSE . NEW YORK
COPYRIGHT 1948 BY RANDOM HOUSE, INC.
* * * * *
For My
FATHER
* * * * *
CONTENTS
1. My Father Meets the Cat 9
2. My Father Runs Away 15
3. My Father Finds the Island 22
4. My Father Finds the River 31
5. My Father Meets Some Tigers 39
6. My Father Meets A Rhinoceros 48
7. My Father Meets A Lion 56
8. My Father Meets A Gorilla 63
9. My Father Makes A Bridge 73
10. My Father Finds the Dragon 79
* * * * *
_Chapter One_
MY FATHER MEETS THE CAT
One cold rainy day when my father was a little boy, he met an old
alley cat on his street. The cat was very drippy and uncomfortable so
my father said, \"Wouldn't you like to come home with me?\"
This surprised the cat--she had never before met anyone who cared
about old alley cats--but she said, \"I'd be very much obliged if I
could sit by a warm furnace, and perhaps have a saucer of milk.\"
\"We have a very nice furnace to sit by,\" said my father, \"and I'm sure
my mother has an extra saucer of milk.\"
[Illustration]
My father and the cat became good friends but my father's mother was
very upset about the cat. She hated cats, particularly ugly old alley
cats. \"Elmer Elevator,\" she said to my father, \"if you think I'm going
to give that cat a saucer of milk, you're very wrong. Once you start
feeding stray alley cats you might as well expect to feed every stray
in town, and I am _not_ going to do it!\"
This made my father very sad, and he apologized to the cat because his
mother had been so rude. He told the cat to stay anyway, and that
somehow he would bring her a saucer of milk each day. My father fed
the cat for three weeks, but one day his mother found the cat's saucer
in the cellar and she was extremely angry. She whipped my father and
threw the cat out the door, but later on my father sneaked out and
found the cat. Together they went for a walk in the park and tried to
think of nice things to talk about. My father said, \"When I grow up
I'm going to have an airplane. Wouldn't it be wonderful to fly just
anywhere you might think of!\"
\"Would you like to fly very, very much?\" asked the cat.
\"I certainly would. I'd do anything if I could fly.\"
[Illustration]
\"Well,\" said the cat, \"If you'd really like to fly that much, I think
I know of a sort of a way you might get to fly while you're still a
little boy.\"
\"You mean you know where I could get an airplane?\"
\"Well, not exactly an airplane, but something even better. As you can
see, I'm an old cat now, but in my younger days I was quite a
traveler. My traveling days are over but last spring I took just one
more trip and sailed to the Island of Tangerina, stopping at the port
of Cranberry. Well, it just so happened that I missed the boat, and
while waiting for the next I thought I'd look around a bit. I was
particularly interested in a place called Wild Island, which we had
passed on our way to Tangerina. Wild Island and Tangerina are joined
together by a long string of rocks, but people never go to Wild Island
because it's mostly jungle and inhabited by very wild animals. So, I
decided to go across the rocks and explore it for myself. It certainly
is an interesting place, but I saw something there that made me want
to weep.\"
[Illustration]
[Illustration]
_Chapter Two_
MY FATHER RUNS AWAY
\"Wild Island is practically cut in two by a very wide and muddy
river,\" continued the cat. \"This river begins near one end of the
island and flows into the ocean at the other. Now the animals there
are very lazy, and they used to hate having to go all the way around
the beginning of this river to get to the other side of the island.
It made visiting inconvenient and mail deliveries slow, particularly
during the Christmas rush. Crocodiles could have carried passengers
and mail across the river, but crocodiles are very moody, and not the
least bit dependable, and are always looking for something to eat.
They don't care if the animals have to walk around the river, so
that's just what the animals did for many years.\"
\"But what does all this have to do with airplanes?\" asked my father,
who thought the cat was taking an awfully long time to explain.
\"Be patient, Elmer,\" said the cat, and she went on with the story.
\"One day about four months before I arrived on Wild Island a baby
dragon fell from a low-flying cloud onto the bank of the river. He was
too young to fly very well, and besides, he had bruised one wing quite
badly, so he couldn't get back to his cloud. "}""",
今までのチャンクと比べるとかなり長く、設定したMax parent token size（デフォルトでは1500トークン）まで、
"AMAZON_BEDROCK_METADATA"の"parentText"に格納されているようです。
ベクトル検索に使用するチャンク自体は、
"AMAZON_BEDROCK_TEXT_CHUNK"に格納されています。
長いので畳みます
"AMAZON_BEDROCK_TEXT_CHUNK": """
The Project Gutenberg eBook of My Father's Dragon
This ebook is for the use of anyone anywhere in the United States and
most other parts of the world at no cost and with almost no restrictions
whatsoever. You may copy it, give it away or re-use it under the terms
of the Project Gutenberg License included with this ebook or online
at www.gutenberg.org. If you are not located in the United States,
you will have to check the laws of the country where you are located
before using this eBook.
Title: My Father's Dragon
Author: Ruth Stiles Gannett
Illustrator: Ruth Chrisman Gannett
Release date: September 18, 2009 [eBook #30017]
Language: English
Credits: Produced by Sankar Viswanathan, Greg Weeks, and the Online
Distributed Proofreading Team at http://www.pgdp.net
*** START OF THE PROJECT GUTENBERG EBOOK MY FATHER'S DRAGON ***
Produced by Sankar Viswanathan, Greg Weeks, and the Online
Distributed Proofreading Team at http://www.pgdp.net
Transcriber's Note:
Extensive research did not uncover any evidence that the
U.S. copyright on this publication was renewed.
MY FATHER'S DRAGON
STORY BY
RUTH STILES GANNETT
ILLUSTRATIONS BY
RUTH CHRISMAN GANNETT
RANDOM HOUSE . NEW YORK
COPYRIGHT 1948 BY RANDOM HOUSE, INC.
* * * * *
For My
FATHER
* * * * *
CONTENTS
1. My Father Meets the Cat 9
2. My Father Runs Away 15
3. My Father Finds the Island 22
4."""
}
"parentText"よりはかなり短く、子チャンク（デフォルト300トークン）の長さで格納されているようです。
次のチャンクも確認してみます。
"AMAZON_BEDROCK_METADATA"は一つ目のチャンクと同じ内容が格納されています。
抜粋
"_source": {
"x-amz-bedrock-kb-source-uri": "s3://***/pg30017.txt",
"AMAZON_BEDROCK_METADATA": """{"source":"s3://***/pg30017.txt",
"parentText":"
The Project Gutenberg eBook of My Father's Dragon
...
全体は長いので畳みます
"_source": {
"x-amz-bedrock-kb-source-uri": "s3://***/pg30017.txt",
"AMAZON_BEDROCK_METADATA": """{"source":"s3://***/pg30017.txt",
"parentText":"
The Project Gutenberg eBook of My Father's Dragon
This ebook is for the use of anyone anywhere in the United States and
most other parts of the world at no cost and with almost no restrictions
whatsoever. You may copy it, give it away or re-use it under the terms
of the Project Gutenberg License included with this ebook or online
at www.gutenberg.org. If you are not located in the United States,
you will have to check the laws of the country where you are located
before using this eBook.
Title: My Father's Dragon
Author: Ruth Stiles Gannett
Illustrator: Ruth Chrisman Gannett
Release date: September 18, 2009 [eBook #30017]
Language: English
Credits: Produced by Sankar Viswanathan, Greg Weeks, and the Online
Distributed Proofreading Team at http://www.pgdp.net
*** START OF THE PROJECT GUTENBERG EBOOK MY FATHER'S DRAGON ***
Produced by Sankar Viswanathan, Greg Weeks, and the Online
Distributed Proofreading Team at http://www.pgdp.net
Transcriber's Note:
Extensive research did not uncover any evidence that the
U.S. copyright on this publication was renewed.
MY FATHER'S DRAGON
STORY BY
RUTH STILES GANNETT
ILLUSTRATIONS BY
RUTH CHRISMAN GANNETT
RANDOM HOUSE . NEW YORK
COPYRIGHT 1948 BY RANDOM HOUSE, INC.
* * * * *
For My
FATHER
* * * * *
CONTENTS
1. My Father Meets the Cat 9
2. My Father Runs Away 15
3. My Father Finds the Island 22
4. My Father Finds the River 31
5. My Father Meets Some Tigers 39
6. My Father Meets A Rhinoceros 48
7. My Father Meets A Lion 56
8. My Father Meets A Gorilla 63
9. My Father Makes A Bridge 73
10. My Father Finds the Dragon 79
* * * * *
_Chapter One_
MY FATHER MEETS THE CAT
One cold rainy day when my father was a little boy, he met an old
alley cat on his street. The cat was very drippy and uncomfortable so
my father said, \"Wouldn't you like to come home with me?\"
This surprised the cat--she had never before met anyone who cared
about old alley cats--but she said, \"I'd be very much obliged if I
could sit by a warm furnace, and perhaps have a saucer of milk.\"
\"We have a very nice furnace to sit by,\" said my father, \"and I'm sure
my mother has an extra saucer of milk.\"
[Illustration]
My father and the cat became good friends but my father's mother was
very upset about the cat. She hated cats, particularly ugly old alley
cats. \"Elmer Elevator,\" she said to my father, \"if you think I'm going
to give that cat a saucer of milk, you're very wrong. Once you start
feeding stray alley cats you might as well expect to feed every stray
in town, and I am _not_ going to do it!\"
This made my father very sad, and he apologized to the cat because his
mother had been so rude. He told the cat to stay anyway, and that
somehow he would bring her a saucer of milk each day. My father fed
the cat for three weeks, but one day his mother found the cat's saucer
in the cellar and she was extremely angry. She whipped my father and
threw the cat out the door, but later on my father sneaked out and
found the cat. Together they went for a walk in the park and tried to
think of nice things to talk about. My father said, \"When I grow up
I'm going to have an airplane. Wouldn't it be wonderful to fly just
anywhere you might think of!\"
\"Would you like to fly very, very much?\" asked the cat.
\"I certainly would. I'd do anything if I could fly.\"
[Illustration]
\"Well,\" said the cat, \"If you'd really like to fly that much, I think
I know of a sort of a way you might get to fly while you're still a
little boy.\"
\"You mean you know where I could get an airplane?\"
\"Well, not exactly an airplane, but something even better. As you can
see, I'm an old cat now, but in my younger days I was quite a
traveler. My traveling days are over but last spring I took just one
more trip and sailed to the Island of Tangerina, stopping at the port
of Cranberry. Well, it just so happened that I missed the boat, and
while waiting for the next I thought I'd look around a bit. I was
particularly interested in a place called Wild Island, which we had
passed on our way to Tangerina. Wild Island and Tangerina are joined
together by a long string of rocks, but people never go to Wild Island
because it's mostly jungle and inhabited by very wild animals. So, I
decided to go across the rocks and explore it for myself. It certainly
is an interesting place, but I saw something there that made me want
to weep.\"
[Illustration]
[Illustration]
_Chapter Two_
MY FATHER RUNS AWAY
\"Wild Island is practically cut in two by a very wide and muddy
river,\" continued the cat. \"This river begins near one end of the
island and flows into the ocean at the other. Now the animals there
are very lazy, and they used to hate having to go all the way around
the beginning of this river to get to the other side of the island.
It made visiting inconvenient and mail deliveries slow, particularly
during the Christmas rush. Crocodiles could have carried passengers
and mail across the river, but crocodiles are very moody, and not the
least bit dependable, and are always looking for something to eat.
They don't care if the animals have to walk around the river, so
that's just what the animals did for many years.\"
\"But what does all this have to do with airplanes?\" asked my father,
who thought the cat was taking an awfully long time to explain.
\"Be patient, Elmer,\" said the cat, and she went on with the story.
\"One day about four months before I arrived on Wild Island a baby
dragon fell from a low-flying cloud onto the bank of the river. He was
too young to fly very well, and besides, he had bruised one wing quite
badly, so he couldn't get back to his cloud. "}""",
"AMAZON_BEDROCK_TEXT_CHUNK"も確認してみます。
長いので畳みます
"AMAZON_BEDROCK_TEXT_CHUNK": """
My Father Runs Away 15
3. My Father Finds the Island 22
4. My Father Finds the River 31
5. My Father Meets Some Tigers 39
6. My Father Meets A Rhinoceros 48
7. My Father Meets A Lion 56
8. My Father Meets A Gorilla 63
9. My Father Makes A Bridge 73
10. My Father Finds the Dragon 79
* * * * *
_Chapter One_
MY FATHER MEETS THE CAT
One cold rainy day when my father was a little boy, he met an old
alley cat on his street. The cat was very drippy and uncomfortable so
my father said, "Wouldn't you like to come home with me?"
This surprised the cat--she had never before met anyone who cared
about old alley cats--but she said, "I'd be very much obliged if I
could sit by a warm furnace, and perhaps have a saucer of milk."
"We have a very nice furnace to sit by," said my father, "and I'm sure
my mother has an extra saucer of milk."
[Illustration]
My father and the cat became good friends but my father's mother was
very upset about the cat. She hated cats, particularly ugly old alley
cats. "Elmer Elevator," she said to my father, "if you think I'm going
to give that cat a saucer of milk, you're very wrong."""
}
一つ目の"AMAZON_BEDROCK_TEXT_CHUNK"の続きが格納されています。
まとめ
チャンキング戦略によって、（そりゃそうですが）チャンキングのされ方が変わる事が確認できました。
特にHierarchicalチャンキングについては、ベクトル検索に使用する子チャンクと検索結果として使用する親チャンクでチャンクの長さを変える事が出来るので、かなり便利に使えそうです。
今回はテキストファイルを使用した関係か日本語ではちょっと微妙で、英語のテキストを使ったのですが、日本語PDFを使って良い感じにした記事は以下