はじめに
ポエムです. 疑問に思っただけです.
ハルシネーションと言ってみると詳しい気になれるかもしれない？しかし, 多くの場合は指示が悪いだけだと思う. 曖昧な指示をされてどうしろと言うのか. その仕様書で貴方はアプリをまともに作れる？
Coding Agentに思考させない
私は, Coding Agentは指示通りに作業をするだけ, を目指している.
次は, あるアプリの改良作業の指示書だ.
まず, 無理のない分割でロードマップを作成する. Phase7ぐらいまである. "1 Phaseに数十時間かかる"ということはない. 思考のループに陥ることは, 今までなかった.
長大だが, 英語なのでトークン効率はいい, LLMの思考を観察するとよいが, コーティングに関しては, プログラミング言語に英語要素が強いので, thinkingは英語になっていることが多い. それでも, 英語でのthinkingに強く圧力をかけるために英語にしている. 理由は, LLMが最も得意なのが英語だからだ. アウトプットに日本語が欲しいなら最後に翻訳させればよい.
軽量のモデルではごくまれに作業を飛ばす, フロンティアだと余計なことをしがちではあるが, タグにある3つのエージェントでは, ほぼ同じ結果に到達すると思う.
1 Phaseが10分ほどで, トータルで1時間かかるが, 曖昧な指示で数十時間かけて, 残念な結果になるより良い. この改良ロードマップの前に, 効果計測の環境を作っているので, 効果が薄いPhaseを無視したり, 戻って再考したりもできる.
Semantic-Aware Review Grouping
# Phase 1 — Semantic-Aware Review Grouping
## Objective
Improve review context quality by replacing the current filename-oriented grouping with a lightweight **semantic-aware grouping strategy**.
This phase must change **group construction only**.
Do not change:
```text
LLM prompts
thinking configuration
Turn 1 behavior
Turn 2 behavior
candidate schema
review formatting
model configuration
number of LLM turns
The purpose of this phase is to determine whether better grouping alone improves detection quality for thinking off.
Background
The current review pipeline already performs adequately with thinking off, but some issues are detected inconsistently.
A likely cause is that related implementations are not always presented together.
Examples of useful relationships include:
Environment::sample
Environment::pdf
Environment::eval
and:
renderSphere
renderMesh
When related implementations are in the same review context, the model can more easily detect inconsistencies such as:
width vs height
row vs column
different argument order
different indexing convention
different use of the same API
The existing grouping based primarily on base filenames is useful for simple header/source pairing, but it does not capture enough semantic relationships.
Phase 1 should improve grouping without introducing a new review architecture.
Current Architecture
Preserve the existing review flow.
Conceptually:
GitLab MR
↓
Fetch changed files
↓
Build file groups
↓
Build Turn 1 prompt
↓
Turn 1
↓
Build Turn 2 prompt
↓
Turn 2
↓
Final review
Only this part should change:
Build file groups
Everything after group construction should continue working as before.
Target Architecture
Replace simple filename grouping with:
Changed files
↓
Semantic relation extraction
↓
Semantic group construction
↓
Existing review pipeline
The grouping should consider several signals.
Recommended priority:
1. Same class / struct
2. Header / source pair
3. Direct changed-symbol relationship
4. Shared direct callee or constructor
5. Same module / directory
6. Existing base-filename affinity
These signals should be used to place strongly related changed files or changed symbols into the same review group.
1. Preserve Existing Header / Source Pairing
Do not remove the useful behavior of the current grouping.
For example:
foo.h
foo.cpp
should still naturally belong to the same group.
Similarly:
include/render/image.h
src/render/image.cpp
should be grouped when the existing project structure or semantic information makes the relationship clear.
Do not regress simple header/source grouping while adding semantic behavior.
2. Prefer Same-Class Relationships
Changed methods belonging to the same class or struct should strongly prefer the same review group.
Example:
Environment::sample
Environment::pdf
Environment::eval
should belong to the same semantic group when practical.
Reason:
These methods frequently share assumptions such as:
coordinate conventions
sampling conventions
field meaning
width / height semantics
resource ownership
This is one of the strongest grouping signals.
3. Group Related Implementations
Functions implementing closely related behavior should be placed together when the relationship can be derived deterministically.
Example:
renderSphere
renderMesh
may be related because they:
call the same constructor
use the same rendering settings
perform equivalent pixel indexing
construct equivalent camera coordinates
The implementation does not need sophisticated code similarity in Phase 1.
Use semantic relationships already available from the AST or existing analysis infrastructure.
Good signals include:
same direct callee
same constructor call
same important referenced type
same containing class
same directly referenced field family
Avoid fuzzy or expensive similarity analysis in this phase.
4. Use Direct Changed-Symbol Relationships
If one changed symbol directly calls or references another changed symbol, they should strongly prefer the same group.
Examples:
A -> B
where both A and B are changed.
Or:
A reads Foo::bar_
B writes Foo::bar_
when the relation is already available from the existing semantic layer.
Prefer direct relationships over broad module proximity.
5. Use Shared Direct Callees as a Secondary Signal
Two changed functions that invoke the same important direct callee may benefit from being reviewed together.
Example:
renderSphere -> Sampler::Sampler
renderMesh -> Sampler::Sampler
This relationship may expose inconsistent argument construction.
However, do not group every function that happens to call a very common utility.
Avoid grouping based on generic functions such as:
std::move
std::min
std::max
logging helpers
basic allocation helpers
Use judgment or a simple filtering rule if required.
The grouping must remain useful rather than becoming a large connected component through common utility calls.
6. Use Module / Directory Affinity Only as a Weak Signal
Directory proximity may be used as a fallback.
Example:
src/render/image.cpp
src/render/image_loader.cpp
src/render/environment.cpp
However:
same directory
must not by itself force a very large group.
Treat directory/module affinity as weaker than:
same class
header/source
direct symbol relation
shared meaningful callee
7. Build Groups From Changed Code Only
Phase 1 grouping should primarily organize changed files / changed symbols.
Do not expand the review context with arbitrary unchanged dependency files.
This phase is about reorganizing the existing changed review context.
Do not yet implement:
AST-selected unchanged dependencies
caller expansion
callee body expansion
candidate-driven context retrieval
Those belong to later phases.
8. Avoid Giant Semantic Groups
Semantic grouping must not collapse an entire subsystem into one group.
Introduce a simple group budget.
Use existing token estimation utilities if available.
Possible limits may include:
maximum files per group
maximum changed symbols per group
maximum estimated tokens
Choose values consistent with the current review system and existing context limits.
Do not introduce arbitrary aggressive limits if the project already has relevant configuration.
When a group exceeds its budget, preserve the strongest semantic relationships first.
Suggested retention order:
1. same class
2. header/source pair
3. direct changed-symbol relation
4. shared meaningful callee
5. base filename
6. module/directory affinity
Weak relationships should be the first ones removed when splitting a group.
9. Prefer Deterministic Grouping
For the same set of changed files and semantic relationships, grouping should be stable across runs.
Do not use:
LLM-based grouping
randomized grouping
embedding similarity
non-deterministic clustering
Phase 1 grouping should be deterministic.
Stable grouping is necessary for benchmark comparison.
10. Keep Group Ordering Stable
Preserve deterministic ordering of:
groups
files within groups
changed symbols within groups
when practical.
Recommended ordering can be based on:
repository path
source location
existing change order
Do not allow hash-map iteration order or other unstable implementation details to affect prompt construction.
Benchmark runs must be comparable.
11. Do Not Change Prompt Content Beyond Group Composition
The existing prompt templates must remain unchanged.
Do not modify:
review1.en.md
review2.en.md
unless a tiny mechanical change is strictly required by the grouping implementation.
Do not add semantic hints such as:
These functions are related because...
Compare width and height...
Compare sibling implementations...
in this phase.
The experiment must measure whether context composition alone improves review quality.
12. Do Not Add a Second Detection Pass
Phase 1 must remain one primary detection pass.
Do not implement:
Recovery Detection
Second Turn 1
Additional candidate search
Unreported-region review
The review flow must still use the existing number of turns.
Recovery detection belongs to a later phase after semantic grouping has been benchmarked independently.
13. Do Not Change Candidate Limits
Keep the current Turn 1 output constraints unchanged.
In particular:
Maximum 10 candidates
confidence must be high or medium
no low-confidence candidates
must behave exactly as before.
Grouping changes must not modify candidate selection rules.
14. Reuse Existing AST / Semantic Infrastructure
Use the existing AST and semantic information where possible.
Do not create a second parallel parser or semantic model.
The expected source of grouping information should be existing structures representing concepts such as:
changed symbols
declarations / definitions
containing type
direct callers / callees
referenced fields
referenced types
header/source relationships
Use only relationships that are reliable enough for deterministic grouping.
If some desired relationship is not currently available, prefer a smaller Phase 1 implementation rather than building a large new analysis subsystem.
15. Suggested Grouping Model
A simple weighted relation graph is acceptable.
Conceptually:
ChangedSymbol A
ChangedSymbol B
│
└── relation weight
Example weights could follow the relative ordering:
same containing class very strong
header/source pair very strong
direct call/reference strong
shared meaningful callee medium
same base filename medium
same module/directory weak
The exact numeric weights are implementation details.
Do not over-engineer graph clustering.
A simple deterministic merging strategy is sufficient if it produces sensible groups.
16. File-Level vs Symbol-Level Grouping
Prefer building semantic relationships at the symbol level, but the final review pipeline may continue consuming file-oriented review contexts if that is how the existing architecture works.
For example:
Foo::Open in foo.cpp
Foo::Close in foo.cpp
Bar::UseFoo in bar.cpp
may establish that:
foo.cpp
bar.cpp
belong to the same review group.
Do not require a complete rewrite of the prompt builder into symbol-only contexts.
Adapt semantic information to the existing review context abstraction.
17. Avoid Duplicate File Inclusion
A file should not be duplicated unnecessarily across many groups.
If the current architecture requires each changed file to belong to exactly one group, preserve that constraint.
If controlled overlap is already supported, keep overlap minimal.
Do not create large token duplication merely to expose every possible relationship.
Prefer the strongest semantic group assignment.
18. Add Grouping Diagnostics
Because this is a benchmark-driven change, add lightweight diagnostics showing how groups were formed.
Use the existing logger.
For each review group, log something similar to:
Semantic group 0:
files:
src/renderer.cpp
include/renderer.h
changed symbols:
Environment::sample
Environment::pdf
Environment::eval
relations:
same containing type: Environment
Another example:
Semantic group 1:
changed symbols:
renderSphere
renderMesh
relations:
shared callee: Sampler::Sampler
Do not put these diagnostics into the LLM prompt.
They are only for benchmark/debugging.
Keep logs concise.
19. Extend Existing Benchmark Logging if Needed
If temporary benchmark logging is already present, record enough information to identify the grouping used for each run.
For example, add:
{
"groups": [
{
"group_id": "g0",
"files": [
"src/renderer.cpp"
],
"changed_symbols": [
"Environment::sample",
"Environment::pdf"
]
}
]
}
Do not add automatic TP/FP scoring.
The benchmark recorder should still only capture execution data.
20. Benchmark Against the Existing Baseline
After implementation, compare:
A. Existing grouping
thinking off
existing two-turn pipeline
B. Semantic grouping
thinking off
same two-turn pipeline
Everything except grouping should remain identical.
Use the same seeded-bug benchmark and the same model configuration.
Run multiple trials because detection behavior is not completely deterministic.
Record:
Turn 1 detected candidates
Turn 2 final findings
candidate count
false positives
per-bug detection frequency
Turn 1 latency
Turn 2 latency
total latency
input tokens
output tokens
group count
files per group
estimated tokens per group
21. Important Benchmark Question
The primary question for Phase 1 is:
Does better semantic grouping improve thinking-off detection
without materially increasing false positives or latency?
Do not attempt to answer:
Does a second detection pass help?
Does additional AST context help?
Does a different prompt help?
in this phase.
Those are separate experiments.
22. Success Criteria
Phase 1 is successful if:
-
The existing review architecture remains unchanged.
-
thinking offremains the review mode used for the benchmark. -
Related changed symbols are grouped more meaningfully than with base-filename grouping alone.
-
Header/source pairing does not regress.
-
Groups remain deterministic across identical runs.
-
Large connected groups are prevented or split using a simple budget.
-
Existing Turn 1 and Turn 2 prompts are unchanged.
-
No additional LLM calls are introduced.
-
No unchanged dependency expansion is introduced.
-
Benchmark logging clearly identifies resulting semantic groups.
-
Review quality does not regress materially.
-
Latency does not increase materially due only to grouping overhead.
23. Non-Goals
Do not implement any of the following:
second detection pass
recovery turn
verification turn
three-stage review pipeline
candidate-driven context expansion
unchanged caller/callee source expansion
full-file context redesign
prompt heuristic changes
new confidence rules
severity changes
automatic benchmark scoring
LLM-based semantic grouping
embedding-based similarity
repository-wide semantic clustering
These are explicitly outside Phase 1.
Suggested Implementation Order
Implement in this order:
1. Identify the current grouping implementation.
2. Preserve the existing base-filename/header-source behavior.
3. Expose reliable semantic relationships already available
from the existing AST / semantic layer.
4. Add same-class grouping.
5. Add direct changed-symbol relation grouping.
6. Add meaningful shared-callee grouping.
7. Add module/directory affinity as a weak fallback.
8. Add deterministic group budgeting / splitting.
9. Add grouping diagnostics.
10. Add grouping information to benchmark logs if useful.
11. Run the existing tests.
12. Run the same thinking-off benchmark against
the pre-Phase-1 baseline.
Do not proceed into recovery detection as part of this task.
Design Principle
The central idea of Phase 1 is:
Do not ask the LLM to search harder.
Put related changed code closer together so the existing
non-thinking review can make better comparisons.
The expected benefit should come from improved context organization, not from more reasoning, more prompts, or more LLM calls.
Keep this change small, deterministic, benchmarkable, and easy to roll back.
まとめ
ハルシネーションは, 今はほぼなく, 使う側が足りていないと思う.