Intellics Labs

A Kabuken (opens in a new tab) experiment by Intellics K.K. (opens in a new tab)

Do simulated investors read a filing the way their own rules say they should?

We gave five Japanese annual securities reports to simulated investor personas, each with a written investment rule, and checked three things. Do they cite the filing's facts exactly? Do they act differently from one another, in line with their own rules? And when we change a key fact, do they move in the direction their rule implies?

These personas are simulated. They are not real investors, and nothing here is a forecast or investment advice.

Method

• Filings. Five annual securities reports (有価証券報告書) of large listed Japanese companies, shown here as Company A to Company E.
• Evidence. For each filing we built a fixed evidence pack from Kabuken's public MCP server, which serves data taken from EDINET. No price data, no news.
• Personas. Twenty personas, each with a short written rule (for example a passive index holder, a dividend-focused holder, an active manager). Each had to answer with one action (hold, add, trim, exit or no reaction) and list every fact it relied on, citing the exact source row.
• Changed facts. In a test version of each pack we changed key facts, strongly and consistently (for example a large fall in revenue, a loss, lower equity and a deep dividend cut; for one company, a return to profit instead). Company names were removed from these test packs.
• A test fixed in advance. Before the runs we wrote down which personas should move, and in which direction, if they follow their own rules, and how many had to do so for the result to count. We did not change the rule after seeing results.
• Models. We ran the first experiment on Claude Haiku 4.5. We then ran the second experiment on Claude Haiku 4.5 and, on the same 400 persona-and-condition cells, on Claude Sonnet. A small 20-cell pilot also tried Gemma 4 26B. Both Claude models were run with extended thinking on.

Example: personas and a changed fact

Three of the twenty personas and one kind of edit, to show how the test works. All personas are simulated. The rules below are short paraphrases, not the full persona files.

One worked change. One of the edits we used was a deep dividend cut. On one pack, the facts looked like this:

• Before: the dividend per share as filed.
• After: the dividend per share cut deeply, with the other key facts moved consistently with it.

Shape of an answer (shape only, values left blank):

Action:      one of hold, add, trim, exit, no reaction
Cited fact:  concept = ...  | context = ...  | document = ...  | value = ...
Reason:      one or two sentences that use only the cited facts

The test checks whether the personas whose rule depends on the changed fact move the way their rule implies, and whether the passive personas stay put.

Findings

1. Haiku: the pre-fixed test was not met. With Claude Haiku 4.5, in the first experiment (small single-fact edits) the test was not met for any of the filings. In the second experiment (strong, consistent changes) fewer than half of the personas expected to move did so in the expected direction, and the verdict under our rule was "mixed".
2. Sonnet: the same test was met. With Claude Sonnet on the same 400 cells, under the same rule, the test was met: a clear majority of the expected movers moved in the direction their rule implies, none moved the opposite way, and personas expected to stay put stayed put.
3. Most of the gap is in how each model reads the unchanged filing. Under strong changes both models mostly chose to trim. They differed more in what they chose before any change was made. We do not read the difference as Sonnet "understanding" shocks better.
4. Citations resolved exactly, for both Claude models. Nearly every fact the personas cited matched its source row exactly. Claude Sonnet also cited almost every number it used in its reasoning; Claude Haiku left noticeably more numbers uncited.
5. Personas differ, and passive ones stay passive. Actions were spread across personas in line with their rules, and personas with a passive rule did not react to the changes.
6. Gemma 4 26B. On a small pilot it cited facts less reliably than the two Claude models, so we did not use it further.

Limits

• Simulated personas reading fixed data. The results say nothing about what real holders do.
• Not a forecast, a prediction, a ranking of companies, or investment advice. Kabuken has no stock prices.
• Five large companies only; results may not hold for smaller companies, for realistic news or for other models.
• The changes to the packs are strong, synthetic and internally consistent, which is easier than real filings.
• Anonymisation removed names and codes but not the size of the numbers, so a model may still recognise a very large company. We did not test this.
• In the second experiment each cell was repeated only twice, and Sonnet's repeats were nearly identical, so its effective sample is smaller than it looks.
• Sampling settings were not controlled; repeats are independent samples.
• Personas that already rest at "trim" can show a further negative reaction only by moving to "exit", which makes the test harder for them. A second rule for these personas, written down before the Sonnet run, gave only weak support.
• In the first experiment, one explanatory note field reached the personas in garbled form. We fixed it for later runs and did not re-run the first.
• The Gemma pilot was small, and one of its calls timed out.

What's next

We are exploring collaborative use: for example, teams reading the same filing and comparing their readings side by side. This is exploration only. There is no date and no promise, and nothing described here is a product feature today.

Attribution

Source: EDINET (Financial Services Agency, Japan), Public Data License 1.0. Processed by Kabuken.

Blog article: https://intellics.ai/blog/post/25 (opens in a new tab)

Kabuken: https://kabuken.intellics.ai/ (opens in a new tab)