What and why FinSkillBench evaluates
When comparing AI agents for investment management, it's easy to get carried away by the choice of language model: more parameters, newer architecture, higher scores on standard tests. But FinSkillBench offers a different perspective: how well an agent can actually use financial procedural skills — step-by-step instructions and executable components that turn knowledge into action.
The benchmark covers three key areas of investment work:
- portfolio construction
- risk management
- fundamental analysis
Within these areas, the authors assembled 12 subtasks and 2,603 episodes. Each episode is a task with data as of a specific point in time, a hidden correct answer, and a formulation the agent must turn into action. In other words, what's tested is not the model's memory, but its ability to apply the right procedure at the right moment.
Three agent operating modes
To understand what exactly a skill provides, the authors compared three variants of agent behavior.
- Without skills. The model solves the task with "bare" knowledge, without external hints on how to act.
- With curated skills. The agent receives a set of procedural documents and executable components selected by experts. This is effectively a ready-made action library: rules, formulas, algorithms.
- With self-generated skills. The agent writes its own procedures on the fly and tries to reuse them within the same episode.
The key question was whether well-prepared procedures compensate for model limitations — or whether a smart model can manage on its own.
The results were telling. Across nine models, adding curated skills consistently improved final scores: the average result rose from 0.366 to 0.528. The effect was strongest in portfolio construction and risk management — areas where sequence of actions and explicit rules matter most.
Self-generation barely helps
Intuition suggests: give the model the ability to create its own procedures — and it will adapt to the task better than with someone else's templates. The experiment did not confirm this. Self-generated skills yielded only marginal improvement while requiring noticeably more compute. The agent spent time creating and rewriting procedures, but the result barely changed.
Naive self-generation is like reinventing the wheel in every task: costly and with no quality guarantee. A ready-made, proven procedural skill turns out to be more reliable than a model's attempt to derive a method from scratch each time.
Reproducibility on another framework
Any benchmark can be "fitted" to a specific environment. So the authors additionally ran the evaluation on an independent agent framework, Hermes Agent: 8 models, 5,280 episodes, the same three areas. The overall pattern held — ready-made skills improve agent performance across all directions.
That said, the magnitude of the effect varied: it depended both on the subtask and on how the agent "pipeline" is structured. This is an important caveat: skills are not a magic button, but they do provide a systematic advantage.
Takeaway for those building investment agents
The main practical lesson of the study: in investment management, access to reliable procedural skills can be no less important than the choice of the base model. If an agent has a clear procedure and can follow it, it often outperforms a stronger model that acts blindly.
For product teams, this means not putting all effort solely into model selection. Attention should be paid to building a library of proven financial procedures: how to recalculate portfolio weights, how to account for a risk factor, how to analyze financial statements. The authors separately emphasize: procedures should be curated, not created by the agent on the fly.
A final detail: the authors released both the benchmark itself, the evaluation tools, the curated skill packages, and the full trajectories of agent actions. This makes it possible to reproduce results and develop your own agent systems without building everything from scratch.
