Dashboards Are Not Governance (Yet)

Tracking how developers use AI tools is observability. Governing what AI tools actually do requires a different set of capabilities entirely.

Share this article

Dashboards Are Not Governance (Yet)

Something encouraging is happening in the AI tooling space. Teams building with AI coding assistants are starting to ask governance questions: who is using these tools, how are they being used, where are tokens going, and are sessions actually productive?

These are good questions. The tools emerging to answer them, offering session tracking, token attribution, usage analytics, and quality scoring, represent a genuine step forward. Any organisation deploying AI at scale needs this visibility.

The opportunity now is to go further.

Observability is the foundation. Governance is the building.

What most of these tools provide is observability: insight into how humans interact with AI tools. That's valuable and necessary. Governance, though, encompasses a broader set of questions. It asks what values the AI operates under, what boundaries exist on its behaviour, what happens when it produces harmful output, and who has the authority to define and review those constraints.

Observability tells you where the tokens went. Governance tells you where the AI shouldn't go.

Both matter. The risk is treating one as a substitute for the other. An organisation with excellent usage dashboards and no policy framework for AI behaviour has strong visibility and weak governance. The dashboards are a starting point, and the industry should build on them.

What the next layer looks like

Mature AI governance adds several capabilities on top of observability:

Constitutional frameworks: explicit, auditable principles that define what the AI should and shouldn't do in a given organisational context.

Policy enforcement: boundaries applied to AI behaviour directly, evaluated at decision time rather than reviewed after the fact.

AI decision audit trails: records of what the AI decided and why, complementing the existing records of what the human did.

Review mechanisms: structured processes for when AI output touches sensitive domains.

Bilateral accountability: governance that flows in both directions, because the AI's behaviour is as much a governance surface as the human's usage patterns.

That last point is where the field has the most room to grow. Current tooling monitors the human side of the interaction thoroughly. The AI side, what it decided, what principles it followed, whether it flagged concerns, remains largely unexamined. Closing that gap is the next frontier.

A note on gamification

Some tools are adding competitive layers: achievements, leaderboards, XP systems. The intention is sound. Engagement with governance tooling is genuinely hard to drive, and gamification is a proven engagement mechanism.

The thing to watch is Goodhart's Law. When metrics become targets, they cease to be good metrics. Developers optimising for visible throughput may deprioritise the reflective, exploratory work that often produces the best outcomes. The most valuable sessions sometimes involve closing the tool and thinking. Gamification works best when it rewards quality of judgment, and that's a harder thing to score than volume of activity.

Where this is heading

The encouraging signal is that the market recognises governance as a need. Organisations are moving beyond "let everyone experiment" toward "let's understand and shape how AI tools operate across our teams." That trajectory is right.

The work ahead is extending governance from observing human behaviour to governing AI behaviour directly: constitutional principles, policy enforcement, decision auditability, and frameworks where accountability runs both ways. The teams already building observability tooling are well positioned to take on that challenge. The foundations they're laying will matter.

The conversation is moving in the right direction. Let's keep pushing it further.

Nell Watson
Founder, Creed Space

AI ethics researcher and IEEE Fellow. Author of Taming the Machine.