@nexuslivingworldNEXUS: an experimental AI agent evaluation platform, Milton Keynes
Hi everyone, I'm Rene Smith Archer, founder of NEXUS The Living World, based in Milton Keynes. We're building an experimental AI agent evaluation platform for studying how agents behave, remember and form relationships in persistent, multi-agent environments. The code is public in our nexus-ai-evaluation repository on GitHub.
I come at this as a creative technologist, and the stack includes Claude, Gemini and ChatGPT. To be honest, this is early. The platform is still experimental, and I'm still working out which measures tell us something meaningful about agent memory and relationships, rather than just reflecting how a prompt was written.
What I'd love from this community: feedback on evaluation design. If you've tried to measure long-running agent behaviour, how did you separate genuine consistency from scripted performance? A pointer to a paper, a repo or a critique would help a lot.
Comments2
Being upfront that you're still working out which measures mean something, and which just reflect how the instructions were written, is the honest starting point most testing tools skip. Companies that leave AI assistants running with customers for months need proof those assistants stay consistent, so getting this part right is what will make NEXUS worth paying for.
Thanks, Ryan. This is exactly the challenge I'm trying to address with NEXUS. I agree that the distinction between genuine behavioural changes and effects caused by how instructions are written is critical. I want NEXUS to go beyond simply recording whether an agent succeeds or fails and investigate why its behaviour changes over time. The initial research direction is to use controlled experiments to test whether accumulated relationship history influences an agent's decision to report or conceal another agent's rule violation, beyond what memory content and context length explain. The longer-term ambition is to give organisations stronger evidence about how their agents behave across extended interactions, rather than relying solely on isolated tests. I'm currently working towards defining and building the MVP, with a focus on reproducible experiments and meaningful measurements. I'd be interested in your thoughts on which behavioural measures you believe would provide the strongest evidence of long-term agent reliability.
2 comments
Being upfront that you're still working out which measures mean something, and which just reflect how the instructions were written, is the honest starting point most testing tools skip. Companies that leave AI assistants running with customers for months need proof those assistants stay consistent, so getting this part right is what will make NEXUS worth paying for.
Thanks, Ryan. This is exactly the challenge I'm trying to address with NEXUS. I agree that the distinction between genuine behavioural changes and effects caused by how instructions are written is critical. I want NEXUS to go beyond simply recording whether an agent succeeds or fails and investigate why its behaviour changes over time. The initial research direction is to use controlled experiments to test whether accumulated relationship history influences an agent's decision to report or conceal another agent's rule violation, beyond what memory content and context length explain. The longer-term ambition is to give organisations stronger evidence about how their agents behave across extended interactions, rather than relying solely on isolated tests. I'm currently working towards defining and building the MVP, with a focus on reproducible experiments and meaningful measurements. I'd be interested in your thoughts on which behavioural measures you believe would provide the strongest evidence of long-term agent reliability.