Without wishing to jump on the bandwagon, like the rest of the IT world it has to be the large scale adoption of generative AI in all parts of our industry, and even the whole online world. The proliferation in the security environment of AI products is impacting everything from the development and deployment cycle to the identification and remediation of vulnerabilities.

Organisations are rushing to roll out some sort of AI tooling or products like chat bots without always realising or adequately assessing the potential risk exposure. Other orgs are using AI to build software products they’d never have the capability of releasing in the past because AI “can build it for them”. As a result, there are AI chat bots with direct access to sensitive internal documentation or processes without proper guard rails. Other products are deployed which are inherently insecure because the AI that “built it” wasn’t guided by an engineer or developer who knew the appropriate prompts and development lifecycle to reduce or prevent vulnerabilities. Even huge industry leaders are getting caught out by this. See the example of Meta recently where their AI support bot was used to take over thousands of accounts because it had insufficient guard rails and too much access to internal systems.

At the other end of the security spectrum are the attackers who can now rapidly deploy exploit systems they previously wouldn’t have the knowledge or infrastructure to use. When you can jump into an agentic coding interface and just say “build me some phishing infrastructure”, or “build a tool to scan the web for and exploit this vulnerability” the barrier to entry becomes pretty low. The fact these exploit systems are low skill and the attacker may be caught in 12 months doesn’t help the organisations who are being exploited today.

And somewhere in the middle are the security teams, red, blue and purple, who have to rush to keep up with the fast pace of change and make some very difficult decisions about how to, or even whether they should integrate AI into their tooling. Like all other organisations and teams they need to decide, do we deploy this neat new AI security tool, not knowing how it will hallucinate, how it might expose our or our customers data inadvertently? Do we put AI in our process flow even though we know it produces invalid results some times?

For us at Red Cursor the hardest part of this is trying to take on the best benefits of AI without sacrificing our clients’ security and information privacy. Using 3rd party AI providers with any of our clients’ data is a non-starter as we have no way of knowing where that data will go. No way of ensuring it doesn’t come out in some hallucination and expose itself to the world. So, we have to build up in-house capabilities with locally deployed models and hardware, keep up with the rapid pace of change in AI tech, all the while also keeping up with the now even more rapid vulnerability landscape.