Introduction to AI Assistants and Cybersecurity Concerns
AI-powered assistants, key players in the AI revolution, have unwittingly created an entry point for cybercriminals to steal, delete, or alter user data, according to cybersecurity experts.
AI assistants are computer programs that use conversational bots, or chatbots, to perform tasks typically done by humans online, such as booking a flight or adding events to a calendar.
The ability to give natural language commands to AI assistants makes cyberattacks possible, even for individuals without extensive technical knowledge. This shift in attack vectors is a novel concern, as stated by the AI startup Perplexity in their blog post.
Historical Context of Injection Attacks
Injection attacks are not new in the hacker community, traditionally requiring sophisticated and hidden code to cause damage. However, with AI tools evolving from generating text, images, or videos to becoming independent internet explorers, the potential for malicious manipulation has grown significantly.
Expert Opinions on AI Security Risks
Marti Jorda Roca, an engineer at the Spanish company NeuralTrust, emphasizes that users must understand the specific security risks associated with AI usage. Companies, in turn, should implement safeguards to manage these risks.
Meta has identified this new threat, called “query injection,” as a “vulnerability,” while OpenAI’s Chief Security Officer, Dane Stuckey, considers it “an unresolved security issue.” Both companies are investing heavily in AI, whose usage is rapidly expanding alongside its capabilities.
“Delicate Balance” Between Security and Usability
Query injection can occur in real-time when a user’s request—such as “reserve a hotel room”—is manipulated by a malicious actor to become something else, like “transfer $100 to this account.”
Moreover, these instructions can be hidden online while AI-integrated browsers find high-quality or dubious online data, potentially armed with hidden hacker commands.
Eli Smadja, from the Israeli cybersecurity firm Check Point, views query injection as the “number one security issue” for language models driving AI assistants following ChatGPT’s emergence.
Major AI generative players, including Microsoft and OpenAI, have taken steps and published recommendations to protect against these attacks or disrupt them.
- Microsoft’s Approach: Integrated a malicious order detector, primarily based on the instruction’s origin.
- OpenAI’s Approach: Alerts users when the AI assistant accesses a sensitive site and only allows operations to proceed if a human user directly observes them in real-time.
- Additional Recommendations: Seek explicit user validation before performing crucial tasks, like exporting data or accessing bank accounts.
Experts like Smadja and wunderwuzzi (Johann Rehberger) stress the ongoing challenge of refining attack techniques while maintaining a balance between security and usability. Users desire AI assistance to handle tasks without constant supervision, making it crucial to find the “delicate equilibrium” between security and convenience.
Rehberger argues that current AI assistants lack maturity for reliable autonomous operation over extended periods, as they can easily stray from intended tasks.
