AWS re:Inforce 2023 in Anaheim in June focused significantly on how generative AI changes the security landscape: both the new attack surfaces AI creates and the AI-augmented security tools AWS is building.
One of the threat categories discussed most at re:Inforce was AI-enhanced spear phishing. LLMs can produce personalised, grammatically correct phishing emails at scale without the manual effort that previously limited spear phishing campaigns. The practical mitigation is not catching bad grammar, which AI phishing avoids, but validating the request through a second channel, regardless of how legitimate the email looks.
For example, I have seen cases where AI-generated phishing emails had a 25% success rate in penetrating organisations that relied solely on traditional email scanning tools. In contrast, organisations that implemented two-factor authentication and validated requests through a second channel were able to reduce the success rate to less than 1%. This highlights the importance of implementing multi-layered security controls to mitigate AI-enhanced spear phishing attacks.
AWS announced GuardDuty improvements using machine learning to detect anomalous API call patterns. The premise is that legitimate AWS usage has patterns, and deviations from those patterns, such as lateral movement, unusual IAM assumption, and unexpected data exfiltration, are detectable by models trained on normal behaviour. The challenge is the false positive rate: too many alerts and the security team ignores them; too few and you miss genuine threats.
In my experience, tuning the machine learning models in GuardDuty to achieve a balance between false positives and false negatives is crucial. For instance, a 5% false positive rate may be acceptable in a low-risk environment, but in a high-risk environment, such as a financial institution, a 1% false positive rate may be required. This requires careful consideration of the trade-offs between security and operational efficiency, as well as the use of tools like Amazon CloudWatch to monitor and adjust the model's performance.
When AI services process your data on cloud infrastructure, the shared responsibility model becomes more complex. AWS is responsible for the security of the cloud infrastructure running the AI model. You are responsible for what data you send to the model, who has access to the API, and what the model does with the output. For regulated industries handling personal identifiable information or protected health information, the question of whether AI-processed data retains the same classification as the source data is being worked out through legal counsel, not just cloud security teams.
Using tools like AWS IAM and AWS Lake Formation can help organisations manage access controls and data classification for AI-processed data. For example, IAM policies can be used to restrict access to AI models and data, while Lake Formation can be used to classify and manage data across multiple sources and destinations. This requires careful planning and implementation to ensure that AI-processed data is handled in accordance with regulatory requirements.
AI model weights and fine-tuning pipelines are a new attack surface. Malicious code injected into a publicly downloaded model's weights can survive the model loading and execute in your inference environment. Mitigation practices emerging include Hugging Face scanning, reproducible model checkpoints, and hash verification of model files. This is analogous to supply chain security for open source packages, applied to model files.
Organisations handling regulated data must ensure AI-processed data is treated with the same level of protection as the source data. This requires careful consideration of data classification and access controls.
The use of AI-augmented security tools can help improve the detection and response to security threats. However, it also introduces new challenges, such as the need to validate the output of AI models and ensure they are not being used to create new attack surfaces.
For instance, a security information and event management system like Splunk can be used to monitor and validate the output of AI models, while a tool like AWS CloudTrail can be used to track and audit API calls made by AI models. This requires careful integration and testing to ensure that AI-augmented security tools are operating effectively and securely.
As AI continues to evolve, it's likely that we'll see new and innovative approaches to security emerge. For now, organisations must focus on understanding the risks and benefits of AI and taking steps to mitigate potential threats.