Safety is one of the top concerns with AI. Organizations have seen the incredible power the technology wields and the many use cases it can support – and they’re eager to begin leveraging it. But they’re also worried about the risks, from data leakage to cyberattacks and many other threats.
That’s why the National Institute of Standards and Technology’s (NIST) release of its Dioptra tool is so important. The introduction of the tool marks a significant milestone in advancing the security and resilience of machine learning (ML) models. As cyber threats become increasingly sophisticated, Dioptra provides a helpful framework for addressing vulnerabilities such as evasion, poisoning, and oracle attacks. These attacks pose distinct risks, from manipulating input data to degrading model performance and uncovering sensitive information.
By equipping developers with the means to simulate these scenarios and test various defenses, Dioptra enhances the robustness of AI systems. The ultimate beneficiaries are the companies looking to employ these systems. Many organizations, especially larger ones, are still stuck in the evaluation phase when it comes to AI adoption, and safety considerations are a major reason why. As those worries are alleviated, they can ramp up deployments into full production and start to drive new business value.
Supporting AI safety initiatives
Created to meet goals laid out in President Bident’s Executive Order on AI safety, Dioptra was built to assist organizations in evaluating the strength, security, and trustworthiness of ML models. The tool is part of NIST’s broader efforts to improve the understanding and mitigation of risks associated with the deployment of AI and ML systems.
Dioptra provides a platform for conducting a variety of tests on ML models that can support:
- Adversarial Robustness Testing: Assesses how ML models perform when subjected to adversarial inputs, which are intentionally crafted to deceive the model.
- Performance Evaluation: Measures how well ML models generalize to new data, particularly in the presence of noise or perturbations.
- Fairness Testing: Analyzes ML models to ensure they do not exhibit bias or unfair treatment of certain groups based on attributes like race, gender, or age.
- Explainability: Provides insights into how ML models make decisions, helping users understand the reasoning behind specific predictions.
Flexibility is central to Dioptra. The tool is designed to be highly extensible, allowing researchers and developers to integrate new types of tests and evaluations as the field of AI security evolves.
The open-source nature of Dioptra is also commendable, as it fosters collaboration within the AI community. AI is a fast-changing and growing field. Dioptra being an open-source and agile platform will allow it to keep pace, especially if it gains popularity in the AI research community. By making the tool available on GitHub, NIST encourages a collective effort to improve AI security.
Next steps for safer AI
Looking ahead, I hope that we will see platforms such as Dioptra provide targeted features across more specialized AI subfields – especially in generative AI, where safety is already a paramount concern. While the US federal government hasn’t introduced any major AI regulations, states are moving forward with their own. For example, California’s SB-1047 introduces major requirements for AI developers to safeguard their models.
To meet the growing regulatory requirements of AI, I expect companies to additionally adopt real-time protections around AI models. For example, real-time protection of genAI systems can be met using LLM Firewalls, a new breed of inline, natural-language systems designed to inspect and protect against attacks present in LLM prompts, retrievals, and responses, mitigating the significant threats from data exposure and prompt injections, as well as acting as a guardrail against risks from prohibited topics and harmful content.
Initiatives like Dioptra are vital in ensuring AI technologies are developed and used ethically, reinforcing the commitment to safeguarding AI systems while promoting innovation. AI Governance is an increasingly important topic for enterprises leveraging AI. Dioptra and other tools that can support AI Governance efforts will allow organizations to deploy AI with more confidence and begin reaping the rewards while mitigating the biggest risks.
