CAMLIS 2026
Agenda
Wednesday, October 21
Add to Calendar
Registration
Check in, pick up your badge, and connect with the CAMLIS community before the main program begins.
Add to Calendar
Conference Opening & Welcome
Conference Co-Chair Welcome & Keynote Introduction
Add to Calendar
AI Security: A 10+-year Perspective
Presenter
Olsen Bicentennial Professor of Engineering and Professor of Computer Science
University of Virginia
Add to Calendar
Break
Add to Calendar
CTI-REALM: A Benchmark to Evaluate Agent Performance on Security Detection Rule Generation Capabilities
CTI-REALM (Cyber Threat Real World Evaluation and LLM Benchmarking) is a benchmark designed to evaluate AI agents’ ability to interpret cyber threat intelligence (CTI) and develop detection rules. The benchmark provides a realistic environment that replicates the security analyst workflow. This enables agents to examine CTI reports, execute queries, understand schema structures, and construct detection rules. Evaluation involves emulated attacks of varying complexity across Linux systems, cloud platforms, and Azure Kubernetes Service (AKS), with ground truth data for accurate assessment. Agent performance is measured through both final detection results and trajectory-based rewards that capture decision-making effectiveness. This work demonstrates the potential of AI agents to support labor-intensive aspects of detection engineering. Our comprehensive evaluation of 16 frontier models shows that Claude Opus 4.6 (High) achieves the highest overall reward (0.637), followed by Claude Opus 4.5 (0.624) and the GPT-5 family. An ablation study confirms that CTI-specific tools significantly improve agent performance, and a variance analysis across repeated runs demonstrates result stability. Finally, a memory augmentation study shows that seeded context can close 33% of the performance gap between smaller and larger models.
Presenter
Staff Software Security Engineer
Databricks
Add to Calendar
Citadel: A Maintained, Queryable Grounding Layer for the SOC
A Security Operations Center depends on an accurate map of its own telemetry: which log sources exist, what is in them, and whether they are current. However, that map rarely exists as a single trustworthy artifact, and the emerging alternative of letting an agent search the telemetry on demand tends to scope narrowly and query the wrong indices, because practical log management diverges sharply from the tidy inventory such an agent assumes. We present Citadel, a git-native catalogue that makes this map cheap enough to generate continuously rather than maintain by hand. Citadel holds one Markdown summary per log source per log platform, pairing human-readable prose with a machinecheckable block of facts, so that newly appearing sources, drifting volumes, and content that no longer matches a source’s name are caught as soon as the catalogue is built. We report its adoption in production, decompose the cost of building it, and measure its reproducibility across independent rebuilds. As deployed across a production estate of three log platforms carrying roughly 450 TB of telemetry per day, it holds 1,536 summaries and, within two weeks of release, was used by 58 analysts and engineers (roughly 31% of the SOC) with sustained, repeat adoption. An automated pipeline builds a single platform’s worth of summaries for roughly $600 at public list-price rates, a cost dominated by the live probing of each source; and across independent rebuilds we find the surprising result that the primary disagreement occurs in the probing stage rather than in the language model. We argue that Citadel is one instance of a general pattern: a grounding layer of durable, versioned artifacts that agents and analysts read as ground truth, and that making such artifacts cheap enough to generate turns SOC knowledge maintenance from a reconciliation problem into an automatable process.
Presenter
Architect, CyberSecurity Operations Center
Salesforce
Add to Calendar
Robustness in the Gray Zone: Securing AI Agents via Semantic and Episodic Memory against Boundary-Search Attacks
As AI agents and Large Language Models (LLMs) are increasingly deployed in security operations centers (SOCs) to triage alerts and automate decision-making, their reliability becomes a critical security concern. A significant vulnerability emerges in the form of model inconsistency: identical inputs yield contradictory outputs across different runs. This phenomenon is particularly pronounced for data points residing near the decision boundary (the “gray zone”), which are inherently ambiguous. In this extended abstract, we frame LLM inconsistency as an adversarial robustness issue, drawing parallels to how modern LLM jailbreaks and malware evasion techniques exploit boundary geometry. We present a lightweight, memory-augmented architecture to mitigate this vulnerability, grounded in the cognitive science distinction between episodic and semantic memory. By equipping AI agents with semantic memory (explicit domain knowledge) and episodic memory (retrieved past similar cases), we can significantly sharpen the decision boundary without the computational overhead of fine-tuning. We demonstrate that this approach not only reduces prediction flip-flops by over 60%, but also acts as a natural defense against jailbreak optimization algorithms that target the boundary, thereby enhancing the overall trustworthiness of AI-augmented SOC applications.
Presenter
Tech Lead
Datadog
Add to Calendar
RuleForge: Automated Generation and Validation for Web Vulnerability Detection at Scale
Security teams face a challenge: the volume of newly disclosed Common Vulnerabilities and Exposures (CVEs) far exceeds the capacity to manually develop detection mechanisms. In 2025, the National Vulnerability Database published over 48,000 new vulnerabilities (National Institute of Standards and Technology, 2025), motivating the need for automation. We present RuleForge, a system that automatically generates detection rules—JSON-based patterns that identify malicious HTTP requests exploiting specific vulnerabilities—from structured Nuclei templates describing CVE details. Nuclei templates provide standardized, YAML-based vulnerability descriptions that serve as the structured input for our rule generation process. This paper focuses on RuleForge’s architecture and operational deployment for CVErelated threat detection, with particular emphasis on our novel LLM-as-a-judge (Large Language Model as judge) confidence validation system and systematic feedback integration mechanism. This validation approach evaluates candidate rules across two dimensions— sensitivity (avoiding false negatives) and specificity (avoiding false positives)—achieving AUROC of 0.75 and reducing false positives by 67% compared to synthetic-test-only validation in production. Our 5×5 generation strategy (five parallel candidates with up to five refinement attempts each) combined with continuous feedback loops enables systematic quality improvement. Our lessons learned highlight critical considerations for applying LLMs to cybersecurity tasks, including overconfidence mitigation and the importance of domain expertise in both prompt design and quality review of generated rules through human-in-the-loop validation.
Presenter
Senior Applied Scientist, AWS Security
Amazon Web Services
Add to Calendar
Lunch
Add to Calendar
Trouble With Time: Compile Timestamps and First-Seen Date Regression in EMBER2024
Colloquially, it is well recognized and understood that malicious executables modify their “compiled on” timestamps for evasive and duplicitous purposes. As such, using a “first seen” data from external sources is preferable. But if the “first seen” date was itself predictable, we could minimize our dependence on external sourcing and enable academics to build better test/time splits. In this work, we investigate this from a machine learning perspective and find that regressing “first seen” is challenging, but determining that a COFF timestamp is unreliable is within reason. Our work provides this initial exploration and defines a new path of potential research for others.
Presenter
Senior Director, Cyber Superintelligence Lab
CrowdStrike
Add to Calendar
Latent Undertow: How Ordinary Typos Break Probes
LLMs handle ordinary typing variation fluently: a typo or missing punctuation leaves both user intent and the model’s response substantively unchanged. Yet probes that detect malicious prompts by reading the model’s hidden states tell a different story: the same edit rotates the readout vector by 43◦–56◦ at the perturbed token, decaying below 15% within ≈10 downstream tokens. Stacked common typos cut a single-position prompt-injection probe’s TPR@FPR=1% by 12.0pp, a gap recalibration alone cannot close. Multi-position aggregation cures localized perturbations (≤ 0.5pp loss) but only attenuates distributed ones, where even attentionand max-based aggregators still drop ∼3.8pp. For singleposition probes, we introduce a KV-cache fork: a short fixed suffix appended after the user message lets the probe read a few tokens downstream of the perturbation, exploiting its rapid spatial decay. This closes 95% of the gap (−0.6pp residual), an order of magnitude better than perturbation-augmented training (−3.7pp). The rotation-and-decay geometry replicates on Llama-3.1-8B, Qwen3-8B, and Gemma-4-E4B; probe evaluation is on Llama3.1-8B. Code: https://anonymous.4open.science/r/how_typos_break_probes-E77B. Keywords: Activation Probes, LLM Safety, Prompt Injection, Robustness, AI Security
Presenter
Senior AI Researcher
Zenity Labs
Add to Calendar
Euclidean Embedding of TLSH Digests
The Trend Micro Locality Sensitive Hash (TLSH) is popular and effective for finding nearduplicate items, using a compact hash function that can be compared with other hash functions. This comparison produces a dissimilarity score that does not meet the technical requirements for being a metric. The non-metric nature of TLSH and its incompatibility with most machine learning tools/libraries are hindrances to scalability and more effective application. We resolve these issues by introducing a deterministic, fixed Euclidean embedding algorithm that maintains approximate matching scores to the original TLSH, providing full metric support and compatibility with modern machine learning tools.
Presenter
Applied Machine Learning Engineer
CrowdStrike
Add to Calendar
Inherited Circuits, Learned Semantics: How Fine-Tuning Creates Evasion Vulnerabilities Invisible to Standard Evaluation
LLMs are increasingly fine-tuned for specific security classification tasks, and the resulting models are typically evaluated on held-out examples from the same distribution as their training data. We show that this evaluation practice misses a class of vulnerability introduced by fine-tuning itself: a model trained on a corpus where specific indicator tokens are strongly correlated with maliciousness can learn tokenlevel semantics that preserve canonical accuracy while failing under behavior-preserving transformations. In our PowerShell setting, these transformations include alias substitution, command reconstruction, string construction, execution indirection, and case mutation. To understand the mechanism, we study a natural base/finetuned model pair, Llama-3.1-8B-Instruct and Foundation-Sec8B-Instruct, on matched PowerShell classification cohorts. Using causal interventions, we localize the classification circuit to a small late-attention bundle and show that this route is inherited from Llama rather than created by fine-tuning. Fine-tuning concentrates and semantically specializes this inherited structure, adding associations between canonical command indicators and malicious classification. Those associations improve baseline classification behavior but create transformation-sensitive attack surfaces: a three-tier evasion benchmark finds Foundation-Sec misses across multiple transformation categories that Llama does not share. From this analysis we derive a practical pre-deployment monitoring method. A linear probe from base-model activations at the classification boundary predicts Layer-12 head contributions with r = 0.80–0.87, supporting cheap family-level drift monitoring after fine-tuning. A complementary indicator-token sign test compares how each model’s confidence changes when canonical indicators are ablated and identifies families where the token role has flipped from malicious driver to suppressor. Together, these signals produce a ranked list of command families for targeted red-team variant generation
Presenter
AI & Security Researcher
Cisco Talos
Add to Calendar
Conference Program Day One Concludes
Conference Co-Chair Recap and Conclusion
Add to Calendar
Speaker/Volunteer Dinner
By Invitation Only
Thursday, October 22
Add to Calendar
Registration
Check in, pick up your badge, and connect with the CAMLIS community before the main program begins.
Add to Calendar
The Breakfast Club
An exclusive morning session, co-hosted with align. Requires seperate registration.
Add to Calendar
Conference Opening & Welcome
Conference Co-Chair Welcome & Keynote Introduction
Add to Calendar
AI Security and the Future of Responsible Innovation
Presenter
Director of UK’s AI Security Institute and former advisor in the Department for Science, Innovation and Technology. Visiting fellow to the Oxford Martin AI Governance Initiative and Said Business School at the University of Oxford.
UK’s AI Security Institute · University of Oxford
Add to Calendar
Break
Add to Calendar
Set an Agent to Catch an Agent: Automated Vulnerability Discovery in AI Coding Tools
AI coding agents now run with full user privileges - accessing the filesystem, network, and credential stores - making even benign implementation issues high-impact security risks. We present a methodology for finding vulnerabilities in AI coding agents using an AI coding agent as the primary research tool. We applied this approach to three opensource AI coding agents (30k–200k+ GitHub stars) and found 20 vulnerabilities including remote code execution, privilege escalation, credential exposure, and server-side request forgery. Four findings received CVE identifiers and multiple vendors shipped patches within days. Our results show a recurring pattern: absent trust boundaries, insecure defaults, and unsandboxed execution treated as intended design.
Presenter
Senior Threat Researcher
CrowdStrike
Add to Calendar
How Much Does the Harness Matter, and for Whom? A Multi-Model Study of Context in LLM-Driven Vulnerability Discovery
Large language models can now find real security vulnerabilities in production code, and growing evidence suggests that the agent harness (tools, context, bundled infrastructure) substantially affects model capability (Big Sleep Team, 2024; DARPA, 2025). How much this matters, how that varies across models, and which model-and-harness pairings are worth their cost remain unclear, since most studies use one top-tier model on public, already-patched vulnerabilities it may have seen in training (Wang et al., 2025a). We address this with a multi-model, cost-aware study using an open-source agent harness (Ubl, 2026) that statically screens code, investigates candidates with an LLM agent, and validates findings through self-revalidation, deduplication, and proof-of-concept (PoC) confirmation where available. We run twelve models on a real production codebase (codename Conduit1 ) whose vulnerabilities were non-public at scan time. Three findings emerge. First, no single model is sufficient: across 280 validated vulnerabilities, even the largest single-model two-run set covers only about 60%, and more than a third are found by exactly one model, so coverage comes from combining models rather than picking one. Second, the value of structured program context depends on model capability: injecting a CodeQL-derived call graph raises criticality-weighted yield by 78–105% for smaller and mid-tier models but only 1% for the frontier model, with the same direction of effect appearing on a held-out CyberGym subset. Third, the frontier model buys the lowest false-report rate, not uniquely higher raw yield: a strong low-cost model matches the frontier model’s raw yield at a fifth of the cost. Together, these results argue against a one-size-fits-all recipe: the value of the context and tools an agent is given is a property of the model and task together, and the largest returns come from matching them to the model at hand, especially for the cheaper models that security teams are most likely to deploy.
Presenter
Graduate Student · AI & Security Research
University of Toronto · Georgian AI Lab
Add to Calendar
Learning to Attack and Defend: Adaptive Red Teaming of Language Models via GRPO
Language model safety must continually adapt to evolving attacks. Recent works have demonstrated that reinforcement learning can be used to train stronger attacker and defender models in tandem by applying PPO-style self-play and DPO-style online preference optimization. In this work, we explore the efficacy of GRPO in this setting. Co-training can be challenging because it requires jointly optimizing multiple properties of both the attacker and defender. We therefore shape model outputs using multiple LLM judge-based reward channels and compute advantages with GDPO, which prevents any single channel from dominating. Our method uses a curriculum that progresses from attacker-only single-turn and multi-turn training to co-training, where attacker and defender models are updated in alternation. We show that this method produces highly effective and transferable attacks, and that co-trained defenders reach competitive safety while preserving general utility. Through a controlled ablation, we further identify which components of our training pipeline most affect the resulting balance between safety and utility. Finally, we find that GRPO tends to collapse attacker diversity over training and discuss possible ways to address this limitation.
Presenter
Senior Researcher, AI Red Team
Microsoft
Add to Calendar
An Attacker-Refiner Framework for Automated Generation of Prompts Leading to Denial-of-Service in Large Language Models
In this paper we study how one can automatically generate prompts that can cause excessive resource consumption on a given target LLM. We propose a framework which makes use of an Attacker LLM and a Refiner LLM and is able to refine initial attack prompts from Attacker LLM to obtain refined prompts that can cause greater resource consumption on the target LLM (as compared to the initial prompts). We measure the results on many different target LLMs (Gemini 2.5 Pro, Claude Sonnet 4.5, GPT 5, Llama 4 Scout, Mistral Large, Qwen 235B) and show up to 17x increase in latency and up to 4x increase in output tokens on average. The prompts generated by our method do not make use of any optimization methods thus making this attack easily accessible to potential adversaries. Moreover, the attack prompts resemble normal prompts (thus can evade detections like perplexity) and are very compact (using only 100-200 tokens on average)
Presenter
Senior Security Researcher
TrendAI
Add to Calendar
Lunch
Add to Calendar
Nomos: Verifiable Council-Based Authorization for Autonomous Agent Actions
Autonomous LLM agents now propose and execute privileged actions such as shell commands, infrastructure changes, repository edits, and internal-service calls. We present Nomos, an actionlevel authorization kernel that gates each proposed action before it runs. Nomos combines deterministic policy checks with a multi-model LLM Security Council for semantic risk assessment, then binds approvals into single-use, timeand scope-bound Ed25519-signed writs verified by a separate executor before sandboxed execution. We evaluated Nomos on 948 synthetic benign and attack actions across Kubernetes administration, GitHub pull-request management, and SOC alert triage agents. Nomos blocked 99.7% of adversarial actions, including decoy and rationalized attacks, while achieving 89.2% approve recall with 23.9s median end-to-end authorization decision latency. Nomos also auto-approved 86.2% of 542 low-risk, real-world coding-agent commands in a small-scale preliminary validation. These results suggest that action-level authorization can preserve useful agent autonomy while preventing high-risk actions before execution.
Presenter
System Software Engineer
NVIDIA
Add to Calendar
CaMeLoT: CaMeL orchestrated with Temporal logic for static verification and liveness
LLM-based agents generate and execute multi-step plans that invoke external tools which can access private data or execute commands. In this setting, security is a property of the entire execution that a plan creates, not just any single step. The plan itself is a critical artefact that captures the tool calls, control flow, and data dependencies. We present CaMeLoT, a complement to CaMeL, an existing defence against prompt injection in toolusing LLM agents. CaMeLoT extends CaMeL by adding a static verification layer that checks an agent’s plan before any tool is invoked. CaMeLoT translates a generated plan into a finite-state transition system, labels it with tool calls, provenance and taint information, and checks it against temporal policies expressed in CTL using the nuXmv model checker. Because verification happens before execution, unsafe plans are rejected without using LLM calls or tool calls, saving tokens that runtime could have cost, as well as the need to unwind changes or teardown temporary sandboxes. When a verification fails, the model checker returns a counterexample to give feedback to the agent to repair the plan. We evaluate CaMeLoT on policies derived from the AgentDojo benchmark, SOC workflows, and promptextraction experiments, showing that it verifies a broad class of temporal properties before execution while preserving CaMeL’s runtime-checkable coverage.
Presenter
PhD Candidate
University of Edinburgh
Add to Calendar
Evaluating Agentic Learning Harness Capabilities Without Labels via the Scaling Hypothesis
Agentic “Continual Learning Harnesses”, systems that pair an LLM with retrieval or memory to improve from feedback without retraining, have shown growing value in cybersecurity. But their value is conventionally measured by gains against labeled benchmarks, an approach that often fails in operational security settings. Benchmark labels are scarce, stale, and unrepresentatively sampled, so a practitioner often cannot tell whether a given harness helps at all, or which of two is better for their task. Traditional LLM-as-a-judge offers little signal because it is no stronger than the agent it evaluates, and distillation is unreliable on scarce, sporadic, and biased labels. We propose a framework for evaluating learning harnesses end to end without a labeled benchmark, grounded in the scaling hypothesis. A stronger teacher model provides sparsely sampled corrections to a smaller student with a continual learning harness. We score a harness by how much its student converges toward the teacher over time. Across security tasks, model families, and harness designs, we show that improvement relative to the teacher correlates with improvement relative to a held-out gold standard, validating teacher-relative lift as a proxy for true harness uplift when labels are absent. We further show that LLMas-a-judge between similarly powered models yields no usable signal. These results suggest that a teacher-sized model can be improved through the same harness when humans provide the same kind of sparse, high precision corrections.
Presenter
Machine Learning Researcher
Sublime Security
Add to Calendar
Conference Program Day Two Concludes
Conference Co-Chair Recap and Conclusion
Add to Calendar
Poster Session & Happy Hour
Discover emerging ideas and research during the CAMLIS poster session, and close the day with a rooftop happy hour.
View poster 17
-
View PDF
A Large-Scale Sanitized Dataset of Windows API Behavioral Reports from VirusShare Malware
-
View PDF
A Perplexity-Aware Perturbation Approach to Guard Against Prompt Injection Attacks
-
View PDF
Alignment of Autonomous Cyber Defense Agents via Adaptive Policy Shaping
-
View PDF
AttnDA: Attention-Based Classification for Indirect Prompt Injection
-
View PDF
Automated Adversarial Testing for LLMs: Guided Fuzzing and Browser-Native Red Teaming
-
View PDF
Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents
-
View PDF
Detection as Geometry: On-Device Semantic Policy Enforcement via Signed Directional Alignment
-
View PDF
Evaluating LLM Security Review Agents with Operationally Sourced Ground Truth
-
View PDF
LLM-PassEval: A Practical Framework for Evaluating LLM-Assisted Passphrase Generation
-
View PDF
Operationalizing Enterprise AI Security: An Open-Source Framework for AI Cyber Risk Management
-
View PDF
ScopeJudge: Cost-Aware Pre-Execution Gating for Offensive Security Agents
-
View PDF
Secret Scanner Agent: Extracting Secrets and Access Context from Unstructured Documents
-
View PDF
Show, don’t tell: Calibrating Agentic SOC Analysts via Hierarchical Evidence Graphs
-
View PDF
Temporal Splitting with Near-Duplicate Samples: Measuring Memorization versus Generalization in Malware Classification
-
View PDF
TRACE: Threat Recognition through Administrative Correlation and Extraction
-
View PDF
Training-Free Refusal of MCP Exploits via Retrieval-Augmented Generation
-
View PDF
Workspace Topology as an Attack Vector in Agentic Coding Assistants
Friday, October 23
Add to Calendar
Registration
Check in, pick up your badge, and connect with the CAMLIS community before the main program begins.
Add to Calendar
Conference Opening & Welcome
Conference Co-Chair Welcome & Keynote Introduction
Add to Calendar
NXDirection: Bringing the NX-bit to the Residual Stream
Indirect prompt injection attacks exploit the absence of any enforced boundary between instructions and data in the shared token stream of a language model. We show that such a boundary is nevertheless encoded as a linear direction in the residual stream, recoverable by per-token difference-in-means on hidden states. Intervening on externally-sourced (untrusted) tokens at inference time, via forward-pass hooks at a small number of layers, yields a training-free defense against indirect prompt injection, which we call NXDirection by analogy to the NX-bit in operating-system memory. The defense relies only on token provenance to identify external-data spans, and assumes no knowledge of whether or where an injection appears within them; all external-data tokens are untrusted by construction. The projection along this direction is removed from every such token — gated to fire only on tokens already projecting onto the instruction side. We evaluate on 9 instruct models across 3 families and 3 sizes (Llama-3, Qwen-2.5, Gemma-2; 2B–72B). On average, the defense lowers attack success rate from 38.1% to 0.6% on BIPIA and from 28.6% to 9.2% on AgentDojo, while clean-task utility (Uc) is largely preserved on both benchmarks (BIPIA 78.4% → 76.4%, AgentDojo 37.9% → 37.0%). Because the intervention runs inside the forward pass — modifying residual-stream activations rather than rewriting the prompt beforehand or filtering the generation afterwards — it is orthogonal to prompt-engineering and guardrail-based defenses and stackable with either.
Presenter
Senior AI Security Researcher
NVIDIA
Add to Calendar
Hardening ML Malware Classifiers Against Adversarial Evasion: Systematic Benchmarking and Covariate Shift Correction
Static machine learning malware classifiers are increasingly targeted by semantics-preserving adversarial file modifications that reduce model decision values without altering malicious behavior. We present a study of adversarial robustness for production ML-based malware detectors spanning Windows PE, macOS MachO, Linux ELF, PDF, scripting formats including JavaScript and Python, and memory injection buffers. Our contributions are: (1) a modular adversarial attack framework supporting combinatorial semantics-preserving modifications across all major binary and script formats encountered in production; (2) systematic evasion benchmarking revealing critical model-specific and subpopulation-specific vulnerabilities; and (3) a covariate shift correction technique that improves adversarial robustness without the cost of adversarial corpus augmentation.
Presenter
Senior Applied ML Engineer, Data Science
CrowdStrike
Add to Calendar
What Counts as PII? Constitutional Targets for Broad, Efficient Identifier Detection
Personally identifiable information (PII) detectors and public benchmarks can apply different criteria for what should be recognized, even within the same PII categories, making benchmark results difficult to interpret and real-world behavior difficult to predict. To make detection criteria explicit, we introduce constitutional specifications for 12 PII categories and use them to relabel public datasets into a unified corpus of approximately 1.1 million samples. We then train a compact 140M-parameter single-pass encoder on this shared detection target and evaluate it on the Ai4Privacy and NVIDIA Nemotron-PII benchmarks under both constitutional and native annotation policies. Under the constitutional policy, our detector achieves the highest low-FPR partial AUC across all comparable categories while delivering 4–33× higher throughput than public baselines. Comparing constitutional and native evaluations further shows that benchmark rankings can reverse when native annotation policies omit privacy-relevant disclosures, such as identifiers embedded within larger spans. We plan to release the constitutional specifications, unified corpus, and efficient detector as open source to support more transparent development and evaluation of PII detection systems.
Presenter
Machine Learning Engineer, AI Defense
Cisco
Add to Calendar
kNNGuard: Turning LLM Hidden Activations into a Training-Free Configurable Guardrail
Large language models (LLMs) are increasingly deployed in domains requiring guardrails to detect unsafe, off-topic, or adversarial prompts. Existing guardrails predominately rely on fine-tuning to build classifiers, which often suffer from low generalization and high inference latency. We present kNNGuard, a training-free guardrail that utilizes the activation space of an off-the-shelf LLM. Given a small bank of 50 safe and unsafe prompts, kNNGuard extracts hidden activations and performs multi-layer kNN fusing activation-space and embedding-space scores for classification. Across six domains spanning topical and security prompts, kNNGuard achieves competitive or superior F1 compared to fine-tuned state-of-the-art guardrails while running 2.7×faster than the best comparable guardrail, and 10×faster than a fine-tuned safety classifier without gradient updates or fine-tuning. Domain adaptation requires only updating the labeled bank, which can be constructed in under 10 seconds and several orders of magnitude faster than established guardrails. We also analyze the impact of system prompts, layer selection, and integration into production LLM pipelines as a configurable, low-latency guardrail.
Presenter
PhD Student, AI Security
Lancaster University
Add to Calendar
Conference Program Day Three Concludes
Conference Co-Chair Recap and Conclusion