Input Security Review Model

Input Security Review Model

The Input Security Review Model is a pre-flight classifier for enterprise AI. It reviews prompts, document snippets, and AI tool requests before they are sent to a downstream LLM or executed by an agent, then returns a structured risk assessment and a practical handling recommendation — allow, redact, ask the user, require approval, quarantine, or block and alert.

It is built on a Qwen3 0.6B base model adapted with LoRA over a 7-batch cumulative synthetic dataset (840 chat-format examples). The first experiments used MLX-LM LoRA locally on Apple Silicon; the final experiment moved the same task to a CUDA host with ModelScope Swift, producing a materially stronger adapter on the same eval, test, and diagnostic splits. The goal is not to replace policy engines or human review, but to give security, compliance, and platform teams a fast, auditable first opinion they can wire into AI gateways, RAG ingestion, document upload, and agent tool-use flows.

This article covers what the model does, where it fits, the output schema, the MLX and CUDA/Swift training runs, generation-based evaluation, known weak areas, and how to think about production deployment.

---

Table of contents

1. The problem: inputs are the new perimeter
2. What the model does
3. Where it fits in an AI stack
4. Output schema
5. Risk levels and recommended actions
6. Dataset and training setup
7. MLX training results
8. Generation-based evaluation: MLX baseline
9. Diagnostic comparison: 5-batch vs 7-batch
10. CUDA/Swift final adapter
11. Failure analysis
12. Production deployment guidance
13. Limitations and honest caveats
14. Further reading

---

The problem: inputs are the new perimeter

Enterprises adopting AI assistants, RAG pipelines, and autonomous agents have moved a lot of risk to the input side of the LLM. The most common incidents are not model hallucinations — they are humans and systems pushing the wrong thing into the model:

• Employees pasting API keys, tokens, or connection strings into a chat box.
• Regulated personal, healthcare, financial, or government data flowing into a hosted model with no data-use agreement.
• Prompt injection hidden inside a retrieved web page, an uploaded document, or an email attachment that quietly overrides the agent's instructions.
• An agent tool call that, if executed, would modify a production record, send a file externally, or shut down a logging pipeline.

Traditional DLP and WAF tools were not designed for this. They look at network traffic and file movement, not at the semantic intent of a prompt or the risk profile of a tool request. The Input Security Review Model fills that gap with a small, fast, structured classifier that runs in front of the LLM rather than around it.

---

What the model does

Given a user prompt, a document snippet, or an AI tool request, the model returns a single JSON object with six core decision fields plus audit fields:

1. risk_level — none, low, medium, high, or critical.
2. data_categories — the sensitive data types present (personal info, credentials, source code, government data, trade secret, model asset, …).
3. user_intent — what the user is actually trying to do (normal business use, summarization, jailbreak, prompt injection, data exfiltration, privilege escalation, …).
4. attack_types — if an attack is present, which kind (direct/indirect prompt injection, jailbreak, system-prompt extraction, tool-call manipulation, obfuscation, social engineering, multi-turn setup, …).
5. matched_spans — the exact substrings that triggered each category or attack label, copied verbatim from the input.
6. recommended_action — the handling decision: allow, allow_private_model_only, redact_then_allow, ask_user_to_remove_sensitive_data, human_review, block, block_and_alert, quarantine_attachment, or require_approval_for_tool_action.

A short explanation and a set of reason_codes accompany the structured fields so downstream policy engines and auditors can trace why a decision was made, not just what it was.

The model is intentionally a classifier, not a generator. It is trained to emit only the assistant JSON object, which makes the output cheap to parse, validate, and act on deterministically.

---

Where it fits in an AI stack

The model is designed to sit in front of the LLM or agent, not beside it. Typical insertion points:

User input ──► Input Security Review Model ──► policy gate
                                                  │
                          ┌───────────────────────┼───────────────────────┐
                          ▼                       ▼                       ▼
                       allow                redact / ask user        block / alert / quarantine
                          │                       │
                          ▼                       ▼
                  downstream LLM / agent   revised or gated request

Concrete deployment targets:

• AI gateways / LLM proxies — classify every inbound prompt before routing to a hosted or on-device model.
• RAG ingestion — review retrieved chunks and uploaded documents for hidden indirect prompt injection before they enter the context window.
• Agent tool-use gates — inspect a proposed tool call and its arguments before execution; force require_approval_for_tool_action for irreversible operations.
• Document and attachment upload — quarantine untrusted external content with quarantine_attachment until reviewed.
• Compliance evidence — log the structured decision (risk, categories, action, reason codes) as an auditable record of input review.

Because the model is small (0.6B plus a compact LoRA adapter) and returns a fixed JSON shape, it can run close to the gateway rather than as a separate heavy service. The MLX path is convenient for local Apple Silicon experiments; the CUDA/Swift path is a better fit for server-side batch evaluation and deployment-oriented inference testing.

---

Output schema

Each training and evaluation row uses chat fine-tuning format with exactly three messages: system, user, and assistant. The assistant content is a strict JSON object:

{
  "risk_level": "none | low | medium | high | critical",
  "data_categories": ["none | personal_info | sensitive_personal_info | government_data | confidential_government_info | internal_business_info | trade_secret | financial_info | healthcare_info | legal_info | source_code | credential | security_config | model_asset | metadata | public_info"],
  "user_intent": ["normal_business_use | summarization | translation | rewriting | classification | data_analysis | code_review | debugging | compliance_review | redaction_request | sensitive_data_processing | credential_use | data_exfiltration | privilege_escalation | jailbreak | prompt_injection | tool_abuse | unknown"],
  "attack_types": ["none | direct_prompt_injection | indirect_prompt_injection | jailbreak | instruction_override | system_prompt_extraction | data_exfiltration_attempt | tool_call_manipulation | obfuscation | social_engineering | multi_turn_setup"],
  "matched_spans": [{"text": "exact substring", "type": "category or attack type"}],
  "recommended_action": "allow | allow_private_model_only | redact_then_allow | ask_user_to_remove_sensitive_data | human_review | block | block_and_alert | quarantine_attachment | require_approval_for_tool_action",
  "reason_codes": ["NO_RISK_DETECTED | PUBLIC_INFORMATION | LOW_SENSITIVITY_INTERNAL_CONTEXT | PERSONAL_DATA_PRESENT | SENSITIVE_PERSONAL_DATA_PRESENT | GOVERNMENT_DATA_PRESENT | CONFIDENTIAL_GOVERNMENT_DATA_PRESENT | TRADE_SECRET_PRESENT | CREDENTIAL_PRESENT | SECURITY_CONFIG_PRESENT | SOURCE_CODE_PRESENT | MODEL_ASSET_PRESENT | PROMPT_INJECTION_DETECTED | JAILBREAK_DETECTED | SYSTEM_PROMPT_EXTRACTION_ATTEMPT | DATA_EXFILTRATION_ATTEMPT | TOOL_ABUSE_RISK | PRIVILEGE_ESCALATION_RISK | OBFUSCATION_DETECTED | HUMAN_REVIEW_REQUIRED | REDACTION_REQUIRED | BLOCK_REQUIRED"],
  "explanation": "brief explanation"
}

matched_spans carry only text and type. Character offsets were intentionally excluded after earlier experiments showed that exact offset matching caused unnecessary schema failures without adding decision value.

---

Risk levels and recommended actions

Risk levels are calibrated against handling severity, not just data sensitivity:

Recommended actions map a risk to a concrete next step. They are deliberately broader than a binary allow/block so policy teams can express nuance:

---

Dataset and training setup

The training set is a synthetic, seven-batch cumulative dataset of 840 chat-format examples, split into 560 train, 140 eval, and 140 test rows. All examples are fictional; names, agencies, companies, IDs, emails, tokens, and keys are fake, and credential-shaped strings are intentionally invalid — included only to teach risk-labeling behavior.

Distribution across risk groups:

Language coverage is roughly 55–60% Chinese, 35–40% English, and 5% mixed Chinese-English. The eval set is stratified across the major categories; the test set deliberately overrepresents known weak areas (prompt injection, tool abuse, exfiltration, credential-shaped placeholders).

Each batch targeted specific failure modes observed in the prior batch's analysis — for example, batch 7 added high-risk healthcare and disability records that earlier versions downgraded to medium, government records with case identifiers contrasted against truly public civic text, and credential-vs-security-configuration boundary cases that should trigger block_and_alert.

The first training path used MLX-LM locally, which is convenient on Apple Silicon and easy to iterate against the same repository artifacts. After the 7-batch MLX run exposed semantic calibration issues, the final experiment repeated the adapter training on a remote CUDA host with ModelScope Swift. That run used the same base family and the same 560/140/140 split, but produced a stronger LoRA checkpoint.

MLX training settings:

ADAPTER_PATH=adapters/input-review-qwen3-0.6b-lora-7batch ITERS=500 train-input-review-qwen3-0.6b-lora.sh

CUDA/Swift final adapter:

The final Swift checkpoint contains the expected adapter files, including adapter_model.safetensors, adapter_config.json, args.json, trainer_state.json, and trainer state files. The training log reached step 980/980, completing epoch 7.

---

MLX training results

Validation loss declined steadily through the run, with the best observed checkpoint at iteration 450. The final checkpoint at iteration 500 was slightly higher but still close to the minimum.

The final MLX test loss was 0.133 with perplexity 1.143, indicating the model is well-fit to the schema and label distribution without being overconfident.

---

Generation-based evaluation: MLX baseline

Generation evaluation uses deterministic sampling (temp=0.0), Qwen3 thinking disabled where supported, schema validation, and exact-match comparison for risk level, recommended action, data categories, and attack types. The table below is the MLX 7-batch baseline, retained because it is the useful local-training reference point for the later CUDA/Swift run.

The headline takeaway: format reliability is now essentially solved, the remaining work is semantic calibration.

• JSON validity is 100% across eval, test, and diagnostic generation runs.
• Schema validity is strongest on the cumulative test split at 98.6%.
• Risk and action accuracy sit materially below format validity, which tells us the bottleneck is judgment, not parsing.
• Data-category exact match is the weakest diagnostic metric after full-data retraining.

These are internal benchmark results, not a production SLA. Production use should combine the model with deterministic policy rules, monitoring, and human review for high-risk decisions.

---

Diagnostic comparison: 5-batch vs 7-batch

The batch-7 diagnostic set targets known weak areas: credentials, prompt attacks, tool exfiltration, hard negatives, protected action boundaries, and high-risk data handling. Comparing the 5-batch and 7-batch MLX adapters on the same 120-row slice shows where the extra data helped and where it hurt before the CUDA/Swift run.

The 7-batch model improved schema validity, risk exact match, action exact match, and attack-type exact match on the diagnostic slice. Category accuracy regressed sharply — the additional data improved threat and action behavior while making category assignment noisier. That is a useful signal: the next batch should focus on category-boundary examples (public-info plus sensitive-context rows, internal-business vs security-config boundaries) rather than adding more attack variety.

---

CUDA/Swift final adapter

The final experiment moved training and full generation evaluation to a remote CUDA host using ModelScope Swift. The purpose was not to change the task or the dataset, but to test whether a CUDA/Swift LoRA run would produce better semantic calibration than the MLX 7-batch baseline.

The Swift inference output format differs from the MLX prediction files: Swift writes raw records with response, labels, logprobs, and messages. For apples-to-apples analysis, each raw response was converted back into the existing analyzer schema while preserving the original raw_prediction. The converter stripped leading Qwen <think> blocks, parsed the first JSON object, validated it with the same review_schema.py checks, and computed exact-match fields with the same logic used for MLX.

Swift inference command family:

swift infer \
  --model Qwen/Qwen3-0.6B \
  --use_hf true \
  --adapters /workspace/input-review/output/qwen3-0.6b-lora-swift/v0-20260624-074408/checkpoint-980 \
  --max_new_tokens 192 \
  --temperature 0.0 \
  --top_p 1.0 \
  --max_length 1024 \
  --write_batch_size 20

Raw Swift output row counts:

Converted analyzer-ready files:

Side-by-side results against the MLX 7-batch baseline:

The deltas are large enough to change the interpretation of the project. The MLX run proved the schema and workflow, but the CUDA/Swift adapter is the stronger candidate for downstream testing.

High-impact failure counts also improved:

The diagnostic split is the clearest win. It was intentionally built from previous failure modes, and the Swift adapter reaches 97.5% risk exact, 91.7% action exact, 89.2% category exact, and 92.5% attack-type exact with perfect schema validity. That does not make it production-ready by itself, but it shows the targeted examples were learnable rather than inherently ambiguous.

Two practical notes from the CUDA run:

• Swift/Qwen still emitted empty <think> wrappers before many JSON objects, even when the desired task was JSON-only classification. This was harmless after conversion, but production inference should strip wrappers or use a stricter serving template.
• One test row produced malformed JSON and one eval row emitted an invalid action enum (data_exfiltration_block_and_alert). A production gateway should always validate and repair or reject outputs before applying policy.

---

Failure analysis

After the CUDA/Swift run, the remaining failures are even more concentrated in semantic policy decisions, not broad format reliability.

Observed failure modes:

• High-severity underclassification. The most important residual misses are isolated but serious: a credential-like MySQL URL downgraded from critical to high, a deployment password routed to private-model-only handling, and an approval-workflow abuse prompt predicted as safe.
• Protected action confusion. block_and_alert, quarantine_attachment, and require_approval_for_tool_action are still occasionally swapped, even when the model correctly recognizes critical risk.
• Exact span brittleness. Schema errors now mostly come from non-exact copied spans, case changes, paraphrased attack phrases, Chinese phrase snippets, or quoted code strings.
• Enum discipline is good but not perfect. One eval row emitted data_exfiltration_block_and_alert, which is outside the allowed action set.
• Hard-negative overblocking is mostly controlled. The Swift test run had one notable overblock: a placeholder credential example was predicted as critical / block_and_alert.
• Credential detection improved on the diagnostic and test splits, but one eval credential category miss remains.

The pattern is encouraging for production: the model is now much better at recognizing known high-risk patterns, but the remaining mistakes still matter because they appear on exactly the rows where a gateway must be conservative. The next improvement should be a rule-plus-model policy layer, not just more raw generation benchmarking.

---

Production deployment guidance

The model is designed to be one component in a layered review pipeline, not a standalone decision-maker. Recommended production shape:

1. Run the model on every inbound prompt, retrieved chunk, uploaded document, and proposed tool call.
2. Apply deterministic policy rules on top of the structured output for high-confidence cases. These rules are cheap, auditable, and remove the model's remaining under-classification on the most dangerous patterns.
3. Route ambiguous cases (human_review, require_approval_for_tool_action) to a human or a stronger LLM judge.
4. Log the full structured decision — risk, categories, intent, attack types, matched spans, action, reason codes — as compliance evidence.
5. Monitor drift in action distribution, schema validity, and review-queue volume.

High-value deterministic rules that pair well with this model:

• Credential-like pasted secret → force critical + block_and_alert.
• System-prompt extraction or developer-instruction disclosure → force critical + block_and_alert.
• External exfiltration through tools → force critical + block_and_alert.
• Irreversible internal tool action without exfiltration → force require_approval_for_tool_action.
• Untrusted document, web, email, or RAG hidden instruction → force quarantine_attachment.

This rule-plus-model pattern is what should be evaluated for production readiness, not raw model generations alone. The Swift adapter is a better starting point than the MLX 7-batch adapter, but the deployment posture is the same: validate the JSON, apply deterministic overrides for high-confidence dangerous patterns, and escalate uncertain protected actions.

---

Limitations and honest caveats

• The dataset is small and synthetic. It should be expanded with organization-specific, legally approved, properly de-identified examples before production use.
• The strongest current adapter is the CUDA/Swift run, with 86.4% risk exact and 72.9% action exact on the test split. That is useful as a first opinion but still not sufficient as the only control on high-risk paths.
• The diagnostic split improved dramatically under Swift, but it is only 120 rows and intentionally targets known historical failures. It should not be treated as a broad production benchmark.
• Data-category exact match is materially better with Swift than with the MLX baseline, but category boundaries still need post-processing for credentials, security configuration, public information, and internal-business context.
• The model was trained on a fixed schema. New categories, actions, or attack types require retraining.
• The model can still emit malformed JSON, non-exact spans, or invalid enum values under generation. A production gateway must validate every output before acting on it.
• All numbers here are internal benchmarks, not a production SLA.
• Production deployments should include policy review, privacy impact assessment, red-team testing, monitoring, and human escalation workflows.

The honest framing: this model is a fast, cheap, structured first opinion that collapses a free-text judgment into an auditable JSON decision. It is not a replacement for policy, red-teaming, or human review on regulated paths.

---

Further reading

• Companion product sheet and technical report in the training repository.
• Qwen3 model cards on Hugging Face / ModelScope.
• MLX-LM documentation for LoRA fine-tuning on Apple Silicon.
• ModelScope Swift documentation for CUDA LoRA training and inference.
• OWASP LLM Top 10 and MITRE ATLAS for prompt-injection and agent-abuse taxonomies.

---

The Input Security Review Model acts as a pre-flight classifier for enterprise AI. It reviews prompts and AI tool requests before they run, identifies sensitive data or attack attempts, and returns a structured handling recommendation — so teams can adopt AI faster with stronger, more auditable security controls.