FEAT Add generic GCG model adapters and an open-weight SLM compatibility matrix

[romanlutz]

Is your feature request related to a problem? Please describe.

GCG loads models through AutoModelForCausalLM, but its gradient path still hard-codes exact classes for GPT-2/GPT-J, GPT-NeoX, Llama, Mistral/Mixtral, and Phi-3. A newly supported Hugging Face causal model can therefore load successfully and then fail when GCG tries to obtain its input embeddings.

Prompt slicing also assumes a tokenizer with a usable chat template and character-to-token mapping. There is no explicit preflight check or maintained compatibility matrix, so unsupported tokenizer behavior may fail late or silently compute incorrect goal, control, or target slices.

We want GCG to support the leading open-weight small language model families without losing its multi-model, multi-prompt, held-out evaluation, or progressive optimization capabilities.

Describe the solution you'd like

Introduce small model and tokenizer adapter interfaces for GCG.

The model adapter should:

• use the standard Hugging Face get_input_embeddings() contract instead of architecture-specific isinstance branches where possible;
• expose the embedding matrix, device, dtype, vocabulary size, and relevant cache capabilities;
• validate that the model supports differentiable causal-LM input-embedding optimization;
• produce actionable capability errors before worker processes and long-running optimization begin.

The tokenizer adapter should:

• render conversations using the tokenizer's chat template;
• locate goal, control, assistant-role, and target token slices robustly;
• validate decode/encode round trips and vocabulary compatibility;
• expose a stable tokenizer/vocabulary identity that later multi-model aggregation work can use safely;
• allow narrowly scoped family-specific behavior only when the standard Hugging Face interfaces are insufficient.

Create and document an initial compatibility matrix for representative causal text models/tokenizers from these open-weight SLM families: Qwen, Phi, Gemma, Llama, Mistral/Mixtral/Nemo, SmolLM, and applicable DeepSeek-distilled models. Keep the matrix architecture-based so new checkpoints can be added without copying model-specific attack code.

Acceptance criteria:

• The hard-coded embedding helper branches are replaced by the adapter contract or reduced to documented exceptions.
• Every supported family has tokenizer/template/slice tests using a small representative tokenizer and a tiny or randomly initialized compatible model where practical.
• At least one opt-in GPU smoke test exercises the complete gradient and candidate-scoring path for representative modern architectures.
• Unsupported models fail during preflight with a clear explanation of the missing capability.
• Adding support for another standard Hugging Face causal architecture normally requires a compatibility test or adapter registration, not edits to the core GCG optimization loop.
• Existing single-model, arbitrary multi-model, single-prompt, multi-prompt, held-out evaluation, and progressive model/goal behavior are preserved.
• Support must not be restricted to a single model, exactly two models, or a single shared prompt.

Describe alternatives you've considered, if relevant

We could continue adding architecture-specific isinstance branches, but that does not scale with the Hugging Face model ecosystem and leaves tokenizer assumptions implicit. Replacing PyRIT's implementation with a single-model GCG library would simplify compatibility but would lose PyRIT's multi-model, multi-prompt, and progressive optimization properties.

Additional context

Track this as a focused child of #960. Coordinate performance-related cache work with #962, but keep prefix caching, quantization, model sharding, and heterogeneous-tokenizer gradient aggregation as separate changes so this issue remains reviewable.

[devangpratap]

I would like to take this.

I reproduced the embeddings half on main (9616789) first, so here is what I am working from.

The three accessors in pyrit/executor/promptgen/gcg/attack/base/attack_manager.py (get_embedding_layer, get_embedding_matrix, get_embeddings) each dispatch through an isinstance chain over six hard-coded classes and end in raise ValueError(f"Unknown model type: {type(model)}"). Since GCG loads through AutoModelForCausalLM, anything outside that list loads fine and then fails at the gradient step.

Building tiny random configs for a spread of architectures, no weights or GPU needed:

model_type       has get_input_embeddings   GCG accessors
gpt2             True                       layer=ok          matrix=ok          embed=ok
llama            True                       layer=ok          matrix=ok          embed=ok
qwen3            True                       layer=ValueError  matrix=ValueError  embed=ValueError
gemma3_text      True                       layer=ValueError  matrix=ValueError  embed=ValueError
olmo2            True                       layer=ValueError  matrix=ValueError  embed=ValueError
starcoder2       True                       layer=ValueError  matrix=ValueError  embed=ValueError
mpt              True                       layer=ValueError  matrix=ValueError  embed=ValueError

Every one of them implements PreTrainedModel.get_input_embeddings(), so the information GCG needs is already reachable through the public API.

Proposed approach

1. Fall back to model.get_input_embeddings() in all three accessors instead of raising, so any AutoModelForCausalLM works by default. Keep the existing branches where there is a genuine quirk rather than a naming difference: GPT-2/GPT-J and GPT-NeoX currently call .half() in get_embeddings while the Llama/Mistral/Phi-3 paths do not, and I do not want to silently change that dtype behaviour for existing users.
2. Add unit tests over several architectures using tiny configs like the table above, so the generic path is covered on CPU in CI rather than only for the six listed classes.
3. Preflight check and compatibility matrix as described, once the accessor part is agreed.

Two questions before I open anything:

• Would you prefer this as one PR, or the accessor fix first and the preflight plus matrix separately? The accessor change stands alone and is easy to review in isolation.
• On the .half() asymmetry: is that deliberate for the older models, or incidental? If it is incidental I would rather leave it exactly as-is in this PR and raise it separately than fold a behaviour change into a compatibility fix.

[romanlutz]

I think either way is fine. Heads up it may be a couple weeks before we get to reviewing this because of an upcoming release.