Reference implementation for the manuscript "SpecLLM: LLM-Based VNIR Spectral Interpretation".
SpecLLM turns a hyperspectral reflectance spectrum into a compact textual description that a large language model can classify. A spectrum is smoothed, transformed e.g. to log(1/R), reduced to summary statistics plus its most prominent peaks and valleys, and then serialised either as a sentence of natural language or as structured JSON. The evaluation compares those compact representations against encodings of the full numerical spectrum (CSV, JSON, XML) and against Random Forest reference classifiers, on two scenes of the public HyperBlood dataset.
This repository contains everything needed to reproduce the paper's main experiment: the spatial sampling that builds the splits, the feature extraction, the class-archetype construction, the prompt templates, the LLM client, the response parser, and the evaluation metrics.
Scope. Only the two HyperBlood scenes evaluated in the paper (Scene F and
Scene E), the main-experiment split (split_00), the six representations, and
the XML/scratchpad prompt are supported. Ablation studies and the supplementary
statistical analyses are not part of this release.
Python 3.11 is recommended (the experiments were run on 3.11).
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txtThe spectra come from the public HyperBlood dataset, which is not included here:
M. Romaszewski, P. Głomb, M. Cholewa, A. Sochan, HyperBlood, https://doi.org/10.5281/zenodo.3984905 (CC BY 4.0)
Download and extract it, then point the configuration at the extracted
directory. Only two of the fourteen scenes are used, F_1 and E_1; the
reader expects this layout:
<hyperblood_root>/
data/
F_1.float F_1.hdr # ENVI BIL float32; the header reports
E_1.float E_1.hdr # 696 samples, 520 lines, 128 bands
anno/
F_1.npz E_1.npz # ground-truth class maps, key 'gt'
The loader applies the dataset's own cleaning conventions: it removes the damaged sensor line (row 445) and the 15 noisy bands. Each scene therefore arrives in memory as a 519 × 696 × 113 cube (rows × columns × bands) covering roughly 400–1010 nm.
Copy the example configuration and edit the two blocks marked EDIT ME:
cp configs/example_config.yaml configs/my_config.yamlpaths.hyperblood_root— the directory you extracted in step 2.endpoints.models— one OpenAI-compatible base URL per model. The paper used a separate llama.cpp server per model, but any OpenAI-compatible deployment works (llama-server, vLLM, Ollama, a hosted API).
The four keys under endpoints.models are short names; the paper evaluated
these specific builds, and reproducing its numbers means serving them:
| Name in this repository | Model and quantisation evaluated |
|---|---|
Llama4-scout |
Llama-4-Scout-17B-16E-Instruct, IQ4 |
Qwen14b |
Qwen2.5-14B-Instruct, Q4 |
gpt-oss-20b |
gpt-oss-20b, F16 (unquantised) |
Mistral7b |
Mistral-7B-Instruct-v0.2, Q4 |
Pointing a name at a different model, or at the same model at a different quantisation, is a methodological deviation rather than a reproduction, and the results should be reported as such. The names are otherwise free: they are labels for the output files and the result tables, and nothing in the code checks what a server actually loaded.
Credentials come from the environment, never from the configuration file:
export SPECLLM_API_KEY=... # ignored by most local serversEverything else in the file is a scientific setting of the paper experiment; change it only if you intend to deviate. Two are worth knowing about:
llm.temperatureandllm.top_parenull, so no sampling parameters are sent and the serving backend's own defaults apply. This matches the configuration recorded for the paper run, which means the effective decoding settings are a property of your server, not of this code. Set them explicitly if you want fixed decoding.llm.max_test_per_class: 100sets how many test spectra per class are sent to each model. Lower it to try the pipeline cheaply.
The configuration is validated on load. Unknown keys, unsupported scenes or representations, and a model with no configured endpoint all fail immediately with a message naming the offending value.
run.py is the only entry point. The configuration file is always given
explicitly.
# 1. build the spatially separated train/test splits for both scenes
python run.py --config configs/my_config.yaml --mode prepare-data
# 2. class archetypes, Random Forest baselines, and the LLM evaluation
python run.py --config configs/my_config.yaml --mode experiment
# 3. recompute the reported metrics from the stored predictions
python run.py --config configs/my_config.yaml --mode analyze
# or all three in sequence
python run.py --config configs/my_config.yaml --mode allSelecting a single scene or model without editing the configuration:
python run.py --config configs/my_config.yaml --mode experiment \
--scene hyperblood_scene_f --model gpt-oss-20b--scene and --model are repeatable. Other useful flags: --force to
regenerate splits that already exist, and --rebuild-archetypes to rebuild the
class descriptions instead of reusing the stored ones. (Archetypes are also
rebuilt automatically when the feature or preprocessing settings change.)
split_id is fixed at "00", the split the paper's main experiment uses; any
other value is rejected on load.
Scale. A complete run of the paper experiment is 2 scenes x 4 models x 6
representations x 700 (Scene F) or 600 (Scene E) spectra = 31,200 requests.
With concurrent_models: true the models run in one process each, so four
requests are in flight at a time; expect hours to days depending on your
serving hardware. Lower llm.max_test_per_class to try the pipeline first.
Responses are cached under outputs/responses/ keyed by a hash of the exact
prompt pair, so an interrupted run resumes without re-querying, and editing a
prompt makes the run miss the old cache entries (they are left on disk rather
than deleted). Failed requests are never cached as results: a single failure is
recorded and retried on the next run, and llm.max_consecutive_errors
consecutive failures abort the run rather than producing a table of empty
predictions.
A deliberately small live end-to-end check: one scene, one model, all six representations, two test spectra each — twelve classification calls, preceded by a one-token request that checks the endpoint answers at all, so the server sees thirteen requests in total. It exercises loading, preprocessing, feature extraction, archetype and prompt construction, the request, caching, parsing, and prediction and metric writing, then verifies that every expected file exists and that each representation produced at least one valid parsed label with a confidence. It also checks that malformed, out-of-vocabulary and empty responses are recorded as invalid rather than crashing the run, and it fails if any request failed at the transport level. It is not a reproduction of the paper's numbers.
Create a configuration listing exactly one model, then:
python run.py --config configs/my_smoke_config.yaml --mode smoke-testThe mode narrows whatever you give it to one scene, all six representations and
two spectra, and redirects its results into <output_root>/smoke_test/, so a
full configuration works too as long as llm.models has a single entry and
your real results are never overwritten. It exits non-zero on failure.
Offline checks that need neither the dataset nor a network:
python -m pytest tests -q # or: python tests/test_offline.pyFor the authors: the local gpt-oss-20b configuration lives at
configs/local_gpt_oss.yaml. It is matched by configs/local_*.yaml in
.gitignore and must stay out of the published repository and the archive.
python run.py --config configs/local_gpt_oss.yaml --mode smoke-testoutputs/
configs/resolved_config.yaml the settings the run actually used
archetypes/<scene>/split_<id>/ class descriptions built from training data
prompts/<scene>/split_<id>/<model>/<representation>/
responses/<scene>/split_<id>/<model>/<representation>/ cache, one file per query
predictions/<scene>/split_<id>/ one CSV per model and representation
metrics/<scene>/split_<id>/ per-run metrics and confusion matrices
smoke_test/ results of --mode smoke-test, kept apart
reports/results_summary.{csv,tex,md} the reported tables
The summary reports BACC, BACC(IAW), ACC, macro-F1 and the invalid rate, for
the multiclass task and for the binary blood-versus-others task. BACC(IAW)
counts every unparseable output as wrong; the other columns are computed over
the valid outputs only, and invalid gives the rate they were computed at.
A failed request is not an invalid response: the model produced no output at
all. Such a spectrum is recorded in the request_error column of the
predictions CSV and counted as n_request_errors in the metrics, and it is
excluded from invalid and n_invalid, whose denominator is the number of
responses actually received (n_responses). It still counts as unclassified in
BACC(IAW), so request failures cannot inflate a score either. A streak of
llm.max_consecutive_errors failures aborts the run.
Data preparation is deterministic. Seeded with random_seed: 12345, the split
generator reproduces the exact spectra and pixel coordinates used for the
paper.
Determinism of the split does depend on library versions, because patch
placement uses sklearn.cluster.KMeans.
LLM inference is not deterministic. No generation seed is supplied, each spectrum is queried once, and the effective sampling parameters belong to the serving backend. Different servers, quantisations and runs will produce different responses, different invalid rates and slightly different scores. Reproducing this work means running the same pipeline and obtaining comparable results — the same ordering of representations and the same broad accuracy levels — not obtaining identical responses. Every prompt and raw response is saved so that any individual result can be inspected.
The Random Forest baselines are deterministic given a fixed seed and scikit-learn version.
run.py single entry point
configs/example_config.yaml
specllm/
config.py settings, YAML loading, validation
hyperblood.py ENVI and annotation reader for the public dataset
sampling.py spatially separated split generation with guard zones
data.py split loading, log(1/R), Savitzky-Golay smoothing
features.py extremum detection, canonical spectrum object
representations.py the six textual representations
archetypes.py class-archetype feature selection and descriptions
prompts.py system and user prompt templates
parsing.py response parsing and invalid-output handling
llm_client.py OpenAI-compatible chat completion client
runner.py evaluation loop with response caching
baselines.py Random Forest reference classifiers
metrics.py metric computation and summary tables
pipeline.py stage orchestration
smoke.py live end-to-end smoke test
tests/test_offline.py deterministic checks, no dataset or network needed
If you use this code, please cite the SpecLLM manuscript:
M. Romaszewski, A. Strzoda, M. Halama, P. Głomb, P. Masłowski, SpecLLM: LLM-Based VNIR Spectral Interpretation.
and the underlying dataset:
M. Romaszewski, P. Głomb, M. Cholewa, A. Sochan, A dataset for evaluating blood detection in hyperspectral images, and the accompanying data deposit https://doi.org/10.5281/zenodo.3984905
Apache License 2.0 — see LICENSE.
The HyperBlood dataset is distributed separately by its authors under CC BY 4.0 and is not covered by this licence.