feat(testcase): annotate testcase links with results

docs/how-to/test_to_doc_links.rst

@@ -20,21 +20,44 @@ Reference Docs in Tests
 This guide explains how to annotate test cases so that
 docs-as-code automatically creates traceability links between tests and requirements.
 
-Details of implementation
--------------------------
+How to annotate tests
+---------------------
 
-The mechanism is language-agnostic:
-The ``score_source_code_linker`` extension parses ``test.xml`` files, extracts test metadata
-(name, file, line, result, and verification properties),
-and creates backlinks on the referenced requirements.
+To link a test to the requirements it verifies, add the test metadata using
+the mechanism provided by its test framework:
 
-The extension will look for ``test.xml`` files in both ``bazel-testlogs/``
-as well as a folder named ``tests-report/``. This folder can be created manually if the tests
-require some pre-run step or are matrix tests or similar.
+Python (pytest)
+^^^^^^^^^^^^^^^
+   Use the ``@add_test_properties`` decorator. The test docstring supplies
+   the ``Description`` metadata.
+
+C++ (gTest)
+^^^^^^^^^^^
+   Use ``RecordProperty``. Put shared properties in ``SetUp()`` and
+   per-test properties inside each ``TEST_F``.
+
+Rust
+^^^^
+   There is currently no provided official way to add this metadata in Rust.
+   Use the advanced JUnit XML path below until Rust support is available.
 
+See the `Verification Templates <https://eclipse-score.github.io/reference_integration/main/_collections/score_process/process/process_areas/verification/guidance/verification_templates.html>`_
+for complete examples and the required metadata.
 
-Required Properties
--------------------
+Advanced usage: JUnit XML for other languages
+----------------------------------------------
+
+This section is only relevant when your language or test framework does not
+have one of the integrations above. In that case, produce JUnit XML with the
+metadata described below. The generated test results are processed
+automatically and create GitHub links from the requirements to the testcases.
+
+The extension looks for files named ``test.xml`` under ``bazel-testlogs/`` or
+``tests-report/`` at the workspace root. Create ``tests-report/`` manually when
+the test framework needs a separate pre-run step or produces matrix results.
+
+Required properties
+^^^^^^^^^^^^^^^^^^^
 
 Every linked test must declare the following properties
 (see :need:`gd_guidl__verification_specification` for detailed values):
@@ -50,13 +73,12 @@ Every linked test must declare the following properties
 
 ``Description``
    A human-readable explanation of the test objective and expected outcome.
-   *In Python tests, a docstring takes the place of the description attribute.*
 
 What should a test.xml look like?
----------------------------------
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Each testcase has file as well as it's line number as additional attributes.
-Then as described above, each testcase also has properties inside the XML.
+Each testcase must include its source file and line number as attributes,
+along with the verification properties.
 
 .. code-block:: xml
 
@@ -80,35 +102,22 @@ Then as described above, each testcase also has properties inside the XML.
      </testsuite>
    </testsuites>
 
-If you are not working in Python and using the provided pytest plugin, please ensure that the xml that is written in the end
-looks like this, otherwise the extension will not be able to parse the xml correctly.
-
-What happens when properties are missing
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+When properties or attributes are missing, testcases are still generated,
+but a testcase can only be linked to requirements if either ``PartiallyVerifies``
+or ``FullyVerifies`` is filled.
 
-When properties or attributes are missing for some or all tests inside the test.xml, testcases are still generated.
-However, testcases can only be linked to the requirements if either Partially- or FullyVerifies is filled.
-
-Language-Specific Annotations
------------------------------
-
-Each language uses a different mechanism to attach properties to test cases,
-but all produce the same JUnit XML output that the linker consumes.
-
-C++ (gTest)
-   Use ``RecordProperty`` — shared properties go in ``SetUp()``, per-test properties
-   inside each ``TEST_F``.
-   See :need:`gd_req__verification_link_tests_cpp`.
-
-Rust
-   Currently there is no provided official way to do this in Rust.
-   We are working with the rust community to figure this out.
-
-Python (pytest)
-   Use the ``@add_test_properties`` decorator; the docstring serves as ``Description``.
-   See :need:`gd_req__verification_link_tests_python`.
+Testcase result annotations
+---------------------------
 
-See :need:`gd_temp__verification_specification` for code templates.
+GitHub test links are decorated with their result, for example ``(passed)``,
+``(failed)``, ``(skipped)``, or ``(disabled)``. The annotation is applied to
+rendered links that target a testcase need, including the ``testlink`` entries
+shown on requirements. The status text inherits the surrounding theme colour
+and uses CSS classes with colours selected for the current S-CORE light and
+dark themes. These colours are not configurable for arbitrary Sphinx themes.
+Testcases without a result are left unchanged; links whose GitHub URL
+identifies multiple testcases are also left unchanged because their result
+would be ambiguous.
 
 
 Running Tests and Building Docs
@@ -120,7 +129,7 @@ Running Tests and Building Docs
 
       bazel test //...
 
-2. Build the documentation — the linker picks up ``bazel-testlogs/`` automatically:
+2. Build the documentation — the generated test results are picked up automatically:
 
    .. code-block:: bash
 
@@ -183,4 +192,4 @@ Limitations
 -----------
 
 - Tests must be executed by Bazel before building docs so ``test.xml`` files exist.
-- Not compatible with Esbonio / live preview (no ``bazel-testlogs/`` available).
+- Not compatible with Esbonio / live preview because generated test results are unavailable there.

src/extensions/docs/source_code_linker.md

@@ -381,6 +381,7 @@ score_source_code_linker/
 ├── needlinks.py                 # CodeLink dataclass & JSON encoder/decoder
 ├── testlink.py                  # DataForTestLink definition & logic
 ├── xml_parser.py                # Parses XML files into test case data
+├── testcase_annotations.py      # Adds result annotations to GitHub testcase links
 ├── tests/                       # Testsuite, containing unit & integration tests
 │   └── ...
 ```

src/extensions/score_source_code_linker/__init__.py

@@ -51,6 +51,9 @@
     load_repo_source_links_json,
     store_repo_source_links_json,
 )
+from src.extensions.score_source_code_linker.testcase_annotations import (
+    annotate_testcase_results,
+)
 from src.extensions.score_source_code_linker.testlink import (
     DataForTestLink,
     load_data_of_test_case_json,
@@ -323,6 +326,10 @@ def setup_once(app: Sphinx):
     # Priority=515 to ensure it's called after the test linker & combined connection
     app.connect("env-updated", inject_links_into_needs, priority=525)
 
+    # sphinx-needs resolves NeedIncoming and need metadata links on the same event.
+    # Run after those resolvers so the GitHub references are available to annotate.
+    app.connect("doctree-resolved", annotate_testcase_results, priority=800)
+
 
 def setup(app: Sphinx) -> dict[str, str | bool]:
     # Esbonio will execute setup() on every iteration.

src/extensions/score_source_code_linker/testcase_annotations.py

@@ -0,0 +1,211 @@
+# *******************************************************************************
+# Copyright (c) 2026 Contributors to the Eclipse Foundation
+#
+# See the NOTICE file(s) distributed with this work for additional
+# information regarding copyright ownership.
+#
+# This program and the accompanying materials are made available under the
+# terms of the Apache License Version 2.0 which is available at
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# SPDX-License-Identifier: Apache-2.0
+# *******************************************************************************
+"""Decorate rendered GitHub testcase links with their execution result.
+
+Testcase needs are external needs. Their ``external_url`` is therefore also
+the URL used by the ``testlink`` metadata rendered on requirements. The same
+URL is used for incoming ``fully_verified_by``/``partially_verified_by`` links.
+This hook handles both kinds of references: resolved need links can identify a
+testcase by ``refid``, while GitHub ``testlink`` references can be matched by
+their ``refuri``.
+
+An ``external_url`` points to a repository, file, and source line. It is not
+guaranteed to identify one testcase, because multiple testcases can share a
+source location. ID-based references are therefore preferred; URL-only
+references are annotated only when exactly one testcase matches.
+"""
+
+from __future__ import annotations
+
+from html import escape
+from typing import Any
+
+from docutils import nodes
+from sphinx_needs.data import SphinxNeedsData
+
+# Known result values get semantic classes so the stylesheet can provide
+# readable colours for the current S-CORE light and dark themes.
+RESULT_CLASSES = {
+    "passed": "score-testcase-result--passed",
+    "failed": "score-testcase-result--failed",
+    "skipped": "score-testcase-result--skipped",
+    "disabled": "score-testcase-result--disabled",
+}
+# Keep an unknown result visible, but do not assign it the meaning of a known
+# status such as ``passed`` or ``failed``.
+_FALLBACK_CLASS = "score-testcase-result--unknown"
+# The event handler is expected to be idempotent for a doctree. This marker
+# prevents a second invocation from appending the same status again.
+_ANNOTATED_ATTR = "score_source_code_linker_testcase_result_annotated"
+
+# The styles are inserted into a document only when that document contains an
+# annotation. Keeping them in one block avoids repeating inline style rules on
+# every testcase link and lets the same classes handle theme changes.
+_TESTCASE_STATUS_CSS = """
+<style>
+.score-testcase-result {
+  font-weight: bold;
+}
+.score-testcase-result--passed {
+  color: #146c2e;
+}
+.score-testcase-result--failed {
+  color: #b42318;
+}
+.score-testcase-result--skipped {
+  color: #8a5300;
+}
+.score-testcase-result--disabled,
+.score-testcase-result--unknown {
+  color: #5f6368;
+}
+html[data-theme="dark"] .score-testcase-result--passed {
+  color: #7ee787;
+}
+html[data-theme="dark"] .score-testcase-result--failed {
+  color: #ff7b72;
+}
+html[data-theme="dark"] .score-testcase-result--skipped {
+  color: #d29922;
+}
+html[data-theme="dark"] .score-testcase-result--disabled,
+html[data-theme="dark"] .score-testcase-result--unknown {
+  color: #c4cad2;
+}
+@media (prefers-color-scheme: dark) {
+  html:not([data-theme="light"]) .score-testcase-result--passed {
+    color: #7ee787;
+  }
+  html:not([data-theme="light"]) .score-testcase-result--failed {
+    color: #ff7b72;
+  }
+  html:not([data-theme="light"]) .score-testcase-result--skipped {
+    color: #d29922;
+  }
+  html:not([data-theme="light"]) .score-testcase-result--disabled,
+  html:not([data-theme="light"]) .score-testcase-result--unknown {
+    color: #c4cad2;
+  }
+}
+</style>
+"""
+
+
+def _resolved_target_need_id(ref: nodes.reference) -> str | None:
+    """Return the need ID encoded in a resolved reference, if available.
+
+    Depending on how sphinx-needs created the reference, the target is stored
+    either as ``refid`` or as the fragment of a local ``refuri``. The caller
+    can then perform an exact lookup before falling back to URL matching.
+    """
+    refid = ref.get("refid")
+    if refid:
+        return refid
+
+    refuri = ref.get("refuri")
+    if refuri and "#" in refuri:
+        return refuri.rsplit("#", 1)[-1]
+
+    return None
+
+
+def _testcases_by_external_url(needs: Any) -> dict[str, list[dict[str, Any]]]:
+    """Group testcase needs by their rendered GitHub URL without losing duplicates.
+
+    The URL identifies a source location, not a testcase identity. Keeping a
+    list is important because parameterized tests or multiple reported tests
+    can point to the same file and line.
+    """
+    testcases_by_url: dict[str, list[dict[str, Any]]] = {}
+    for need in needs.values():
+        if need.get("type") != "testcase":
+            continue
+        external_url = need.get("external_url")
+        if external_url:
+            # Do not overwrite an earlier testcase with the same location.
+            # _testcase_for_reference() uses the list to detect ambiguity.
+            testcases_by_url.setdefault(external_url, []).append(need)
+    return testcases_by_url
+
+
+def _testcase_for_reference(
+    ref: nodes.reference,
+    needs: Any,
+    testcases_by_url: dict[str, list[dict[str, Any]]],
+) -> dict[str, Any] | None:
+    """Resolve a reference by exact need ID or by an unambiguous external URL.
+
+    A need ID identifies one testcase even when its source URL is shared. A
+    URL-only reference is used only when it has exactly one testcase candidate;
+    choosing one of several candidates could display the wrong result.
+    """
+    target_id = _resolved_target_need_id(ref)
+    if target_id:
+        target_need = needs.get(target_id)
+        if target_need is not None and target_need.get("type") == "testcase":
+            return target_need
+
+    refuri = ref.get("refuri")
+    if refuri:
+        candidates = testcases_by_url.get(refuri, [])
+        if len(candidates) == 1:
+            return candidates[0]
+
+    return None
+
+
+def annotate_testcase_results(app, doctree, docname):
+    """Append a coloured result annotation using the S-CORE theme palette.
+
+    The handler runs after sphinx-needs' own ``doctree-resolved`` handlers.
+    It therefore sees both regular resolved references and the external
+    references generated for GitHub ``testlink`` metadata.
+
+    References without a testcase match or without a result are left as they
+    are. This keeps the post-processing safe when test reports are incomplete
+    or when a source URL is shared by multiple testcases.
+    """
+    needs = SphinxNeedsData(app.env).get_needs_view()
+    testcases_by_url = _testcases_by_external_url(needs)
+    # CSS applies to the whole document, so one style block is enough even if
+    # the document contains many annotated references.
+    css_added = False
+
+    for ref in list(doctree.findall(nodes.reference)):
+        if ref.get(_ANNOTATED_ATTR):
+            # A repeated event invocation must not append another badge.
+            continue
+
+        testneed = _testcase_for_reference(ref, needs, testcases_by_url)
+        if testneed is None:
+            continue
+
+        result = testneed.get("result")
+        if not result:
+            # A missing result is not a status and must not render as ``()``.
+            continue
+
+        result_text = str(result)
+        # Preserve unexpected report values as text, but use a neutral class
+        # instead of presenting them as one of the known statuses.
+        result_class = RESULT_CLASSES.get(result_text, _FALLBACK_CLASS)
+        status_html = (
+            f'<span class="score-testcase-result {result_class}"> '
+            f"({escape(result_text, quote=True)})</span>"
+        )
+        if not css_added:
+            doctree.insert(0, nodes.raw("", _TESTCASE_STATUS_CSS, format="html"))
+            css_added = True
+        # Preserve the existing link label and append the result annotation.
+        ref.append(nodes.raw("", status_html, format="html"))
+        ref[_ANNOTATED_ATTR] = True