feat: hard cut-over — rename LogStream→AgentStream, Metric→Evaluator

[adityamehra]

Summary

Hard cut-over rename of SDK domain entities. No backward compatibility shims — old names are gone. Migration is handled by a separate migration guide.

Old	New
LogStream	AgentStream
LogStreams	AgentStreams
Metric	Evaluator
LlmMetric	LlmEvaluator
CodeMetric	CodeEvaluator
LocalMetric	LocalEvaluator
SplunkAOMetric	SplunkAOEvaluator
BuiltInMetrics	BuiltInEvaluators
Metrics (service)	Evaluators
get_log_stream()	get_agent_stream()
list_log_streams()	list_agent_streams()
create_log_stream()	create_agent_stream()
enable_metrics()	enable_evaluators()
delete_metric()	delete_evaluator()
create_custom_llm_metric()	create_custom_llm_evaluator()
get_metrics()	get_evaluators()

The underlying API endpoints (/log_streams, /scorers) are unchanged — this is a client-side rename only. Server-side rename is tracked separately.

New API

from splunk_ao import AgentStream, Evaluator, LlmEvaluator, LocalEvaluator

# Agent Streams
stream = AgentStream(name="prod-traces", project_name="my-project").create()
stream = AgentStream.get(name="prod-traces", project_name="my-project")
streams = AgentStream.list(project_name="my-project")
stream.enable_evaluators([SplunkAOMetrics.correctness, "toxicity"])

# Evaluators
ev = Evaluator.get(name="factuality-checker")
llm_ev = LlmEvaluator(name="quality", prompt="Rate 1-10: {input}").create()

# Project methods
project = Project.get(name="My AI Project")
stream  = project.create_agent_stream(name="Production Traces")
streams = project.list_agent_streams()
for s in project.agent_streams:
    print(s.name)

# Module-level convenience functions
from splunk_ao.agent_streams import get_agent_stream, list_agent_streams, create_agent_stream, enable_evaluators
from splunk_ao.evaluators import create_custom_llm_evaluator, delete_evaluator, get_evaluators

Files changed

Source — old files deleted, new files are the full implementation (not wrappers):

Deleted	Replaced by
src/splunk_ao/log_stream.py	src/splunk_ao/agent_stream.py
src/splunk_ao/log_streams.py	src/splunk_ao/agent_streams.py
src/splunk_ao/metric.py	src/splunk_ao/evaluator.py
src/splunk_ao/metrics.py	src/splunk_ao/evaluators.py
src/splunk_ao/__future__/log_stream.py	(removed)
src/splunk_ao/__future__/metric.py	(removed)
src/splunk_ao/__future__/agent_stream.py	(removed — was a shim)
src/splunk_ao/__future__/evaluator.py	(removed — was a shim)

Other source files updated:

• src/splunk_ao/__init__.py — removed deprecated __getattr__ shims; exports only new names
• src/splunk_ao/project.py — removed create_log_stream, list_log_streams, logstreams; keeps create_agent_stream, list_agent_streams, agent_streams
• src/splunk_ao/__future__/__init__.py — updated to new class names
• src/splunk_ao/export.py, src/splunk_ao/logger/logger.py — updated to AgentStreams
• src/splunk_ao/types.py — MetricSpec now references Evaluator

Tests renamed and updated:

Old	New
tests/test_log_stream.py	tests/test_agent_stream.py
tests/test_log_streams_metrics.py	tests/test_agent_streams_evaluators.py
tests/test_log_streams_pagination.py	tests/test_agent_streams_pagination.py
tests/test_metric.py	tests/test_evaluator.py
tests/test_metric_types.py	tests/test_evaluator_types.py
tests/test_metrics.py	tests/test_evaluators.py

All @patch paths, import statements, assertions, and parameter names updated across the full test suite.

Verification

Check	Result
poetry run invoke type-check	✅ Success — 112 source files, 0 errors
poetry run invoke test	✅ 1,913 passed / 15 failed (all pre-existing) / 3 error (pre-existing)

Related

• PR refactor: rename Galileo OTel symbols to Splunk AO (GalileoSpan, galileo-tracer, galileo-otel) #90 — OTel Galileo → Splunk AO rename

[fercor-cisco]

🤖 This review was generated by the Astra agent (claude-opus-4-8). It may contain mistakes.

Verdict: request_changes — Public convenience functions for agent streams return broken objects; deprecation shims silently never fire.

General Comments

• 🟠 major (testing): The test plan lists a manual import/deprecation smoke test but no automated tests were added, and the checked items don't cover the module-level convenience functions (get_agent_stream/list_agent_streams/create_agent_stream) or the module-level deprecation aliases — which is exactly where the bugs are. A rename PR advertising "full backward compatibility" should include tests that (1) round-trip an AgentStream through each convenience function and assert .name/.id/is_synced(), (2) assert each deprecated module symbol emits DeprecationWarning, and (3) assert Evaluator.get()/.list() return the expected type. All three would have caught the issues flagged here.

Follow-ups

Suggested follow-up work that could be tracked as Shortcut stories:

• src/splunk_ao/evaluators.py:15-20: Import ordering: from galileo_core.schemas.logging.step import StepType (line 19) is placed after the splunk_ao.* imports, which violates the isort/ruff grouping used elsewhere in the repo. Run ruff/isort to reorder (third-party before first-party).
• src/splunk_ao/agent_streams.py:164-170: enable_evaluators does from splunk_ao.log_streams import LogStreams inside the function body even though LogStreams is already imported at module top (line 17). Drop the redundant local import.
• src/splunk_ao/project.py:1139-1157: The deprecated create_log_stream/list_log_streams docstrings were rewritten to say "Create/List a new agent stream", which reads oddly for a method named *_log_stream. Consider keeping the summary line referring to log streams and letting the .. deprecated:: note point to the agent-stream replacement, so the deprecation intent stays clear.

[fercor-cisco]

🔴 critical (bug): This produces a broken AgentStream. get_log_stream() returns the service-layer splunk_ao.log_streams.LogStream (a subclass of LogStreamResponse, decorated with @attrs.define → slots). Its fields (id, name, project_id, ...) live in slots, so result.__dict__ does not contain them — stream.__dict__.update(result.__dict__) copies essentially nothing. Combined with AgentStream.__new__ bypassing __init__ (so _sync_state, project_name, etc. are never set), the returned object raises AttributeError on stream.name, stream.id, str(stream), is_synced(). The two LogStream classes also have incompatible schemas. AgentStream already inherits a working get() from the object-centric LogStream that returns AgentStream instances via cls._from_api_response; delegate to it instead.

Suggested change

	result = get_log_stream(name=name, project_id=project_id, project_name=project_name)
	if result is None:
	return None
	stream = AgentStream.__new__(AgentStream)
	stream.__dict__.update(result.__dict__)
	return stream
	return AgentStream.get(name=name, project_id=project_id, project_name=project_name)

🤖 Generated by the Astra agent

[fercor-cisco]

🔴 critical (bug): Same defect as get_agent_stream: list_log_streams() returns slotted service-layer LogStream objects, so s.__dict__.update(ls.__dict__) copies nothing and AgentStream.__new__ skips initialization. Every returned object is a broken AgentStream that raises AttributeError on attribute access. Delegate to the inherited classmethod, which already returns AgentStream instances.

Suggested change

	log_streams = list_log_streams(
	project_id=project_id,
	project_name=project_name,
	limit=limit,
	starting_token=starting_token,
	)
	result: builtins.list[AgentStream] = []
	for ls in log_streams:
	s = AgentStream.__new__(AgentStream)
	s.__dict__.update(ls.__dict__)
	result.append(s)
	return result
	return AgentStream.list(
	project_id=project_id,
	project_name=project_name,
	limit=limit,
	starting_token=starting_token,
	)

🤖 Generated by the Astra agent

[fercor-cisco]

🔴 critical (bug): Same defect: create_log_stream() returns a slotted service-layer LogStream, so stream.__dict__.update(ls.__dict__) copies no fields and __init__/_sync_state are never set. Use the object-centric create path instead, which returns a fully-initialized AgentStream.

Suggested change

	ls = create_log_stream(name=name, project_id=project_id, project_name=project_name)
	stream = AgentStream.__new__(AgentStream)
	stream.__dict__.update(ls.__dict__)
	return stream
	return AgentStream(name=name, project_id=project_id, project_name=project_name).create()

🤖 Generated by the Astra agent

[fercor-cisco]

🟠 major (bug): These top-level imports make Metric, LlmMetric, CodeMetric, LocalMetric, SplunkAOMetric, and BuiltInMetrics real module globals, so the deprecation __getattr__ at line 164 is never invoked for them (module __getattr__ only fires when normal attribute lookup fails). Accessing splunk_ao.evaluator.Metric returns the class with no DeprecationWarning, contradicting the module docstring and the PR's stated backward-compat contract. To make the shim work, import these under private names (e.g. from splunk_ao.metric import Metric as _Metric) so the deprecated public names resolve only via __getattr__, and reference the private names in the class bases below.

🤖 Generated by the Astra agent

[fercor-cisco]

🟠 major (bug): create_custom_llm_metric, delete_metric, and get_metrics are imported as module globals here, so the deprecation __getattr__ at line 180 never fires for them — accessing splunk_ao.evaluators.get_metrics returns the function with no warning. Import them under private aliases (e.g. create_custom_llm_metric as _create_custom_llm_metric) so the deprecated names are only reachable through __getattr__.

🤖 Generated by the Astra agent

[fercor-cisco]

🟠 major (bug): create_log_stream, get_log_stream, and list_log_streams are imported as module globals, so the deprecation __getattr__ at line 193 only ever fires for enable_metrics (the one name not imported here). The other three deprecated aliases never emit a warning. Import them under private names so the deprecated symbols resolve only via __getattr__.

🤖 Generated by the Astra agent

[fercor-cisco]

src/splunk_ao/evaluator.py:843-855 (line not in diff)

🟠 major (bug): Evaluator.get()/Evaluator.list() are inherited from Metric and route through Metric._create_metric_from_type, which hardcodes LlmMetric/CodeMetric/SplunkAOMetric — never the *Evaluator subclasses. So Evaluator.get(name="correctness") returns a SplunkAOMetric, and this docstring's assert isinstance(ev, SplunkAOEvaluator) is false. The rename doesn't surface new types on read paths. Either override _create_metric_from_type in Evaluator to return the Evaluator subclasses, or correct the docstrings to state that get()/list() return Metric-family instances.

🤖 Generated by the Astra agent

[fercor-cisco]

@adityamehra as discussed in the ticket, let's not add a deprecated shim layer, the plan is to do a pure renaming. The migration guide / tool will be used for customers that need to migrate their code. Backward compatibility is provided via the galileo SDK.

[fercor-cisco]

🤖 This review was generated by the Astra agent (claude-opus-4-8). It may contain mistakes.

Verdict: request_changes — Code cut-over is sound, but the newly-added doc file documents the abandoned backward-compat design and contradicts the implementation.

Follow-ups

Suggested follow-up work that could be tracked as Shortcut stories:

• src/splunk_ao/shared/column.py:62-62: Pre-existing docstrings across shared/ (column.py:62,525; query_result.py:65) still use LogStream.get(...) in examples. Not introduced by this PR, but worth a sweep to align all docstring examples with the AgentStream/Evaluator names now that the old names are gone.
• src/splunk_ao/agent_stream.py:47-99: AgentStream/AgentStreams class and method docstrings throughout still say "log stream"/"metrics" in prose and reference splunk_ao.log_streams imports. Functionally harmless, but a follow-up doc pass to rename the user-facing prose to "agent stream"/"evaluators" would complete the rename intent.

[fercor-cisco]

🟡 minor (documentation): Evaluator.evaluators.correctness is wrong — the built-in accessor attribute is named metrics (with a legacy scorers alias); see evaluator.py:145,148. There is no evaluators attribute, so this raises AttributeError. Even after the larger doc rewrite, use Evaluator.metrics.correctness.

Suggested change

	ev = Evaluator.evaluators.correctness
	ev = Evaluator.evaluators.completeness
	# Access built-in evaluators
	ev = Evaluator.metrics.correctness
	ev = Evaluator.metrics.completeness

🤖 Generated by the Astra agent

[fercor-cisco]

src/splunk_ao/agent_stream.py:470-478 (line not in diff)

🟡 minor (documentation): set_metrics docstring example imports and uses Metric (from splunk_ao import Metric, Metric.metrics.correctness, Metric.get(...)), which no longer exists after the cut-over. Update to Evaluator. Line 78 similarly shows project.create_log_stream(...) which was renamed to create_agent_stream.

Suggested change

	from splunk_ao import Evaluator, AgentStream
	log_stream = AgentStream.get(name="Production Logs", project_name="My Project")
	# Set metrics (replaces existing)
	log_stream.set_metrics([
	Evaluator.metrics.correctness,
	Evaluator.metrics.completeness,
	Evaluator.get(id="metric-from-console-uuid"), # From console
	])

🤖 Generated by the Astra agent

[fercor-cisco]

🟠 major (documentation): This doc, added by this PR, describes the abandoned shim/backward-compat design rather than the hard cut-over that was actually implemented. It is broadly and actively misleading:

• Header says "Status: Implemented with full backward compatibility" and the whole "Backward Compatibility" section (lines 144-186) claims from splunk_ao import LogStream/Metric/LlmMetric/... and project.create_log_stream()/list_log_streams()/logstreams still work with DeprecationWarning. In reality the old modules and __getattr__ shims were deleted and project.py no longer defines create_log_stream/list_log_streams/logstreams — every one of these snippets now raises ImportError/AttributeError.
• The "Files Changed" table (lines 194-201) lists src/splunk_ao/__future__/agent_stream.py and __future__/evaluator.py as "New — deprecation shim"; those files do not exist (the PR body itself says they were removed). It also describes AgentStream as a "subclass of LogStream" and Evaluator via a PEP 562 __getattr__, neither of which matches the code.
• The "Design Decisions" section (lines 206-229) documents subclassing-LogStream and __getattr__-deprecation rationales that no longer apply.

This doc should be rewritten to describe the hard cut-over (old names gone, no shims, migration via the separate migration guide / galileo SDK) or removed in favor of the migration guide referenced in the PR body.

🤖 Generated by the Astra agent

[fercor-cisco]

🟡 minor (documentation): Class/method docstrings here still reference deleted names, so copy-pasting the examples fails after this hard cut-over. Lines 63 from splunk_ao.metric import Evaluator (module deleted), 118 & 1174 from splunk_ao import ... LogStream and 120 & 1176 LogStream.get(...) (LogStream no longer importable — use AgentStream). Update these examples to the new names as the ticket requests ("we also want the examples and code snippets to use the renamed entity names").

🤖 Generated by the Astra agent

[fercor-cisco]

🤖 This review was generated by the Astra agent (claude-sonnet-5). It may contain mistakes.

Verdict: approve — The hard cut-over is implemented consistently and correctly at HEAD — the old shim-based issues raised in earlier review rounds (broken slotted-object copies, imports bypassing getattr deprecation shims, Evaluator.get() returning the wrong subclass) have all been resolved by fully reimplementing the new files rather than wrapping the old ones. Remaining issues are minor/doc-scoped.

General Comments

• 🟡 minor (documentation): This is a breaking, hard cut-over rename (old top-level names LogStream, Metric, LlmMetric, etc. and modules splunk_ao.log_stream(s)/splunk_ao.metric(s) are gone entirely), but CHANGELOG.md has no entry for it. Given the PR explicitly states 'No backward compatibility shims — old names are gone,' users upgrading in-place will hit ImportError/AttributeError with no changelog signal. Add an Unreleased/Breaking Changes entry pointing to the migration guide.