feat: add prefix to component tool names - #1432
Conversation
|
@akihikokuroda, we should create a full proposal for this. I don't know if we want to prefix component tools with an index. I think we need to ensure that performance isn't impacted. We also need to make sure that we can clearly link tools to any given component, which might require more testing / planning. |
|
Small note on just an initial pass, I think this will have conflicts will #1430 adding |
|
@jakelorocco Here is a proposal. Proposed Solution: Object ID-Based PrefixingDesignUse Python object identity as the stable component identifier: component_id = hex(id(component))[-8:] # e.g., "a1b2c3d4"
prefixed_name = f"component_{component_id}.{original_tool_name}"Properties
Example: Multi-Turn Scenario with ID-Based PrefixingAdvantages Over Index-Based1. Multi-Turn Stability
2. Prevents Parallel Tool Collisions
3. Component Context Independence
4. Better Observability
5. Integrates with PR #1430 (Tool Tracing)
Performance ImpactAnalysis
Example with 10 tools:
Token CostName length comparison:
Example with 10 tools in JSON schema:
Conclusion: Performance impact is negligible. No performance concerns.
About concern:
Better Solution: Metadata on TemplateRepresentationInstead of reverse mapping, store component metadata on TemplateRepresentation: @dataclass
class TemplateRepresentation:
obj: Any
args: dict[...]
tools: dict[str, AbstractMelleaTool] | None = None
tool_name_mapping: dict[str, str] | None = None
# NEW: Component identity metadata
component_id: str | None = None # e.g., "a1b2c3d4"
component_type: str | None = None # e.g., "SearchAgent"
component_description: str | None = None # Optional Why This is Better
Usage Example# At tool registration time:
if isinstance(action, Component):
tr = action.format_for_llm()
if isinstance(tr, TemplateRepresentation):
tr.component_id = hex(id(action))[-8:]
tr.component_type = type(action).__name__
# tool_name_mapping already set
# All metadata in one place ✅
# At trace time:
span.set_attribute("mellea.component.id", tr.component_id)
span.set_attribute("mellea.component.type", tr.component_type)
# Metadata available without reverse mapping ✅
# In error messages:
error = f"Tool {tool_name} failed (component: {tr.component_type})"
# Clear attribution without reverse mapping ✅
# For debugging:
# User reads error message, sees component type, can correlate
# No need to reverse-map tool name back to component ✅ |
|
@akihikokuroda, I think this is still a good idea but we probably need some evaluations to indicate that the model can properly track the difference between components. Especially since we don't print component ids when outputting them currently. |
Signed-off-by: Akihiko Kuroda <akihikokuroda2020@gmail.com>
Signed-off-by: Akihiko Kuroda <akihikokuroda2020@gmail.com>
Signed-off-by: Akihiko Kuroda <akihikokuroda2020@gmail.com>
Signed-off-by: Akihiko Kuroda <akihikokuroda2020@gmail.com>
Signed-off-by: Akihiko Kuroda <akihikokuroda2020@gmail.com>
Signed-off-by: Akihiko Kuroda <akihikokuroda2020@gmail.com>
|
@jakelorocco I added examples. docs/examples/components/pattern2_context_and_tools.py shows the component tools renaming, executing and telemetry output. |
Signed-off-by: Akihiko Kuroda <akihikokuroda2020@gmail.com>
planetf1
left a comment
There was a problem hiding this comment.
Thanks for tackling the tool-name-collision problem (#95) — the underlying idea (auto-prefix component tools so they don't clobber each other) is sound. A few things worth resolving before this merges, most importantly a correctness bug that would break real OpenAI/Azure usage but is currently invisible to CI because the only exercising examples are Ollama-gated.
Two non-code notes:
- The PR description says tools are prefixed with
component{N}.(sequential index, e.g.component0.search), but the shipped code usescomponent_{hex(id(action))[-8:]}.(object-identity hash). Worth updating the description so reviewers/future readers aren't misled. - Small thing, but since this will likely be squash-merged and the PR title becomes the commit message: "prfefix" → "prefix" in the title itself would be good to fix before merging.
|
|
||
| for original_tool_name, tool_instance in tr.tools.items(): | ||
| # Auto-prefix tool name using component ID to avoid collisions | ||
| prefixed_name = f"component_{component_id}.{original_tool_name}" |
There was a problem hiding this comment.
BLOCKER: OpenAI/Azure OpenAI function names must match ^[a-zA-Z0-9_-]{1,64}$ — periods aren't allowed. This . gets copied verbatim into function.name by convert_tools_to_json and sent to any OpenAI-compatible backend, which will 400 on real OpenAI/Azure. It only "works" here because the two new examples are Ollama-gated and Ollama's endpoint is lenient about the character.
| prefixed_name = f"component_{component_id}.{original_tool_name}" | |
| prefixed_name = f"component_{component_id}__{original_tool_name}" |
Changing the separator ripples into metrics.py's tool.split(".", 1) (line 992) and both test regexes (component_[0-9a-f]{8}\.) — those need updating too if this lands.
There was a problem hiding this comment.
"." changed to "__" everywhere.
| # Track mapping from original to prefixed names | ||
| name_mapping = {} | ||
|
|
||
| for original_tool_name, tool_instance in tr.tools.items(): |
There was a problem hiding this comment.
This prefixes every component's tools unconditionally, even when there's no collision (e.g. a lone Table.to_markdown gets renamed too). Was that intentional — favoring consistency/predictability over leaving non-colliding names untouched — or would it be worth detecting an actual collision first and only prefixing the colliders? The tradeoff is longer/hash-like names and extra prompt tokens for every existing single-component caller, even ones that never hit a naming conflict. Curious about the reasoning either way.
There was a problem hiding this comment.
We should likely not update the names of existing tools. As a result, we should probably always prefix component tools.
There was a problem hiding this comment.
Investigation Summary
The question asks whether unconditional prefixing (prefixing all component tools) is intentional or whether conditional prefixing (only prefix when collisions are detected) would be better.
Design Decision: Unconditional Prefixing (Intentional)
Yes, this was intentional. The current implementation prefixes all component tools unconditionally, and here's the reasoning:
Current behavior:
- Every component's tools get prefixed with its component ID (e.g., component_2dfb1a00__to_markdown)
- Single-component contexts pay a cost: +20 chars per tool name
- Multi-component contexts gain stability and clarity
Why Unconditional is Better Than Conditional
I found the analysis in ISSUE_1432_PROPOSAL.md, which addresses exactly this concern. The key tradeoffs:
Conditional Prefixing (proposed alternative) ❌
Pros:
- Shorter names in single-component cases (saves ~60 chars per component)
Cons:
- Non-deterministic naming: Same tool has different names depending on context
- Multi-turn fragility: Adding/removing components changes existing tool names
- Complex logic: Requires two-pass collision detection at tool registration time
- Observability risk: Tool names don't reliably indicate component source
- Breaking semantics: Existing tool name assumptions break when context changes
- Model confusion: LLM sees search in one turn, component_xyz__search in the next
Unconditional Prefixing (current) ✅
Pros:
- Deterministic: Same component always produces same tool names
- Multi-turn stable: Tool names never change across turns, even if context changes
- Simple logic: No upfront collision detection needed
- Clear observability: All component tools are identifiable and traceable
- Model consistency: LLM always sees the same tool names for the same component
Cons:
- Token cost: ~20 chars per tool × 3 tools × 1000 calls =
15k tokens ($0.01 at scale)
Token Cost Analysis
The per-component overhead is minimal:
component_2dfb1a00__to_markdown (26 chars)
vs
to_markdown (11 chars)
──────
Overhead: ~15 chars per tool
With 3 tools per component: ~45 chars/component/call
Context: This overhead is negligible compared to:
- Full model output (~2000 tokens)
- Tool descriptions and schemas
- Benefits from observability (ability to trace tool failures back to components)
- Stability across multi-turn interactions
Multi-Turn Stability (Critical for Agents)
With conditional prefixing, this scenario breaks:
Turn 1: [Database, Search] components → tools: [db.search, search.search]
Turn 2: [Database, Search, Cache] added → tools: [db.search, search.search, cache.search]
→ Now same tool has DIFFERENT NAME depending on turn order!
With unconditional prefixing, it's stable:
Turn 1: tools: [component_abc__search, component_def__search]
Turn 2: tools: [component_abc__search, component_def__search, component_ghi__search]
→ Same names, just added one more
Observability & Traceability
Current implementation enables:
- Tool failure linkage: Given prefixed name component_2dfb1a00__query, quickly identify which component owns it
- Telemetry: Metrics system extracts component ID from tool name for observability
- Multi-component debugging: Error messages can pinpoint which of N components failed
Recommendation
Keep unconditional prefixing. The design is sound because:
- Token cost is negligible (~$0.01 per 1000 calls)
- Multi-turn stability is critical for agents doing multi-turn reasoning
- Observability gains outweigh token overhead
- Simplicity beats optimization when overhead is minimal
- Determinism matters more than saving 45 chars per component
The only valid concern would be if there's a hard constraint on context length for specific models, but that would apply equally to both strategies (just affects budget allocation elsewhere).
| if name_mapping and tr.tool_name_mapping is None: | ||
| tr.tool_name_mapping = name_mapping |
There was a problem hiding this comment.
tr.tool_name_mapping is written here but never read anywhere in the codebase — convert_tools_to_json derives the JSON function name purely from the tools_dict key, not from this mapping. Same for tr.component_id/component_type/component_description (lines 399-401): tr is a fresh object returned by action.format_for_llm() on every call, so these mutations are discarded before the loop's next iteration even starts. This directly contradicts the PR description's claim that the mapping is "necessary … for JSON schema generation." Either wire these fields into something that reads them, or drop them along with the new TemplateRepresentation fields in base.py.
There was a problem hiding this comment.
These are removed.
| if tool.startswith("component_"): | ||
| parts = tool.split(".", 1) | ||
| if len(parts) == 2: | ||
| component_id = parts[0].replace("component_", "") | ||
| attributes["component_id"] = component_id |
There was a problem hiding this comment.
component_id is hex(id(action))[-8:] — effectively per-instance, per-process-run. Adding it as a counter attribute on mellea.tool.calls turns a bounded metric label into an unbounded one; on a long-lived server this is a cardinality-explosion pattern for whatever backend scrapes these metrics (Prometheus/OTLP). Worth reconsidering before this ships, independent of the naming-scheme fix above.
cc @ajbozarth — flagging given the telemetry/metrics work you've been driving, in case this cardinality concern is already covered by a convention I'm missing.
There was a problem hiding this comment.
I'll create a follow up issue when this PR is merged.
Answer: "Component Tracing via Spans/Logs" is NOT Implemented
No, the component tracing infrastructure is incomplete. The statement was aspirational, not accurate to the current state.
What Exists ✅
- OpenTelemetry tracing infrastructure: mellea/telemetry/tracing.py with full span support
- Tool span functions: start_tool_span(), finish_tool_span_success/error()
- Span attributes: tool name, type, description, arguments, execution time
- Export backends: OTLP and console exporters configured
What's Missing ❌
- No component_id extraction in spans
- No component_type in span attributes
- ModelToolCall doesn't store component metadata
- Span attributes don't parse component_id from tool name prefix
- No structured logging of component information
The Gap
When we removed component_id from metrics (correct decision), we said component tracing belongs in spans. But spans don't currently capture it:
Current tool spans:
gen_ai.tool.name: "component_abc123__search" ← component_id buried here
gen_ai.tool.type: "query"
mellea.tool.execution_time_ms: 123
What's missing:
mellea.component.id: "abc123" ← Not extracted from prefix
mellea.component.type: "Table" ← Not captured
The Problem
Information vacuum: Component identification is now:
- ❌ Not in metrics (we removed it)
- ❌ Not in spans (infrastructure doesn't extract it)
- ❌ Not in logs (no structured logging)
- ✅ In tool name prefix only (hidden, requires parsing)
Users cannot easily trace tool failures back to source components.
What Needs to Happen
To actually implement component tracing in spans, add to mellea/telemetry/tracing.py:start_tool_span():
def start_tool_span(tool_invocation_id, model_tool_call, *, is_control_flow, attach_context=True):
attrs = get_tool_call_attrs(model_tool_call)
# Extract component_id from tool name prefix
if model_tool_call.name.startswith("component_"):
parts = model_tool_call.name.split("__", 1)
if len(parts) == 2:
component_id = parts[0].replace("component_", "")
attrs["mellea.component.id"] = component_id
attrs["mellea.tool.is_control_flow"] = is_control_flow
return _start_application_span(...)
Recommendation
The metric removal was correct and necessary. But this gap should be:
- Documented — span attributes don't capture component info (yet)
- Tracked — create a follow-up issue to implement component attributes in spans
- Implemented — add extraction logic to start_tool_span() (5 lines of code)
|
|
||
| --- | ||
|
|
||
| ### `duplicate_tool_names_experiments.py` |
There was a problem hiding this comment.
This section documents duplicate_tool_names_experiments.py with full run instructions and "key findings," but the file doesn't exist in this PR (only duplicate_tool_names.py and pattern2_context_and_tools.py do). Please add the file or drop this section.
|
|
||
| --- | ||
|
|
||
| ### `telemetry_tool_calling_demo.py` |
There was a problem hiding this comment.
Same here — telemetry_tool_calling_demo.py is documented with run instructions and expected output but doesn't exist in this PR.
| from mellea.core.base import AbstractMelleaTool | ||
| from mellea.formatters import TemplateFormatter | ||
| from mellea.stdlib.context import ChatContext | ||
| from mellea.stdlib.functional import _call_tools |
There was a problem hiding this comment.
This imports the private mellea.stdlib.functional._call_tools directly, and it's used nowhere else outside functional.py itself (only transform() calls it internally, e.g. functional.py:469). Docs/examples are what people copy from, so this is teaching users to reach into an implementation detail. If the goal is just "call the LLM and run any tools it invokes," this example can go through m.transform(...) / act(..., tool_calls=True), which already does this internally. If there's a genuine need to trigger tool execution manually outside of transform/act, that's a gap worth closing with a public helper rather than exporting from examples via a private import.
There was a problem hiding this comment.
It is not true that the act() is calling _call_tools.
| Function | File | Line | Calls _call_tools? | Executes Tools? |
|---|---|---|---|---|
session.act() |
session.py | 482 | ❌ No | ❌ No |
mfuncs.act() |
functional.py | 89 | ❌ No | ❌ No |
aact() |
functional.py | 576 | ❌ No | ❌ No |
backend.generate_from_context() |
backends/*.py | - | ❌ No | ❌ No |
transform() |
functional.py | 426 | ✅ Yes (line 469) | ✅ Yes |
query() |
functional.py | - | ✅ Yes | ✅ Yes |
| React framework | frameworks/react.py | 123 | ✅ Yes | ✅ Yes |
| from mellea.core.base import AbstractMelleaTool | ||
| from mellea.formatters import TemplateFormatter | ||
| from mellea.stdlib.context import ChatContext | ||
| from mellea.stdlib.functional import _call_tools |
There was a problem hiding this comment.
Same private-import concern as in duplicate_tool_names.py:33 — see that comment.
| Attributes recorded: | ||
| - gen_ai.tool.name: Full tool name (semantic convention) | ||
| - status: Execution status | ||
| - component_id: Extracted from tool name if available (e.g., "203e1b50" from "component_203e1b50.query") |
There was a problem hiding this comment.
NIT: "Attributes recorded:" isn't a standard Google-style docstring section — worth using Note: or folding into the existing description instead, per AGENTS.md §5.
It's more an issue of us ensuring this approach works long term and is resilient. If we go this route, we need data / benchmarks to prove that this approach works. We also have no way to correlate the components that are being output as messages with the prefixed tools still. |
|
@jakelorocco This should be a bug fix. Can we leave this bug as is?
I don't have idea what data / benchmarks needs. Please advise. The component doesn't have name. How is it identified?
I wonder what this correlation is used for? Thanks! |
I think these things should be addressed before merging. I think potential issues related to the prefixes are large enough that we ought to ensure the approach works across a wide range of situations before committing to it. Tools that come from components usually relate to that component. So the fact that we don't have a way to match a tool prefix to its serialized component message seems problematic. The fact that we don't know if this holds up for more complicated examples where many components are in scope / being added. For instance, if we ask a model to transpose all charts, is it smart enough to make that same tool call multiple times with the different prefixes. Do we need to add language to a system prompt to explain how to use the component-prefixed tools? I'm asking these questions because this likely isn't the only way to implement a feature like this. For example, instead, we could force tool calls to require an object identifier as a parameter instead of appending multiple of the "same" tools. An approach like that might actually work better by potentially reducing the context required to encode all the tools. |
|
I'll hold off further PR review until we address the design questions Jake raised, feel free to loop me in if you have a call to discuss things |
Signed-off-by: Akihiko Kuroda <akihikokuroda2020@gmail.com>
Signed-off-by: Akihiko Kuroda <akihikokuroda2020@gmail.com>
Signed-off-by: Akihiko Kuroda <akihikokuroda2020@gmail.com>
|
Prototyped the "we could force tool calls to require an object identifier as a parameter" It hits an issue and here is the analysis. The issue is that only ONE tool is in the tools dict (['query']), so the LLM doesn't know there are two different "query" tools from different components. It just sees one tool and calls it. This reveals a fundamental problem with the parameter approach when tools have duplicate names:
With multiple components having the same tool name, we need a different strategy:
The parameter approach I implemented assumes one-to-one tool names. To truly support multiple components with identical tool names, we'd need either:
Here is the list-based implementation analysis: Tool Structure: Where It's Defined and UsedCurrent Structure1. DefinitionFile: class TemplateRepresentation:
tools: dict[str, AbstractMelleaTool] | None = NoneType:
2. Tool ExtractionFile: def add_tools_from_context_actions(
tools_dict: dict[str, AbstractMelleaTool],
ctx_actions: list[Component | CBlock | ModelOutputThunk] | None,
):
for action in ctx_actions:
tr = action.format_for_llm()
for tool_name, func in tr.tools.items():
wrapped_tool = wrap_tool_with_component_id(func, component_id)
tools_dict[tool_name] = wrapped_tool # ← OVERWRITES duplicates!3. Tool UsageFile: def convert_tools_to_json(tools: dict[str, AbstractMelleaTool]) -> list[dict]:
return [t.as_json_tool for t in tools.values()]Converts dict values to JSON array for LLM. 4. Backend Tool PassingExample: chat_response = self._async_client.chat(
model=self._model_id,
messages=conversation,
tools=[t.as_json_tool for t in tools.values()], # ← Dict values to list
...
)The Problem with Dict StructureWhen multiple components have the same tool name:
SolutionsOption A: Keep Dict, Add Prefixes (Current PR #1432 approach)tools_dict["component_abc123__query"] = tool_1
tools_dict["component_def456__query"] = tool_2Pros: Works, but tool names become verbose Option B: Change to List-Based Structure@dataclass
class TemplateRepresentation:
tools: list[tuple[str, AbstractMelleaTool]] | None = NonePros: Preserves all tools, natural names + parameters work Option C: Dict with Composite Keystools_dict[(component_id, "query")] = toolPros: No overwrites, clean structure Option D: Dict of Liststools_dict["query"] = [tool_1, tool_2, ...]Pros: Simple, preserves all tools Files That Need ChangesIf changing to list-based (Option B):
Current Parameter Approach Impact:
Recommended FixTo make parameter approach work with multiple tools: |
|
Makes sense; I was just trying to describe that there are multiple ways to do something like this. And this is a big enough change that we should test out multiple options and ensure the chosen approach works in many different scenarios. We don't quite have a process for that right now; but moving this to a discussion might help. We can also probably figure out what performance metrics we are looking for with a change like this. |
|
Discussion (#1455) is created. |
Pull Request
Issue
Fixes #95
Description
Testing
Attribution
Adding a new component, requirement, sampling strategy, or tool?
If your PR adds or modifies one of the types below, check the matching box. A checklist of type-specific review items will be posted as a comment.
NOTE: Please ensure you have an issue that has been acknowledged by a core contributor and routed you to open a pull request against this repository. Otherwise, please open an issue before continuing with this pull request.