Bug: metrics.set_default_dimensions emits PowertoolsUserWarning: Dimension 'Key' has already been added. The previous value will be overwritten.

[ericbn]

Expected Behaviour

When adding new default dimensions to a Metrics instance using metrics.set_default_dimensions(Key=value) there should be no WARNING logs if they were not set before.

Current Behaviour

When adding new default dimensions to a Metrics instance using metrics.set_default_dimensions(Key=value) there are two WARNING logs being emitted:

[WARNING] aws_lambda_powertools/metrics/metrics.py:124: PowertoolsUserWarning: Dimension 'Key' has already been added. The previous value will be overwritten.
  self.provider.add_dimension(name=name, value=value)

[WARNING] aws_lambda_powertools/metrics/provider/cloudwatch_emf/cloudwatch.py:571: PowertoolsUserWarning: Dimension 'Key' has already been added. The previous value will be overwritten.
  self.add_dimension(name, value)

Code snippet

metrics = Metrics()
    metrics.set_default_dimensions(Environment=os.environ["STAGE"], Resource="MyJob")
    metrics.add_metric(name="MyJobDuration", unit=MetricUnit.Seconds, value=42)
    metrics.flush_metrics()

Possible Solution

powertools-lambda-python/aws_lambda_powertools/metrics/metrics.py

Lines 176 to 201 in 8db13c7

	def set_default_dimensions(self, **dimensions) -> None:
	self.provider.set_default_dimensions(**dimensions)
	"""Persist dimensions across Lambda invocations
	Parameters
	----------
	dimensions : dict[str, Any], optional
	metric dimensions as key=value
	Example
	-------
	**Sets some default dimensions that will always be present across metrics and invocations**
	from aws_lambda_powertools import Metrics
	metrics = Metrics(namespace="ServerlessAirline", service="payment")
	metrics.set_default_dimensions(environment="demo", another="one")
	@metrics.log_metrics()
	def lambda_handler():
	return True
	"""
	for name, value in dimensions.items():
	self.add_dimension(name, value)
	self.default_dimensions.update(**dimensions)

In the Metrics.set_default_dimensions method, these two code snippets seem to be doing duplicated work:

        self.provider.set_default_dimensions(**dimensions)

and

        for name, value in dimensions.items():
            self.add_dimension(name, value)

as both end up calling add_dimension in the provider, which will happen twice. The solution is not totally clear to me as I didn't go deeper into understanding the self.dimension_set, self.default_dimensions, self.provider.dimension_set and self.provided.default_dimensions relationship. Does it make sense to have these structures duplicated in self and self.provider?

Steps to Reproduce

1. Set the following environment variables in the Lambda function:

   POWERTOOLS_METRICS_NAMESPACE: MyNamespace
   POWERTOOLS_PARAMETERS_MAX_AGE: 900
   POWERTOOLS_PARAMETERS_SSM_DECRYPT: true
   POWERTOOLS_SERVICE_NAME: MyService
   STAGE: dev
   

2. Add the code snipped to the Lambda function code.
3. Run the Lambda function.
4. Observe the logs containing the WARNING entries for the Environment and Resource keys.

Powertools for AWS Lambda (Python) version

3.34.0

AWS Lambda function runtime

3.14

Packaging format used

Lambda Layers

Debugging logs

[vishwakt]

Confirmed on latest develop, no Lambda needed: calling metrics.set_default_dimensions(Environment="dev", Resource="MyJob") on a fresh Metrics instance emits one warning per dimension, and a second call emits two per dimension.

Root cause is the duplicated work suspected above. Metrics.set_default_dimensions first calls self.provider.set_default_dimensions(**dimensions), which adds each dimension to dimension_set and records it in default_dimensions. It then loops self.add_dimension(name, value) over the same keys. At that point every key is already present, so the check in AmazonCloudWatchEMFProvider.add_dimension warns "already been added" even when nothing was set before.

The two warning locations in the report correspond to cold vs warm invocations. On the first call only the redundant Metrics-level loop warns (metrics.py:124). On later calls, with the keys already in default_dimensions, the provider's internal loop warns as well (cloudwatch.py:571).

Two related observations:

• With the default provider, self.default_dimensions.update(**dimensions) in Metrics.set_default_dimensions is also redundant: Metrics.default_dimensions and provider.default_dimensions are the same dict object, wired together in __init__.
• The docstring in Metrics.set_default_dimensions sits after the first statement, so Python does not treat it as a docstring.

Happy to pick this up. The minimal fix is to drop the redundant add_dimension loop in Metrics.set_default_dimensions and delegate to the provider. One question on intended semantics: should calling set_default_dimensions again with the same keys (warm starts) still warn about overwriting, or should re-registering default dimensions be silent?

[vishwakt]

Confirmed on latest develop (8db13c7), no Lambda needed: calling metrics.set_default_dimensions(...) on a fresh instance emits one warning per dimension. It is also worse than reported: with defaults set once at module level, the recommended pattern, the warning fires for every default dimension on every flush, because clear_metrics re-registers defaults through the same path.

Two causes:

1. The duplicated work suspected above. Metrics.set_default_dimensions calls provider.set_default_dimensions(**dimensions) and then re-adds the same keys through add_dimension, so the second pass always finds them registered and warns. This is the metrics.py:124 entry in your logs.
2. The warning condition in AmazonCloudWatchEMFProvider.add_dimension fires for any name already present in dimension_set or default_dimensions, so the internal re-registration paths (clear_metrics after each flush, repeated set_default_dimensions on warm invocations) warn as well. This is the cloudwatch.py:571 entry in your logs.

Opened #8403 with a fix: remove the redundant loop, and warn only when a dimension is overwritten with a different value, matching the warning message and the intent of #5653. Re-registering unchanged defaults is now silent; changing the value of an existing dimension still warns once.

[ericbn]

To add to the analysis:

I see there's another redundant call: self.set_default_dimensions(**self.default_dimensions) in the clear_metrics method in the provider, which causes the add_dimension method to be called a second time.

Also, I think it might be worth investigating this whole code better. I don't get why Metrics keeps its own data and I don't understand the explanation in the comment in

powertools-lambda-python/aws_lambda_powertools/metrics/metrics.py

Lines 77 to 81 in 8db13c7

	# NOTE: We use class attrs to share metrics data across instances
	# this allows customers to initialize Metrics() throughout their code base (and middlewares)
	# and not get caught by accident with metrics data loss, or data deduplication
	# e.g., m1 and m2 add metric ProductCreated, however m1 has 'version' dimension but m2 doesn't
	# Result: ProductCreated is created twice as we now have 2 different EMF blobs

There's even a side effect here: since Metrics passes references of all its data structures to the provider, they're both (the Metrics and the provider) sharing the same instances of data. In this case mutating them from Metrics and from the provider is also redundant. And this does not happen if a Metrics is initialized with a provider instance given in the constructor by the user. So there's a lot of confusing complexity here, I'd say most of it possibly unintentional.

(Disclaimer: this was all reviewed and written without the use of AI)

[vishwakt]

@ericbn Replied in detail on the PR to keep the discussion in one place: #8403 (comment)

[ericbn]

Here's my take on this: #8404