aws orig error in aks environiment in otel auto instrumentation python setup

[tejaaa45]

What happened?

I'm setting up distributed tracing and end-to-end transaction monitoring for my services, which are hosted in an AKS (Azure Kubernetes Service) cluster and built using the Python framework. Since we use a microservices architecture, I'm implementing OpenTelemetry (OTel) with auto-instrumentation. This involves deploying the OTel Operator in AKS along with an OTel Collector and using Jaeger for tracing and ELK as backend storage

As part of the setup, I added the necessary Python-related annotations to enable tracing for four of my services. However, while I can see trace data for two of the services, the other two are not sending any data. Instead, I see an error message:

Could not find the ORIG_HANDLER or _HANDLER in the environment variables. This instrumentation requires the OpenTelemetry Lambda extension installed.

Since our environment is in Azure and not AWS, I don't understand why this error is appearing. What could be causing this issue, and how can I resolve it?

Steps to Reproduce

setup open telemetry by deploying otel operator in aks with this command

helm install opentelemetry-operator open-telemetry/opentelemetry-operator --namespace opentelemetry-operator --create-namespace --set "manager.collectorImage.repository=otel/opentelemetry-collector-k8s"
--set admissionWebhooks.certManager.enabled=false
--set admissionWebhooks.autoGenerateCert.enabled=true

and deploy otel collector and instrumentation.yaml wity the following

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
name: traces
spec:
mode: deployment
config:
receivers:
otlp:
protocols:
grpc: {}
http: {}
processors:
memory_limiter:
check_interval: 1s
limit_percentage: 75
spike_limit_percentage: 15
batch:
send_batch_size: 10000
timeout: 10s
exporters:
otlp:

endpoint: "jaeger-collector:4317"

endpoint: "jaeger-collector.observability.svc.cluster.local:4317"

endpoint: "jaeger-collector.jaeger.svc.cluster.local:4317"

tls:
insecure: true
debug: {}
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [debug, otlp]

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: otel-instrumentation
spec:
exporter:
endpoint: http://traces-collector.opentelemetry-operator.svc.cluster.local:4317/
propagators:

• baggage
• b3
  sampler:
  type: parentbased_traceidratio
  argument: "1"
  python:
  env:
• name: OTEL_EXPORTER_OTLP_ENDPOINT
  value: http://traces-collector.opentelemetry-operator.svc.cluster.local:4318/v1/traces
• name: OTEL_PYTHON_DISABLED_INSTRUMENTATIONS
  value: aws-lambda,metrics
• name: OTEL_METRICS_EXPORTER
  value: none
• name: OTEL_LOG_LEVEL
  value: "debug"
  image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest

Expected Result

need to get traces exported from the services of aks to the jaeger ui backend but not able to get that

Actual Result

i'm getting this error

Could not find the ORIG_HANDLER or _HANDLER in the environment variables. This instrumentation requires the OpenTelemetry Lambda extension installed.

this is aws labda error which is not related to aks or azure but i'm getting this error in the logs of services which i auto instrumenetd with these annotations

instrumentation.opentelemetry.io/inject-python:true
instrumentation.opentelemetry.io/otel-python-platform:musl
instrumentation.opentelemetry.io/container-names

Additional context

No response

Would you like to implement a fix?

None

[tejaaa45]

@swiatekm @TylerHelmuth @opentelemetrybot @pmcollins can any of you guys please help me on this?
it's very important for us!

[xrmx]

Are you installing the aws-lambda instrumentation? If you are using the operator it's a yes. Also it's just a log so it won't mess with the rest.

[tejaaa45]

Are you installing the aws-lambda instrumentation? If you are using the operator it's a yes. Also it's just a log so it won't mess with the rest.

@xrmx i'm not explictly installing aws-lambda instrumentation i'm deploying otel operator,will it automatically installs aws-lambda instrumentation, if yes,then what should i do now i'm getting aws error in azure aks environiment i think it showing error because traces are not exporting to jaeger with this and if i added this envi variable -- - name: OTEL_PYTHON_DISABLED_INSTRUMENTATIONS
value: aws-lambda-- and then the error log is gone but still traces are not exporting what should i do ?? could you please help me on this ??

[xrmx]

@tejaaa45 I'm trying to tell you that the aws-lambda message is harmless and unrelated to whatever other problem you have with your setup.

[tejaaa45]

@tejaaa45 I'm trying to tell you that the aws-lambda message is harmless and unrelated to whatever other problem you have with your setup.

@xrmx but traces are not exporting to jaeger is that not due to this aws-lambda error ?

[tejaaa45]

okay @xrmx then not sure why i'm not getting traces exported to jaeger we have python fastapi bulit services deployed in aks and i have used these instrumentation
instrumentation.opentelemetry.io/inject-python:true
instrumentation.opentelemetry.io/otel-python-platform:musl
instrumentation.opentelemetry.io/container-names
these are enough to get the traces through otel auto instrumentation right?? or am i missing anything in my setup
?