Skip to content

Debugging Probe Failure

🕵️‍♂️ Troubleshooting Probe Failures in US

Probe failures in the US environment typically indicate IL SIM card validation issues. Follow these steps to systematically diagnose and resolve probe failures.

1️⃣ Initial Detection

Probe Failure Detected

A probe failure usually means the IL SIM card validation did not succeed.

2️⃣ Investigate via Grafana Dashboards

  • US Dashboard: US Unibeam Dashboard

  • Open the US dashboard and navigate to Unibeam → PROBE.

  • Locate failed probe logs and expand them for details.

3️⃣ Extract Request ID

Find the requestId from the error log. Example:

[ERROR] 2025-10-19 16:09:06.538 [c88fe1973d3f7511cbdf0f1ef37df692] [83bf0eec0a74f840] [lettuce-epollEventLoop-5-1] c.u.m.p.s.ProbeService:184 - Reason: Unexpected callback status: NOT_AVAILABLE

Log Fields

The requestId is the second bracketed value in the log entry.

4️⃣ Cross-Reference in IL Dashboard

Search for the requestId in sia-service logs.

5️⃣ Trace Transaction Across Services

Once you have the transactionId (first field in the log):

  • Search for it in sia-service, sim-service, and sms-service logs.

If no errors are found, continue with:

  • Search for sequenceId in sms-service logs.
  • Search for smscMessageId in sms-service logs.

6️⃣ Analyze Delivery Receipt Status

Delivery Receipt Status

The delivery receipt status in the logs provides insight into the root cause of the probe failure.

📝 Summary Table

Step Action Tool/Service
1 Detect failure US Grafana Dashboard
2 Find requestId US Probe logs
3 Cross-reference IL Grafana Dashboard
4 Trace transactionId sia-service, sim-service, sms-service
5 Search sequenceId/smscMessageId sms-service
6 Review delivery receipt status sms-service logs

Further Investigation

If the above steps do not reveal the issue, escalate to the engineering team with all gathered IDs and log excerpts.