Mobile Core Engineer

About Telnyx

Telnyx leads innovation in global connectivity by developing a private worldwide multi-cloud IP network and delivering edge technology through user-friendly APIs. We aim to modernize outdated systems, automate manual processes, and solve real connectivity challenges with pioneering solutions. As a financially strong and profitable firm, we reinvest in technology and foster team growth. We envision enabling limitless innovation through borderless connectivity, inviting passionate individuals to join our journey.

Role Overview

You will join the Core Wireless Squad, an eleven-member team managing and enhancing Telnyx's mobile core EPC infrastructure across multiple data centers with diverse roaming partners and vendor platforms. This role focuses on hands-on operational tasks involving in-depth protocol-level troubleshooting of subscriber issues and practical platform management.

Key Responsibilities

• Diagnose subscriber connection failures involving protocols such as Diameter (S6a), MAP (SS7), GTP-C, and GTP-U.
• Analyze pcap trace captures from DRA, STP, and GTP proxies to identify signaling problems.
• Track specific IMSI sessions across partner networks and protocols to locate session interruptions.
• Troubleshoot routing discrepancies in DNN/APN configurations, IMSI-pinned APN issues, and private wireless gateway settings.
• Manage end-to-end customer SIM connectivity investigations from report through resolution.
• Administer and resolve issues in EPC components including HSS/HLR, DRAs (utilizing freeDiameter with custom Python transforms), STPs, PGWs, SMFs, and GTP proxies (Go).
• Operate configuration tools based on etcd for steering/blocking such as forbidden PLMNs and 3G restrictions.
• Maintain the GTP proxy routing database, including management of IMSI groups and cache invalidation.
• Engage in vendor management by coordinating escalations and resolutions with vendors and roaming partner NOCs.
• Support roaming agreements and sponsored roaming partnerships by investigating registration failures, NRTRDE gaps, and routing issues.
• Collaborate with partner NOCs on protocol incidents, including exchanging pcap data and joint root-cause analysis.
• Participate in a weekly on-call rotation to handle alert responses related to various EPC failures and incidents.
• Lead incident resolution, coordinating efforts among internal teams and external vendors, and document incident learnings.
• Enhance and utilize pcap analysis and protocol decoding tools to streamline troubleshooting.
• Contribute to configuration management and automation tasks using Ansible and Kubernetes for EPC infrastructure.
• Improve observability through Grafana dashboards, alerts, and detailed protocol metrics.
• Make minor code modifications as needed to fix or diagnose issues, while primarily focusing on operations.

Qualifications and Experience

• Minimum three years of experience operating or engineering telecom infrastructure such as EPC, HSS/HLR, DRA, STP, PGW, or SMF.
• Strong expertise in mobile core signaling protocols including Diameter (S6a, S6d, Gx), SS7/MAP/TCAP, GTPv2-C, and GTP-U.
• Proficiency in interpreting pcap traces using tools like tshark/Wireshark, filtering by IMSI/AVP/transaction, and analyzing protocol hierarchies.
• Solid Linux system administration skills with experience in infrastructure configuration and automation (e.g., Ansible).
• Ability to manage vendor relationships and collaborate with external roaming network operations centers to resolve issues.

Additional Skills (Preferred)

• Ability to interpret code and scripts to track signaling flows.
• Experience with scripting/programming languages such as Python, Elixir, or Go for developing diagnostic tools.
• Familiarity with freeDiameter and custom Attribute-Value Pair (AVP) transforms.
• Knowledge of Kubernetes and containerized EPC deployments.
• Understanding of roaming agreements, including NRTRDE, TAP, sponsored roaming, and ORS blacklists.
• Experience with multi-IMSI SIM applets and over-the-air provisioning.
• Working knowledge of etcd-based configuration systems.
• Familiarity with relevant 3GPP standards (TS 29.272, 29.060, 29.244, 29.002) for analyzing protocol-level issues.