Description
Agent Observability & Tracing: Debug Production LLM Systems. This course is a comprehensive and specialized program that teaches software and artificial intelligence engineers how to use tracing and observability tools to troubleshoot and resolve errors and incorrect behavior of intelligent agent systems and language models in a real environment. The course “Agent Observability and Tracing: Troubleshooting Language Model Systems in the Production Environment” is a completely practical and measurement-based guide to solving complex intelligent agent problems. In many cases, intelligent agents provide incorrect or erroneous answers while all initial tools and tests report the system state as completely healthy and error-free. By providing 71 dedicated training sessions, this course aims to bridge the gap between initial assessments and the actual behavior of the system in the production environment, and all data and numbers presented in it have been measured based on real systems. During this course, learners will implement a real-world intelligent agent capable of calling various tools such as search, fetch, calculation, and summarization, and track it with OpenTelemetry. They will also deploy two trace management platforms, Aries Phoenix and LangFuse, using Docker, and compare them on the same data. The training program will examine silent errors; errors that do not generate any error messages but cause the agent to behave incorrectly, such as a search tool that returns empty results or documents that contain outdated data. In addition, learners will encounter real-world security and technical incident scenarios and learn how to detect false positives, aberrant signals, and prompt injection attacks, calculate costs and delays per part, and implement observability structures professionally in development teams.
What you will learn
- Equipping a real intelligent agent with open telemetry tools to examine structures, properties, events, and context transitions
- Setting up Aries Phoenix and Langfuse platforms using Docker and comparing their performance on the same traffic
- Identify and discover silent errors without error messages such as empty results, hidden retries, and incorrect responses
- Calculate latency and cost per section, per feature, and per user and understand why quality degradation is hidden in averages
- Building an evaluative language model on live traffic and connecting the evaluations to tracking segments
- Detecting prompt injection attacks and degraded response quality from tracking patterns
- Troubleshoot unknown traffic under time pressure and match user reports with system execution flow
- Complete management of a real multi-factor incident including planted errors, deviant signaling, and post-incident reporting
- Accurately calculate large-scale observability costs and make scientific decisions for traffic sampling
- Implementing and deploying an observability system within the team by defining ownership, reviewing codes, and creating implementation guidelines
This course is suitable for people who:
- Engineers who have deployed systems that rely on language models and are looking for the exact cause of unknown errors
- People whose intelligent agents are successful in initial tests but misbehave in the real environment
- Support and infrastructure engineers who are familiar with performance management tools but have not tracked complex agent traffic
- Machine learning and artificial intelligence engineers who write evaluation systems and look for live traffic analysis
Course details Agent Observability & Tracing: Debug Production LLM Systems
- Publisher: Udemy
- Instructor: Frank Robotics Lab
- Training level: Beginner to advanced
- Training duration: 7 hours and 35 minutes
- Number of lessons: 71
Course topics
Prerequisites for the Agent Observability & Tracing: Debug Production LLM Systems course
- Comfortable Python: functions, classes, dictionaries, reading a traceback
- Any OpenAI-compatible API key. Total spend across the whole course is a few dollars
- Docker and Docker Compose installed, for the two tracing backends
- No prior OpenTelemetry or observability experience needed. It is built from the first span
Course images
Sample course video
Installation Guide
After Extract, view with your favorite player.
Subtitles: None
Quality: 1080p
Download link
Rapidgator link
File(s) password: www.downloadly.ir
File size
3.9 GB