Episode 9: The Architect – Designing Basic and Advanced LLM Workflows

(Narrator): Welcome to Chapter 9: LLM Workflows. In Chapter 8, we discussed conversational agency, where a human and an assistant collaborate through dialogue to accomplish tasks. While conversational agents equipped with tools can be quite capable within a domain, they may not be sufficient for complex workflows that require more structure. Chapter 9 moves beyond conversational agents to explore how LLMs can be used as components within defined workflows.
(Narrator): Basic LLM workflows are introduced as a way to handle more complex tasks. These workflows are relatively easy to reason about because they are composed of a finite set of tasks connected in a fixed pattern. This makes them dependable and easier to evaluate and optimize. However, a limitation of basic workflows is that they are rigid and cannot adapt to scenarios outside of their design.
(Narrator): Building a basic workflow involves several steps:
•
Define goal: Identify the purpose of the workflow and the desired outcome.
•
Specify tasks: Break down the workflow into a set of tasks that achieve the goal when executed in order. For LLM-based tasks, consider the tools each task will need and identify their inputs and outputs.
•
Implement tasks: Build each task, ensuring clear input and output definitions.
(Narrator): When specifying tasks, it's important to clearly define the inputs and outputs. This includes determining if inputs and outputs are structured (like defined schemas) or free-form text. An example uses schemas for describing a Shopify plug-in as input and the resulting email as output.
(Narrator): Implementing LLM-based tasks can be done using different approaches.
•
Templated prompt approach: This involves building a prompt template for the specific task, filling in inputs, and parsing the completion for outputs. This approach uses techniques like gathering, ranking, trimming, and assembling content, as discussed earlier in the book. Post-processing the completion is critical to extract the required output values.
•
Tool-based approach: For tasks that extract structured content from input (like scraping a webpage), models with tool-calling capabilities can be used. You define a tool with the desired output structure as arguments and set up the prompt to call this tool. The model will then call the tool with the structured information it extracts from the input.
(Narrator): If initial tasks don't produce high-quality results, you can add more sophistication. Techniques like chain-of-thought reasoning or ReAct, which encourage the model to "think out loud" before taking action, can significantly improve results. It's also noted that not all tasks need to use LLMs; mechanical tasks or those better suited to traditional software (like a web crawler or a BERT-based classifier) can be implemented without them, as they are often faster and more dependable.
(Narrator): Optimizing workflows can involve incorporating corrective feedback. This can happen at the task level (e.g., evaluating output and prompting for improvements) or workflow level (sending failed items back with details). Using offline harness tests before production and A/B testing in live traffic are important for evaluating and ensuring quality. Frameworks like DSPy and TextGrad can even use input-output examples to automatically optimize prompts based on a metric.
(Narrator): While basic workflows are useful, advanced LLM workflows are needed for scenarios where rigidity is a limitation and adaptation is required. These can involve more complex structures, such as stateful task agents that interact and update based on external events. Libraries like AutoGen can be used to build teams of conversational agents working together.
(Narrator): Chapter 9 illustrates how LLMs, fundamentally text completion engines, can be orchestrated within structured workflows to tackle complex problems, moving beyond simple conversational interactions towards more autonomous task execution.