Introduction
The scaffolding layer that developers once needed to ship LLM applications—indexing layers, query engines, retrieval pipelines, and carefully orchestrated agent loops—is collapsing. According to Jerry Liu, co-founder and CEO of LlamaIndex, this is not a problem but rather a pivotal shift. As models become capable of reasoning over massive unstructured data, self-correcting, and performing multi-step planning, the need for heavy frameworks diminishes. Instead, context emerges as the real differentiator. This guide will walk you through the steps to adapt your approach, building powerful LLM applications without relying on collapsing scaffolding.
What You Need
- Basic understanding of LLMs and retrieval-augmented generation (RAG)
- Access to a modern LLM (e.g., Claude, GPT-4, or open-source equivalent)
- A coding environment (e.g., Python, Node.js) with ability to use AI coding assistants
- Familiarity with document formats (PDF, images, etc.) and parsing techniques
- Optional: Model Context Protocol (MCP) tools or similar plug-in systems
Step-by-Step Guide
Step 1: Understand the Shift from Frameworks to Context
Recognize that traditional deterministic workflows are no longer necessary. LLMs now possess incremental reasoning capabilities that allow them to handle complex tasks without extensive manual orchestration. As Liu notes, agent patterns have consolidated toward a managed agent diagram—a simple harness layer combined with tools, MCP connectors, and skills plugins. Your first step is to unlearn the old framework-dependent mindset and embrace a context-first approach.
Step 2: Prioritize High-Quality Context Extraction
Context becomes your moat. Focus on accurately extracting data from diverse file formats—PDFs, images, scanned documents—using agentic document processing with optical character recognition (OCR). LlamaIndex has invested heavily in this area because “the thing that they all need is context,” says Liu. Invest in parsing that provides higher accuracy and cheaper extraction. The better your context, the better your agent’s performance, regardless of the underlying model.
Step 3: Leverage Natural Language as Your Programming Language
With coding agents like Claude Code or OpenAI Codex, you can interact using plain English instead of writing elaborate code. Liu reveals that about 95% of LlamaIndex code is now AI-generated. “Engineers are not actually writing real code—they’re all typing in natural language.” Start describing your retrieval logic, integration steps, or parsing rules in natural language prompts. This collapses the barrier between programmers and non-programmers, making development faster and more accessible.
Step 4: Adopt a Managed Agent Diagram Approach
Instead of building custom orchestration for every workflow, use a managed agent harness that combines tools, MCP connectors, and skills plugins. This modular setup allows models to discover and use tools without requiring separate integrations for each one. Configure your agent to self-correct and perform multi-step planning autonomously. The goal is to reduce manual composition of deterministic pipelines and let the model handle the orchestration.
Step 5: Use Simple Primitives for Retrieval
Liu points out that it’s now “way easier for people to build even relatively advanced retrieval with extremely simple primitives.” You no longer need complex indexing layers or query engines. Start with basic retrieval methods—vector search, keyword matching—and rely on the model’s reasoning to refine results. Three years ago, these simple approaches would break the agent; today, they work effectively because the models can self-correct and understand context deeply.
Step 6: Keep Your Stack Modular to Avoid Lock-In
While frameworks like LlamaIndex are still useful, the trend is toward modularity. Use open protocols like MCP to connect tools and data sources independently. Avoid deep integration with any single vendor. This ensures you can switch between models (e.g., Claude vs. GPT) without rebuilding your entire application. Context and data connections should be portable, not tied to a specific orchestration layer.
Step 7: Test, Iterate, and Rely on AI Feedback
Because development is now faster, you can iterate rapidly. Use prompt engineering and AI-generated code to build prototypes, test them with real data, and refine based on output. The collapse of scaffolding means you can fail fast and cheaply. Liu encourages leveraging the model’s ability to reason over large datasets to spot errors and improve accuracy—treat the AI as both a developer and a tester.
Tips for Success
- Context over complexity: The single biggest differentiator is the quality of context you feed the agent. Invest in parsing and extraction, not in elaborate pipelines.
- Embrace natural language: The new programming language is English. Train your team to think in prompts, not code.
- Use agentic document processing: For files stuck in PDFs, images, or proprietary formats, apply OCR and structured extraction to unlock data.
- Stay vendor-agnostic: Adopt MCP and other open protocols to future-proof your stack.
- Monitor model improvements: Each LLM release expands reasoning capabilities; adjust your approach accordingly—you may need even less scaffolding over time.
- Safety and alignment: As agents become autonomous, ensure you have guardrails for self-correction and tool usage to prevent errors.