The Structural Problem of AI Agents That Cannot Prevent Token Explosion — and "ORBIT," a Custom Harness Aiming to Reduce Costs
In recent years, AI coding agents such as Claude Code, Cline, and Kilo Code have been gradually penetrating development environments. However, when I look at the business models adopted by these commercial tools, as an engineer, I cannot help but sense a certain "unhealthiness."
In this article, I first point out the structural distortion of incentives in current AI coding agents, and then explain the architecture and design philosophy of a proprietary agent harness called "ORBIT (Operational Runtime Bridge for Iterative Tasks)" — a project I am developing personally to counter this situation.
1. The Trap of the API Reseller Business Model: Why "Token Efficiency" Does Not Improve
Many of today’s AI coding tools adopt a business model in which they rebrand and resell APIs from cloud LLMs (Claude Opus, ChatGPT, GLM 5.2, Kimi K2.7 Code, etc.) under their own brand, profiting from the margins generated by bulk purchasing — or a tiered subscription model (fixed monthly fee + pay-as-you-go for usage exceeding the cap).
At first glance, this appears to offer convenience to users, but this model has a fatal problem: there is no incentive on the developer side to conserve tokens.
- In the pay-as-you-go model: The more tokens users consume like running water, the higher the vendor’s revenue. Since profits increase when the tool feeds unnecessarily long contexts or spins trial-and-error loops, there is no motivation for the tool side to optimize prompts or compress context.
- In the tiered subscription model: The optimal profit strategy is to ensure users do not work too comfortably — hitting usage limits (such as the Fast request cap) at precisely the right moments — thereby nudging them to upgrade to a higher-tier plan.
In other words, the vendor’s interests and the user’s interests (obtaining high-accuracy code with minimal tokens) are in direct opposition.
Furthermore, how many tokens are being consumed behind the scenes is a black box. Users have no way to verify whether the API cost incurred for a given task is truly justified.
2. User-Side Self-Defense: A Hybrid of Local LLM and Cloud LLM
To escape this unhealthy structure, one strategy available to users is to build a custom harness (execution framework) that performs context preprocessing locally and sends only the minimal prompt to the cloud.
Currently, I am developing a proprietary harness called "ORBIT α2.0" that uses a combination of a local LLM and a cloud LLM. The central theme of ORBIT is: "How much can we leverage a low-cost but less powerful local LLM (30B class) while keeping costs down and still reaching a solution?"
However, when attempting to run a 30B-class local LLM autonomously, you run into the following walls:
- Context Overload: Exceeding the 16K limit renders the model unable to fix even a single-character typo.
- SEARCH/REPLACE Collapse: The LLM hallucinates, fails to accurately reproduce the target code, and no edits are applied at all.
- Lack of State Management: The model cannot manage which parts of the overall task are complete and which have failed, falling into infinite loops at the same location.
The approach of using a local LLM with a single prompt for end-to-end generation broke down quickly. In response, ORBIT fundamentally shifted its architecture from "capability-dependent" (relying on the LLM) to "process-dependent" (the system orchestrates).
3. ORBIT’s Architecture: Maximizing the Local LLM Through Process
ORBIT decomposes tasks into single-file units (under 200 lines) and controls the dependency graph (DAG) on the system side. The LLM is instructed to sequentially perform the task of "creating or fixing one given file." The core is a six-layer structure:
① Planner Layer & Scheduler Layer
Automatically generates a skeletal DAG (dependency graph) from spec.json. The Scheduler prioritizes executing "high-risk tasks" (those involving external API dependencies, async processing, etc.) among tasks whose dependencies have been resolved. This allows costly backtracking tasks to be handled first, preventing wasteful retries.
② Executor Layer & Contract Validator
Instructs Gemma 4 (the local LLM) to generate or fix a single target file. At this point, the entire application context is not provided — only the task instructions and the type definitions of dependency files are injected.
Additionally, before running a build, contract validation via AST analysis is performed. This allows early detection of integration failures such as "the file was generated but no functions are defined," skipping unnecessary test executions.
③ Controller Layer & Memory Layer
Handles "differential rollback" and "retry control" upon failure. Before task execution, a snapshot of the target file is saved; upon failure, only that part is restored and retried from a clean state.
Furthermore, failure patterns and successful fix patterns are automatically accumulated in failure_db.jsonl for reuse in subsequent similar errors.
④ Escalation Layer: Cost Suppression Through Gradual Retries
When the local LLM cannot solve a problem, rather than immediately granting full authority to the cloud LLM, a gradual retry within Level 1 is performed with finer granularity:
- L1.0: Standard retry by Gemma 4
- L1.5: DeepSeek V4 Pro summarizes the StackOverflow DB QA (described later) and injects it into Gemma 4
- L1.7: GLM-5.2 (cloud) generates a fix directionality from web search and the StackOverflow DB, and injects it into Gemma 4
- L1.8: Harness-driven perspective-shift prompt for Gemma 4 to retry
- L2/L3: Upon reaching limits, GLM-5.2 performs task splitting or DAG restructuring
By consistently having the local LLM perform the core code execution and using the cloud LLM only as "assistance," the explosion of API costs is prevented.
4. StackOverflow QA DB Integration as Search-Based RAG
One area I focused on particularly was building a RAG that functions even in offline environments and under the 16K context limit. From a 99 GB StackOverflow Data Dump, I extracted approximately 860,000 high-quality QA pairs and built them as a locally hosted SQLite database (approximately 5.4 GB) with FTS5 full-text search.
When an error occurs, a pinpoint search is performed against this database using the error message or API name, and a summary snippet is injected into the local LLM’s context. This enables obtaining stable error-resolution hints without any network connection.
5. The Harness Is the "Vehicle," and the LLM Is the "Engine"
While developing a harness like ORBIT from scratch, I honestly have moments of doubt — "Am I just building a rule-based debugger?" or "Won’t local LLM performance catch up before this reaches a practical level?" Indeed, the current 30B class has limitations when it comes to autonomously running the build-test-debug loop.
However, I am convinced that this approach represents the future of AI coding.
The harness is the "vehicle," and the LLM is the "engine." Even if the engine is underpowered, the design of the vehicle’s suspension, brakes, and navigation (DAG management, rollback, RAG) is not wasted. When a better engine becomes available, simply swapping the module will allow us to reach the goal faster and with fewer tokens. I don’t know how many years it will take, but as local LLM performance improves, hybrid harnesses will inevitably reach a practical level.
On the other hand, no matter how intelligent an LLM becomes, loading a codebase exceeding 1,000 lines all at once will compress the context and cause API costs to skyrocket. The "process-dependent" approach — decomposing tasks into single files and passing only dependency contracts — retains its essential value: the smarter the LLM becomes, the more efficiently it can output correct answers.
I am technically a member of the engineering community, but having spent many years on the system planning side, I expect ORBIT itself will likely end at the "plaything" level. I would be delighted if excellent developers are inspired by my prototype and use it even slightly as a reference for next-generation harness development.
6. Conclusion: The Need for Cost-Performance Benchmarks
In the current AI coding agent community, only "whether the task was solved (Accuracy)" gets attention, while efficiency — "how much cost and time were spent to solve it" — is ignored. Under these conditions, a "power play" (using tokens like running water) that is convenient for vendors goes unchecked.
Going forward, for users to defend themselves, I believe an indispensable metric is something like a "harness benchmark" that compares the token cost required to solve the same task using the same model.
Not being at the mercy of the API reseller business model, but rather extracting high accuracy with as few tokens as possible. The architecture that pursues this "ultimate in cost efficiency" is, I believe, the form of the tool that the next generation of developers truly desires.