There is a shift playing out in enterprise AI, and most roadmaps remain too focused on models rather than implementation. Foundation models are widely available, including GPT-5, Claude, Gemini, Mistral, and others. Companies can access them through APIs and platforms with relatively little effort. The difficult work is integrating those models into the workflows and systems that run the business, in ways that create measurable value and improve outcomes.
Put simply: the LLM is the interface. Integration is the product. Finally, efficient integration is the business.
Enterprise AI is moving into core business systems
Many enterprise AI applications can already read, summarize, and answer questions. The current push is toward agents and automated workflows that connect activity across CRM, call-center, R&D, and other operating systems. Enterprises expect the next gains to come from this shift, but it also makes integration considerably harder.
Those systems were rarely designed to support modern AI. Companies are extending older integration patterns into faster, more dynamic workflows that cross several systems and require reliable execution. The result is often brittle automation that produces useful analysis but cannot carry a business process through from beginning to end.
The surrounding architecture often determines whether enterprise AI works
Inside large organizations, model selection, fine-tuning, RAG design, and evaluation still matter. Some implementations underperform because they use the wrong model or fail to adapt it to the specific use case. However, the barriers that prevent enterprise AI from scaling or operating reliably often are determined by the surrounding architecture:
- Legacy systems with limited or unusable APIs. The system of record cannot be accessed in the way a modern workflow requires.
- Data scattered across ERPs, CRMs, and departmental platforms, with incompatible formats and inconsistent metadata.
- Security controls that interrupt context and execution. The controls are necessary, but they were rarely designed for AI-driven workflows.
- No reliable real-time access to core systems. Many organizations still depend on batch jobs and aging middleware.
- AI permitted to analyze but not execute. Testing, permissions, governance, monitoring, rollback, and model quality may all remain unresolved.
- Costs that rise quickly at scale. Token usage, repeated retrieval, inefficient orchestration, and poorly designed workflows can make an otherwise useful system uneconomical.
Many of the GenAI failures I am called into involve both model and architecture issues, but the architecture often becomes the limiting factor. Executives want automation and decision support. Delivering either requires reliable access to systems of record, production-grade controls, and integration engineered around the business process. Prompt design represents only one part of that work.
Token usage becomes an architecture issue at scale
The excitement around AI agents and workflow orchestration makes sense. Every step involving an LLM call consumes tokens, and costs can rise quickly as workflows add agents, longer contexts, retries, branching, or greater autonomy. Without clear limits and monitoring, usage becomes difficult to predict and control.
A workflow may appear inexpensive during a limited rollout with a small number of users and transactions. The economics can change quickly when the same workflow supports hundreds or thousands of users, carries larger context windows, retries failed steps, and makes several model calls for each transaction. By the time the application reaches production volume, token usage can exceed the original budget even though the workflow itself has not changed substantially.
Efficient architectures use smaller or specialized models where appropriate, discriminative models when generation is unnecessary, caching, routing, bounded context, and firm limits on retries and agent autonomy. They also measure token usage by workflow and transaction before production deployment. Integration is the product. Efficient integration is the business.
Where leaders should focus
If your AI initiatives are stalling, start by examining the integration layer and the model together:
- Modernize API access for the systems that matter most.
- Build a reliable data layer with consistent metadata so high-value data can be reused across applications and workflows.
- Integrate security and compliance into AI development from the start rather than adding them near deployment.
- Use hybrid model patterns to control cost, and fine-tune models with proprietary data when the use case supports it. I have built these models in biopharma, healthcare, and telecom, and the right approach can materially change both performance and economics.
- Design AI with the permissions, controls, and integrations required to take action. Analysis has value, but many of the largest gains depend on connecting AI to operational workflows and systems of record.
Organizations making meaningful progress are improving the architecture around their models while continuing to evaluate model quality, retrieval, fine-tuning, and cost. This engineering work determines how reliably enterprise AI can operate at scale.