The AI Black Box: Why Claude, Gemini, GPT and Others Still Can’t Explain Themselves
- Balaji Anbil
- Jun 26
- 4 min read
Updated: Jun 30
Introduction
AI systems are becoming more fluent, more useful, and more embedded in everyday work. But beneath that surface lies a real problem: we still can’t explain how they actually arrive at their answers.
People often assume tools like GPT-4, Claude, Gemini, or Perplexity are reasoning systems. In reality, they’re advanced pattern-completers. They generate plausible responses, not logic. And if you ask why they reached a particular conclusion, you’ll find yourself staring into a black box.
For organisations working in defence, public safety, or critical infrastructure, that’s not good enough.
What You'll Learn
How traditional systems differ from AI-based models
Why leading LLMs can't provide verifiable explanations
What Chain of Thought prompting does (and doesn’t) solve
Why explainability matters for trust and accountability
How this problem sets up the case for a different architectural approach
1. First Principles: How IT Systems Were Originally Designed
Before the rise of generative AI, most computer systems operated on a clear and deliberate foundation. Business analysts would define what should happen in various scenarios. Developers would then translate these conditions into rules and algorithms. The computer’s job was simply to execute them.
For example, if a user missed three payments, the system would flag the account. If a form was incomplete, it would trigger a rejection. Every outcome had a clearly defined cause, grounded in human logic.
This rule-based architecture made traditional systems easy to audit. You could trace an output back to a specific rule or line of code. While these systems weren’t perfect, they had a vital quality: they were explainable.
In regulated industries, this clarity is non-negotiable. If a system makes a decision that affects someone’s financial standing, legal status, or physical safety, you need to know exactly why that decision occurred. Traditional systems allowed for that.
2. AI Systems Work Differently
With the introduction of large language models (LLMs), the architecture has changed completely. Tools like OpenAI’s GPT-4, Anthropic’s Claude, Google’s Gemini, and wrappers like Perplexity no longer follow hand-crafted logic. Instead, they generate outputs based on statistical prediction.
These models are trained on enormous datasets. Rather than applying human-defined rules, they predict the next word or token that is most likely to follow based on patterns seen during training. They do not understand the data or apply consistent logic. They are, fundamentally, completing patterns.
The result often appears intelligent, but the internal process is inaccessible. These models do not reason. They do not apply a known structure. There is no list of steps to audit, because no such steps exist in the traditional sense.
3. The Illusion of Explainability
Some newer models simulate reasoning by generating step-by-step outputs. This is commonly called Chain of Thought prompting. It looks like a structured solution process. The model might show several steps leading to a final conclusion.
However, each of those steps is still a statistically likely guess. The model doesn’t maintain an internal logical structure across the steps. It is not solving a problem in the way a human would. It is simply continuing the pattern.
If you phrase the input differently, it may give a completely different explanation. If you ask twice, it may contradict itself. What you’re seeing is not logic. It’s a well-trained surface pattern.
4. Model Comparison: Different Brands, Same Limitation
Although providers present their models differently, the underlying limitation is shared. None of the major systems offers verifiable, structured reasoning.
GPT-4 (OpenAI) Highly capable across a wide range of tasks. Performs well with Chain of Thought prompting, but still operates on probabilistic output generation. Offers no access to the internal reasoning process.
Claude (Anthropic) Designed to be more aligned with ethical principles through a technique called Constitutional AI. Still a language model. Still generates predictions. Still not able to show why it reached a particular conclusion.
Gemini (Google) Built for multimodal input. Uses a Mixture-of-Experts approach that allows the system to route different parts of a task to different sub-networks. While this improves efficiency, it does not improve explainability. The system remains untraceable.
Perplexity AI Designed as a search assistant. Appears more transparent because it provides source links. However, the citations are chosen for presentation, not necessarily used in the model’s reasoning. The appearance of explainability is not the same as actual provenance.
All four are excellent at language generation. None of them provides a provable logic path. They produce outputs, not explanations.
5. Why This Matters
In consumer applications, this may not be a problem. If an AI writes a helpful email or summarises a news article, you do not need to audit the process.
In enterprise, public sector, or safety-critical environments, that tolerance disappears. When an AI suggests a medical decision, flags a cyber threat, or rejects a visa application, the organisation must be able to explain what happened and why.
Current AI models cannot provide that explanation. They cannot show the input data path. They cannot demonstrate compliance with a policy or legal standard. They cannot guarantee that a decision was fair, justified, or repeatable.
6. Returning to First Principles
Traditional systems were designed using human logic, encoded through software, and executed with full traceability. We knew what happened and why.
In contrast, LLMs generate outcomes through opaque statistical inference. No analyst wrote the rules. No engineer can unpack the decision after the fact. No one can reconstruct the steps that led to a specific output. The logic lives only in probability weightings across billions of parameters.
We have traded explainability for apparent intelligence. And in many domains, that trade-off is simply not acceptable.
What Comes Next
Part of this series, we’ll explore in a counter parallel universe, what happens as we build a lot of AI Agents based on these black-box models?
How JEDAI addresses this problem differently. It doesn’t try to decode the black box. It avoids building one in the first place.
JEDAI restores the first principles of system design: clarity, context, and traceability. It builds AI workflows around structured actor roles and data access paths. It generates insight that can be explained, defended, and audited.
Because real trust in AI doesn’t come from a confident answer. It comes from being able to prove why the answer was reached.
