LLMs are powerful observability tools. This is true even if you avoid using LLMs to write code.
Codebases (and many other things!) are high-dimensional objects. LLMs provide a way to project down to a lower-dimensional space of interest (e.g. bugginess, security, a subsystem 1, etc.).
I think of these different lower-dimensional spaces as being "angles" at which you are viewing the codebase to identify opportunities for improvements.
Some angles are more valuable in terms of the insights they provide, and confidence they inspire. Prompt engineering for observability is partly about maximizing an angle's value.
One particularly valuable angle is that of a retrospective: for an LLM agent which has undertaken a change, to identify potential improvement and opportunities, e.g. architectural changes, unrelated bugs, etc. The context window populated after implementing a feature, fixing a bug, etc. often contains very valuable insights. And those are useful for observability even if you immediately throw away the agent's code! Getting an LLM to make changes to a codebase can therefore be a pure observability tool.
To extend the geometric analogy, the "angle" corresponds to the topical focus of analysis, whereas the existing context window and framing is like an "offset". Starting with an empty vs. filled context window can lead to quite different outputs to the same prompt.