“If you give AI bad requirements, it's going to write bad code, and it's going to say the code is good because it's testing itself.”
Vu Lam
Katalon CEO Vu Lam gave that warning when he sat down with Alex Martins and Mush Honda for the latest discussion in The Future of Software Testing series. It comes as the industry reaches back for disciplines it spent two decades leaving behind. GitHub's Spec Kit puts a written spec in front of the coding agent and checks each phase before the next begins. Whether that’s waterfall’s return or just a better-structured iteration was part of what the panel argued about.
Specifications. Stage gates. Independent validation. None of it is new.
Nobody on the panel wanted 18-month release cycles back. Nobody dismissed the question either. What emerged wasn’t a verdict on waterfall but a diagnosis: AI has removed coding as the bottleneck, and the constraint has moved upstream to requirements. The panel split on what to do about it, and both answers ask more of testers, not less.
The bottleneck moved upstream
Early in his career at Bell Labs, Vu worked on the 5ESS telephone switch, where releases ran 18 months and six or seven of those months went to requirements. That discipline had a price tag behind it: Writing software was expensive, and finding a fundamental mistake late was even more expensive. Agile changed the bet. As Alex described it, “Let’s just do a little bit, put it in front of the customers and get feedback.”
AI is shifting economics again.
“Previously, developers were the bottleneck. Nobody had enough development bandwidth. Now you have enough development bandwidth. So I think the bottleneck moves further up front.”
Vu Lam
When code can be generated quickly, discovering late that you've built the wrong thing can wipe out the speed advantage. Imagine an AI agent misinterpreting a business rule. It builds the feature, writes tests based on the same misunderstanding, and reports that everything passes. The mistake surfaces only during user acceptance testing or after release.
Much enterprise software work, Vu argued, involves modernizing or rebuilding existing systems. The business logic may already be understood by people within the organization, but much of that knowledge isn't captured in a way AI can use. “It's Jira, it's Confluence, it's Google Docs, and it's spread all over,” he said. Engineers remember why certain exceptions exist. Testers know business rules that never made it into a user story. Unless that context is documented and made accessible, an AI agent may never see it.
Thus, for Vu, writing requirements is “a skill set we used to have that's no longer prevalent, but we may need to go back and learn how to do that.”
The problem isn't just the requirements. It's us.
“We are inherently messy and ambiguous. We cannot be very clear on what we want.”
Alex Martins
Alex accepted that requirements matter. He rejected the idea that people can ever write them completely, and expertise doesn't change that. Two experts in the same domain, he argued, will describe the same rule differently, in good faith.
"I think it's very naive to expect that now we spend a lot of time curating requirements, refining them, and then giving them to AI and it will be the end of it," Alex said. "Because the fundamental issue is us. It's the humans. It's our brains."
His answer is to embrace the uncertainty, use AI to work through it faster, and pay for that speed with a lot more thorough testing, a lot more frequent testing, testing that is continuous throughout the process. But on one point they agreed. "If you don't have those requirements upfront, how are you going to test what AI produced?" Alex asked. "You're still accelerating the problem, not resolving it."
Spec-driven isn't the same as waterfall
“Was it really agile for them to begin with? Or was it more just shorter waterfalls to a certain extent?”
Mush Honda
Mush questioned whether some teams ever truly moved beyond waterfall. Even with Agile, testing often piled up at the end of each sprint. Development cycles became shorter, but testing practices didn't necessarily change.
For Mush, the debate over agile versus waterfall misses a bigger issue. Both are approaches to organizing development, but neither guarantees software quality. As AI accelerates code generation, the challenge is ensuring that testing and validation keep pace. What matters isn't just how quickly teams build software, but how they establish confidence in what they deliver.
He also pushed back on treating today's spec-driven development as waterfall's return. When specifications continue evolving during development, it's "just another flavor of the iterative approach." What's worth preserving is "the real value and essence of waterfall, the pros that came with it."
That leaves testers with a hard question: how do you establish correctness when requirements keep evolving? Specs that never change can't keep up with what teams learn. Specs that change without clear controls can leave tests checking outdated expectations. Either way, when requirements move, test expectations need to move with them. And testers still need sources beyond the spec, such as test history, comparable products, user expectations and domain rules, to judge whether behavior is right.
A passing test isn't proof
“Better requirements mean better code, but better requirements also mean better tests.”
Vu Lam
When AI writes both the code and the tests based on the same misunderstood requirement, everything can pass while the software still fails to meet the actual business need. Alex was blunt about agent demos that claim otherwise: "It didn't test, because it's not looking at requirements. It's basically making assumptions."
Using a second AI agent doesn't automatically solve the problem. If both agents work from the same incomplete requirements, they may share the same blind spots. "Acceptance criteria should be written by humans, not by the agent that generates the code," Mush emphasized. Independence draws on domain expertise, real user behavior and business rules: evidence that can challenge the assumptions behind the code.
Vu recalled that mindset from the system validation teams at Bell Labs: "I don't know who you are. I don't care who you are. I don't care how good you are. I'm going to assume that there are bugs in there."
And some of the most important defects are the ones nobody thought to specify. Mush recalled being particularly proud of finding "those issues that had not been pre-specified" — unexpected failures that emerge when conditions interact in ways no one anticipated. "That is still a very human aspect," he said.
What testers should do differently
Better specs don't push testing to the end. They pull it forward. "If you have more specs earlier, you can get started on your test planning, test design, and test strategy sooner rather than later," Vu said.
As AI accelerates development, testers have opportunities to influence quality throughout the process:
"I don't feel it's replacing me. It's actually accelerating," Mush said.
A new shape, not an old process
“You're not trying to get everybody back to waterfall with an 18-month cycle, but shift the human effort more up front.”
Vu Lam
When Alex pushed for numbers, Vu described a model being proposed in software development work: a 12–16 month project compressed to six. Three months go to requirements, one to generating code, two to testing and validation. It illustrates where effort moves; it is not a benchmark.
How much upfront definition teams need depends on two things. The first is novelty: "If you're creating software that never existed in the world, then you're vibing, iterating, agile, whatever you want to call that," Vu said. The second is criticality: "More waterfalls as you get close to mission critical." Alex agreed that high-stakes systems demand greater rigor. Agile's build-deliver to users-and-adjust approach works for many applications, "but in heavily regulated environments, mission-critical systems, and all that, you can't do that." That doesn't necessarily rule out iterative development, but it does raise the bar for requirements, traceability and validation.
So, is waterfall coming back?
Not exactly. Not the waterfall most teams spent years moving away from.
What's coming back is waterfall's discipline: clear requirements, deliberate sequencing, and validation that assumes the build is wrong until proven otherwise. "I'm not going to let the AI do any coding until I feel pretty good about the requirements," Vu emphasized.
But Alex's objection still stands: no amount of upfront refinement can eliminate inherent human ambiguity. Requirements will evolve, and teams need room to learn and adjust. Mush adds a third condition: Whatever process teams follow, they still need independent evidence that the software behaves as intended.
Generating code faster doesn't help if you're building the wrong thing. Generating more passing tests doesn't help if they're checking the wrong expectations.
The question isn't whether to bring waterfall back. It's how much to define upfront, when to iterate, and what evidence teams need before trusting what AI builds.
Are stronger upfront specs cutting rework on your team, or is faster experimentation with continuous testing paying off more? Tell us what you're seeing.
The future of testing won’t be decided by one company — agree, push back, or add what we missed.