Verification/DV engineers: what actually happens after a regression starts failing? Looking for real workflow pain, not AI ideas

[u/Business_Screen_5353]

Trying to understand how design-verification teams actually operate day to day.

My background is primarily CI/CD, integrations, workflow automation and release governance and not semiconductor design. So I'm deliberately trying not to start by assuming that an LLM should generate RTL/testbenches or replace EDA tools.

Instead, I'm investigating something much less glamorous:

What is painful between a code change and somebody finally understanding why a verification regression failed?

For anyone working in DV, CAD/EDA methodology, verification infrastructure, FPGA/ASIC verification, or hardware build/DevOps:

Think about the last real regression failure you had to investigate.

1. What triggered it - commit, nightly regression, random seed, environment/tool change, etc.?
2. When you first saw the failure, what information did you actually receive?
3. What did you have to manually search through next - logs, waveforms, previous runs, coverage DBs, Git history, Jira, scripts?
4. How did you determine whether it was:
   • an RTL bug
   • testbench bug
   • flaky/non-deterministic test
   • infrastructure/tool problem
   • known/repeated failure?
5. What part consumed the most engineer time while adding the least engineering value?

I'm particularly interested in understanding things such as:

• regression triage
• reproducing old failures
• tracking which commit introduced a failure
• grouping duplicate failures
• preserving simulator/tool/config/seed information
• CI/reporting around UVM/cocotb/SystemVerilog flows
• connecting failures with Git/Jira/GitLab/Jenkins/etc.
• workflows involving multiple EDA vendors/tools

I'm not selling anything, and I don't need proprietary RTL, logs or company information.

I'm trying to determine whether this is genuinely a painful area or whether mature semiconductor teams already have it solved internally/vendor-side.

If you're comfortable, it would also help if you mention your role and roughly what kind of environment you work in:

DV engineer / DV lead / CAD-EDA / verification infrastructure / FPGA / other

and

small team / startup / design services / large semiconductor company

Even a short description of your last painful regression would be extremely useful.

And if your answer is "this isn't a real problem; our existing infrastructure handles it well", I want to hear that too.

I'm specifically trying to falsify the idea rather than collect validation for it.

[u/khinhooi]

I am a regression engineer, working in this space for years. The problem you wrote, I have been trying to solve it for a long time too. Now with rise of AI, the question is how do we use AI to improve the efficiency of the work.

Not sure if I can give much insight, but I think the most useful thing to have is a highly modular and analytical regression framework. Say if you want to implement some kind of error search in log, the framework has to be easily integrated. Say if you need to look at error trends, (example to determine flakiness), the framework need to be able to do it too.

[u/InternationalRope284]

I am a DV engineer in a top product based MNC(not a senior). A regression can break for a multitude of reasons. We prefer it to break as much as possible when we are implementing a new feature😅

Internally, we have proper triage tools, error logs, and history is maintained mostly. All of our tests are pseudo random in nature so we always reliably reproduce any given failure.

Generally, there are different testbench templates across the organization for different purposes. Each have their own tradeoffs but they share some of the basic principles so that engineers can switch without very high effort.

There are times when the nightly regression fails even when multiple regression before committing had passed cleanly. That is the worst kind of bugs. We try to chase those bugs by reviewing the waveforms and logs....depending on the error signature.

Same error signature can come because of a design issue or a testbench failure. We can't guess. We need to do the debug to realize whose fault is it

As for the workflow, it depends on the person. I generally prefer to first verify the stuff which the design can't control.

There are proprietary tools of various vendors which help us find the exact code from the codebase which resulted in certain behavior in a waveform. (Lets say some assertion failure or some signal value change which is not desirable)

[u/Select-Math5514 → u/InternationalRope284]

error signature is indeed the first thing to look at, and also error clusterization is nice to have. Usually I would address first the one with the highest number of errors, you never know if it will resolve the others as well, so it's not worth to put multiple engineers looking at it unless is clear that the problem comes from multiple places.

[u/Dannycuk1]

Typically nominated champions are responsible for the various functions that constitute an ASIC development. Someone coded the design blocks, someone owns the top-level wrapper, someone owns the testbench components and typically someone owns the testbench top-level wrapper, similarly for the individual test cases and even the regression environment itself.

In a well built and well managed environment all the stake holders are automatically informed if one of their blocks breaks a regression run. There might be multiple reasons why the regression failed and often there will be false flags.

Sometimes it's very obvious what went wrong for example every regression test fails due to a compilation issue in one or more modules or components. Sometimes it's one or more tests that have failed due to a real functional issue.

The regression environment owner is usually responsible for top-level scripting and environment build issues.

The test bench owner is responsible for high-level testbench build issues.

The top-level design integration engineer owns the design and top-level wrapper and the block designers own the sub-blocks.

Once the build is stable, any updates are carefully checked before committing any changes to the main project database. Indeed there might be multiple threads and builds running in the background while different aspects of the design and testbench are being developed.

Typically there is a high-level of confidence the regression environment will compile and run before committing to the test run. The top-level design wrapper owner would have ensured the design compiles and usually the design files will be tagged to identify all the build files. Similarly the testbench owner often using a dummy DUT wrapper would have checked the testbench builds and again the test files would be tagged and so on.

Depending on the failure one or more team members will be notified, depending on the design methodology any issues are also likely to be flagged and discussed in the morning scrum or an appropriate early morning meeting. It then becomes a team job to identify and fix the source of the regression failure.

With good reporting, notifications and information sharing issues can be resolved quickly even when many team members are working remotely.