Integrating advancements into open source processes makes sense. The friction often comes in the how and the usefulness. While AI holds immense potential for revolutionising triage and review processes, it’s crucial to acknowledge its current limitations and look to where it can evolve, learn, and grow to its full capacity. This post aims to explore this potential, offering a realistic perspective. It’s not a post that claims to solve everything; instead, it will show how we can get started and regain some hours to focus on other needed areas of the project.
Before we delve into this topic, it’s important to note that while AI can achieve a great deal, it also has its limitations. This post will focus on high-level inferences rather than direct proposals, starting with small steps. I don’t have all the answers, but together, we can find them. My perspective is informed by extensive theme reviews and ongoing triage work, so I won’t cover areas where I lack knowledge.
What can it do today
In simple terms, the things we likely shouldn’t be doing manually include reviews, first-pass triage, testing and routine checks. Historically, there have been areas where we have been slowly adding more processes over time before the term AI was widely used. That’s one of the key things: often, having a robust process, a path trodden and documented, means the parsing is simple to learn, and therefore you can hand it over more easily.
Triage is already being done incredibly effectively in areas like healthcare. There it is improving accuracy and saving lives, as noted in an article by Reuters:
“Our study provides the first multinational evidence that artificial intelligence can help enhance accuracy” in determining HER2 clinical categories, “potentially closing critical diagnostic gaps and enabling more patients to have access to new therapies, said Dr. Marina De Brot of the A.C. Camargo Cancer Center in Sao Paulo, Brazil, who led the study.
“Until recently, most of these patients would have not been offered these options,” she said.”
Link: Reuters
A triage first-pass
The first area is improving first-pass triages. What do I mean by this? In simple terms, doing a lot of the manual work like duplicate label removal, closing after a specific time, checking for details and even duplicate issue detection. Pattern-matching tasks are where these systems excel, and a lot of this is simply a matter of pointing to our learnings during triage and matching them accordingly.
This is basic; it’s not AI as much as it’s processing. It’s one step above issue templates. The following piece is where it gets interesting: the model for triage needs to learn comprehension of the issue contents. We need to add features like dashboards to the flow because the first stage could be a catch-all that, over time, allows it to triage more to a buffer dashboard, recommending specific states.
All of these approaches are an excellent way of training, as it provides the model with some freedom to make mistakes, be corrected, learn from them, and refine its patterns. We also collectively don’t have to worry about labels being added and things closed. Whilst those of us doing triage, get to benefit early on and help train.
The impact of triage processing
As far as triage goes, we will be far from achieving full automation of this for quite some time. My instinct is that this is more of a help in clearing a percentage and making it easier for those working on issues. It can do that in significant numbers, to the extent that over 50% of the time spent in triage could be recovered I’d suggest sooner over later. I do see this going up over time, but how much depends on the quality of the modelling and our trust as a project.
Theme review first-pass
Just as triage, I suggest adopting a similar model for theme reviews. In this case, too, if we are honest, reviewing what we do and don’t want in reviews would be beneficial. I will be bold here, but nearly 100% of theme reviewing could be done through a process without manual interaction fairly soon.
How do we get there? We achieve this through a balance of reviewing the theme review guidelines and, similarly, triage training. These reviews are very formulaic and matching. This is what models thrive on. Suppose we gain trust in this approach and implement the process. In that case, we can then focus all our effort on manual reviewing in various areas, including raising the quality of the core product, documentation, education, and our theme offerings as a project.
Testing
We have, over the years, even before the recent AI tech wave, been leaning into less manual testing. Issues and tickets needed less manual testing. We require testing for regression across all areas in our project. That’s not AI, but we can take it even further, adopting a more updated approach to our testing.
The reality of the world today is that you can share a screenshot with code editors and get an application mocked up that looks like that. We also need to evolve our systems to test for differences and minimise the need for multiple humans to be involved in each ticket.
What will we do if it does this?
I share these perspectives as someone who performs triage and frequently engages in this manual work. Some may be sceptical or wonder how their sponsored time will be allocated. The point is that there is a great deal that needs to be done in this project. These systems to implement today’s technology need people. The models require training by experts. Then they need to be nurtured and grown by those same people. There will always be a need for contributors.
At this time, with our capabilities in automation and AI in a world of agentic flows, we can do better than relying solely on manual processes. We must do so in order not to be left behind. It is also not resourceful to use humans this way. We can use our scarce human resources more sensibly. Our time is finite, and this project requires us to accomplish many tasks. There will be a balance between what we automate and utilise as resources and what we provide to a system. That, though, is something we need to start working out, and we do that by realising we need to find a balance in how we do things and measure the costs of our processes today and in the future.