Most people read a tool’s claim and believe it. A program committee reviewer reads the claim and runs the experiment. When Myroslav Mishov sat down to evaluate a submission called JobTruth, a tool whose entire pitch was catching AI-generated job postings, he did not score the README. He went and found real AI-generated job postings, fed them in, and watched the tool wave them through with confidence scores of 100 and 91. The detector built to catch slop had just certified slop as genuine, and it had done so with complete confidence. That gap, between what a tool says it does and what it does when you actually test it, is the whole discipline Mishov brought to the panel.
Myroslav Mishov is a Lead Enterprise Architect and a CNCF Kubestronaut who reviews submissions for the KubeCon Security Track program committee, where his job is to distinguish substantive technical work from work that is merely well-packaged. He judged the AI Slop Scan Hackathon the way he judges a conference abstract: by refusing to take the claim on faith and testing whether the thing survives contact with a real adversarial input.
AI Slop Scan, organized by Hackathon Raptors, asked teams to build tools that detect, measure, or mitigate AI-generated low-quality content across code review, documentation, marketplace reviews, and general writing. Forty-three teams shipped. The event had a recursive trap built into it that most entrants did not notice: a tool that claims to detect low-effort content is itself a technical claim that can be low-effort, and the only way to know which is to test it adversarially. For a judge who spends his professional life reviewing security submissions for a major conference, that recursion was not a complication. It was the entire point.
Mishov evaluates the way a program committee teaches you to evaluate. A polished abstract means nothing until you check whether the method holds. “On a program committee you learn very quickly that presentation quality and substance are independent variables,” he says. “Some of the worst submissions are the most beautifully formatted, and some of the best look rough. So you stop reading the claim and start testing it. With these detectors I did the same thing. I did not ask whether the tool said it could catch slop. I asked whether it caught slop when I handed it slop on purpose.” That single habit, testing rather than trusting, sorted his entire batch.
The detector that certified slop as genuine
JobTruth, from team beTheNoob, made a clean and confident promise: paste in a job posting and learn, in under five seconds, whether a human who knows the role wrote it or an AI generated it and published without review. The interface was polished, the pitch was sharp, and the demo worked. By the standard most people apply, it passed.
Mishov did not apply that standard. “Didn’t really catch AI generated job posting,” he wrote in his evaluation. “I tried 2 and got results 100 and 91.” The numbers are the whole story. He took postings he knew were machine-generated, the exact input the tool exists to flag, and the tool scored them as overwhelmingly authentic. He scored its detection accuracy at zero, and he is unsentimental about why. “A detector has one job. Not a beautiful interface, not a fast response time, not a clever tagline. One job: catch the thing. If I hand it the thing and it says everything is fine, then everything else is theater. And the dangerous part is the confidence. It did not hedge. It told me 100. A tool that is wrong and certain is worse than no tool, because now the user has a number telling them to stop paying attention.”
This is the failure mode he is trained to find. A submission that is convincing on the surface and hollow underneath does more damage than one that is obviously weak, because it gets trusted. “The reason program committees test methods instead of reading abstracts is that the polished-but-wrong submission is the expensive one,” he says. “It passes review, it gets cited, people build on it, and the error compounds. A slop detector that confidently misfires is the same hazard aimed at a different audience. It teaches people to trust exactly the content they should have questioned.”
Same problem, two detectors, one that survived the test
What made JobTruth’s failure legible rather than anecdotal is that another team in Mishov’s batch attacked the identical problem and held up under the same scrutiny. Ghost Job Detector, from team coding ninja, also targeted fraudulent and AI-generated job listings, surfacing a verdict as an overlay directly on a LinkedIn or Indeed posting. He scored its detection materially higher than JobTruth’s.
The contrast is the point, and it is the kind of comparison a reviewer lives for. “Two teams, same domain, same claim, and the only thing that separates them is whether the detection actually fires,” he says. “That is exactly the situation a program committee exists to adjudicate. You cannot tell these apart from the pitch. Both pitches are good. You can only tell them apart by feeding both the same adversarial input and seeing which one holds. One did and one didn’t, and that single fact should dominate every other consideration in the score, because it is the only one the user will feel.” For Mishov, the existence of a working version in the same batch removed every excuse. The problem was tractable. JobTruth simply had not solved it, and a sharper presentation could not close that gap.
The team that asked a question it could actually answer
If the job-posting pair showed detection succeeding or failing, two other projects impressed Mishov by being honest about what is even checkable in the first place. The highest score in his batch went to SlopGuard, from team Team Batman, and his respect for it was structural rather than aesthetic.
SlopGuard refuses the question most detectors ask. Instead of trying to determine whether text was AI-generated, an unanswerable and unfair question in its own framing, it scores whether writing shows signs of human thinking. To a reviewer, that reframing is not a marketing move. It is intellectual honesty about the limits of the method. “The question ‘was this written by AI’ is unfalsifiable, and a good reviewer flinches at an unfalsifiable claim,” Mishov says. “SlopGuard did the thing I wish more submissions did. It admitted the real question cannot be answered and replaced it with one that can: does this show evidence of thought. That is a checkable property. You can argue about the threshold, but you can actually measure signal density and reasoning. They chose a question they could defend, which is the first sign of a serious method.”
He saw the same instinct in Parakh, from team TeamHM, which checks marketplace reviews for a specific, objective failure: claiming features the product does not actually have. AI-generated reviews routinely hallucinate category-standard features, attributing noise cancellation to earbuds that lack it, and Parakh tests that claim against the product rather than guessing at authorship. “This is my favorite kind of detector, because it is grounded,” Mishov says. “It is not asking a vibe question. It is asking ‘does this review assert something false about this specific product,’ and that has a real answer you can verify. A detector built on a verifiable question can be tested, debugged, and trusted. A detector built on an unfalsifiable one can only be believed, and belief is not something I extend to a tool.”
How a reviewer reads a detector
Mishov’s scores resolve into a short protocol, the same one he applies to a conference submission, adapted to a tool that claims to find slop. It is worth stating plainly, because it is the discipline most of the field skipped.
Test it adversarially before you read its pitch. Hand the detector the exact input it exists to catch, sourced independently, and see whether it fires. JobTruth’s confident miss on real AI postings was invisible from the README and obvious within two tests.
Treat confidence without correctness as a defect, not a feature. A tool that is wrong and certain is more dangerous than one that admits uncertainty, because it switches off the user’s own judgment at the worst moment.
Prefer a falsifiable question to an impressive one. SlopGuard and Parakh scored well because they chose questions with real answers: does this show evidence of thought, does this review claim a feature the product lacks. A method you can check is a method you can trust.
Use the batch as a control group. When two teams attack the same problem, the one whose detection actually fires sets the bar, and a polished presentation from the team that missed cannot argue its way back over it.
Separate packaging from substance on purpose. They are independent variables. The most formatted submission is not the most correct one, and a reviewer’s job is to keep from confusing the two.
The verdict, and the discipline the field is missing
For Mishov, AI Slop Scan rewarded the same thing his program committee work rewards, which is intellectual honesty under test. The teams that did well were not the ones with the best taglines. They were the ones whose tools survived having their central claim checked against a hostile input.
“The irony of this whole event is hard to miss,” he says. “We are building tools to catch low-effort, confident, unverified output, and some of those tools were themselves low-effort, confident, and unverified. A detector that has not been tested against real slop is slop with a nicer interface. That is not an insult. It is just the same standard applied to the tool that the tool is applying to everyone else, and a good detector should welcome that, because surviving it is the entire value proposition.”
He sees the lesson extending well past a hackathon, into a market that is about to fill with detection and trust tooling of every kind. As AI-generated content saturates code, documentation, reviews, and hiring, the demand for tools that flag it will explode, and most of them will be exactly what he warns against: confident, polished, and untested. “The next few years are going to produce thousands of these,” he says. “Slop detectors, AI graders, authenticity checkers, trust scores. And the same failure I saw twice in one batch is going to scale to an entire category. People will deploy a confident detector, trust its number, and stop checking, and the detector will be wrong in ways no one audits. The engineers who matter will be the ones who treat their own tool as a submission to be reviewed, who try to break it before they ship it. The reviewer’s instinct, test it, do not trust it, is not a hackathon nicety. It is about to be the core competency of building anything that claims to tell truth from noise.”
AI Slop Scan was organized by Hackathon Raptors, a Community Interest Company supporting innovation in software development. The event challenged 43 teams to build tools that detect, measure, and mitigate AI-generated low-quality content across code review, documentation, marketplace reviews, and general writing. Myroslav Mishov, a Lead Enterprise Architect and CNCF Kubestronaut who reviews for the KubeCon Security Track program committee, served as a judge evaluating submissions for detection accuracy, practical usefulness, technical execution, innovation, and presentation.
