Overview
Scam Sentinel is an AI-powered Discord security project I developed to investigate moderation behavior and help online communities assess suspicious content. My work spans exploratory model testing, human-feedback workflows, community-specific context, and the practical question of when a model judgment should lead to action. The project also publishes threat-intelligence writing.
Discord confirmed that the application had been added to 75 servers on August 20, 2025. That is a historical installation milestone. The research and evaluation described here address a separate question: how reliably the system recognizes harmful content and supports moderation decisions.
The research questions
- How can a moderation system recognize scams and harmful content without penalizing legitimate discussion?
- When context is incomplete, what additional evidence does the system need?
- Does human feedback actually reach and inform later decisions?
- Can a model explain what an obfuscated message means, or is it reacting to the presence of encoding alone?
Exploratory model evaluation
I tested benign and adversarial moderation inputs across saved Gemini and Gemma configurations, including encoded harmful language and content whose meaning depended on context. I repeated selected prompts and examined both the risk assessment and the explanation.
In one decoding test, I asked the model to translate an encoded message to check whether its classification reflected an understanding of the underlying text. The returned interpretation was incorrect, showing why a risk label alone was insufficient to evaluate that response.
Two saved trials with the same complete prompt and saved settings produced opposite high-confidence assessments: risk 0 in one and risk 100 in the other. This made consistency across repeated inputs a concrete question to investigate.
Other examples show conflicting judgments about a benign descriptor and responses truncated before producing complete JSON. These observations separate three evaluation concerns: consistency, understanding of meaning, and usable structured output.
Scope of the evidence: The retained material contains 30 moderation-prompt submissions across four distinct message contents. These exploratory tests support specific observations about inconsistent outputs and decoding failures. They do not establish population accuracy, a model ranking, or a measured improvement in detection.
Feedback as a mechanism
I specified a human-feedback workflow that includes both moderation actions and non-actions. Reviewing only removed messages would reveal false positives while overlooking content the system should have acted on. I also required ordinary server reports to reach central review before being accepted as feedback.
I questioned whether a negative label would influence a later decision when examples were sampled rather than included every time. I also examined whether many repetitive benign examples could crowd out more informative cases. These questions concern label quality, coverage, and how reviewed examples reach the model.
The implementation supplies feedback examples in the model’s context. The retained examples are selected, so they cannot provide a representative accuracy, precision, or recall estimate. The evaluation question is whether the feedback mechanism changes subsequent behavior in the intended way.
Context and moderation decisions
I investigated how community-specific context affects false positives. A discussion that is legitimate within one community can look suspicious when judged in isolation. My development questions asked how to avoid flags caused by insufficient context while retaining detection effectiveness.
This work informed exploration of additional context and server-specific instructions. The available records do not measure the resulting change in precision or recall. A useful evaluation needs to check the model’s judgment, the explanation, and the resulting moderation action.
Public threat-intelligence communication
The project’s research page publishes practical threat analysis for online communities. Its March 31, 2026 brief, “Discord False Report Scam Analysis,” follows an attack from a fake or compromised friend and an urgent account-ban claim to fake support and requests for an email change, credentials, or cryptocurrency.
The brief translates that sequence into recognizable warning signs: urgency, impersonation, and unusual requests. Its advice includes verifying the friend through another channel, using official support, avoiding suspicious verification links or scripts, protecting credentials and tokens, and enabling two-factor authentication.
The article is published under the Scam Sentinel Staff byline. It adds a public communication component to the project by explaining how the scam works and what community members can do. Read the published brief · Browse the research page
System implementation
The project source includes a Node.js/Discord.js security application, Gemini text and image analysis, configurable moderation policies, human feedback, persistence, and an Express dashboard. Retained source also includes audio-analysis components and companion interfaces.
These components connect model behavior to the decisions moderators and communities experience. The research questions cover both the AI assessment and the software that turns it into a review or moderation action.
What the work demonstrates
- Repeated model probes can reveal contradictions that a single confident response would hide.
- Semantic understanding and valid structured output are separate evaluation concerns.
- Human feedback needs a defined path from a reported problem to a reviewed example and a later decision.
- Practical threat analysis can turn an attack sequence into clear guidance for community members.
Evidence and next steps
This account draws on saved model probes, documented feedback and context requirements, retained project source, the public threat-analysis brief, and Discord’s historical installation notice. Private community data and original harmful test strings are omitted.
A next evaluation would use a consistently labeled set of benign and harmful examples, fixed resolved prompts, repeated trials, controlled comparisons of feedback and context settings, and separate checks of model decisions and downstream actions. That is a proposed next step.