Test SigWise on your own data

Before a verdict can block a message or hold a payout, check it against cases your team has already judged. This page describes a shadow test: you send historical conversations, read the answers, and compare them with your labels. Nothing reaches your users, and no integration is needed beyond an export.

1. Choose what to ask

Start with two or three signals, written the way you would brief a new analyst. Concrete evidence steers the analysis far more than words like "suspicious". See Signals, and the use cases for marketplace scams, rental scams and resale marketplaces.

2. Build two groups

Group	What it holds	Why
Confirmed bad	Users your team actioned for the scam you are testing	Measures what is caught
Confirmed good	Users who were not actioned, including long-standing trusted users and users who looked suspicious but were fine	Measures false alarms. This group decides whether you can trust the tool

A few hundred users per group is enough to see a pattern. Include the kinds of users you would be embarrassed to block: a seller with hundreds of clean sales who asked an odd question once.

Cut each conversation at the point your team would have had to decide. Leave out anything that happened after the scam was detected, such as the messages that followed a ban notice, or the test will look better than it is.

3. Send the conversations

Use inline mode so each answer comes back in the response, and use a prefix on object IDs so you can delete the test data afterwards:

curl -X POST "$API/v1/objects/shadow-user-1042/events?api_key=$SIGWISE_API_KEY" \
  -H "Authorization: Bearer $(sign)" -H 'Content-Type: application/json' \
  -d '{"wait":true,"object_type":"user","signals":["is_scammer"],
       "events":[
         {"type":"message","content":"Is the room still available?"},
         {"type":"message","content":"I am abroad, I can send the deposit today and my agent will collect the keys"}
       ]}'

To judge a message on its own, without the user's history, set "include_history": false. To check how a conversation is judged without recording anything, use the Playground.

4. Measure

Pick a threshold on the answer (for a yes/no signal, a probability) and count:

	Flagged	Not flagged
Confirmed bad	caught	missed
Confirmed good	false alarm	correct

• Recall: caught divided by all confirmed bad.
• False-alarm rate: false alarms divided by all confirmed good. On long-standing trusted users this is the number to look at first.
• Precision: caught divided by everything flagged. It depends on how common the scam is in your real traffic, so a balanced test set flatters it. Re-weight the groups to your real mix before you quote it.

Do not copy a threshold from an example. Probabilities for clear scams often sit well below 0.9, so a cut-off taken from a guide can miss most of them. Look at where your confirmed-bad users land before you fix one.

Try several thresholds, and look at how many alerts each would send a reviewer per day. A tool that finds every scam by flagging a tenth of your users is not usable.

Compare with what you run today

The number that decides a pilot is how many confirmed-bad users SigWise flags that your current rules did not, or flagged late, at a false-alarm rate you can accept. Record whether your current system flagged each user, and report on that subset separately.

Example script

import json, sys
from sigwise import SigWise

sigwise = SigWise()  # reads SIGWISE_API_KEY and SIGWISE_SECRET
SIGNAL = "is_scammer"

rows = [json.loads(line) for line in open(sys.argv[1])]  # {"id", "label": "bad"|"good", "events": [...]}
scores = []
for r in rows:
    verdict = sigwise.events.ingest(
        f"shadow-{r['id']}",
        object_type="user",
        wait=True,
        signals=[SIGNAL],
        events=r["events"],
    )
    answers = verdict.get("answers") or []
    scores.append((r["label"], answers[0].get("noul", 0) if answers else 0))

bad = [s for l, s in scores if l == "bad"]
good = [s for l, s in scores if l == "good"]
for t in (0.5, 0.7, 0.9):
    caught = sum(s >= t for s in bad)
    false_alarms = sum(s >= t for s in good)
    print(f"threshold {t}: recall {caught / len(bad):.0%}, false alarms {false_alarms / len(good):.0%}")

# Remove the test data afterwards.
for r in rows:
    sigwise.objects.delete(f"shadow-{r['id']}")

Run it on a small sample first, because every analysis is billed. See Billing and usage for how to read the cost per signal.

5. Go live carefully

1. Keep SigWise in alert-only mode first: a rule that sends a Slack message when a signal crosses your threshold, with no automatic action.
2. Review what it flags for a week and adjust the signal's wording and your threshold.
3. Only then move one path, such as a new user's first message, to inline checks, and decide how to handle failures with Reliability.