# Test SigWise on your own data

Before a verdict can block a message or hold a payout, check it against cases
your team has already judged. This page describes a shadow test: you send
historical conversations, read the answers, and compare them with your labels.
Nothing reaches your users, and no integration is needed beyond an export.

## 1. Choose what to ask

Start with two or three signals, written the way you would brief a new
analyst. Concrete evidence steers the analysis far more than words like
"suspicious". See [Signals](https://sigwise.ai/docs/guide/signals.md), and the use cases for
[marketplace scams](https://sigwise.ai/docs/use-cases/marketplace-scam-detection.md),
[rental scams](https://sigwise.ai/docs/use-cases/rental-scams.md) and
[resale marketplaces](https://sigwise.ai/docs/use-cases/resale-marketplaces.md).

## 2. Build two groups

| Group | What it holds | Why |
|---|---|---|
| **Confirmed bad** | Users your team actioned for the scam you are testing | Measures what is caught |
| **Confirmed good** | Users who were not actioned, including long-standing trusted users and users who looked suspicious but were fine | Measures false alarms. This group decides whether you can trust the tool |

A few hundred users per group is enough to see a pattern. Include the kinds of
users you would be embarrassed to block: a seller with hundreds of clean sales
who asked an odd question once.

Cut each conversation at the point your team would have had to decide.
Leave out anything that happened after the scam was detected, such as the
messages that followed a ban notice, or the test will look better than it is.

## 3. Send the conversations

Use inline mode so each answer comes back in the response, and use a prefix on
object IDs so you can delete the test data afterwards:

```bash
curl -X POST "$API/v1/objects/shadow-user-1042/events?api_key=$SIGWISE_API_KEY" \
  -H "Authorization: Bearer $(sign)" -H 'Content-Type: application/json' \
  -d '{"wait":true,"object_type":"user","signals":["is_scammer"],
       "events":[
         {"type":"message","content":"Is the room still available?"},
         {"type":"message","content":"I am abroad, I can send the deposit today and my agent will collect the keys"}
       ]}'
```

To judge a message on its own, without the user's history, set
`"include_history": false`. To check how a conversation is judged without
recording anything, use the [Playground](https://sigwise.ai/docs/guides/moderation.md#trying-signals-first).

## 4. Measure

Pick a threshold on the answer (for a yes/no signal, a probability) and count:

| | Flagged | Not flagged |
|---|---|---|
| **Confirmed bad** | caught | missed |
| **Confirmed good** | false alarm | correct |

- **Recall**: caught divided by all confirmed bad.
- **False-alarm rate**: false alarms divided by all confirmed good. On
  long-standing trusted users this is the number to look at first.
- **Precision**: caught divided by everything flagged. It depends on how common
  the scam is in your real traffic, so a balanced test set flatters it. Re-weight
  the groups to your real mix before you quote it.

Do not copy a threshold from an example. Probabilities for clear scams often sit
well below 0.9, so a cut-off taken from a guide can miss most of them. Look at
where your confirmed-bad users land before you fix one.

Try several thresholds, and look at how many alerts each would send a reviewer
per day. A tool that finds every scam by flagging a tenth of your users is not
usable.

### Compare with what you run today

The number that decides a pilot is how many confirmed-bad users SigWise flags
that your current rules did **not**, or flagged late, at a false-alarm rate you
can accept. Record whether your current system flagged each user, and report on
that subset separately.

## Example script

```python
import json, sys
from sigwise import SigWise

sigwise = SigWise()  # reads SIGWISE_API_KEY and SIGWISE_SECRET
SIGNAL = "is_scammer"

rows = [json.loads(line) for line in open(sys.argv[1])]  # {"id", "label": "bad"|"good", "events": [...]}
scores = []
for r in rows:
    verdict = sigwise.events.ingest(
        f"shadow-{r['id']}",
        object_type="user",
        wait=True,
        signals=[SIGNAL],
        events=r["events"],
    )
    answers = verdict.get("answers") or []
    scores.append((r["label"], answers[0].get("noul", 0) if answers else 0))

bad = [s for l, s in scores if l == "bad"]
good = [s for l, s in scores if l == "good"]
for t in (0.5, 0.7, 0.9):
    caught = sum(s >= t for s in bad)
    false_alarms = sum(s >= t for s in good)
    print(f"threshold {t}: recall {caught / len(bad):.0%}, false alarms {false_alarms / len(good):.0%}")

# Remove the test data afterwards.
for r in rows:
    sigwise.objects.delete(f"shadow-{r['id']}")
```

Run it on a small sample first, because every analysis is billed. See
[Billing and usage](https://sigwise.ai/docs/guides/billing.md) for how to read the cost per signal.

## 5. Go live carefully

1. Keep SigWise in alert-only mode first: a [rule](https://sigwise.ai/docs/guides/rules.md) that sends a
   Slack message when a signal crosses your threshold, with no automatic action.
2. Review what it flags for a week and adjust the signal's wording and your
   threshold.
3. Only then move one path, such as a new user's first message, to
   [inline checks](https://sigwise.ai/docs/guides/moderation.md), and decide how to handle failures with
   [Reliability](https://sigwise.ai/docs/guides/reliability.md).