Events, webhooks and observability for integrations
What changed
Every content update is versioned. Select a version to compare it with the one before.
Version 1 → 2
Update5 lessons updatedduration
Content update. Enrolled learners were notified.
Duration changed from 13 to 17 minutes.
A webhook definition in Monosign is bound to one audit type. Each audit of that type creates one execution, and the webhook service sends the request. Two questions come up in every project: why did my test arrive empty, and what happens when the message queue is down?
::diagram api-webhook-delivery
### Tokens come from the audit
On the audit path Monosign always supplies five keys, plus the data keys of that audit type:
| Token | Value |
| --- | --- |
| `{{Audit.UserId}}` | The user the event is about |
| `{{Audit.ActorUserId}}` | The user who performed the action |
| `{{Audit.SessionId}}` | The session |
| `{{Audit.ApplicationId}}` | The application |
| `{{Audit.CreatedDate}}` | When the audit was written |
Tokens are replaced in `URL`, `Headers` and `Body`. A token that is neither one of these five nor a data key of the audit is replaced with nothing. A body copied from an example with `{{UserId}}` therefore reaches the receiver with an empty value. Write `{{Audit.UserId}}`.
A small definition for a login feed looks like this:
```
Type: USER_LOGIN_SUCCESS
URL: https://siem.example/hook
Method: POST
Headers: {"X-Actor-Id": "{{Audit.ActorUserId}}"}
Body: {"user": "{{Audit.UserId}}", "at": "{{Audit.CreatedDate}}"}
```
### What `Execute` proves, and what it does not
`Execute` sends the request with an empty data set. It proves that the URL is reachable, that the method and headers are accepted and that the receiver answers in time. It proves nothing about your tokens, because every token renders as an empty string. To test the tokens, cause a real audit event of that type, for example a sign-in, and look at the definition's `Items` for a row with `Send Success`.
### Retries
A failed send is tried up to five times, with a fixed wait of one second between attempts, and then the row is marked `Max Try Reached` and appears under `Dead Letters`. This is why the receiver has to treat a repeated event as harmless.
### When the message queue is down
::diagram obs-webhook-outage
In normal operation the execution travels through the message queue. While the broker is unusable, the **service host** delivers webhooks directly from the database in bounded batches. Delivery continues, and it also continues when Redis is down.
The practical consequence concerns the network. During the outage the outbound connection to your receiver is opened by the service host itself. On a site with per-process or per-host firewall rules, the service host needs an egress rule to the webhook receiver. If events stop exactly when the broker fails, check that rule first.
Management, System, Services, `Dead Letters` is the place where work that stopped short of delivery waits for a person. An integration engineer has to know what the list contains, what it leaves out, and what each button really does.
::diagram obs-dead-letters
### What is listed
| Kind | A row appears when | Shown |
| --- | --- | --- |
| `Notifications` | Dispatching the notification itself failed and its retries are used up | The last 30 days |
| `User Provisions` | A provisioning row is in an error state, for example after 48 hours without completion | Up to 200 rows |
| `Webhooks` | A webhook reached its maximum number of attempts | Up to 200 rows |
The list always belongs to the tenant you are signed in to.
### An empty list is not proof of health
Three limits make a quiet tab a weak signal:
- A notification whose mail, SMS or push failed is **not** listed. The failure is recorded on that channel's history row, and the notification itself still ends `Completed`. Only an error at dispatch level parks a notification here.
- You see one tenant, and for notifications only the last 30 days.
- The page does not alert, mail or monitor. A row is visible only when someone opens the page.
Treat the list as a work queue to review on a schedule, and combine it with channel history and with monitoring of the receiving systems.
### What the buttons do
| Kind | `Requeue` | `Discard` |
| --- | --- | --- |
| Notification | Back to waiting, with a fresh 24-hour expiry | Final discarded state |
| User provision | Back to its waiting state, picked up by the provisioning service | Final discarded state, nothing is sent |
| Webhook | The row leaves the list with its attempt count reset, but it is not sent again | Final discarded state |
Both buttons act on one row at a time. A requeued item can fail again and return to the list. A discard keeps the row and its last error in the database but cannot be undone from the screen.
The webhook line deserves attention: because a requeued webhook is not delivered, do not use `Requeue` to replay a lost event. Trigger the business event again, or have the receiver fetch the current state through the API.
### Who did it
Neither action writes audit columns on the row. Each writes one line to the Management log:
```
Dead letter requeued: webhook 7d0c... by 41aa...
Dead letter discarded: userprovision 92be... by 41aa...
```
The line carries the type (`notification`, `userprovision` or `webhook`), the row ID and the user ID. For a discard this line is the only record. If an auditor may ask who retired a row, keep the Management logs long enough to answer.
Monosign offers two families of health endpoints, without authentication, on the API and Management applications. They answer different questions, and picking the wrong one causes either blind spots or needless restarts.
### The two families
| Endpoint | What it checks | Answer |
| --- | --- | --- |
| `GET /ping/hc` | A checklist: `Database`, `Mq`, `Cache`, `ServerGC`, `License` | 200 `{"IsHealthy":true}`, or 400 with the failing entries |
| `GET /ping/hc/mq` (API only) | One dependency; `db`, `cache` and others work the same way | The verdict of that one check |
| `GET /probes/ready` | The database only. Cache state is reported but not judged | 200, or 503 `not ready` |
| `GET /probes/live`, `/probes/startup` | For orchestrators | 200 while the process serves |
### Which one for which consumer
- **An orchestrator that restarts a container on failure** should use `/probes/ready`. A broker problem makes `/ping/hc` unhealthy, and restarting the API container does not repair a broker. Readiness looks at the database only, so it does not trigger restarts for a fault a restart cannot fix.
- **A load balancer** should use the full `/ping/hc`, where any failing check flips `IsHealthy` and the node leaves rotation.
- **A monitor for one dependency** should use the targeted form, for example `/ping/hc/mq`.
::diagram obs-health-probes
### Give the monitor at least six seconds
The `Mq` check is a real round trip through the broker, with a probe timeout of five seconds. A degraded broker makes the whole answer take that long. An external monitor with a timeout below six seconds reports a timeout of its own before Monosign can report the actual reason.
### Message queue monitoring from v2026.10
Two things changed in v2026.10. Health failures are counted per minute and shown on the `Status` page as failing minutes per day. And the message queue is no longer checked centrally, because the broker belongs to each data center. A Monosign monitor that watches the central `mq` service therefore stays silent during a broker outage. Each node's `/ping/hc` still carries the `Mq` entry, so point the monitoring at the nodes.
When a REST call fails or a push does not arrive, you need the log of the right component. Monosign gives you three ways to get it, and each has a clear limit.
| Way | Covers | Limit |
| --- | --- | --- |
| Container output | Every component, continuously | You need your own log pipeline |
| Management, System, `Logs` | Management, and service processes that are connected to it | Not API, not Account; kept in memory only |
| Management, System, `Tracing & Log` | The applications you select, for a set number of minutes | Some applications have no collector |
::diagram obs-log-paths
### Container output and JSON lines
Each process writes its log to stdout as plain text. From v2026.10, setting `MONOSIGN_LOG_FORMAT=json` on a component makes every stdout line of that process one JSON object, which is what a JSON-lines pipeline expects. Three details matter when you set it up:
- The variable is read per component, so set it on every container you collect from.
- It covers **stdout only**. stderr is unchanged, so the runtime's crash report still arrives as plain text. A pipeline has to tolerate those few lines.
- A value other than `json` or `text` falls back to text with one warning line. Startup never fails because of it.
### The Logs page
The page streams events of the Management process, and of service processes that have `MONOSIGN_LOG_HUB_URL` pointed at Management. The API and Account applications never send their events there, and the `Apps` checkboxes do not filter anything. If you are debugging failed REST calls, this page cannot show the API side.
The events live in the memory of one Management instance, up to 100 000, and are gone after a restart.
`Configure` contains a setting that reads the wrong way round. `Default Level` **drops** events at or below the chosen level:
| `Default Level` | What you see |
| --- | --- |
| `Verbose` | Everything above Verbose, which is the most the page can show |
| `Warning` (default) | Error and Fatal |
| `Error` | Fatal only |
To see errors, the level must be lower than Error. The setting filters the page only. It does not change what the processes log.
### Tracing & Log
A trace captures the full log output of the selected server applications for `Duration` minutes and offers the result as one zip. This is the tool for the API and Account applications.
Select only applications that can answer. `Monosign - Access Gateway` and `Monosign - RADIUS Server` appear in the list but have no collector, so a trace that names them never completes: it stays `Started`, and `Download` reports that applications are in progress. Use `Download Partial` to get what was collected, and leave those two out next time. `Force Start` and `Interval` on the form are stored but not used, and traces are not removed by data retention, so delete old ones yourself.
> Tip: Start from the component, not from the tool. For the API or Account side use `Tracing & Log` or the container output. The `Logs` page can only show Management and the service processes connected to it.
Monosync has four signals. Each answers one question well and stays silent on others, and an integration engineer is expected to know which is which.
| Signal | Question it answers | What it does not tell you |
| --- | --- | --- |
| `/ping/hc` | Is the installation up, and are service nodes alive? | Whether a particular integration works |
| `Integration Logs` | What did one run do to one record? | Anything across runs, or outside Monosync |
| `Monitoring` | Did an integration fail or stop running? | Nothing, when its mail path is not configured |
| `Audits` | Who changed configuration in the web UI? | Changes made by other paths |
### The health answer has two fields
```
GET /ping/hc
{"IsHealthy":true,"InstanceCount":3}
```
`InstanceCount` is the number of service nodes that are reporting in. Those nodes run the integrations. An answer of `{"IsHealthy":true,"InstanceCount":0}` is still HTTP 200, yet no integration can run, for example because the service role is disabled on every node. A probe that looks only at `IsHealthy` misses this, so alert on a count of zero as well.
### Reading a run
The logs page drills down from the run to its actions, to the objects an action touched, and to the changed values of one object. When a job fails for some users only, start here: the reason is on the object row, in `Status`, `Message` and `Exception`, and mapping failures are collected under `Mapping error(s) found!`.
Two limits matter for a security operations team:
- `Export All` exports the changed values of the run. It does not export errors.
- Monosync does not push its logs anywhere. A SIEM, or any view across runs, needs a read-only query on the log tables. Read them before log retention purges the rows.
### Monitors need the Monosign API
A monitor opens an issue when an integration has errors or has not run for a set time, and sends a `[DOWN]` notice, reminders and an `[UP]` notice on recovery. Monitors have no mail server setting of their own. Mail and push go through the Monosign API, configured on the UI node with `MONOSIGN_API_URL`, `MONOSIGN_APPID` and `MONOSIGN_APPKEY`.
If any of the three is empty, the send is skipped, but the issue is still marked as notified. Those notices are lost for good. After you add the variables, expect mail only for new state changes and reminders, and tell the operations team that the earlier DOWN notices will not arrive.
::diagram obs-monitor-notify
### An empty Audits list is not proof
Monosync writes an audit row for configuration changes made in the web UI. Several paths write none:
- Changes through REST API v1, such as a source setting or a variable value.
- A settings import.
- Create and update commits in the UI itself, when the installation runs with `MONOSYNC_COMMIT_DISABLE=true`.
When a source's settings changed overnight and `Audits` shows nothing, the conclusion is not that nobody changed it. Check who holds the API key and whether an import ran.