← Back to Blog

What's in 9.10.0

9.10.0 is a maintenance release. It is almost entirely correctness and reliability work: a number of things behaved differently from the way the documentation described them, and this release brings the two back into line. There is one security fix, and one known issue that is not fixed here and is worth reading if you use config backup.

Config push and rollback

Config restores and template pushes now run inside a RouterOS safe-mode session. The change is applied inside the session, TOD dials a new connection to confirm the device still answers, and only then commits. If the device does not answer — or if TOD crashes, or loses its network mid-push — the session drops and RouterOS reverts every change made in it, on its own, without rebooting. This replaces the previous rollback mechanism entirely. Verified on real hardware; device uptime ran continuously through every test.

One limit worth knowing: safe mode's undo buffer holds 100 actions, and past that RouterOS silently keeps the changes while still reporting that it is in safe mode. A push over that size is now refused rather than applied without a safety net. Split a large template into stages.

This does not cover direct edits in the config editor. Those still apply immediately with no rollback, and the toggle that implied otherwise has been relabelled.

WinBox sessions

Abandoned WinBox sessions were not being cleaned up promptly, so they held one of ten concurrency slots — and an authenticated connection to your router — far longer than intended. Disconnects are now detected reliably, a disconnected session enters a 30-second grace period, and sweeps run every 30 seconds. A slot comes back in about a minute instead of hours, and a reconnect inside the grace window cancels the teardown. A crashed session also used to leave processes behind while the worker reported its slot free; the worker now reaps its own orphans. Verified against real hardware.

Device onboarding

Onboarding now completes a real protocol handshake — TCP, then TLS, then an actual API login — rather than only checking that a port accepts connections. The practical difference: a device that answers on api-ssl but cannot negotiate a usable session used to onboard green and then fail every poll afterwards. It now fails at adoption instead, and says why in those terms rather than "unreachable". There is also a test-connection endpoint so you can re-run the same probe against a device you have already added.

Free tier: 250 devices

The free tier is 250 devices, which is what the licence says. Some documentation quoted a higher number; the licence is the number that counts. The docs are corrected, and there is now a test that fails the build if the licence, the code and the documentation ever disagree.

Security

A validation error could echo submitted credentials back in the response. Tracked as GHSA-89xh-5xwx-pvw6 — moderate, affects 9.9.0 and earlier, fixed in this release. An HTTP 422 from a credential-related endpoint could include the request body it was rejecting. It required an authenticated account with permission to create or update credentials, but any validation error would trigger it.

The 422 handler now keeps only which field failed and why, and drops the submitted input entirely. If you have been calling the API and logging response bodies, those logs are worth a look, and rotating any credential that went through a failed create or update is the safe move.

Bug fixes

Under the hood

Mostly CI and operational plumbing. The TypeScript build now genuinely checks the frontend sources, and the errors that surfaced are fixed rather than silenced. Container vulnerability scans can now fail a build on CRITICAL and HIGH. The WinBox worker is linted and has a session-lifecycle test suite. Postgres and Redis carry a restart policy, so they come back on their own after a host reboot instead of leaving the API and poller retrying against nothing. And the poller's health check probes each dependency and returns a 503 naming the one that failed, instead of always reporting healthy.

Documentation

A pass over the website and docs to bring the claims into line with the code — the config push description most of all. Several pages also described the product as open source; the licence is BSL 1.1, which is source-available, and it converts to Apache 2.0 in March 2030. None of this changed what the software does. It changed what the site claims it does, which had drifted further than I would like.

Known issue: a stale SSH host key blocks config backup

One defect found while working on this release is not fixed in it. A proper fix — a pin-reset action in the product — is scheduled for the next release.

A stale SSH host key permanently blocks config backup for that device. TOD pins a device's SSH host key the first time it connects, and nothing in the product can clear that pin afterwards. There is no endpoint and no UI for it, and the one piece of code that writes the column only runs when no fingerprint is stored yet — so it will never overwrite a stale one.

That matters if a device's host key legitimately changed: RouterOS reinstalled, hardware swapped, keys regenerated, or an IP now answered by a different device. Backup for that device stops and never resumes. It does not self-heal, and restarting the poller does not help — the block is held in memory, so a restart clears it and the next attempt immediately re-blocks.

How to spot it. Config backup for one device goes quiet while everything else keeps working. There is a metric label that means this and nothing else:

mikrotik_config_backup_total{status="skipped_hostkey_blocked"}

Any non-zero rate on that label is this bug. If you alert on one thing from this post, alert on that.

To see what the device is presenting now:

ssh-keyscan <ip> | ssh-keygen -lf -

That prints the same SHA256: format TOD stores, so it is directly comparable to the value in the database.

Before you clear anything, read this. A host key mismatch is also exactly what a man-in-the-middle attack looks like. Clearing the pin tells TOD to trust whatever the device presents next. Confirm out-of-band that the device really was reinstalled or replaced. If you cannot account for why the key changed, investigate that instead — the blocked backup is the system working.

Recovery is manual SQL. There is no nicer way to say it:

UPDATE devices
SET ssh_host_key_fingerprint   = NULL,
    ssh_host_key_first_seen    = NULL,
    ssh_host_key_last_verified = NULL
WHERE id = '<device-uuid>';

Run it as the postgres superuser. The application role has no update privilege on those columns at all — only the poller does — and devices carries FORCE ROW LEVEL SECURITY, so a tenant-scoped connection will not do it either. Match on id, not hostname: hostnames are not unique across tenants and you can silently clear the wrong device's pin.

Leave the fingerprint NULL rather than pasting in the new value. The poller re-pins it automatically on the next cycle and backups resume with no restart. A typo in a hand-written fingerprint pins a value no device will ever present, which reproduces this same fault in a form that is harder to recognise.

The Other Dude is source-available MikroTik fleet managementdocumentation · GitHub