🤖 Remote task agents

← Developer wiki

ZuloOne in the cloud owns the cron and the queue. On the customer's machine runs the ZuloOne Agent (ZuloOne.Agent): it initiates the connection by itself (HTTPS + SignalR) and executes the TaskScriptBase scripts Core has handed it.

There are no inbound ports at the store.

How to connect one#

1. In the UI, Administrator → Agents, create an agent (a name, tags). An existing one is edited in the same place: the name (on blur) and the tags. Tags reach the live connection immediately — the agent does not have to be reconnected. A change of name has to match Agent:Name in zuloone.agent.json on the machine, otherwise the next connect will not find the record.
2. Pick Windows or Linux.
3. Download ZIP (Windows) or Download .deb (Linux). Chrome complains about an MSI by itself as "unusual", so Windows is a zip. Linux is the .deb directly — wrapping it in a zip doubled the wait. The URL of this ZuloOne and the enroll key are already baked into it.
4. On the customer's machine, unpack the archive:
   • Windows: open the MSI (administrator rights are needed). The ZuloOne Agent service is installed into Program Files\ZuloOne\Agent and starts automatically.
   • Linux: sudo dpkg -i zuloone-task-agent-….deb (the systemd unit is zuloone-agent).
5. On /agents the agent goes online.

The click does not compile the agent. Core already has the published binaries in agent-dist (from the image). Download patches the enroll slots and compresses the tree; Linux gzip of a self-contained publish is the slow part and can take a minute. The office banner Preparing the installer is that wait, not a from-source build.

That wait is also why the button has two states. While the server is still building the package there is nothing to be a percentage OF, so the button spins and the banner says Preparing the installer; once the first byte arrives the banner goes and the button counts Download Windows — 42%. The request carries no timeout at all (timeout: 0): the house default of 180 s aborted a ~100 MB download on anything slower than about 5 Mbit, which reads as a server fault and is not one.

Note

Print agents have no percentage, and that is correct. Their zip and .deb are DIRECT links to a public GitHub release, so the browser downloads them itself and draws its own progress. Fetching them through the page instead would need the asset to allow this origin — it sends no Access-Control-Allow-Origin — and would pull 93 MB through the tenant and hold it in memory to gain a bar that already exists. A percentage in our UI means "the file comes from us and cannot be a navigation": the task agent, which is built per agent and bakes an enroll key. The workspace mirror has no session cookie, so a plain href 401s; it mints a 15-minute signed URL and the browser downloads the zip itself — native progress, no SPA heap.

Downloading again issues a new key — the previous installer stops connecting. While the key from the moment of creation is still in the browser session, the first download reuses it.

The address in the package is taken from PublicBaseUrl. Download from localhost and the store will connect to its own localhost. For the cloud, set a public HTTPS URL.

Core serves the installers out of Agent:DistributionPath (/app/agent-dist in the image): the win-x64 tree (a self-contained zip, or an MSI out of the wixl template) and linux-x64 (the .deb is built from it on each download). Windows is a zip: Chrome marks an unsigned MSI by itself as "unusually downloaded". Linux is the .deb. The image builds both RIDs on Linux CI (EnableWindowsTargeting). The Windows button is always there in the UI; there is nothing to download until the image has a win-x64 payload.

Automatic updates#

An installed Windows service or Linux systemd agent checks for an update when it opens a new session, before it joins the SignalR hub. With the default 8-hour session lifetime, this normally bounds update pickup to about 7.5 hours after a Core release. A console run and the workspace mirror do not update themselves.

The agent uses its session token to request an update for its own supported runtime. Core serves a payload only to an enabled agent session, only when the published agent-dist version is newer, and only when that payload version matches Core. The archive contains binaries, not zuloone.agent.json, the enrollment key, or the manual install scripts. The response includes a SHA-256 digest; the agent verifies it and the staged assembly version before starting the updater. TLS and the agent session authenticate the source; the digest detects transfer or staging corruption, it is not a separate code-signing key.

An update check runs before the agent opens its task hub. If a prior task is still running when a session is renewed, the agent skips that update attempt and retries at the next session. The updater runs as a separate process, stops the service, backs up the installed directory, preserves the local enrollment configuration, and then replaces the binaries. It starts the service and restores the backup if the new service does not start. The next agent process removes staging directories older than two days.

Set Agent:AutoUpdate to false in zuloone.agent.json or the environment variable Agent__AutoUpdate=false to opt out. Missing or invalid values mean enabled. Changing this setting takes effect on the next agent process start.

An older installation that predates the updater cannot download its first update. Install one current package once from Administrator → Agents; later versions arrive automatically. Linux still needs a systemd service and the standard /opt/zuloone/agent installation. Windows updates work from the standard Program Files\ZuloOne\Agent service installation.

The fallback path (enrolling by hand)#

The key is shown once. On a machine that already has ZuloOne.Agent.exe:

ZuloOne.Agent.exe --enroll https://zulo.example.com zuloone-agent.…
ZuloOne.Agent.exe --install-service

Or set ZuloOne:BaseUrl, Agent:Name and Agent:EnrollKey in zuloone.agent.json.

A job's target#

Offline: one slot in the queue, a TTL (15 minutes by default), then Expired.

Outbound integration through an agent#

A local operator (M.E.Doc on an accountant's PC) is unreachable from the cloud: there are no inbound ports on the LAN, and a loopback from Core is SSRF. So a connection may name an agent tag (TaxAuthorityConnection.AgentTag / UaBankConnection.AgentTag). The queue still lives on the host, and the last HTTP hop is made by the agent — including http://127.0.0.1:….

An agent carrying that tag has to be online. An older agent with no SendOutbound handler returns an error, and the document does not count as sent. Cloud metadata (169.254.169.254) is refused by the agent exactly as it is by the host.

The erpn-send channel does not hard-code a tag: one client's operator is in the cloud, another's is a service on a PC. It is a field on the connection card.

Agent HTTP responses return as SignalR RPC completion messages. AgentHub has a bounded 32 MiB receive limit through AgentHubMessagePolicy; other hubs keep the default 32 KiB limit. The larger envelope allows the agent's bounded HTTP body plus JSON escaping and headers. Leaving the agent hub at the browser default disconnects it when a response exceeds 32 KiB: a small connectivity probe can succeed while an OData metadata response repeatedly disconnects and reconnects the same agent. This is a transport failure, not an authentication failure. AgentHubMessagePolicyTests exercises both a 256 KiB RPC completion and a 4 MiB body requiring JSON escaping, twice on the same connection, and verifies that the default hub limit is unchanged.

Updated agents read complete HTTP bodies up to 4 MiB through AgentResponseBody. The bound counts bytes, not characters. A larger response fails explicitly; partial XML or JSON is never returned as a successful response. Older agents silently truncate at 1,048,576 characters: upgrading Core alone fixes the disconnect but cannot fix that truncation. Update the installed agent as well. No credential change is required. AgentResponseBodyTests checks complete metadata above 1 MiB and rejection of oversized and multibyte responses.

Permissions#

The agent has a role of its own. "Run as" is limited to the users on the agent's allow-list. A short run token travels with the task; the enroll key cannot impersonate people.

The run token is inert on its own. Core mints it bound to the enroll session of the machine that received the job (zulo_session on both JWTs). Assemblies, GetService RPC, and complete must send that session JWT in X-ZuloOne-Agent-Session. The session never rides on the RunJob offer, so a stolen run token cannot invoke. A new enroll session invalidates in-flight run tokens of the previous one. Compromising the agent process is having the machine: the enroll key lives there.

GetService<T>() on the agent is REST POST /api/agent/runtime/invoke under the run token plus the session header. The script body runs on the customer's machine (disk, LAN, 1C). Each GetService call executes on Core through that RPC. It works for two kinds of interface:

• a platform service on the allow-list (IDataService, IDocumentManager, IMetadataService, IOutboundCall… in AgentRuntimeInvoker);
• a model service (ZuloOne.Services.Contracts.IXxx). Core resolves it through the service registry, the way an in-process script's GetService<IXxx>() does, against the live contract generation.

Not on the list: IPermissionService, ISchemaSynchronizer, ISqlService, wipe. Those skip the HTTP gates.

Looking a model service up in DI answers "Service IXxx is not registered": model services are never there — they live only in the service registry.

Runtime service proxies#

Model service interfaces belong to collectible runtime assemblies. The agent implements these interfaces with AgentCollectibleServiceProxy, using a RunAndCollect assembly that forwards each call to the existing AgentServiceProxy HTTP dispatcher. The factory cache is scoped to the runtime load context, supports concurrent first use, and removes its entry when that context unloads. Platform interfaces in ordinary assemblies still use DispatchProxy; the HTTP endpoint, run token, and session header are unchanged.

Warning

Do not use the ordinary AgentServiceProxy type as a DispatchProxy base for a collectible interface. On .NET 9 the base assembly's load context controls whether the generated proxy is collectible, causing A non-collectible assembly may not reference a collectible assembly. Moving only a bridge base into the runtime context fixes creation but still leaves that context retained by the DispatchProxy cache in the regression scenario. Direct interface implementations avoid that cache entirely.

RPC serialization and deserialization use independent JSON metadata resolvers so shared serializer options do not retain runtime DTO types. .NET 9 also has a short-lived member-accessor cache: stale delegates are evicted on subsequent metadata activity, not immediately on Unload(). AgentRemoteServicesTests checks eventual collection after that normal eviction, as well as repeated runtime generations, concurrent creation, inherited methods, generic methods, and DTO results. The compiler must release its own references before its context can unload; completing a job alone does not unload the shared compiler.

How to tell it is alive#

On /agents: the online/offline badge and Last seen. On a job's card — the run history (Queued / Running / Success / Expired). In the trigger column — Scheduled or Manual (TriggeredBy), and next to it who executed it (the agent or core).

online is the memory of the current Core process. Updating the container clears that registry; the agent has to reconnect by itself.

Reconnecting#

The agent holds SignalR with a short AutomaticReconnect (packet loss, the minutes of a Core restart). When the hub has closed, three heartbeats in a row have failed, or the JWT is about to expire, it does POST /api/agent/session again and brings the hub back up. Restarting the service is not needed for that.

In the service log: Session opened / Connected to / Agent hub closed; opening a new session / Agent session failed; reconnecting in 10s. A run of Expired: no eligible agent in a job's history means Core saw no live connection until the agent opened a new session.

In Administrator → Journal the agent's rows carry CallStack (the Activity chain Agent.Session / Agent.RunJob plus ZuloOne.* frames) and TraceId. Older events without those fields stay as they are — the stack only appears once both the agent and Core have been updated.

Where next#

• Platform architecture
• Chain of Command