Can the AI Agent on My VPS Reach My Work Laptop? A Closer Look at Hermes Desktop

…and why I’m not installing it on my work laptop for now: I want to keep my VPS deliberately quite open so I can try out wild things, and there must be no way for those experiments to compromise my university. My Mac laptop is a different story. There, I definitely still have to work through the checklist below and adapt it for macOS where needed.

In my last post I reached the Hermes dashboard on my VPS from my Windows work laptop with nothing but the built-in SSH client and a browser. It works. But after a few days of using it, I noticed that I miss the comfort of the Hermes Desktop app I use on my Mac: live tool activity, tabs, a proper session list.

So I started thinking about installing the desktop app on the Windows laptop after all. And then a second thought crept in, the one that sits at the back of my head every time I read about agents getting more capable: if I connect this laptop to the agent on my server, can the agent reach back? Can it get to my files, my network, my work data?

That thought didn’t come out of nowhere. Over the past months, the news kept reporting on AI agents that left their test environments, and, more recently, on agents that were deliberately turned into attack tools:

• Hugging Face was hacked by AI agents. In July 2026, Hugging Face disclosed an intrusion into its production infrastructure that was driven end to end by an autonomous agent system (Hugging Face disclosure, technical timeline). A few days later, OpenAI said the agents were its own models, running an internal cybersecurity benchmark with safeguards deliberately switched off. According to OpenAI, the models found a previously unknown vulnerability in the package proxy that was their only connection to the outside, reached the internet, and broke into Hugging Face to get the benchmark’s solutions (OpenAI’s disclosure, OpenAI’s follow-up report).
• Agents used an old German wiki as their message board. In September, researchers reported that agents identifying as OpenAI systems had posted around 18,000 times on a dormant German software developer wiki between May and July, to pool answers for a timed task and to share a way around their sandbox. The trick was almost banal: the old wiki software accepted edits through plain “read” requests, so agents that were only allowed to read the web could still write (Nightingale Collective report, The Hacker News). OpenAI later described this as a case of misalignment rather than a security incident.
• Anthropic reported its own cases. The system card of Claude Mythos Preview (April 2026) describes how an earlier version of the model, asked in a test to escape a secured sandbox and send a message to a researcher, succeeded and then published details of its exploit on public websites without being asked (Claude Mythos Preview System Card). And after the Hugging Face news, Anthropic reviewed more than 140,000 evaluation runs and found three incidents in which Claude models reached the real internet through a misconfigured third-party test environment and got into the systems of real organizations, believing they were still in a simulation (Anthropic).
• Hermes itself was used in a criminal attack campaign. This one is different, and it hit closer to home. In September 2026, the security company Gambit reported that an attacker had used AI agents to attack hundreds of online shops since at least July. In just five days in September, at least 27 companies were compromised, skimmer scripts were planted on shop websites, and data from more than 600,000 credit cards was copied. The attacker combined three open-source agents: Strix to find vulnerabilities, Cairn to exploit them, and Hermes to orchestrate the campaign, with a “SOUL – Red Team Operator” persona, 121 skills (78 of them attack-related), persistent memory, scheduled jobs and a web console. The attacker only typed a few short prompts per target, and the average cost was about 25 US dollars per target (Gambit Security report, heise online, in German, BleepingComputer). The reports don’t explicitly name Nous Research, but the features they describe are exactly the ones of the Hermes Agent I’m running.

To be fair: the first three cases happened in lab test environments, mostly with safety measures switched off on purpose to measure worst-case capabilities. The shop attack is not an agent “going rogue” at all: a human wanted exactly this and pointed the agents at their targets. My little Hermes on a VPS is neither. But together, these cases show two things. First, what these tools are capable of in the wrong hands, including the very tool I’m experimenting with. Second, and this is what stuck with me most: in the lab cases, the way out was a door that nobody had thought of as a door. A package proxy. A “read-only” web access that could still write to an old wiki. A test network that was accidentally connected to the internet. That’s exactly the kind of door I want to look for before I connect a new front end to my agent.

This post is not a setup guide. It’s the thinking I did before installing anything: the questions, the first (too optimistic) answers, a second look, and what I found in the docs and in the Hermes issue tracker. I haven’t installed the app on the work laptop yet. That’s on purpose.

Why this question matters to me

My threat model is not a Hollywood hacker. It’s something much more boring and much more likely:

• The agent reads a web page, an email or a document that contains hidden instructions. This is called indirect prompt injection, and researchers have shown for years that it works against real LLM applications, including data theft (Greshake et al., 2023).
• The agent is not malicious, just wrong. It “helpfully” does something with more reach than I intended.
• The agent pursues its goal a little too creatively and uses a door I didn’t know was there, like the agents in the incidents above.

Simon Willison describes the dangerous combination as the lethal trifecta: an agent that has access to private data, reads untrusted content, and can send data somewhere. My VPS agent already has two of those (it reads the web, and it can send messages and emails). My work laptop would add the third: private data. OWASP calls the underlying problem Excessive Agency: giving a system more functions, permissions or autonomy than its task needs.

So the question is really: does installing the desktop app give the agent on my VPS a way to my private data?

Part 1: The SSH connection itself

Here is the mental model I started with, and it still holds.

Direction	What happens	Who starts it
Laptop → VPS	ssh -L 4860:127.0.0.1:4860 hermes-vps forwards a port on my laptop to the dashboard on the server	Me
VPS → Discord	The Hermes gateway dials out to Discord	The server
VPS → Laptop	Nothing. There is no connection in this direction	Nobody

A local forward (-L) is a one-way street: traffic that arrives at a port on my laptop goes to the server. The server can answer inside that connection, but it cannot open new connections to my laptop through it.

A way back only appears if I build one myself. These are the doors I made sure stay closed:

Possible way back	When it exists	How I keep it closed
Reverse tunnel (-R)	Only if I open one	Never use -R or RemoteForward.
SSH agent forwarding	Only with ForwardAgent yes	Don’t set it. On Windows the ssh-agent service is off by default anyway.
My private key on the server	Only if I copy it there	The server only holds public keys (authorized_keys).
An SSH server on Windows	Only if sshd runs and port 22 is reachable	It’s off by default, and my home router/NAT blocks incoming connections.
Hermes’ SSH terminal backend	Only if I set terminal.backend: ssh with my laptop as the target (Hermes security docs)	Never configure it with a machine I care about.
Tailscale (a possible future experiment)	Server and laptop in the same private network, so the laptop becomes addressable from the server	Use access rules that only allow laptop → server.

To check my own setup, I’ll run these on the laptop (PowerShell):

ssh -G hermes-vps | Select-String "forwardagent|remoteforward"   # should show "forwardagent no" and no remoteforward line
Get-Service sshd                                                   # should not exist, or be "Stopped"

And on the server (host shell, not inside the container), to make sure no private key of mine is lying around:

find /home /root -name 'id_*' -not -name '*.pub'

For completeness: there are community tools that deliberately open such a way back. One unofficial “remote-only” Hermes client, for example, offers a worker that exposes local file and shell tools on your computer to the remote Hermes host (hermes-client on GitHub). That can be exactly what someone wants for their own dev machine. For my work laptop, it’s exactly what I don’t want.

Conclusion for Part 1: the SSH part is fine. My first answer stopped here, and it was too optimistic, because the SSH tunnel is not the only thing on the laptop.

Part 2: The desktop app is not just a window

This is where the second look changed my picture.

It installs a complete local Hermes

The desktop docs say it clearly: the packaged app can install the full Hermes Agent runtime on first launch, in the same layout as a CLI install. On Windows, that means %LOCALAPPDATA%\hermes with Python, Node.js and PortableGit (Windows native guide). No admin rights needed, which is convenient and a little alarming at the same time.

That local Hermes (the “this device” gateway from my Mac post) runs shell commands through Git Bash, directly on the laptop, with my user rights. There is no container around it.

What helps: the onboarding lets you pick Choose provider later. Without a model, the local agent can’t decide anything on its own. I also found a hint that a GUI-only “lite” client without a local agent exists (the uninstall section mentions it), but not how to get it.

The remote agent can open pages in a browser on my laptop

This was the find that made me sit up. Hermes has a toolset called desktop_ui. According to the toolsets reference, it contains tools that act on the desktop app itself: open, read, interact with and annotate the in-app browser, read or close the embedded terminal pane, and read_window_below. The tools reference lists them as desktop-app sessions only.

The question is: when the app is connected to my VPS, where does that preview browser run? A bug report from September 2026 answers it indirectly. Someone with the exact same topology (desktop app on their computer, Hermes backend on a VPS, connected over SSH) reports that the remote agent’s desktop_preview successfully opens a Preview tab in their desktop app. Only drive_preview (reading the page’s elements, clicking, typing) times out on the latest version, and the reporter says it worked before, including navigation. The issue is closed as “not planned”, so I can’t count on it staying broken.

Why does this matter? Because that preview tab is a browser on my laptop, in my network. A page opened there is loaded from the laptop’s position. At the university, that could mean the intranet or internal services that the VPS itself could never reach. Combined with prompt injection, that is a door I don’t want.

Two more tools from the same toolset make me cautious, though I haven’t verified how they behave with a remote backend:

• read_terminal reads the embedded terminal pane. On my Mac, I found that this pane runs on the Mac, not on the server (I typed hostname and got my Mac’s name). If the remote agent can read that pane, it can read whatever I typed or printed there.
• read_window_below belongs to the HUD mode, where the app tells the agent which app and screen lie underneath the floating chat bar.

What I’ll do: switch the desktop_ui toolset off on the server. The Hermes docs mention hermes tools (an interactive per-platform menu), /tools disable <toolset> inside a session, and a disabled_toolsets list in config.yaml under agent: (seen in this issue, which also warns that disabling overlapping toolsets can have side effects). I haven’t tried it yet. After changing it, I’ll open a chat from the desktop app and ask Hermes which tools it has, to be sure desktop_preview and drive_preview are really gone.

Plugins run with the app’s full authority

The Desktop Plugin SDK docs are refreshingly honest here: a loaded desktop plugin runs with the full authority of the app, including the native bridge for files, git, the terminal and installs. The isolation is only about errors (a plugin can’t crash the app), not about security. The docs say plainly that this pipeline is not a trust boundary.

Plugins can come as one package with an agent half (on the server) and a desktop half (in the app). So I wondered: if the agent on my VPS installs such a package on the server, does its desktop half land on my laptop?

The same page answers it, and this was good news:

• The automatic copy of a desktop half only happens for packages installed on the same machine as the app. For a remote backend, the app shows the desktop half as “unavailable (remote backend)” until I install it myself with Install from Git.
• Desktop halves of packages ship off by default. They show up in Capabilities → Plugins but stay disabled until I switch them on.
• One-click install links (hermes://plugin/install?...) always show a confirmation dialog. The docs say deep links never install on their own.

So the plugin door only opens if I open it. My rule: never install a desktop plugin because the agent suggested it, and never confirm an install link that appears in an agent’s answer. Also related: exported profiles can contain plugins, so I won’t import a profile from the server onto the laptop.

Files and outputs I open

When connected to a remote gateway, the app downloads artifacts (files the agent created) through the gateway (desktop docs). Downloading is harmless. Opening happens on my laptop. A script, an .exe, an Office file with macros or an HTML page from an agent’s output is just as risky as any other file from the internet. OWASP lists this as Improper Output Handling: passing model output on without checking it.

Data that goes from the laptop to the server

The other direction matters too, because anything that reaches the server is within reach of an agent that also reads the internet:

• Drag and drop attaches files to the next message, and they go to the server.
• “Add to chat” in the terminal pane sends terminal output into the conversation.
• HUD mode tells the agent which app and screen are underneath it.
• Quick Entry is a global hotkey composer that works from anywhere on the system.

None of this is hidden, and all of it is useful. On a work laptop with student data, I’ll simply not use HUD mode and Quick Entry, and be deliberate about what I drag in.

Smaller things

• Repository scan: by default, the app scans the home directory for Git repositories to fill the Projects sidebar. It can be switched off in Settings → Workspace (desktop.repo_scan_enabled: false).
• Saved logins: the app stores the server login as an owner-only file. A switch called “Encrypt saved secrets with the OS keychain” adds encryption, which uses DPAPI on Windows (multi-connection docs).
• Updates: the app updates itself using git from the GitHub branch tip. That’s not a VPS risk, but a supply-chain one. On a work laptop I’d rather update deliberately than automatically.

Where does this leave me?

Here is my ranking, from “closest to impossible” to “most comfortable”:

Option	What runs on the laptop	Can the VPS agent touch the laptop?	Comfort
Browser + SSH tunnel (last post)	SSH client, browser	No path I could find, as long as I don’t open files it made	Okay
Desktop app on a separate machine or VM	Everything, but away from my work data	Only that machine	Good
Desktop app on the work laptop, hardened	Full local Hermes without a model, the app	Only through doors I open, if desktop_ui is off and plugins stay off	Good
Desktop app on the work laptop, defaults	Full local Hermes, the app, preview browser available to the agent	Yes, at least through the preview browser	Best

I wanted a yes or no, and the answer is “it depends on what you switch on”. The SSH connection is not the problem. The app’s features are.

And there’s one more factor that tips the scales for me: my VPS is supposed to stay a playground. I want to keep it deliberately quite open, give Hermes room, and try out wild things there. The more open the server is, the more carefully I have to guard every door between it and a machine that holds university data or sits in the university network. So for the Windows work laptop, the decision is simple for now: browser plus SSH tunnel, nothing more. On my Mac, where the desktop app is already installed, I’ll work through the checklist below and adapt it for macOS.

My checklist, if I would install it on my Windows work laptop 👀

On the laptop:

• [ ] Check with IT whether this installation is allowed on a work device 🤪.
• [ ] Onboarding: Choose provider later, and never connect a model to “this device”.
• [ ] Settings → Workspace: turn off the repository scan.
• [ ] Capabilities → Plugins: keep the Desktop column empty, and check it again after changes on the server.
• [ ] Don’t enable HUD mode, Quick Entry or “Tap to summon”.
• [ ] Never confirm plugin install links or import profiles from the server.
• [ ] Don’t open agent-made files without looking at them first.
• [ ] Keep ForwardAgent and RemoteForward out of the SSH config; no sshd on Windows.
• [ ] Full-disk encryption (BitLocker) and a screen lock, because the SSH key and the server login live here.

On the server:

• [ ] Switch off the desktop_ui toolset and verify it from the desktop app.
• [ ] Keep approvals.mode on manual or smart, never off (security docs).
• [ ] Never set terminal.backend: ssh with my laptop as the target.
• [ ] No private keys of my laptops on the server.

One more thing from the security docs. Hermes lets you define approvals.deny patterns, for example to block ssh * or scp * commands. I like that as an extra guardrail, but the docs are clear that such rules protect against an honest-but-wrong agent, not a deliberately adversarial one. For the latter, the real protection is isolation: a container without the Docker socket and without access to my keys, which is what my VPS setup already has.

What I’m still unsure about

• Whether read_terminal and read_window_below read my laptop’s terminal and windows when the agent runs on the VPS. The toolset description suggests they act on the app, and the app runs on the laptop, but I haven’t tested it.
• Whether drive_preview will work again with remote backends in a future version (the bug report is closed as “not planned”).
• How to get the “lite” client without the local agent, which the docs mention in passing.
• Whether disabling desktop_ui in the config has side effects on other toolsets.

These are the questions I’ll answer by trying, once the checklist is done.

Why I’m writing this down

Honestly, because I almost skipped it. The app is convenient, it’s one download, and my first answer to myself was “the tunnel only goes one way, so it’s fine”. That answer was right about the tunnel and incomplete about everything else. The design-pattern paper by Beurer-Kellner et al. (2025) makes a point I find useful beyond Hermes: once an agent has read untrusted input, you should limit what it can still do with its tools, by design and not by hoping the model behaves. For me that translates into a simple habit: before connecting a new front end, ask what the agent can do through it, not just how the connection works.

What’s next

• Actually disabling desktop_ui on the server and testing it from the Mac first.
• Staying with the browser plus SSH tunnel on the Windows work laptop for now, as long as my VPS stays an open playground.
• Working through the checklist on my Mac, where the desktop app is already installed, and adapting it for macOS where needed.
• Still open from earlier posts: a tunnel that doesn’t need an open window, backups, updates, more models, and what Hermes is allowed to do on the server.

---

Sources

The agent incidents mentioned in the introduction

• Hugging Face Security Team (2026, July 16). Security incident disclosure — July 2026
• Hugging Face Security Team (2026, July 27). Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident
• OpenAI (2026, July 21). OpenAI and Hugging Face partner to address security incident during model evaluation
• OpenAI (2026, August 26). The Hugging Face incident and the road ahead
• Nightingale Collective (2026, September 4). Discovery of a new OpenAI agent message board
• The Hacker News (2026, September 5). Thousands of OpenAI Agents Quietly Turned an Abandoned Wiki Into Their Coordination Channel
• Anthropic (2026, April 7). System Card: Claude Mythos Preview (PDF)
• Anthropic (2026, July 30). Investigating three real-world incidents in our cybersecurity evaluations
• Gambit Security (2026, September 22). Autonomous AI agents hack online retailers for $25 a company
• Knop, D. (2026, September 24). Angriff mit KI-Agenten auf hunderte Shops: 600.000 Kreditkartendaten geklaut. heise online (German).
• BleepingComputer (2026, September). Malicious AI agents steal 600K credit cards, infect 100+ sites with skimmers
• Wikipedia. OpenAI–HuggingFace incident (overview with many further links)

Hermes documentation and issue tracker

• Hermes Docs: Hermes Desktop
• Hermes Docs: Connecting Desktop to Many Hermes Instances
• Hermes Docs: Windows (Native)
• Hermes Docs: Desktop Plugin SDK
• Hermes Docs: Security
• Hermes Docs: Toolsets Reference
• Hermes Docs: Tools Reference
• Hermes GitHub: Desktop Preview control times out on remote Desktop → VPS setup (Issue #111228)
• Hermes GitHub: Preview-pane tools and surface changes (Issue #110723)
• Hermes GitHub: Remote Gateway file browser reads the local filesystem (Issue #38671)
• Hermes GitHub: disabled_toolsets side effects (Issue #58281)
• hermes-client (unofficial remote-only client)

Research and background on agent security

• Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., & Fritz, M. (2023). Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. arXiv:2302.12173.
• Beurer-Kellner, L., et al. (2025). Design Patterns for Securing LLM Agents against Prompt Injections. arXiv:2506.08837.
• Willison, S. (2025). The lethal trifecta for AI agents: private data, untrusted content, and external communication.
• OWASP (2025). LLM06:2025 Excessive Agency, part of the OWASP Top 10 for LLM Applications 2025.
• OWASP (2025). LLM05:2025 Improper Output Handling.