Hermes Local Model Benchmarks: The GPU and Model Combinations I Would Use

For long-context Hermes work, I would start with Qwen3.5 35B on two 16GB RTX 5060 Ti cards. The tested route retained 239,153 input tokens through LiteLLM and passed separate tool, streaming, vision, queued-request, and cancellation checks. Qwen3.6 is my one-card alternative. On the RTX 3060 Ti and RTX 3070 systems, I would start with Qwen3.5 2B or 4B.

Those choices come from running models through the inference paths used by the agent. I tested long-input retention, tool round trips, streamed responses, image questions, and model release. I also tested the modes and memory settings behind the routes, because a model loading successfully says little about its usefulness during a long agent request.

Hermes requires at least 64,000 tokens of context for agent use with tools. I used 65,536 actual input tokens as the long-context qualification threshold. The input count matters more than the context size advertised by the server. ( Hermes Agent provider documentation )

Which GPUs were in each system #

The system letters identify the hardware combinations throughout the results.

Installed card count does not mean every request used every GPU. The Qwen3.5 35B and Flash Next trials used both cards in System A. Qwen3.6 used one card. The successful System E Qwen3.5 4B vision rerun used one selected card from an eight-card pool while a ninth card stayed excluded from AI work.

Treat the durations below as results for the tested model, runtime, and system together. These runs compare usable configurations. They do not isolate GPU architecture as the cause of a speed difference.

Two RTX 5060 Ti cards make Qwen3.5 35B my long-context choice #

Qwen3.5 35B produced the largest retained-input result in these tests. The direct Ollama progression retained 248,279 input tokens inside a 262,144-token runtime context. A separate streamed gateway request retained 239,153 input tokens, returned the opening marker, and ended naturally after 31 output tokens in 148.12 seconds.

The gateway result matters for Hermes because the request passed through the broker and LiteLLM. Separate requests completed a tool round trip, recognized a red image, and returned two correct answers when submitted together. The broker queued those two requests. Closing a streamed request released the broker and unloaded the model within 60.31 seconds.

The progressive direct requests reused an expanding prefix. Their durations describe warm continuation behavior. The 54.215-second result is not an independent cold processing measurement for a fresh 248K prompt.

At the largest direct stage, the model reported about 27.3 billion bytes resident across the two GPUs. Per-card memory peaks reached 14,964 and 12,386 MiB.

Peak temperatures reached 62 and 63 C, and the run recorded no OOM or Xid fault. The input left 13,865 tokens inside the configured context for continuation. The short marker answer used 31 of those tokens.

I would use this route when retaining a large working set matters more than the shortest initial response. The tool and vision checks establish useful agent behaviors, while the long-input test establishes retention. They are separate workloads.

Qwen3.6 is the one-card option #

Qwen3.6-35B-A3B UD-Q4_K_XL retained 131,000 input tokens on one RTX 5060 Ti. The other card stayed idle during the test. This gives the system a useful alternative when dedicating both cards to one model is undesirable.

The 131,000-token request approached the 131,072-token runtime limit. For an agent, I would leave more space for the answer and subsequent tool messages. A near-limit retention result is useful evidence of input handling, rather than a sensible default input budget.

Tool use passed with thinking disabled. Default thinking consumed the 512-token trial allowance before issuing the tool call. The successful pilot also passed quick, streamed, and vision requests. Its initial 65,845-token context check took 139.57 seconds.

Qwen3.8 27B wins the matched 66K response comparison #

For the same 66,007-token marker workload on System A, Qwen3.8 27B returned the answer fastest. I would keep this route available for requests around the tested input size, alongside Qwen3.5 35B for the larger working set.

Qwen3.8 27B took about one third of the matched Qwen3.6 and Flash Next request durations. These are complete-request observations with the tested runtimes, rather than a hardware-only comparison.

Quick responses, streaming, and tool use also passed. Thinking-disabled mode completed the full color sequence and tool round trip, while the thinking trial returned only part of the color sequence within its 512-token allowance.

Flash Next remains a less attractive default on this system because of memory pressure. Its 65,847-token pilot passed in 201.41 seconds at 98,304 runtime context. Two earlier 131,072-context profiles reached their memory limits before readiness.

The later route reached its 54 GiB cgroup cap. I would choose a route with more memory headroom for unattended work.

The route labelled flash-iq2 served the Flash Next IQ4 compatibility backend. Its name does not identify an IQ2 result.

The RTX 2000E Ada handles long input, but model choice changes the wait #

Gemma 3 4B is my preferred tested long-input route on System C. Two paths retained more than 84K input tokens in about 88 seconds. The larger model routes passed useful retention checks too, but their requests took several minutes.

Streaming did not remove Gemma 4’s long-prefill wait on System C. The first streamed content arrived after 277.49 seconds, later than the complete 274.24-second non-stream response. An earlier non-stream attempt timed out before the successful retry. I would reserve this route for work where several minutes of initial processing fits the task.

Gemma 3 4B also shows why I would keep qualification specific to each tested system. System D missed the opening marker in two 65K recall attempts, including a temperature-zero repeat with 67,252 input tokens. The deterministic request ended naturally in 62.07 seconds with full GPU residency.

Fast completion and a normal API response did not make the answer correct. I would use the successful System C route and leave the failing System D route out of Hermes selection.

Thinking mode decides whether Qwen3.5 0.8B is useful #

Qwen3.5 0.8B became a useful narrow-recall candidate on System C after thinking was disabled. The small Q8_0 model retained 75,649 input tokens through the gateway in 65.92 seconds. This result belongs to the tested Q8_0 artifact and nonthinking route, rather than the older 16K-context configurations elsewhere in the table.

The matched diagnostic fell from 66.56 to 7.65 seconds, about 8.7 times faster, while generated output fell from 3,391 to 21 tokens. This is why I would set the reasoning mode explicitly for simple extraction and routing tasks. Default reasoning spent most of the work on output the task did not need.

A short OpenAI-compatible probe missed its marker before the temperature-zero retry passed. The successful route checks used nonthinking mode and natural completion without an output cap. Three other gateways passed short marker requests in 4.22, 4.37, and 3.85 seconds. Those checks establish routing, while the 75,649-token request establishes long-input retention through the gateway.

I would trial this route for bounded recall and extraction. I would choose a larger model for open-ended agent reasoning. Retrieving one synthetic marker is a narrower task than planning, coding, or coordinating several tools.

The RTX 3060 Ti and RTX 3070 systems favor Qwen3.5 2B and 4B #

Qwen3.5 2B and 4B are my practical starting choices for the 8GB cards. Their text context requests on Systems E, F, and G passed with 65,579 input tokens and durations from 30.343 to 93.753 seconds. The tested 4B routes also passed tool, streaming, and image checks.

System E’s 4B vision-context rerun retained both markers and the ordered image colors at 65,637 input tokens in 44.88 seconds. Forward and reversed color-order requests passed too. The rerun used one healthy card after another card suffered an Xid 79 link fault and the system rebooted.

Each of these systems recorded 29 passing checks. The timing spread across the two RTX 3060 Ti systems also shows why I would select a tested system profile rather than rank cards from these request durations.

All seven systems returned to mining after the inference window. System B’s separate post-stop recovery check passed with fresh accepted shares. The faulting card in System E later returned to mining after a reboot and PCIe link hardening, while its AI-pool exclusion stayed in place. Restoring mining at the end of the window is a separate result from automatic hand-back after an individual inference request.

Loading a model does not qualify the model #

I would exclude DeepSeek Coder V2 16B from the tested 65K Hermes choices. Its f16-cache run retained 50,477 input tokens, but the 65K-target run reached the VRAM guard. Q4 and Q8 cache alternatives loaded the weights on the GPUs, then hit host-memory limits or poor long-prompt throughput.

The practical lesson is to benchmark the request you intend to send. GPU residency, a short answer, and a large configured context do not establish retained long input at a useful latency.

Granite 4.2 8B and GPT-OSS 20B improved after route-specific follow-up work on System A. Granite’s Q6_K route retained 66,082 input tokens in 88.636 seconds with zero cached input. GPT-OSS completed its tool round trip after the broker fix, with 9.854 seconds for the call and 9.849 seconds for the follow-up. I would keep these successful route checks distinct from earlier incomplete attempts.

Full benchmark results #

The table contains 295 checks across 59 model and system combinations. The matrix records 273 passes, one failed recall check, and seven incomplete, inconsistent, or incompatible outcomes. Fourteen checks sit outside the 65K qualification scope. Every row keeps its measured outcome visible.

My recommendations apply to the passing routes and workloads. A failed or incomplete row does not qualify the corresponding behavior.

Short-context and embedding passes apply to their own workloads. A 16K or 32K context pass does not qualify a route for the 65,536-input-token Hermes workload. Model names here are route labels. The flash-iq2 label served the IQ4 compatibility backend.

The System A Qwen3.5 35B context row is the final shared-prefix stage from the progression above. Tool durations written with a plus sign show the call and follow-up separately.

The concurrency row gives the two queued request durations. The cancellation row gives release time after the client closed the stream. Supplemental context progressions and Qwen3.5 0.8B route comparisons appear in their own tables above.

System A #

System B #

System C #

System D #

System E #

System F #

System G #

How I tested Hermes viability #

I used a synthetic marker near the beginning of a long prompt and asked for the marker after the filler. The response had to return the expected value. Server-reported input tokens determined the request size. A model answering normally after a large prompt still failed when the opening marker was wrong.

The Qwen3.5 35B progression used numbered filler records, temperature 0, and increasing lengths of 3,800, 6,800, 10,000, and 13,600 records. Those stages returned 69,429, 124,179, 182,579, and 248,279 prompt tokens. The model stayed loaded between stages, which allowed prefix reuse. The separate Qwen3.6 near-limit checks reported zero cached tokens.

The 0.8B long prompt repeated river stone lantern orchard circuit velvet compass marble 8,400 times between an opening answer key and a final recall question. The native run sent think:false, num_ctx:131072, stream:false, and keep_alive:0. The OpenAI-compatible route used reasoning_effort:"none". The broker enforced nonthinking mode for its route, so gateway callers did not need a client override.

Tool, vision, and release checks #

Tool tests checked the request and the answer after a tool result returned. One simple test used add_numbers(7, 11) and expected 18. The Qwen3.5 35B test used lookup_square and expected 49. GPT-OSS used an inventory lookup and had to incorporate the returned inventory result.

Vision checks used red and blue image questions, reversed image order, and long prompts containing image colors plus opening and ending markers. The Qwen3.5 35B test used a 256 × 256 red PNG. The broker needed an image-content conversion fix before this request passed. The System E 4B long vision test required both markers and the ordered colors to survive the input.

I checked GPU memory, model residency, temperatures, host-memory pressure, OOM events, and GPU faults around the long runs. The Qwen3.5 35B guard also checked release after a controlled interruption. Later client-cancellation and queued-request checks exercised the broker path. Two queued requests prove request handling under the tested load, rather than two independent simultaneous GPU backends.

Reading the performance results #

Request durations include long-input prefill and the returned answer. They are not generation tokens per second. First-content delay matters for streaming, and cached-prefix progressions need their own interpretation.

The successful larger System A and System C follow-up runs used dedicated inference windows with mining inactive. Models unloaded after those requests. Mining restoration across all seven systems happened at the end of the test window. These results describe model service during an inference window, rather than simultaneous mining and inference.

Backend profiles used in the tests #

Choose the profile attached to the workload you want to reproduce. Quantization, runtime context, cache type, reasoning mode, and GPU placement all affect the result.

The recorded service profiles describe their own runtime configurations. The separately tested System A f16-cache Qwen3.5 35B profile does not inherit the Q4_0 cache settings of the smaller Ollama systems.

Model fingerprints for the new retained-input runs #

These model identifiers distinguish the tested artifacts from other downloads or aliases with similar names.

Ollama manifest digests and weight-file SHA-256 values identify different objects. Match the identifier to the artifact type in the table.

What I would run with Hermes #

I would make Qwen3.5 35B the main high-context route on the two-card RTX 5060 Ti system. Its 239,153-token gateway retention result, separate tool and vision passes, and verified request release make the choice stronger than a model advertised with a large context window. I would keep Qwen3.8 27B available for the faster matched 66K response and Qwen3.6 available when using one card matters.

On the RTX 2000E Ada systems, I would choose the verified System C Gemma 3 4B route for the tested long-input response workload. I would trial the nonthinking Qwen3.5 0.8B route for narrow recall and extraction. Larger Gemma, Qwen, and GPT-OSS routes suit work where their tested four-to-five-minute initial response is acceptable. I would exclude the failed System D Gemma 3 route from the selection.

For the 8GB RTX 3060 Ti and RTX 3070 cards, I would start with Qwen3.5 2B or 4B. Their 65K text results and the 4B image checks support a useful starting configuration without dedicating the two 16GB cards to every request. Flash Next’s memory pressure and DeepSeek Coder V2’s incomplete 65K behavior make them weaker defaults on the tested profiles.

The tests support these model and system choices for specific workflows. Retained input, a correct tool round trip, image recognition, and clean release each answer a different part of Hermes viability. A dependable setup needs the behaviors used by your agent, with enough context left for the answer and tool traffic.

Sources and related reading #

• Hermes Agent provider setup and context requirements
• NVIDIA GeForce RTX 3060 Ti specifications
• NVIDIA GeForce RTX 5060 Ti specifications
• Hermes Agent overview
• Choosing a local model backend