Three-layer observability stack

Designed and operated a multi-tool observability stack covering network, server, virtualization, storage, and line-of-business systems at a regional HVAC business. It went well past standing up the vendor dashboard and walking away: collectors feeding a time-series database feeding a visualization layer, which is the pattern an enterprise SRE team would build, run here by a single IT lead for a 125-user multi-site company.

The three layers

1. Collection: PRTG sensors, Telegraf-style Windows agents with per-role profiles, and custom bash scripts for anything the off-the-shelf agents didn't cover.
2. Storage: InfluxDB as a unified time-series backend for everything that didn't naturally live in PRTG.
3. Visualization: Grafana dashboards on top of InfluxDB.

Keeping those layers separate is what made the stack extensible. Adding a new metric source meant writing a collector, not adopting another monitoring product with its own dashboard and its own alerting model to reconcile.

PRTG as the primary monitor

Sensor-based monitoring across the network and server estate: SNMP, WMI, ping, port, service, and bandwidth sensors against the switching and routing gear, the perimeter firewall, the Hyper-V host, redundant domain controllers, SQL Server, and the rest of the production endpoints. Alerting on threshold breaches and outages, with historical trending retained for capacity planning.

The value was one pane of glass across a deliberately multi-vendor environment, network gear from three vendors plus NAS, Windows, and VoIP handsets, none of which agree on how they should be watched.

Per-role telemetry, not one template

The Windows agent ran a distinct configuration profile per role rather than a single blanket template: hypervisor health, network telemetry, SQL Server instance metrics, and a domain-controller profile, plus a host-specific profile for the virtualization host and general profiles for ordinary endpoints.

Blanket templates produce two failure modes at once: noise from metrics that don't matter for a given role, and silence on the ones that do. Profiling by role costs more up front and is the reason the alerting stayed trustworthy.

Writing the collector when none exists

The NAS had no turnkey agent, so I wrote one: a bash script polling it over SNMP on a 600-second loop and writing to InfluxDB through the HTTP API with host, disk, and sensor tags. It pulled from the standard HOST-RESOURCES-MIB for drive count, allocation unit size, and total and used capacity, then from the vendor's private enterprise MIB (1.3.6.1.4.1.6574) for per-drive temperature, names, and health status. That fed a storage dashboard directly.

Reaching into a vendor's private MIB is the part worth noting. Standard MIBs tell you a disk is full; the enterprise MIB tells you a specific drive is running hot before it fails, which is the difference between capacity reporting and actual early warning.

What it demonstrated

• Metrics-driven practice at SMB scale, where most operators settle for whatever the network vendor ships.
• Collector authoring in bash against SNMP and an HTTP write API, rather than clicking through a wizard.
• Cross-platform coverage spanning Windows, Linux, and network hardware through one surface.
• Alerting, trending, capacity planning, and proactive failure detection, instead of waiting for a user to report that something felt slow.