pfSense SIEM Stack

Complete SIEM infrastructure for pfSense firewalls — From basic IDS/IPS monitoring to enterprise-grade threat detection with OpenSearch, Graylog, and Wazuh integration options.

🎯 Multi-SIEM Support: Choose your SIEM backend — OpenSearch (current ✅), Graylog (planned 📝), Wazuh (planned 📝) — with unified pfSense integration.

⚠️ Work in Progress: The logging and SIEM components are actively being developed and refined. OpenSearch implementation is production-ready. Graylog and Wazuh support planned based on community contributions. Core functionality is stable, but documentation and automation are continuously improving. Contributions and feedback welcome!

📖 Overview

What started as a simple Grafana dashboard tweak evolved into a comprehensive pfSense SIEM infrastructure project. This repository captures lessons learned from deploying and operating enterprise-grade pfSense security monitoring, with support for multiple SIEM backends to fit your needs.

⚠️ SCOPE NOTICE: This project focuses on pfSense-based security and monitoring. For UniFi equipment (switches, APs, controllers), use UniFi Poller instead - it's purpose-built for UniFi telemetry and integrates excellently with Grafana + InfluxDB.

🎯 Current Implementation (OpenSearch - Production Ready ✅)

Full SIEM Stack: OpenSearch for event storage and pfBlockerNG analytics, Logstash for parsing/enrichment, InfluxDB for time-series system metrics
Comprehensive IDS/IPS Monitoring: Real-time Suricata alerts with GeoIP mapping, signature tracking, and attack visualization
East-West Traffic Detection: Internal network monitoring to detect lateral movement and insider threats
Resilient Data Pipeline: Rotation-aware forwarders, watchdogs, restart hooks, and automated recovery
Production-Ready: Optimized for dual-WAN inline IPS deployments with per-VLAN policy customization

🔮 Future SIEM Backends (Planned)

Graylog 📝 - Easier setup, better web UI, excellent alerting (see roadmap)
Wazuh 📝 - EDR capabilities, compliance reporting, active response (see roadmap)
Compare All Options - Feature matrix and decision guide

📚 Knowledge Base Coverage

Security Hardening: IDS/IPS configuration, blocklist optimization, signature management
Network Monitoring: Multi-WAN setups, VLAN segmentation, interface tracking
Automation: Log forwarding, watchdog monitoring, automated recovery
Performance Tuning: Resource optimization, retention policies, query performance
Troubleshooting: Common issues, debugging techniques, validation procedures
Deployment Guides: Step-by-step installation, configuration templates, best practices

pfSense System and pfBlockerNG monitoring dashboard

WAN-side Suricata IDS/IPS monitoring with attack sources, signatures, and geographic visualization

Per-Interface Suricata monitoring - LAN/VLAN monitoring with dynamic interface sections

�️ Architecture

Data Flow

pfSense runs Suricata on multiple interfaces (WAN inline IPS, VLAN IDS)
Forwarder (forward-suricata-eve.py) tails eve.json logs, enriches with GeoIP, handles rotation
Logstash receives events via UDP, parses/nests under suricata.eve.*, forwards to storage
OpenSearch indexes Suricata events for search and aggregation (geomap, dashboards)
Telegraf on pfSense collects system metrics → InfluxDB AND pfBlockerNG logs → OpenSearch
InfluxDB stores time-series system metrics (CPU, RAM, interfaces, gateways)
Grafana visualizes everything with mixed datasource dashboards (InfluxDB + OpenSearch)
Watchdogs monitor the pipeline and restart components on failure

Key Components

Component	Purpose	Location
Suricata	IDS/IPS engine	pfSense (per-interface)
PfBlockerNG	Upstream blocklists	pfSense
Forwarder	Suricata log shipping + GeoIP	pfSense → SIEM
Telegraf	System metrics + pfBlockerNG logs	pfSense → InfluxDB/OpenSearch
Logstash	Suricata parsing & enrichment	SIEM server
OpenSearch	Suricata events + pfBlockerNG data	SIEM server
InfluxDB	System metrics (CPU/RAM/interfaces)	SIEM server
Prometheus	Metrics scraping (node_exporter, windows_exporter)	SIEM server
Grafana	Visualization (mixed datasources)	SIEM server
Watchdogs	Pipeline monitoring	pfSense + SIEM

📊 Project Status

✅ Production Ready

Suricata Multi-Interface Monitoring - Stable, tested on 15 instances (2 WAN + 13 VLAN)
Log Forwarder - Inode-aware rotation handling, GeoIP enrichment, watchdog monitoring
OpenSearch/Logstash Pipeline - Nested format, index templates, retention policies
WAN Security Dashboard - Attack visualization, signature tracking, geographic mapping
PfBlockerNG Integration - Blocklist optimization, DNSBL whitelisting
Automated Installation - One-command SIEM stack deployment

🚧 Active Development (Functional but Evolving)

Documentation - Continuously improving guides, adding troubleshooting scenarios
SID Management - 219 optimized rules, suppress.conf refinement (testing in production)
LAN Monitoring Dashboard - East-west traffic panels (planned)
Automation Scripts - Additional watchdogs, health checks, recovery procedures
Performance Tuning - Index optimization, query performance, resource usage

📝 Planned Features

Snort Integration - Currently Suricata-focused, Snort support coming
Multi-Firewall Support - Central monitoring for multiple pfSense instances
Advanced Analytics - Machine learning for anomaly detection
Configuration UI - Web interface for easier setup
Mobile Dashboard - Grafana mobile optimization

✨ Features

🔥 IDS/IPS Monitoring (✅ Stable)

Real-time alerts from Suricata with signature details and severity
GeoIP visualization with city-level accuracy on interactive world maps
Multi-interface support monitors all Suricata instances (WAN, VLANs, lagg)
Event analytics by type, protocol, severity, category, and application
Attack tracking: Top signatures, source countries, HTTP hosts, JA3 fingerprints
SID optimization: 219 tuned signatures for reduced false positives

🌐 Network Intelligence (✅ Stable)

Dual-WAN inline IPS with Snort + Emerging Threats on both uplinks
East-West detection for lateral movement across VLANs (🚧 dashboard pending)
Per-VLAN policies: Heavy monitoring on IoT, lighter on trusted VLANs
PfBlockerNG integration for upstream threat filtering

🛠️ Reliability & Operations (✅ Stable)

Boot persistence: FreeBSD rc.d service starts forwarder automatically on pfSense boot
Process management: Uses daemon(8) with PID tracking — no orphaned processes
Log rotation handling: Inode-aware forwarder survives Suricata rotations
Watchdogs: Cron-based health checks restart forwarder on failure via rc.d service
Restart hooks: Ensure proper startup after pfSense/Suricata upgrades
Debug logging: Comprehensive troubleshooting with /var/log/suricata_forwarder_debug.log
Automated setup: One-command deployment scripts for entire stack

� Dashboards & Alerts

WAN monitoring: Attack sources, signatures, protocols, top talkers
LAN monitoring: Per-interface dashboard with dynamic VLAN sections (✅ production ready)- Windows monitoring: Windows Exporter dashboard — CPU, memory, disk, network, services (✅ Prometheus)- Interface distribution: Traffic breakdown by interface/VLAN
Alerting: Grafana alerts + webhook integrations (🚧 refining rules)

🚀 Quick Start

Prerequisites

🚨 CRITICAL HARDWARE WARNING:

DO NOT USE RASPBERRY PI WITH SD CARDS OR SIMILAR SETUPS FOR LOGGING!

High-frequency log writes will destroy SD cards within weeks. At minimum, use a SATA SSD with USB 3.0 adapter. For production logging/SIEM, invest in proper hardware to avoid data loss and constant rebuilds.

SIEM Server (Ubuntu/Debian 22.04+ or similar):

CPU: Dual-core minimum (quad-core recommended for Logstash processing)
- AMD/Intel with SMT/Hyper-Threading strongly recommended
RAM: 16GB minimum, 32GB recommended for full logging stack
- OpenSearch: 8GB+ heap
- Logstash: 2-4GB heap
- InfluxDB: 2-4GB (recommended for time-series metrics)
- Grafana: 1-2GB
- System overhead: 2-4GB
Storage:
- 100GB+ SSD minimum (NVMe preferred for write performance)
- Consider 500GB-1TB for 30+ day retention with moderate traffic
- NO SD CARDS - Use proper SSDs or you'll regret it
Network: Gigabit ethernet minimum
OS: Ubuntu Server 24.04 LTS recommended (tested configuration)
Access: Root/sudo access, static IP

Production Reference: Purism Librem Mini (Intel Core i7-10510U, 32GB RAM, 2TB NVMe) running Ubuntu Server 24.04 LTS handles home lab with significant headroom.

pfSense Firewall:

pfSense 2.7+ (tested on 2.8.1)
For PfBlockerNG only: 4-8GB RAM sufficient (even with many blocklists)
For Suricata IDS/IPS:
- CPU: Quad-core minimum (more cores = more throughput)
  - Intel Atom C3758 (8-core) handles 15 Suricata instances at 25-35% average load
  - Intel Xeon D-1518 (8-core, AVX2) provides significant headroom with hardware AES-NI for VPN offload
  - Expect CPU spikes to 100% for 3-5 minutes during rule reloads
- RAM: 8-16GB for multi-interface deployments
- Stream Memory: Increase to 1073741824 bytes (1GB) per interface (default 256MB causes crashes on multicore systems)
  - Configure in: Services → Suricata → Interface → Stream tab
Suricata package installed and configured
SSH enabled with key-based auth
Python 3.11+ available

Recommended pfSense Packages

These packages enhance functionality and are used/referenced throughout this project:

Package	Purpose	Why Recommended	Installation
pfBlockerNG-devel	IP/DNS blocklists, GeoIP blocking	Essential for upstream threat filtering. Blocks known bad actors before they reach Suricata. The devel version has latest features and GeoIP updates. Dashboard includes pfBlockerNG statistics panels.	System → Package Manager → Available Packages → Install `pfBlockerNG-devel`
Telegraf	Metrics collection agent	Collects system metrics (CPU, memory, interfaces) and sends to InfluxDB. Also collects pfBlockerNG logs and sends directly to OpenSearch via `[[outputs.opensearch]]` plugin. Required for both system monitoring and pfBlockerNG dashboard panels.	System → Package Manager → Available Packages → Install `telegraf`
ntopng	Network traffic analysis	Provides GeoIP database updates automatically (GeoLite2-City.mmdb). The forwarder uses this database for IP geolocation enrichment. Also useful for deep packet inspection and flow analysis.	System → Package Manager → Available Packages → Install `ntopng`
Service_Watchdog	Service monitoring	Monitors critical services (Suricata, Unbound, etc.) and auto-restarts on failure. Complements our forwarder watchdog for complete pipeline reliability.	System → Package Manager → Available Packages → Install `Service_Watchdog`
nut	UPS monitoring	Network UPS Tools for power monitoring. Telegraf can collect UPS metrics for dashboard display. Critical for graceful shutdowns during power events.	System → Package Manager → Available Packages → Install `nut` (if using UPS)

Installation Priority:

pfBlockerNG-devel - Install first, reduces attack surface before IDS/IPS
Suricata - Core IDS/IPS functionality (required for this project)
Telegraf - Required for system metrics dashboard
ntopng - Recommended for auto-updating GeoIP database
Service_Watchdog - Recommended for service reliability
nut - Optional, only if using UPS hardware

Notes:

pfBlockerNG-devel vs pfBlockerNG: Use -devel for latest GeoIP updates and features
ntopng alternative: If not using ntopng, manually update GeoIP database (see GeoIP Setup)
Telegraf configuration: After installation, use included plugins from plugins/ directory
Service_Watchdog: Configure to monitor at minimum: Suricata, Unbound, dpinger

Installation (2 Options)

Option 1: Interactive Management Console (Recommended)

# 1. Clone repository
git clone https://github.com/ChiefGyk3D/pfsense_siem_stack.git
cd pfsense_siem_stack

# 2. Run management console
./pfsense-siem

# Follow the interactive menu:
#   - Option 1: Install SIEM Stack
#   - Option 2: Deploy to pfSense
#   - Option 3: Configure OpenSearch
#   - Option 4: Import Dashboards

Features:

✅ Complete interactive menu system
✅ System health checks
✅ Service management (start/stop/restart)
✅ Log viewing (all components)
✅ Configuration backup/restore
✅ Built-in troubleshooting tools

Full Management Console Documentation →

Option 2: Manual Commands

# 1. Clone repository
git clone https://github.com/ChiefGyk3D/pfsense_siem_stack.git
cd pfsense_siem_stack

# 2. Install SIEM stack (OpenSearch, Logstash, Grafana)
sudo ./install.sh

# 3. Configure environment and deploy to pfSense
cp config.env.example config.env
nano config.env  # Set SIEM_HOST and PFSENSE_HOST
./setup.sh       # Automated deployment

That's it! The setup script:

✅ Configures OpenSearch index templates (Suricata + pfBlockerNG)
✅ Deploys forwarder to pfSense with GeoIP enrichment
✅ Installs rc.d service for boot persistence (auto-start on reboot)
✅ Installs watchdogs for automatic crash recovery
✅ Creates Grafana OpenSearch-pfBlockerNG datasource
✅ Verifies data flow from pfSense → Logstash → OpenSearch

Import Dashboards

Open Grafana: http://<siem-server>:3000 (admin/admin)
Go to Dashboards → Import
Upload dashboards:
- dashboards/pfsense_pfblockerng_system.json - pfSense system metrics (InfluxDB) and pfBlockerNG stats (OpenSearch)
- dashboards/Suricata_IDS_IPS.json - WAN-side security monitoring (OpenSearch)
- dashboards/Suricata_Per_Interface.json - Per-interface LAN/VLAN monitoring (OpenSearch)
- dashboards/windows_exporter.json - Windows host monitoring via Prometheus (CPU, memory, disk, network, services)
Select your datasources:
- InfluxDB (pfsense database) for system metrics panels
- OpenSearch-pfBlockerNG (pfblockerng-* index) for pfBlockerNG panels
- OpenSearch (suricata-* index) for Suricata panels
- Prometheus for Windows Exporter and node_exporter panels
Click Import

Verify Installation

Check everything is working:

./scripts/status.sh

This will verify:

✅ OpenSearch is running and configured correctly
✅ Logstash is listening for events
✅ Forwarder is running on pfSense
✅ Data is flowing and recent
✅ Watchdog is installed
✅ Suricata is generating events

Green checkmarks = everything is working!
Red X's = see the error messages and suggested fixes

📚 Documentation

Getting Started

Quick Start Guide - 15-minute deployment walkthrough
New User Checklist - Step-by-step validation
Documentation Index - Complete guide to all docs

Installation & Configuration

SIEM Stack Installation - OpenSearch, Logstash, Grafana
pfSense Forwarder Setup - Python forwarder deployment
GeoIP Configuration - MaxMind database setup
Configuration Guide - All config.env options

Optimization & Tuning

Suricata Optimization ⭐ ESSENTIAL
- Rule selection strategies
- Performance tuning (inline IPS vs IDS)
- Multi-interface configuration
- Testing and validation
PfBlockerNG Setup - Blocklist strategies
Retention Policies - Index lifecycle management

Troubleshooting

Troubleshooting Guide - Common issues and fixes
Dashboard "No Data" Fix - Datasource and field issues
Log Rotation Fix - Forwarder stuck on old files
Forwarder Monitoring - Health checks

🔧 Advanced Topics

Custom Deployment Scenarios

Inline IPS on WAN + IDS on VLANs:

# WAN interfaces: ix0, ix1 (inline IPS with Snort ET)
# VLANs: lagg1.10, lagg1.14, lagg1.22, etc. (IDS with ET)
# IoT VLANs: Heavy monitoring
# NAS/trusted VLANs: Light monitoring

See Suricata Optimization Guide for per-VLAN policy configuration.

East-West Detection:

Monitor RFC1918 → RFC1918 flows for lateral movement
Separate dashboard for internal traffic analysis
Detect anomalies like SMB brute force, RDP scanning, etc.

See LAN Monitoring Setup for configuration.

Watchdog & Automation

The forwarder uses a multi-layer reliability architecture to survive reboots, crashes, and upgrades:

rc.d Service (/usr/local/etc/rc.d/suricata_forwarder):

FreeBSD native service for boot persistence — forwarder starts automatically on pfSense boot
Uses daemon(8) for proper process management with PID tracking (/var/run/suricata_forwarder.pid)
Enabled via sysrc suricata_forwarder_enable=YES
Standard service commands: service suricata_forwarder start|stop|restart|status
Deployed automatically by setup.sh

Forwarder Watchdog (suricata-forwarder-watchdog.sh):

Monitors forwarder process health via rc.d service status
Restarts on failure or stuck state using service suricata_forwarder restart
Runs via cron every 5 minutes as a safety net

Restart Hooks (suricata-restart-hook.sh):

Ensures forwarder starts after Suricata upgrades
Handles log rotation gracefully
Integrated with pfSense Suricata package

Recovery Chain: rc.d (boot) → watchdog cron (crash recovery) → restart hook (upgrade recovery)

See scripts/README.md for all helper scripts.

🤝 Contributing

Contributions welcome! Areas of interest:

Additional dashboards (firewall logs, Telegraf metrics, pfBlockerNG stats)
Performance optimizations
Docker/container deployment
Ansible playbooks
Additional forwarder integrations (Zeek, Snort3, etc.)

See CONTRIBUTING.md for guidelines.

📝 License

Mozilla Public License 2.0 - see LICENSE for details.

🔗 Related Projects

Repository	Description
siem-docker-stack	Server-side Dockerized SIEM/SOC stack with hot/warm tiering (OpenSearch, Wazuh, Grafana, Logstash, Prometheus)
PiNodeXMR_Grafana_Dashboard	Monero node monitoring dashboard for Grafana

UniFi Network Monitoring

This project does NOT support UniFi equipment (switches, access points, controllers).

For UniFi monitoring with Grafana, use UniFi Poller — I have contributed some dashboard fixes to the project:

Purpose-built for UniFi telemetry collection
Excellent Grafana dashboard integration
Supports InfluxDB and Prometheus
Active community and development
Personal recommendation: Use with InfluxDB for networking metrics

Why InfluxDB for UniFi Poller?

Better compression for time-series data (interface metrics, client counts)
Faster queries for rate calculations
Lower RAM usage than OpenSearch for metrics
Proven combination for network monitoring

See: UniFi Poller Installation

�🙏 Acknowledgments

pfSense - Rock-solid firewall platform
Suricata - High-performance IDS/IPS engine with excellent multithreading
OpenSearch - Powerful search and analytics
Grafana - Beautiful visualization and alerting
MaxMind - GeoLite2 GeoIP database (free tier)
Emerging Threats - Open IDS ruleset
Snort/Cisco Talos - Registered and subscriber rules
Abuse.ch - Feodo Tracker and SSL Blacklist
UniFi Poller - Inspiration for telemetry collection patterns
Community contributors and testers

📞 Support

Issues: GitHub Issues
Discussions: GitHub Discussions
Documentation: docs/

Verify Installation

pfsense_siem_stack/
├── 📄 Quick Start
│   ├── README.md                    ★ START HERE - Project overview & getting started
│   ├── QUICK_START.md               Fast 15-minute deployment walkthrough
│   ├── ORGANIZATION.md              File organization and directory structure
│   ├── CHANGELOG.md                 Version history and recent changes
│   └── CONTRIBUTING.md              Guidelines for contributions
│
├── 🚀 Installation & Setup
│   ├── install.sh                   ★ SIEM stack installer (OpenSearch, Logstash, Grafana)
│   ├── setup.sh                     ★ Automated configuration (one-command deployment)
│   ├── install_plugins.sh           Telegraf plugin installer
│   └── config.env.example           Configuration template
│
├── 📊 Dashboards & Visualization
│   └── dashboards/
│       ├── pfsense_pfblockerng_system.json        ★ pfSense system & pfBlockerNG dashboard
│       ├── Suricata_IDS_IPS.json        ★ WAN-side Suricata security dashboard
│       ├── Suricata_Per_Interface.json            ★ Per-interface LAN/VLAN monitoring
│       └── archive/                               Historical dashboard versions
│
├── 🔧 Scripts & Automation
│   └── scripts/
│       ├── forward-suricata-eve.py            ★ Multi-interface log forwarder (Python)
│       ├── status.sh                          ★ Comprehensive health check
│       ├── check_custom_sids.sh               Suricata SID verification tool
│       ├── restart-services.sh                Service management & recovery
│       ├── configure-retention-policy.sh      Data lifecycle management
│       ├── suricata-forwarder-watchdog.sh     Monitoring & auto-restart
│       ├── suricata-restart-hook.sh           Post-upgrade restart hook
│       └── README.md                          Script documentation
│
│   # Deployed to pfSense by setup.sh:
│   # /usr/local/etc/rc.d/suricata_forwarder  FreeBSD rc.d service (boot persistence)
│   # /usr/local/bin/forward-suricata-eve.py   Forwarder script
│   # /usr/local/bin/suricata-forwarder-watchdog.sh  Cron watchdog
│
├── ⚙️ Configuration Files
│   └── config/
│       ├── logstash-suricata.conf             ★ Logstash pipeline (parsing & enrichment)
│       ├── opensearch-index-template.json     Suricata index template with geo_point mapping
│       ├── opensearch-pfblockerng-template.json  pfBlockerNG index template with keyword mapping
│       ├── dnsbl_whitelist.txt                PfBlockerNG whitelist
│       ├── pfblockerng_optimization.md        Blocklist configuration guide
│       └── sid/                               Suricata signature management
│           ├── disable/disablesid.conf        219 disabled SIDs (performance optimized)
│           ├── suppress/suppress.conf         2 conditional suppressions (IP-specific)
│           ├── README.md                      SID management documentation
│           └── APPLYING_CHANGES.md            Deployment guide
│
├── 📚 Documentation (🚧 Active Development)
│   └── docs/
│       ├── DOCUMENTATION_INDEX.md             ★ Master documentation index
│       ├── architecture.png                   Visual architecture diagram
│       │
│       ├── 🏗️ Installation Guides
│       │   ├── INSTALL_SIEM_STACK.md          OpenSearch/Logstash/Grafana setup
│       │   ├── INSTALL_PFSENSE_FORWARDER.md   Forwarder deployment to pfSense
│       │   ├── INSTALL_DASHBOARD.md           Grafana dashboard import
│       │   └── NEW_USER_CHECKLIST.md          Step-by-step validation
│       │
│       ├── 🔒 Security & IDS/IPS
│       │   ├── SURICATA_OPTIMIZATION_GUIDE.md ★ Rule selection & tuning
│       │   ├── PFBLOCKERNG_OPTIMIZATION.md    Blocklist configuration
│       │   ├── LAN_MONITORING.md              Internal threat detection
│       │   └── SURICATA_FORWARDER_MONITORING.md  Watchdog strategies
│       │
│       ├── ⚙️ Configuration & Tuning
│       │   ├── CONFIGURATION.md               All config.env settings
│       │   ├── GEOIP_SETUP.md                 MaxMind GeoIP database
│       │   ├── MULTI_INTERFACE_RETENTION.md   Index lifecycle policies
│       │   └── OPENSEARCH_AUTO_CREATE.md      Midnight UTC fix
│       │
│       ├── 🐛 Troubleshooting & Fixes
│       │   ├── TROUBLESHOOTING.md             ★ Common issues & solutions
│       │   ├── LOG_ROTATION_FIX.md            Inode-aware rotation handling
│       │   ├── DASHBOARD_NO_DATA_FIX.md       Data flow validation
│       │   └── TELEGRAF_INTERFACE_FIXES.md    Interface monitoring fixes
│       │
│       └── archive/                           Historical documentation
│
├── 🔌 Plugins (Optional Telegraf Metrics)
│   └── plugins/
│       ├── telegraf_pfifgw.php                Gateway status monitoring
│       ├── telegraf_temperature.sh            Temperature sensors
│       ├── telegraf_unbound.sh                DNS resolver stats
│       └── README.md                          Plugin documentation
│
├── 🖼️ Media & Assets
│   └── media/
│       ├── Grafana-pfSense.png                    pfSense system dashboard screenshot
│       ├── Suricata IDS_IPS WAN Dashboard.png     WAN dashboard screenshot
│       ├── Suricata Per-Interface Dashboard.png   Per-interface dashboard screenshot
│       └── streamelements.png                     StreamElements donation icon
│
└── 🧪 Testing & Validation
    └── tests/
        ├── test-multi-interface.sh            Multi-WAN testing
        └── test-panel-compatibility.sh        Dashboard validation

🌟 Essential Files to Get Started:

README.md - Project overview, architecture, features
QUICK_START.md - 15-minute deployment guide
docs/DOCUMENTATION_INDEX.md - Find any documentation
install.sh - One command to install SIEM stack
setup.sh - One command to configure everything
scripts/status.sh - Verify installation and data flow

🚧 Work in Progress:

SIEM logging infrastructure (functional, documentation evolving)
SID management optimization (219 rules tuned, testing in production)
Dashboard improvements (LAN monitoring panels planned)
Automation enhancements (watchdogs, recovery scripts)

📊 Dashboard Panels

Dashboard 1: pfSense System & pfBlockerNG

Purpose: System monitoring, network performance, and pfBlockerNG statistics

Hardware & System

CPU usage, memory, disk I/O
Temperature monitoring
System uptime and load

Network Statistics

Gateway RTT and packet loss
Interface status and throughput
Per-WAN and per-LAN traffic graphs

PfBlockerNG (via OpenSearch)

IP blocks (inbound/outbound)
DNSBL blocked queries
Top blocked sources and destinations
Blocked traffic by country (GeoIP)
Feed performance statistics
Protocol and port distribution

Note: pfBlockerNG panels use the OpenSearch-pfBlockerNG datasource (pfblockerng-* index), while system metrics panels use InfluxDB. This mixed-datasource approach avoids InfluxDB cardinality issues with high-cardinality fields like IP addresses.

Dashboard 2: Suricata WAN Monitoring

Purpose: External threat detection and WAN-side attack analysis

Statistics

Events & Alerts: Combined counter with sparklines and color-coded thresholds
Event Type Distribution: Pie chart showing alert, http, dns, tls, etc.
Protocol Distribution: TCP, UDP, ICMP breakdown

Alerts

Top Alert Signatures: Most triggered IDS rules
IDS Alert Logs: Detailed table with time, signature, IPs, ports, countries
Alert Severity: Critical, high, medium, low classification

Geographic Visualization

Attack Sources Map: Interactive world map with geohash clusters
Top Source Countries: Donut chart of attack origins
Country Statistics: Detailed breakdown with event counts

HTTP Traffic Analysis

Top HTTP Hosts: Most accessed domains
HTTP Methods: GET, POST, etc. distribution

Dashboard 3: Suricata Per-Interface (LAN Monitoring)

Purpose: Per-VLAN/LAN monitoring with dynamically repeating sections

Features

Dynamic Interface Sections: Automatically creates monitoring section for each selected interface
Multi-select Variable: Monitor one, some, or all interfaces simultaneously
Per-Interface Panels (repeated for each interface):
- Events & Alerts counter with thresholds
- Top Alert Signatures
- Alerts timeline graph
- Top Source IPs
- Top Destination IPs
- Complete alert logs table

Use Cases

East-West Detection: Monitor lateral movement between VLANs
IoT Monitoring: Heavy monitoring on untrusted IoT VLANs
Internal Threats: Detect compromised hosts and insider threats
RFC1918 Traffic: Focus on internal network communication patterns

🔧 Configuration

Customize Logstash Input

Edit /etc/logstash/conf.d/suricata.conf:

input {
  udp {
    port => 5140
    codec => json
  }
}

Adjust Data Retention

# Configure index lifecycle policy
scripts/configure-retention-policy.sh

Default retention: 30 days

GeoIP Updates

The forwarder uses MaxMind GeoLite2-City database. To update:

# On pfSense, if using ntopng, it auto-updates
# Manual update:
ssh root@pfsense
fetch -o /usr/local/share/ntopng/GeoLite2-City.mmdb \
  https://github.com/PrxyHunter/GeoLite2/raw/master/GeoLite2-City.mmdb

Logstash Maintenance

⚠️ After Logstash upgrades, the logstash-output-opensearch plugin may be removed. If Logstash enters a crash loop after an upgrade, reinstall the plugin:
sudo /usr/share/logstash/bin/logstash-plugin install logstash-output-opensearch
sudo systemctl restart logstash
Also check /usr/share/logstash/data ownership — it must be owned by logstash:logstash, not root. Fix with:
sudo chown -R logstash:logstash /usr/share/logstash/data
If Logstash reports Gemfile.lock errors, sync it:
cd /usr/share/logstash
sudo -u logstash bin/logstash-plugin install --no-verify

UDP Receive Buffer

For high-volume deployments, increase the kernel UDP receive buffer on the SIEM server to prevent dropped events:

# Check current max
sysctl net.core.rmem_max

# Set to 32MB (recommended)
echo 'net.core.rmem_max=33554432' | sudo tee -a /etc/sysctl.d/99-logstash.conf
sudo sysctl -p /etc/sysctl.d/99-logstash.conf

🔍 Troubleshooting

Quick Diagnosis

Run the status check first:

./scripts/status.sh

This will identify most common problems automatically.

Common Issues

1. Dashboard shows "No Data" ⚠️ MOST COMMON

Symptom: Grafana panels show "No Data" even though Suricata is running
Causes:
- Datasource variable not resolved (${DS_OPENSEARCH} instead of actual datasource UID)
- Nested field structure vs flat field queries (dashboard expecting event_type but data has suricata.eve.event_type)
- Logstash config changed from flat to nested structure
- Dashboard looking at wrong time range (new flat data only exists in recent timeframe)
Fixes:
1. Check datasource configuration: Dashboard must use actual UID, not ${DS_OPENSEARCH} variable
2. Verify field structure: curl -s "http://localhost:9200/suricata-*/_search?size=1" | jq '.hits.hits[0]._source | keys'
3. If fields are nested under suricata.eve.*, update Logstash config to flatten OR update dashboard queries
4. Run ./scripts/status.sh to diagnose data flow issues
5. Adjust time range to match when data started flowing (check index creation timestamps)

2. Alerts not showing but other events are

Symptom: DNS, TLS, HTTP events work, but alert panels empty
Cause: Forwarder only tails from EOF (end of file), missing historical alerts
Fix: Wait for NEW alerts to be generated after forwarder starts
Why: The forwarder uses f.seek(0, 2) to start at end of file, so pre-existing alerts aren't forwarded
Solution: Generate test alerts or wait for real attacks to trigger rules

3. Data structure mismatch

Symptom: Old data works but new data doesn't (or vice versa)

Diagnosis:

# Check if you have both nested and flat structures
curl -s "http://localhost:9200/_search?size=0" -H 'Content-Type: application/json' \
  -d '{"aggs":{"nested":{"filter":{"exists":{"field":"suricata.eve.event_type"}}},"flat":{"filter":{"exists":{"field":"event_type"}}}}}'

Fix:
1. Choose flat or nested structure (flat is simpler)
2. Update Logstash config to match
3. Update dashboard field references to match
4. Optional: Reindex old data to match new structure

4. Forwarder not running

Check service status: ssh admin@<pfsense-ip> 'service suricata_forwarder status'
Restart: ssh admin@<pfsense-ip> 'service suricata_forwarder restart'
Verify enabled at boot: ssh admin@<pfsense-ip> 'sysrc suricata_forwarder_enable'
Run ./scripts/status.sh to check full pipeline status
Check forwarder logs: ssh admin@<pfsense-ip> 'tail -f /var/log/system.log | grep suricata'

5. Data stops at midnight UTC

Cause: OpenSearch auto-create disabled
Fix: Run ./setup.sh (it configures this automatically)
Details: See docs/OPENSEARCH_AUTO_CREATE.md

6. Multiple forwarders running

Stop all and restart cleanly: ssh admin@<pfsense-ip> 'pkill -f forward-suricata; service suricata_forwarder start'
The rc.d service uses a PID file to prevent duplicate instances
Run ./setup.sh to redeploy with clean state

7. Wrong SIEM IP configured

Edit config.env with correct SIEM IP
Run ./setup.sh to redeploy with new config

Detailed Troubleshooting

Dashboard "No Data" Fix: 🔥 Complete guide for the most common issue - panels showing "No Data"
Troubleshooting Guide: Comprehensive troubleshooting for all components

Geomap Not Displaying

The geomap requires proper geo_point mapping. Check:

# Verify mapping
curl http://localhost:9200/suricata-*/_mapping | jq '.[] | .mappings.properties.suricata.properties.eve.properties.geoip_src.properties.location'

# Should show: {"type": "geo_point"}

If not, re-create the index template:

curl -X PUT "http://localhost:9200/_index_template/suricata-template" \
  -H 'Content-Type: application/json' \
  -d @config/opensearch-index-template.json

Forwarder Not Monitoring All Interfaces

The forwarder should automatically detect all Suricata instances. If missing interfaces:

# Check available eve.json files
ssh admin@pfsense 'ls -la /var/log/suricata/*/eve.json'

# Restart forwarder (picks up new interfaces automatically)
ssh admin@pfsense 'service suricata_forwarder restart'

See TROUBLESHOOTING.md for more solutions.

pfBlocker Panels Show No Data

pfBlockerNG data now flows through OpenSearch (not InfluxDB). Common causes:

Telegraf opensearch output not configured: The pfSense Telegraf package needs [[outputs.opensearch]] configured to send pfBlockerNG data to OpenSearch. See Telegraf pfBlocker Setup.

OpenSearch index template not applied: Fields may be mapped as text instead of keyword, preventing aggregation. Re-apply the template:

./scripts/install-opensearch-config.sh

Important: Applying the template only affects new indices. If existing indices were created without the template, their tag.* fields are already mapped as text and cannot be changed in-place. You must reindex the affected indices:

# For each affected index, reindex through a temp index:
curl -XPUT "http://localhost:9200/pfblockerng-YYYY.MM.DD-temp"
curl -XPOST "http://localhost:9200/_reindex" -H 'Content-Type: application/json' \
  -d '{"source":{"index":"pfblockerng-YYYY.MM.DD"},"dest":{"index":"pfblockerng-YYYY.MM.DD-temp"}}'
curl -XDELETE "http://localhost:9200/pfblockerng-YYYY.MM.DD"
curl -XPOST "http://localhost:9200/_reindex" -H 'Content-Type: application/json' \
  -d '{"source":{"index":"pfblockerng-YYYY.MM.DD-temp"},"dest":{"index":"pfblockerng-YYYY.MM.DD"}}'
curl -XDELETE "http://localhost:9200/pfblockerng-YYYY.MM.DD-temp"

auto_create_index missing pfblockerng-*: Check cluster settings:

curl -s "http://localhost:9200/_cluster/settings?filter_path=persistent.action.auto_create_index"

Grafana datasource not created: The dashboard expects an OpenSearch-pfBlockerNG datasource pointing to pfblockerng-*. Run ./setup.sh to create it automatically.
pfBlockerNG filterlog rotation issue:

# Via SSH to pfSense
ssh root@pfsense
php -r 'require_once("/etc/inc/filter.inc"); filter_configure(); system_syslogd_start();'
php -r 'require_once("/usr/local/pkg/pfblockerng/pfblockerng.inc"); pfblockerng_sync_on_changes();'

Prevention & Monitoring: See pfSense Filterlog Rotation Fix for:

GUI configuration options
Automated monitoring setup
Preventive measures
Integration with status.sh

📖 Documentation

Setup Guides

New User Checklist: 🎯 Complete step-by-step checklist for first-time setup
Quick Start Guide: Fast setup for experienced users
SIEM Stack Installation: Detailed OpenSearch/Logstash/Grafana setup
Forwarder Installation: pfSense forwarder deployment
Dashboard Import: Dashboard configuration and customization

Configuration

Suricata Optimization Guide: 🌟 Complete guide for new users - rule selection, performance tuning, IDS vs IPS
GeoIP Setup: MaxMind database installation
Configuration Guide: Advanced settings and tuning
Telegraf Interface Fixes: Universal interface detection
Forwarder Monitoring: Automatic restart and monitoring strategies
Telegraf pfBlocker Setup: pfBlocker panel configuration

Troubleshooting

Dashboard "No Data" Fix: 🔥 START HERE - Fix the most common issue (panels showing "No Data")
Troubleshooting Guide: Common issues and fixes for all components
pfSense Filterlog Fix: Fix for pfBlocker data loss
OpenSearch Auto-Create: Midnight UTC data stoppage fix

🔐 Security Considerations

Firewall Rules: Restrict Logstash UDP 5140 to pfSense IP only
OpenSearch: Bind to localhost or use authentication
Grafana: Change default admin password immediately
GeoIP Data: Contains location information - secure appropriately

pfSense Syslog Format for Wazuh Integration

If you are forwarding pfSense syslog to a siem-docker-stack deployment that includes Wazuh, you must configure pfSense to use RFC 5424 syslog format with RFC 3339 timestamps:

In pfSense, go to Status → System Logs → Settings
Under Remote Logging Options, enable remote logging to your SIEM server IP on port 514/UDP
Set Log Message Format to RFC 5424 (syslog-protocol)
Set Timestamp Format to RFC 3339 with microsecond precision

Why this matters: pfSense's default BSD syslog (RFC 3164) omits the hostname field. Without a hostname, Wazuh's pre-decoder misidentifies the program name as the hostname, and the built-in pfSense pf decoder never matches — resulting in zero firewall alerts despite data flowing. RFC 5424 always includes the hostname (e.g., chiefgyk3d.firewall), and syslog-ng on the SIEM server handles the RFC 5424 → BSD format conversion for Wazuh automatically.

🤝 Contributing

Contributions welcome! Please:

Fork the repository
Create a feature branch
Test changes thoroughly
Submit pull request with clear description

🙏 Credits

Original pfSense Telegraf dashboards by various contributors
Suricata IDS/IPS by OISF
OpenSearch by Amazon
Grafana by Grafana Labs

💬 Support

Issues: GitHub Issues
Discussions: GitHub Discussions
Documentation: Documentation Index

💝 Donations and Tips

If you find this pfSense SIEM Stack useful, consider supporting development:

Donate:


Patreon	StreamElements

Cryptocurrency Tips

	Bitcoin `bc1qztdzcy2wyavj2tsuandu4p0tcklzttvdnzalla`
	Monero `84Y34QubRwQYK2HNviezeH9r6aRcPvgWmKtDkN3EwiuVbp6sNLhm9ffRgs6BA9X1n9jY7wEN16ZEpiEngZbecXseUrW8SeQ`
	Ethereum `0x554f18cfB684889c3A60219BDBE7b050C39335ED`

Made with ❤️ by ChiefGyk3D

Author & Socials


Mastodon	Bluesky	Twitch	YouTube	Kick	TikTok	Discord	Matrix

Name		Name	Last commit message	Last commit date
Latest commit History 53 Commits
config		config
dashboards		dashboards
docs		docs
media		media
plugins		plugins
scripts		scripts
tests		tests
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
MIGRATION_CHECKLIST.md		MIGRATION_CHECKLIST.md
ORGANIZATION.md		ORGANIZATION.md
QUICK_START.md		QUICK_START.md
README.md		README.md
RENAME_CLEANUP_PLAN.md		RENAME_CLEANUP_PLAN.md
REORGANIZATION_SUMMARY.md		REORGANIZATION_SUMMARY.md
ROADMAP.md		ROADMAP.md
config.env.example		config.env.example
install.sh		install.sh
install_plugins.sh		install_plugins.sh
pfsense-siem		pfsense-siem
setup.sh		setup.sh

Folders and files

Latest commit

History

Repository files navigation

pfSense SIEM Stack

📖 Overview

🎯 Current Implementation (OpenSearch - Production Ready ✅)

🔮 Future SIEM Backends (Planned)

📚 Knowledge Base Coverage

�️ Architecture

Data Flow

Key Components

📊 Project Status

✅ Production Ready

🚧 Active Development (Functional but Evolving)

📝 Planned Features

✨ Features

🔥 IDS/IPS Monitoring (✅ Stable)

🌐 Network Intelligence (✅ Stable)

🛠️ Reliability & Operations (✅ Stable)

� Dashboards & Alerts

🚀 Quick Start

Prerequisites

Recommended pfSense Packages

Installation (2 Options)

Option 1: Interactive Management Console (Recommended)

Option 2: Manual Commands

Import Dashboards

Verify Installation

📚 Documentation

Getting Started

Installation & Configuration

Optimization & Tuning

Troubleshooting

🔧 Advanced Topics

Custom Deployment Scenarios

Watchdog & Automation

🤝 Contributing

📝 License

🔗 Related Projects

UniFi Network Monitoring

�🙏 Acknowledgments

📞 Support

Verify Installation

📊 Dashboard Panels

Dashboard 1: pfSense System & pfBlockerNG

Hardware & System

Network Statistics

PfBlockerNG (via OpenSearch)

Dashboard 2: Suricata WAN Monitoring

Statistics

Alerts

Geographic Visualization

HTTP Traffic Analysis

Dashboard 3: Suricata Per-Interface (LAN Monitoring)

Features

Use Cases

🔧 Configuration

Customize Logstash Input

Adjust Data Retention

GeoIP Updates

Logstash Maintenance

UDP Receive Buffer

🔍 Troubleshooting

Quick Diagnosis

Common Issues

Detailed Troubleshooting

Geomap Not Displaying

Forwarder Not Monitoring All Interfaces

pfBlocker Panels Show No Data

📖 Documentation

Setup Guides

Configuration

Troubleshooting

🔐 Security Considerations

pfSense Syslog Format for Wazuh Integration

🤝 Contributing

🙏 Credits

💬 Support

💝 Donations and Tips

Packages