Custom Regex queries for Log Insight

The missing query

Log Insight provides content packs that come chocked full of queries, alarms, and dashboards for users of specific products. They cover networking, security, storage, hardware, servers and more. A recent update to the NSX for vSphere content back saw TCP Protocol removed. I use TCP protocol heavily in my “segmentation approach” when learning applications. As a result I needed it back. This is where custom queries are useful.

Custom queries

The query missing was searching the dfwpkt log file for the INET protocol (L3 DFW) and then what protocol is used. This is handy in determining what type of rule to build such as UDP or TCP services.

  • Name: vmw_nsx_firewall_protocol
  • pre-context: (IN|OUT) (\d+ )?
  • post-context: \s
  • custom-regex: (TCP6?|UDP6?|PROTO6?\d+)
  • additional-context dfwpktlogs INET

These fields are create in a custom field. This is done by highlighting an the desired field on a given log (TCP in my case). Right click and select Extract Field.

screenshot-2016-09-16-17-14-06

This results in my queries and dashboards working as desired again.

screenshot-2016-09-16-17-24-52

Now I can easily see what is talking to and from my apps when segmenting them. Happy days.

NOTE: This was removed in the NSX Content Pack 3.4 due to it being a resource expensive query. This expensive regex slowed down a query and a any dashboard it referenced and was removed.

My VMworld 2016 submissions

VMworld Public voting is on now. There is information about each session and public members are encouraged to vote for interesting sessions. Alas there is not author detail about who is presenting or additional information that was asked for when submitting a session.

My colleagues and I have been working on some cool things that we hope to share with the wider world and VMworld is a great platform for this.

The first session proposal is with Grant Orchard. He and I have built a process to on approaching micro-segmentation of any application that has a virtual endpoint. After buying into the marketing fluff and value proposition how to start? Where do you start? It is daunting and first. This approach provides the method to tackle any workload no matter the nuances and ensure you capture all the traffic correctly and safely.

  • 8500
  • Building and Visualizing Microsegmentation with Log Insight
  • Breakout Session
  • Logs are one of the most powerful resources that we have, but are often overlooked due to their lack of context. Join us as we show you how to use your log data to create accurate microsegmentation policies, and graphically represent them for easy consumption by even the most junior administrator.
  • Session Outline – This session will cover:
    1. Recommendations from the field for controlling and sending meaningful VMware NSX logs to Log Insight
    2. A live demonstration of grouping pertinent vRealize Automation traffic into a practical dashboard
    3. Continuing the live demo we build security policies from your dashboard to protect and microsegment vRealize Automation
    4. Effective visualisation of log data to validate security posture and detect anomalous traffic patterns
  • Log Insight is more than just a basic troubleshooting tool
  • Log Insight and its free content packs provide context for your logs and a starting point for customization
  • NSX Distributed Firewall and Security Polices can be created with confidence and assurance
  • Advanced Technical
  • Software-Defined Data Center
  • Networking and Security
  • NSX
  • Enterprise
  • Technical Support, IT – All, IT – Risk/Compliance/Security, IT – Operations, IT – Network

Vote for this session here

Readers of this blog will be familiar with some work I have done with Nick Bradford on PowerNSX. This presentation seeks to introduce a wider audience to PowerNSX, a PowerShell module that allows CRUD activities for NSX. The object orientated pipeline provides a unique method of administering NSX environments along with integration into the already popular PowerCLI!

  • 7514
  • PowerNSX – Bringing the power of PowerCLI to VMware NSX for vSphere
  • Breakout Session
  • PowerNSX is a PowerShell module that abstracts the VMware NSX for vSphere API to a set of easily used PowerShell functions. Working seamlessly with VMware PowerCLI, PowerNSX brings unprecendented power and flexibilty to administrators of VMware NSX for vSphere environments. In this session you will learn what PowerNSX is and the flexibility and control that it can bring. From quick ad-hoc queries, to interactive administration and even full-blown automation of complete NSX logical topologies, you will discover how easy it is to leverage your existing PowerCLI skills and extend them to include managing your VMware NSX for vSphere environments. This session will provide an overview of PowerNSX architecture and functionality and then focus on PowerNSX usage and workflows through the use of live demonstrations.
    1. PowerNSX architecture and functionality
    2. Live demonstration of seamlessly using PowerNSX in conjunction with PowerCLI
    3. Practical examples that you can apply to your VMware NSX for vSphere environments
    4. Learnings based on customer deployments on how best to take advantage of PowerNSX
  • Learn what PowerNSX is and what functionality is included
  • Learn how PowerCLI and PowerNSX work hand in hand and get exposed to some examples of common PowerNSX workflows
  • How to get started with PowerNSX and how to contribute
  • Advanced Technical
  • Software-Defined Data Center
  • Networking and Security
  • NSX
  • Enterprise
  • IT – All, IT – Network, IT – Operations, IT – Risk/Compliance/Security

Vote for this session here

If you’re interested in seeing these sessions please vote. There is also additional content in the related links section.