Custom Regex queries for Log Insight

The missing query

Log Insight provides content packs that come chocked full of queries, alarms, and dashboards for users of specific products. They cover networking, security, storage, hardware, servers and more. A recent update to the NSX for vSphere content back saw TCP Protocol removed. I use TCP protocol heavily in my “segmentation approach” when learning applications. As a result I needed it back. This is where custom queries are useful.

Custom queries

The query missing was searching the dfwpkt log file for the INET protocol (L3 DFW) and then what protocol is used. This is handy in determining what type of rule to build such as UDP or TCP services.

  • Name: vmw_nsx_firewall_protocol
  • pre-context: (IN|OUT) (\d+ )?
  • post-context: \s
  • custom-regex: (TCP6?|UDP6?|PROTO6?\d+)
  • additional-context dfwpktlogs INET

These fields are create in a custom field. This is done by highlighting an the desired field on a given log (TCP in my case). Right click and select Extract Field.


This results in my queries and dashboards working as desired again.


Now I can easily see what is talking to and from my apps when segmenting them. Happy days.

NOTE: This was removed in the NSX Content Pack 3.4 due to it being a resource expensive query. This expensive regex slowed down a query and a any dashboard it referenced and was removed.

Loops and Power(shell)NSX

There was a request on the internal PowerShell slack channel. It was about how to make over 4,000 Security Groups and probably as many Security Tags for NSX. PowerNSX fits very well here. The requirement was to take the given fields in a CSV file and create objects from there.

Using the import-csv command it is possible to save the content of a comma separated value.

PowerCLI C:\> $csv | ft -auto

------ ----------- -------------
VM-0001 SG-TAG-0001 SG-GROUP-0001
VM-0002 SG-TAG-0002 SG-GROUP-0002
VM-0003 SG-TAG-0003 SG-GROUP-0003
VM-0004 SG-TAG-0004 SG-GROUP-0004
VM-0005 SG-TAG-0005 SG-GROUP-0005
VM-0006 SG-TAG-0006 SG-GROUP-0006
VM-0007 SG-TAG-0007 SG-GROUP-0007
VM-0008 SG-TAG-0008 SG-GROUP-0008
VM-0009 SG-TAG-0009 SG-GROUP-0009
VM-0010 SG-TAG-0010 SG-GROUP-0010

So I can see a list of desired VM names, Security Tag names, and Security group names. Whilst these are an example they could be anything. Whilst the examples don’t necessarily use $csv it does highlight what content is stored and how it’s referenced.

The first example is designed to create a list of Security Groups and have their include membership criteria of a matching Security Tag. This sets up buckets that objects or Virtual Machines can be dropped into.

#This will make based on CSV.
import-csv .\base-example.csv | % {
$st = New-NsxSecurityTag -name $_.SECURITYTAG
$sg = new-NsxSecurityGroup -name $_.SECURITYGROUP -includemember ($st)

It firstly imports the value in the CSV file. The percentage sign, %, is shorthand for the command ForEach-Object. ForEach-Object create a new Security Tag with the value in the SECURITYTAG column. This is stored in the variable $st. Then create a new Security Group from the value in the column SECURITYGROUP. This is repeated for each Object. The script will traverse horizontally and line by line.

$_ that prefixes the column name is representative of “the current object on the pipe” or THIS.

Here is an alternative that adds ability to apply a Security Tag to a given Virtual Machine

import-csv .\base-example.csv | % {
$st = New-NsxSecurityTag -name $_.SECURITYTAG
$sg = new-NsxSecurityGroup -name $_.SECURITYGROUP -includemember ($st)
$vm = Get-Vm -name $_.VMNAME | New-NsxSecurityTagAssignment -ApplyTag $st

The only difference to the first example is application of a Tag to a Virtual Machine. In this example the command will look for a Virtual Machine with the name listed in VMNAME and then apply a new assignment of a Security Tag based on the value stored in $st.

An alternative is to not use any CSV file and create something based on a loop.

(1..10) | % {
[string]$suffix = $_.ToString("0000")
$st = New-NsxSecurityTag -name SG-TAG-$suffix
$sg = New-NsxSecurityGroup -name SG-GROUP-$suffix -includemember ($st)

For each number (1..10) create append the value to the string 0000 and save it as the variable suffix. The next is to create a new Security Tag with the name SG-TAG-$suffix and save it to the variable $st. Then create a new Security Group with the name of SG-GROUP-$suffix and ensure the included member is the Security Tag saved in the value of $st.
Repeat this the number of times listed in the initial integer range.

Cheers to Nick B for optimizing my loops and Iwan H for the request.