Log Insight and Regular Expression

Log Insight gives you the incredibly powerful RegEx searching powered by what Java can support. You can search for fields to extract and correlate from events. The power of combining these together can yield data that is contextual, relevant, and powerful.

Operators

This post will serve as a reference for supported operators.

Characters

\                Escapes a special character
\b             Word boundary
\B             Not a word boundary
\d             One digit
\D             One non-digit
\n             New line
\r             Return character
\s             One space
\S             Any character except white space
\t             Tab
\w            One alphanumeric or underscore character
\W           One non alphanumeric or underscore character

Quantifiers

.              Any character except a new line
*              Zero or more characters as long as possible
?             Zero or one character OR as short as possible
+             One or more
{<n>}             Exactly <n> times
{<n>,<m>}   <n> to <m> times

Quick test

Testing out quantifiers will give you an idea of what you will see. Assume a string of abcde and lets test some operators.

.            a
*           abcde
.*?        abcde
.{4}      d
.{1,5}   ae

Combinations

.*     Anything
.*?   Anything as short as possible before

Quick test

So if I have a string that is app bravo 3 fault storage fault lets see what occurs when we apply some combinations.

app.*        fault app bravo 3 fault storage
app.*?      fault bravo 3

Logic Operators

Logic operators are where power shines through. Let alone parsing data like previously done, a logic operator will allow concatenation of operators and execute them accordingly.

^      Beginning of a line OR not if in brackets
$      End of a line
()      Encapsulation
[]      One character in brackets
|       OR
–       Range
\A    Beginning of a string
\Z    End of string

Quick test

If you were to RegEx for a string that ended in Manager and nothing else what would you use? Remember that $ denoted an end of a line.

Manager$         Would yield results if a string ended in Manager

(distributed)?  Either contains distributed OR does not
(x|y|z)               x OR y OR z
(a-dp)                a OR b OR c OR d OR p
(a-c|j-l)             a OR b OR c OR j OR k OR l

Lookahead operators

Lookahead’s allow you to apply the logic of Does contain or Does not contain. This allows a vast reduction or logs/strings that aren’t required.

?=              Positive lookahead
?!=             Negative lookahead

More examples

Here are some more examples that are useful for searching logs. They are allow control and manipulation of strings.

[xyz]                          x, y, or z
(info|warn|error)   info, warn, or error
[a-z]                          A lowercase letter
[^a-z]                        Not a lowercase letter
[A-Z]                         An uppercase letter
[^A-Z]                      Not an uppercase letter
[a-z]+                       One or more lowercase letters
[a-z]*                        Zero or more uppercase letters
[a-z] {5}                   Exactly five lowercase letters
[a-z|A-Z] {5}           Exactly five upper or lower case letters
[\d]                           A digit
\d+$                         One or more digit at the end of a string

 

Round off

That is a baptism of fire into RegEx. If you’re looking to practice there are more resources out there. If you’re looking for raw strings to practice against you can use any type of data. Networkers can log into LookingGlass BGP servers and perform BGP RegEx to extract the right data. RegEx is very powerful. When using this RegEx queries as a source for visual data it really does shine through.