Log ALL the things
With the importance of logging on the increase you need to ensure you have a distributed log management system that can ingest thousands, if not hundreds of thousands of inputs per second. The Log Insight log appliance from VMware allows the ingestion of many logs. Coming from an ELK background there is a sense of familiarity here. Ingestion, RegEx and search, and awesome visualisation. Keeping the logging service up and running is important. This has been traditionally been solved with an external load balancer.
It is no longer BYO Load Balancer
The architecture of Log Insight is clustered in nature. Whilst you can have a stand alone server processing logs, if you require high log ingest then you need to have additional workers. The master is the active node in which lookups are performed against. Workers take logs in from the External Load Balancer (ELB) or Integrated Load Balancer (ILB). ILB and ELB allow logs to continue to be ingested when a worker or node become unavailable.
The image below highlights the sources of how my Log Insight cluster is set up.
The ILB here is enabled on the IP address of 192.168.110.99. It is also found through the FQDN of log-l-01.corp.local DNS entry. The VIP takes the L2 MAC of the current master and acts as a proxy. In this case the current Master node is 192.168.110.100. The ILB allows log ingestion to be balanced across worker nodes in a LI cluster. When a worker is put into maintenance mode it can be done safely without loss of log ingestion.
The image below shows the LogInsight workers, the Master, and the VIP.
192.168.110.99 with MAC 00-50-56-03-1b-25 shares the MAC of the current master.
Tips from the coal face
There are some things to watch with regards to migrating or changing from ELB to ILB, changing VIPs and adjusting DNS entries.
- Watch the TCP session. It is possible that long, persistant TCP sessions could be still flowing across an ELB.
- If using a FQDN ensure that the DNS TTL is decremented for cut-over.
- There is a L2 requirement of workers, master and the VIP. ARP is used to discover neighbours so factor this into the design.
Note : If using an ELB you have a different source IP of logs. This can cause confusion when looking at log entries. The ILB honors the source IP of the log therefore giving more accurate logs.
Logging
Logging is paramount with the numerous events that occur. Having a robust and stable platform that not only uses logs but allows users to CONSUME their logs is important. Visualisation of data is important and having the correct and accurate data produces valid outcomes. Protecting your cluster’s log and ingestion will help immensely in troubleshooting scenarios.