Log Insight is downloaded as an OVA. This single OVA is the source of the initial install of Log Insight and the OVA used for additional workers in a Log Insight cluster. The size of the deployment should be selected.
There are three sizes – Extra Small, Small, Medium, and Large. The Log rate is measured in GB per day. The number of vCPU require scales with the instance. The number of IOPS generated by the Log Insight appliance is measured. Syslog connections denote the recommended number of end points. The Events per Second should be taken on board to drive size of environment.
XS 3GB/day 2vCPU 4GB 75 IOPS 20 conn. 200 EPS
S 15GB/day 4vCPU 8GB 500 IOPS 100 conn. 1000 EPS
M 37.5GB/day 8vCPU 16GB 1000 IOPS 250 conn. 2500 EPS
L* 112.5GB/day 16vCPU 32GB 1500 IOPS 750 conn. 7500 EPS
* The large instance requires a hardware upgrade after deployment. This is due to the hardware version of the OVA being v7. v7 only supports 8vCPU. Upgrading this to 8 or higher will allow the use of 16vCPU.
First glance but what about scale?
At first glance these numbers are impressive but in a large environment a single node would not be enough. In a previous life I had 80 ASAs that were quite verbose, let alone VM’s, hosts, storage arrays and more. Scaling out the logging system is the only way. A Log Insight cluster will allow a linear scale out. Deploy a new appliance, point to to join the cluster as shown in this post, and you have increased the amount your log solution can ingest.
A simple three medium node Log Insight cluster behind an Integrated Load Balancer (ILB) or External Load Balancer (ELB) (NSX, F5, HAproxy) could ingest 112.5 GB of records a day, sustain 3000 IOPS, handle over 750 connections, deal with 7500 EPS, and have the ability to sustain a failure of the master or a worker.
Here are some tips for enabling a super highly available logging environment.
- Ensure that the Log Insight worker nodes have anti-affinity rules set to ensure a host failure isn’t a substantial amount of logging ability.
- Ensure DRS can re-distribute workloads based on resources available and to the most suitable host.
- Use a FQDN for the VIP as well as an IP to ensure DNS and IP support for syslog. DNS allows a change of IP without drastic re-addressing.
- ILB or ELB – both have their benefits. For a start, use one! 🙂
- ILB is a new feature of LI 2.5 and eliminates the need for a dedicated or slice of an ELB.
The importance of logs cannot be understated. I love the ability to use historic data to identify issues, predict future events, capacity plan, and visualise what is occurring in my environment. Make your environment work for you and show you what is happening.