rsyslog host

[Michael Butash]

Size/quantity matters, significantly here.

I'd start of with your expected GiB per day, or messages per second (mps).
Factor in some per log kb standard sizing, come up with both mps/per day,
and you can probably work back from there in terms of required disk iops to
write them (or buy a product that does).   Most log infrastructure products
go by messages per second, or daily volume, so when you talk to them for
that reason (more mps, more iops), they'll ask for either, and give you
estimated pricing back.  There's a reason there are (expensive) commercial
products like Splunk, Arcsight, Logstash/ELK (open-ish) and other
appliances to do these, often times with some extensive disk (even
san/filer) behind them.

The other thing is, what do you want to do with the logs once you have
them?  Great if you have them, but who uses them, and how much?  Dumping
them in a log dir local and expecting everyone to grep or zcat through log
files/archives isn't always practical, thus again splunk, elastic
search/kibana, and many others exist with pretty web ui's and such for
everyone to access more conveniently.

Otherwise with rsyslog, cpu's and memory won't be much needed for straight
fifo writes from syslog sockets (assuming no iowait queuing writing to
disk), disk and iops is more relevant here for when those logs are written,
something is open to do so without wait.  You want some redundancy, I've
used ipvs and corosync in the past to do this between a few hosts giving
active/passive ability, but best is to just write to two redundant hosts.
Disk redundancy is optimal outside raid, some sort of backup routine if not
synced via some software routine or storage appliance for you.  You can go
as far as you want with it...

I've worked for managed security providers (most of what we did was around
log collection!), various isp's, large enterprises, critical infrastructure
(power, scada), etc, and I've seen pretty extensive (read: expensive) log
infrastructures as critical parts of their businesses they spend
accordingly and happily on.  Another successful dot.com I work with just
has a simple rsyslog server in each coastal DC receiving all logs from all
devices/systems, and it runs on an old centos vm that only the handful of
admins, all linux geeks, are happy to just grep around with.

-mb


On Wed, Dec 12, 2018 at 2:09 PM Snyder, Alexander J <
alex at misteralexander.com> wrote:

> Looking for suggestions on what kind of physical resources would suggested
> to building a central logging server for an enterprise company.
>
> rsyslog is new for the company, so we're looking to "do it right" from the
> ground up.
>
> How many hosts should be needed to log networking and storage appliances?
>
> Advice on memory, CPU, and disk are requested. Will be running CentOS7.
>
> Thanks,
> Alexander.
>
> Sent from my Samsung Galaxy S8+
> ---------------------------------------------------
> PLUG-discuss mailing list - PLUG-discuss at lists.phxlinux.org
> To subscribe, unsubscribe, or to change your mail settings:
> https://lists.phxlinux.org/mailman/listinfo/plug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.phxlinux.org/pipermail/plug-discuss/attachments/20181212/109c95dd/attachment.html>