Overview
When are logs made
Many events in the operation of the FireBrick create a log entry. These are a one-line string of text saying what happened.
This could be normal events such as someone logging in to the web interface, or unusual events such as a wrong password used, or DHCP not being able to find any free addresses to allocate.
In the configuration you can set log=”…” attributes on various parts of the config to say what is logged. Some things have separate attributes for unusual events like log-error or extra detail like log-debug.
The value of the attribute says the name of the log target to which the log is to be sent. You can have different log targets which log things in different ways and then you can see specific log targets later (e.g. via web pages). This makes it easy to filter what you see which is important if you are logging a lot of things very quickly.
Where do logs go?
Every log entry is put in a buffer in RAM. This only holds a certain number of log entries (typically around 1MB of text) and old ones are lost. On reboot or power fail the buffer is lost.
This buffer can be viewed via the web interface or command line which can show the history in the buffer and then follow the log in real-time.
Can I make logs more permanent?
Entries from the buffer can also be sent to other places such as email. You can set these differently for each log target.
There is inevitably a slight lag between the log happening and being sent on, and in some cases, such as email, you can deliberately delay the sending of logs to avoid getting millions of emails.
If it cannot keep up with the rate logs are generated then log entries can be lost. The fastest type of external logging is using Syslog which should manage to keep up in pretty much all cases.
In some cases, it is essential to ensure something is logged even if there is then a power failure. You can flag a log target to log to flash memory where it is stored even when there is a power failure.
Does logging slow things down
The FireBrick can log a lot of information, and adding logs can cause things to slow down a little. The controls in the config allow you to say what you log in some detail.
However, logging to flash will always slow things down a lot and should only be used where absolutely necessary.
Default config
The default config has a top-level log object named default which simply means you can set log=”default” on various aspects of the system. If moving from an old config from old software issue a log object called Syslog is also created with the previous Syslog settings.
The default config also has a log object called fb-support which is put on the system log-panic attribute. This allows the FireBrick to automatically email the support team if there is a panic (crash). Please only set things to log to FB-support if requested by support staff. You can, of course, change or delete this if you prefer.
Logging targets
You can create a number of logging targets, each with a name. These are in the system icon on the config menu, and create log objects at the top level.
Each log target has various attributes and sub-objects defining what happens to log entries to this target. Any logs sent to a target are put in the RAM buffer anyway even if not sent on to any external logging system. This allows the log entries to be viewed on the command line or web interface.
What to log and when
Many parts of the config have a log=”…” attribute which allows you to specify where log entries are to go. If omitted then there is no normal logging for that part of the system. When specified, logs are created and sent to the target you specified.
Some parts of the config have additional log attributes such as log-errors or log-debug. These exist where that part of the system has more types of logging available for unusual events. You can specify the log target for each of these additional attributes separately.
log-errors: This is when things happen that should not. It could be something as simple as bad login on telnet. If log-errors is not set but the log is set then errors are logged to the log target by default.
log: This is normal events. Note that if log-error is not set then this includes errors.
log-debug: This is extra detail and is normally only used when diagnosing a problem. Debug logging can be a lot of information. In some cases, whole packets are logged (e.g. PPP). It is generally best only to use debug logging when needed.
Weblog view
The status page showing logs allows you to select which log target to view, or all targets. It then continues showing log events as they happen on the web page.
This is an open-ended web page which has been known to upset some browsers, but this is rare, however, it does not usually work with any sort of web proxy which expects the page to actually finish.
Simply having a log object allows logging to that object to be viewed in the weblog – it does not need to log to anywhere specific like Syslog or email if you just want to view the log on the web pages.
Command-line log view
The command line allows logs to be viewed, and you can select which log target, or all targets. The logging continues on the screen until you press a key such as RETURN. In addition, anything set to log to console shows anyway but you can turn off the logging when you want with troff.
Flash log
The internal flash logging system is separate from the external logging. It applies if the log target object has flash=”true”. It causes each log entry to be written to the internal non-volatile flash log as it is created. This slows down the system considerably. The flash log is intended for urgent permanent system information only and is visible using the show flash log command on telnet. Only set this where absolutely necessary.
The flash log does have a limit on how much it can hold, but it is many thousands of entries so this is rarely an issue. Oldest entries are automatically discarded when there is no space.
Console log
If any log has console=”true” then as soon as possible the log entries are logged to the console. This works much like flash log in that the logging is done right away, but unlike flash, it is unlikely to cause much delay.
The console is any telnet login or serial connection if present. You can stop the console logging with troff or restart it with tron.
System logs
Some aspects of the system are specified by log attributes in the main system object, such as log-stats which generates a log line every second for debugging. Normally, not setting a log attribute causes no logging, but some of the system events are logged to all logs (such as startup/shutdown) and console and flash by default.
Note that some of the code in the FireBrick still uses an older logging system which means that things are still printed to the system log. Over time these will be converted to explicit log entries.
External logging systems
The log target object on its own simply allows logging to the web view or command line view, but can also log to a number of other external logs.
Each of these has its own config object within the log target object and has attributes to control the logging.
Note that if any of the external logging systems cannot keep up then log entries are lost. I.e. the operation of the FireBrick does not wait for them (unlike internal flash logging).
Profiles
The log target itself does not have a profile, but each of the external logging entries can have a profile. Some types of logging will catch up when their profile comes back on (email) but most are immediate (such as Syslog) and will drop any entries when profiled off.
Syslog
This simply causes log entries to be sent by Syslog. You specify the Syslog server, facility and severity. Syslog logging is very quick as there is no reply. The logging is according to RFC5424 which includes microsecond time stamps and the hostname (from system settings) and module name.
The module name (i.e. what part of the system caused the log entry) is also shown in all other types of logging such as web and console. older Syslog servers will typically show time and hostname twice and will need upgrading.
This allows logs to be sent by email. You specify email addresses and optionally a smart host for the email. You can specify the subject or leave it to be based on the first line being logged.
An important aspect of emailed logs is that they have a delay and hold-off.
The delay means that the email is not sent immediately because often events happen over several seconds and it is sensible to wait for several log lines for an event before emailing. The hold-off is also relevant as you do not want thousands of emails – this means after one email a subsequent email is not sent for that time.
A profile can be used to stop emails at certain times, and when the email logging is back on an active profile it tries to catch up any entries still in the RAM buffer if possible.