With Log Insight 3.3 a new Log Insight Importer utility was made available. In this post, I would like to talk about the Importer. Read on to learn more!
Basics
- What: Command line utility for Linux and Windows used to import offline logs / support bundles
- Where: Separate download on the VMware site (not bundled with server today)
- When: Released alongside Log Insight 3.3
- Why: To import offline log / support bundles (agent is for real-time logs)
- How: Leverages the ingestion API
Agent
- Meant for real-time log collection
- real-time = <10 minutes delta client/server
- Runs as a daemon
- Available for Linux and Windows
- Sends events over ingestion API and syslog
- Supports winlog and filelog collection
- Requires a config section per directory with filelog
- Filelog include option defaults to: *.log; *.txt
- Supports parsers
- Supports client and server side configuration
- Does not require authentication
- Does not support compressed files/directories
Importer
- Meant for historic logs and support bundles
- Delta between client/server does not matter
- Run as a standalone application
- Available for Linux and Windows
- Sends events over ingestion API
- Supports filelog collection
- Supports recursive directory collection
- Filelog include option defaults to: *
- Supports parsers
- Supports client side configuration
- Requires authentication to honor timestamp
- Supports compressed files/directories
Flags
| Parameter | Description |
| –source <path> | Path to file/directory/support bundle (zip/gzip/tar). |
| –server <hostname> | Destination server hostname or IP address. |
| –port <9000|9543> | Port for connection. If not set then 9000 will be used. |
| –logdir <dir> | Path to logs directory. If not set then log to $(LOCALAPPDATA)\VMware\Log Insight Importer\log on Windows and ~/.loginsight-importer/log on Linux. |
| –manifest <filepath> | Path to manifest file (.ini format). If not set importer.ini in source directory will be used. In case there is no importer.ini file found in source directory, Importer will apply the default (hardcoded) manifest and collect all .txt and .log files (include=*.log*;*.txt*), and also apply auto parser (extracts timestamp + kvp). |
| –no_ssl | Don’t use ssl for the connection. |
| –ssl_ca_path <path> | Path to the trusted root certificates bundle file. |
| –tags <tags> | Set tags for all sent events. Syntax example –tags “{ \”tag1\” : \”value1\”, \”tag2\”:\”value2\”}”. |
| –username <username> | Username for authentication. Required for –honor_timestamp. If not set user will be prompted for username. |
| –password [<password>] | Password for authentication. Required for –honor_timestamp. If not set user will be prompted for password. |
| –honor_timestamp | Extract event timestamp from log entry. If timestamp is extracted using parsers, events will have that timestamp applied. If there is an event in the logs file, with no extracted timestamp, then the successfully extracted timestamp from the previous event in the same log file will be applied. If no timestamp is found/parsed in the file then the MTIME of log file will be applied as timestamp. |
| –debug_level <1|2> | Increases the verbosity level of the log file – should only be changed when troubleshooting. |
Summary
As you can see, the importer is a great addition to Log Insight! In my next post, I will show how the importer can be leveraged to ingest logs into Log Insight.
© 2016, Steve Flanders. All rights reserved.
