Collecting log messages and analyzing them with Log Insight is easy, but what about collecting and monitoring other types of events? How about logging and analyzing the weather for example?! Read on to learn how.
NOTE: My good friend and colleague Tomas Baublys did an awesome blog post on how to log the weather with Log Insight here. With his permission, I am doing an english version of the post. The concept, information, and content pack are all his work.
How to collect the weather
First, a script to collect weather metrics (this one requires cURL and logger):
#!/bin/bash
# Variables
LOCATION='BOS'
METRICS='standard'
# Get weather data
METEODATA=`/usr/bin/curl http://api.openweathermap.org/data/2.5/weather?q=${LOCATION}\&units=${METRICS}`
# Log weather data
/bin/logger "$METEODATA"
Next, save and run the script on a schedule. For example, to run the script every five minutes with cron use something like:
*/5 * * * * root /usr/bin/bos_weather.sh
The output of the cURL command is a JSON string like:
{"coord":{"lon":-71.06,"lat":42.36},"sys":{"message":0.0074,"country":"United States of America","sunrise":1384774784,"sunset":1384809566},"weather":[{"id":801,"main":"Clouds","description":"few clouds","icon":"02d"}],"base":"gdps stations","main":{"temp":289.99,"pressure":1001,"humidity":55,"temp_min":289.15,"temp_max":291.15},"wind":{"speed":6.2,"deg":250,"gust":9.8},"rain":{"3h":0},"clouds":{"all":20},"dt":1384792282,"id":4930956,"name":"Boston","cod":200}
The string is sent over the syslog protocol as defined by the logger command. While the logger command can be run without flags, I would encourage looking at the logger man page and consider adding at least a tag to messages to make querying easier. Also, newer versions of the logger command support sending events to a remote syslog destination via flags.
How to send weather data to Log Insight
If the logger command supports sending to a remote syslog destination then this is one way to get the data into Log Insight. An alternative is to have logger log to the local system and then use a syslog agent like syslog-ng or rsyslog to forward events to Log Insight. I covered forwarding events using a syslog agent in this post.
Analyzing weather data in Log Insight
Looking on the Interactive Analytics page of Log Insight you should see weather events coming in based on the schedule you set above. The JSON format of the event makes it easy to extract useful pieces of information such as the wind speed. To do this, highlight the wind speed, select the Extract Field option, and give the field a name (if you used the tag flag on logger you may want to add that information into the pre-context of the field):
Once you extract the fields you care about, you can use the fields in queries and as part of aggregation functions and groupings to change the visual representation of events at the top of the screen. For example, you can select the maximum wind speed over time:
The possibilities of what you can analyze is only limited by the information provided in the JSON. Here is an example of a content pack analyzing the weather:
Here is an example content pack: Weather.vlcp
NOTE: This content pack is based on, but different from the one created by Tomas:
- Generalized queries
- Grouped by coordinates to support multiple locations
- Optimized extracted fields for performance
How about other types of data
This is just one example of collecting and analyzing data with Log Insight. The possibilities are endless! Another example would be to monitor stock prices. Here is a sample script that would do just that:
#!/bin/bash
# Variables
STOCK='VMW'
# Get stock data
METEODATA=`/usr/bin/curl http://download.finance.yahoo.com/d/quotes.csv?s=${STOCK}&f=sl1&e=jason`
# Log sotck data
/bin/logger "$METEODATA"
All the credit goes to Tomas, great work!
UPDATE: I was asked if the weather could be collected and analyzed without using cURL and logger. Basically, you need a way to collect the weather and cURL is one way to do it. Another option would be to use wget. Alternatively, if you have a way to get the data on the filesystem you could use that as well. Logger is used to send the events over syslog either locally or to a remote destination. As an alternative to logger, you could just write the data to a file and then have a syslog agent monitor the file and send the changes over syslog to Log Insight. I covered monitoring a file in this blog post.
© 2013 – 2021, Steve Flanders. All rights reserved.
