vRealize Hyperic: Deployment and Configuration Part 2/2

In part one of my series on vRealize Hyperic deployment and configuration, I discussed an overview of the Hyperic components, downloads available, deployment options, and upgrade procedure. In this post, I want to discuss configuration of the various Hyperic components.

Server

Once you have deployed Hyperic you should connect to the GUI to ensure everything is working properly and then perform some configuration.

Protip: To access the GUI you need to use port 7080, port 80 will not work

Once you log in you will land on the Dashboards page. Unless you already have agents reporting to the server, the Dashboards page will not contain any interesting information. I would recommend going to the Administration tab to ensure Hyperic is configured to your liking. The section to particularly pay attention to is the HQ Server Settings. You will notice at the top of that page it states, “NOTE: Any changes to HQ Server Settings will take effect after the next server restart.” While I have not done extensive testing, I can confirm that NOT all options require a server restart.

Protip: If/When you need to restart Hyperic there is no way to do it from the GUI, you must go the CLI and run: su hyperic; /opt/hyperic/server-current/bin/hq-server.sh restart

For example vCenter Settings take effect immediately.

Protip: Do not change the base URL to an address that cannot be resolved and/or reached. Doing so and restarting Hyperic will result in Hyperic being unable to start with an unhelpful error message in the logs.

Be sure to set the From Email Address as desired as it defaults to localhost.localdom. Also be sure to configure vCenter Settings to get integration and information exchange between vCenter Server and Hyperic.

Protip: No validation of vCenter Settings is done. If you enter an unreachable vCenter or invalid credentials you will not know unless you check the Hyperic logs and/or check for a VCenter UUID from an agent reporting on the Resources tab — more on this later.

In addition to HQ Server Settings, you will also notice licensing information near the bottom of the page. If/When your license expires and/or you have exceeded the limit, you will receive an error banner at the bottom of the page.

Protip: You cannot add or change licenses from the UI. License management is handled on the CLI by editing /opt/hyperic/server-current/conf/hq-server.conf and entering one of the properties under Licensing attributes. You need to uncomment one of the properties and add an = to the end of the configuration option. Once done, you need to restart Hyperic by running: su hyperic; /opt/hyperic/server-current/bin/hq-server.sh restart

Other configuration sections you may be interested in include:

• New User and/or New Role
• Plugin Manager
• HQ Health

Agent

Once you install the Hyperic agent, you need to configure it. Configuration is made up of three parts:

1. Location of Java – the agent requires Java, but does not come with, Java. In addition, the agent init.d file only looks for Java in a specific location (/usr/lib/jvm/java-6-openjdk/jre). In order for the agent to start you may (likely) need to properly set the JAVA_HOME environment variable within the agent init.d file.
   

   Protip: The agent requires, but does not come with, Java and the agent does not respect the JAVA_HOME environment variable

2. Information to access the Hyperic server – the agent communicates with the Hyperic server so it needs to know where the server is and how to connect to it.
3. Additional information to collect – by default, the agent will collect metrics around system health and applications it knows about via plugins. Some plugins require configuration changes on the client devices to expose the interfaces necessary to collect metrics.
   

   Protip: By default the agent does not discover JMX processes.

Let’s discuss each of these in more depth.

Location of Java

The agent init.d script available at /etc/init.d/hyperic-hqee-agent exports the JAVA_HOME environment variable using a static path (/usr/lib/jvm/java-6-openjdk/jre). Given the init script explicitly exports the JAVA_HOME environment variable, one can not simply set the JAVA_HOME variable on the system to get the agent to work. This means any system not using the default path specified by the agent, which I suspect would be most systems, will not be able to start the agent. The solution is to edit the init script and change the JAVA_HOME variable as appropriate or edit the script so it uses the JAVA_HOME variable already set on the system.

Protip: If the agent does not start with an error about finding Java, edit /etc/init.d/hyperic-hqee-agent and set the JAVA_HOME variable to the location of Java on the device.

Information to access the Hyperic server

The agent requires several pieces of information about the Hyperic server including:

• Whether to use unidirectional communication – may be required in some secure environments, defaults to no
• Server IP
• Server port – defaults to 7080
• Server secure Port – defaults to 7443
• Whether to use secure communication – defaults to yes
• Server login username
• Server login password
• Agent IP – defaults to default (typically should not be changed)
• Agent port – defaults to default (typically should not be changed)
• Whether to accept unverified SSL certificates – defaults to no

Phew. The first time you start the agent, if you do not provide an agent configuration the above questions will be presented for you to fill in. If you wish to automatically configure the agent then you need to edit /opt/hyperic/hyperic-hqee-agent/conf/agent.properties. In the file you will find a section with agent.setup. near the top of the file. You need to uncomment and properly configure all of these options in order for the agent to start without manual intervention.

Protip: The are 11 agent.setup. options that need to be uncommented. 10 are grouped together and the 11th is directly below the group of 10, but is proceeded by a comment. Even though you may desire the default option, you still must uncomment the section or automatic configuration will not work.

If this is your first time configuring a Hyperic agent to connect to a Hyperic server using the credentials you wish to specify than in order to automate, you need to enter the password in plain-text for the agent.setup.camPword option. Once the agent starts successfully for the first time the plain-text password will automatically be converted into an encrypted password.

Protip: Only once you have the encrypted password for the Hyperic server in the agent configuration can you securely automate the configuration of the agent on other devices by copying the agent.properties and the agent.scu files to other systems.

After you have configured an agent to connect to a Hyperic server, you need to add the agent to the server from the server UI. To do this, log into the Hyperic server and under Dashboards you will see an Auto-Discovered section. Every configured agent will appear here and need to individually be added to the Hyperic server.

Protip: Any time you make any change to an agent that has already been added to a Hyperic server you will need to add the agent to the Hyperic server again. For a way to handle this automatically, see the API section below.

Also note that if an agent connects to a server, but is not added to the server and then the agent is stopped (e.g. VM is deleted), the agent remains on the server UI. The server UI does not natively have a way to delete an agent and attempting to add an agent this is currently unreachable results in an error. The solution is to go to Administration > HQ Health and from the Database tab selecting Purge AIG Data. Note this will clear all auto-discovered agents. I suspect using the Maintenance tab’s Clear Orphaned Nodes option would work if you only want to remove unavailable agents, but I have not tested.

Protip: Administration > HQ Health has advanced options for performing operations that are not otherwise available in the UI. Note these commands can negatively impact Hyperic server performance and may also lead to data loss so use at your own risk.

Finally, note that the agent relies on the client’s hostname. If the client’s hostname is changed, the agent will not be able to start. The solution is to delete the data directory of the agent and then start the agent:

rm -rf /opt/hyperic/hyperic-hqee-agent/data; /etc/init.d/hyperic-hqee-agent start

Additional information to collect

If you wish to have complete monitoring of your devices, you may need to properly enable the applications on your devices. In addition, if you are running JMX processes then you need to configure the agent to discover and monitor them. Covering how to configure each application is beyond the scope of this post, but let’s look at a common application type: JMX. To enable monitoring of JMX processes you need two things:

1. The JMX application configured to expose jmxremote. To do this, you need to ensure that the JAVA_OPTS for the JMX enable jmxremote. For example, the configuration may look like:

   JAVA_OPTS+=" -Dcom.sun.management.jmxremote \
   -Dcom.sun.management.jmxremote.port=6969 \
   -Dcom.sun.management.jmxremote.ssl=false \
   -Dcom.sun.management.jmxremote.authenticate=false"

   If you were configuring this for Tomcat then you would add the above configuration to <tomcat>/bin/catalina.sh before the Tomcat process gets started.

   Protip: Be sure to restart the JMX process after enabling jmxremote

2. The agent configured to discover JMX processes. There are multiple ways to do this, but assuming you wish to monitor all JMX processes on the system, the easiest thing to do is edit/opt/hyperic/hyperic-hqee-agent/conf/agent.properties and add:

   jmx.sun.discover=true

   Protip: Be sure to restart the Hyperic agent after editing the configuration and then be sure to accept the agent change on the Hyperic server

API

The Hyperic server features an API, but by default the server and agent do not have the API package installed. To get the API package you can go to the Administration section of the Hyperic server and select HQ Web Services Api. From there, you can download the hqapi1-client tarball.
One reason why you might want to leverage the API is because it provides a function for the server to automatically discover new agent or changes to existing agents. This would remove the manual step of needing to add agents via the Dashboards page of the Hyperic server UI.

Protip: The API can be used to approve auto-discovered agents whether new or modified

The command to approve auto-discovered agents is:

hqapi.sh autodiscovery approve

In my environments, I have added a cronjob to approve automatically. On the Hyperic server, I have installed the API and then added to /etc/crontab:

0 * * * * root /opt/hyperic/hqapi1-client-6.0.4/bin/hqapi.sh autodiscovery approve 2>&1

Summary

As you can see, while Hyperic is made of only two primary components, there is a lot to know in terms of configuration and behavior of the product. For the server, basic configuration is required to ensure proper access to the server as well as integration with vCenter Server. For the agents, understanding configuration is key. Be sure to have Java installed, set the JAVA_HOME on the agent init.d script properly and either automate the initial configuration or fill in the prompts on first start. The agent out of the box performs basic system monitoring, but if you require more specific monitoring, like JMX, you need to configure that manually. The agent relies on the client’s hostname and if the client’s hostname is changed the agent’s data directory must be deleted before the agent will start. Also note that new or changed agents must be added the Hyperic server. By default this is done manually, but it can be done through the Hyperic API and a cronjob.

© 2015, Steve Flanders. All rights reserved.