All posts by Jonathan Almquist

Install SCOM Agent on Red Hat Enterprise Linux 6 (linux agent installation)

This is a step-by-step article on installing the SCOM agent on a RHEL6 system, both from an SCOM and Linux administrator perspective. Everything in this demonstration assumes a new installation of RHEL6 with no additional configuration. It is also assumed there are no firewalls between the SCOM management servers and the RHEL system. Windows Firewall is not turned on, and the RHEL system is running with default IPTABLES configuration.

Disclaimer: I do not claim to be an experienced Linux admin. This is intended to familiarize SCOM administrators with the entire process. There may be better practices or methods for configuring Linux – comments are  welcome.

If you do not have an SCOM environment or a RHEL system in your lab, you can download evaluation versions from the links below (these links may be outdated at the time you are reading this).

SCOM 2012 SP1 Evaluation

RHEL 6 Evaluation (download the rhel-server-6.3-x86_64-dvd ISO)

Installing RHEL6 on Hyper-V step-by-step

Here are the tasks that will be addressed in this walk-through:

   1: Create a Linux user account for the OpsMgr service

   2: Sudoers file configuration

   3: Configure iptables (Linux firewall)

   4: Verify DNS – forward and reverse lookup

   5: Create a cross-platform monitoring resource pool

   6: Install certificates on resource pool members

   7: Install required management packs

   8: Runas accounts and profiles

   9: Discovery wizard

 

Configuring Linux

Before attempting to discover and install the SCOM agent, an account needs to be created on the Linux system and the sudoers file must be configured to allow the account to execute certain commands. Most out-of-box monitoring activities are performed using low privileges on the Linux system, but some activities do require elevation using sudo.

In this example, SCOM will leverage a single Linux account to perform all activities: opsmgrsvc.

 

Create the opsmgrsvc account

There are several free tools you can use to establish a SSH session to a Linux system. The most common is Putty, which can be downloaded at http://www.putty.org/.

Launch Putty and connect to the Linux host by entering the host name or IP address of  the RHEL system and clicking Open.

image

 

Login with an account that has privileges to add user accounts and modify the sudoers file, or login with root. Create the opsmgrsvc account using the following command:

useradd opsmgrsvc

Next, set the password for the account using the following command:

passwd opsmgrsvc

You will be prompted to enter the password after issuing the command.

Configure sudoers file

Next you will configure the sudoers file, giving the opsmgrsvc account the ability to elevate and execute privileged commands without requiring a password (this is a SCOM requirement).

While still logged into the Linux SSH session as root, open the sudoers file editor by typing visudo and then:

1. Move down the file until you find the line Defaults requiretty, and enter Defaults:opsmgrsvc !requiretty just after it. This setting allows the service account to login without requiring a TTY session.

image

 

2. Move down the file until you find the line root ALL=(ALL) ALL, and enter opsmgrsvc ALL=(root) NOPASSWD: ALL just after it. This setting allows the service account to elevate without requiring a password.

image

 

3. Hit the ESC key to finish editing. To save and close the sudoers file, enter :wq. If you made a mistake in editing the file, you can close without saving your work by entering :q!.

 

NOTE!!! Your Linux administrator will likely want more granular control over which commands the SCOM service account can execute. Below are example sudoers file entries which would enable the opsmgrsvc account to only execute the command necessary for agent installation, maintenance tasks, and baseline monitoring.

More information about sudoers file requirements and examples for other cross-platform systems can be found at http://social.technet.microsoft.com/wiki/contents/articles/7375.configuring-sudo-elevation-for-unix-and-linux-monitoring-with-system-center-2012-operations-manager.aspx.

NOTE: Copy/Paste the following – there is no word wrap, so some of it may float off the screen!

#----------------------------------------------------------------------------------- 
#User configuration for Operations Manager agent: opsmgrsvc

#General requirements for OpsMgr service account
Defaults:opsmgrsvc !requiretty

#Agent discovery, installation, certificate signing, and maintenance tasks
opsmgrsvc ALL=(root) NOPASSWD: /bin/sh -c cp /tmp/scx-opsmgrsvc/scx.pem /etc/opt/microsoft/scx/ssl/scx.pem; rm -rf /tmp/scx-opsmgrsvc; /opt/microsoft/scx/bin/tools/scxadmin -restart
opsmgrsvc ALL=(root) NOPASSWD: /bin/sh -c sh /tmp/scx-opsmgrsvc/GetOSVersion.sh; EC=$?; rm -rf /tmp/scx-opsmgrsvc; exit $EC
opsmgrsvc ALL=(root) NOPASSWD: /bin/sh -c cat /etc/opt/microsoft/scx/ssl/scx.pem
opsmgrsvc ALL=(root) NOPASSWD: /bin/sh -c rpm -e scx
opsmgrsvc ALL=(root) NOPASSWD: /bin/sh -c /bin/rpm -F --force /tmp/scx-opsmgrsvc/scx-1.[0-9].[0-9]-[0-9][0-9][0-9].rhel.[0-9].x[6-8][4-6].rpm; EC=$?; cd /tmp; rpm -rf /tmp/scx-opsmgrsvc; exit $EC
opsmgrsvc ALL=(root) NOPASSWD: /bin/sh -c /bin/rpm -U --force /tmp/scx-opsmgrsvc/scx-1.[0-9].[0-9]-[0-9][0-9][0-9].rhel.[0-9].x[6-8][4-6].rpm; EC=$?; cd /tmp; rpm -rf /tmp/scx-opsmgrsvc; exit $EC

#Log file monitoring
opsmgrsvc ALL=(root) NOPASSWD: /opt/microsoft/scx/bin/scxlogfilereader -p

#Custom shell command monitoring example. Replace <shell command> with the correct command string
#opsmgrsvc ALL=(root) NOPASSWD: /bin/bash -c <shell command>

#Daemon diagnostic and restart recovery tasks (example using cron)
#opsmgrsvc ALL=(root) NOPASSWD: /bin/sh -c ps -ef | grep cron | grep -v grep
#opsmgrsvc ALL=(root) NOPASSWD: /usr/sbin/cron &

#Add sudo logging
Defaults logfile=/var/log/sudolog

#End user configuration for Operations Manager agent
#-----------------------------------------------------------------------------------
 
Configure Linux Firewall (iptables)

The firewall is enabled in a default installation of RHEL6. It comes preconfigured to allow SSH connections, ICMP on local subnet, and very basic network service functionality (these protocol ports may need to be manually configured on other distributions). To see the firewall configuration, run the following command in your SSH session:

iptables --list

image

SCOM discovery and installation uses SSH to copy installation files to the target computer and execute shell scripts, such as GetOSVersion.sh, but before this can happen SCOM needs to connect to the SCX CIM provider on the remote host using WinRM. This occurs over port 1270, which is not open in the default iptables file.

To allow WinRM to do it’s job, add the iptables rule with the following command (this is one line):

iptables -I INPUT 1 -p tcp -s 192.168.2.0/24 --dport 1270 -m state --state NEW,ESTABLISHED -j ACCEPT

Save iptables configuration with the following command:

/sbin/service iptables save

Take a look at the ruleset to make sure it saved correctly.

iptables --list

image

Important: You will need to save your iptables configuration and configure the network interface to restore iptables rules at startup. Otherwise, those rules will disappear the next time the computer restarts. Check your distribution for saving/restoring iptables.

 

iptables command explained

-I INPUT 1 : insert new rule at the beginning of the ruleset.

-p tcp : protocol to match.

-s 192.168.2.0/24 : accept requests from specified host (or hosts). Note: this must be changed for your environment or can be removed from the command to accept from anywhere.

--dport : destination port to match on the host.

-m : match specific connection states.

--state NEW, ESTABLISHED : connection states to match.

-j ACCEPT : accept connection and stop processing the ruleset.

 

DNS Configuration

If you host is not resolvable in DNS – forward and reverse – then read this section. Otherwise, you may skip to the next section about creating a resource pool.

Forward lookup zones are required for SCOM in general, and reverse DNS lookup zones are minimally required for a management server to establish communication with UNIX/Linux agents. connections with the management group. Some DNS systems do not automatically create DNS records, so in this example we will walk thru creating these records and zones in Windows DNS Manager.

 

Create forward lookup record

To register the Linux host in DNS, open DNS Manager and create a new A record in the forward lookup zone of your domain. If you have a reverse DNS lookup zone configured in your domain, select Create associated point (PTR) record. This should automatically create the reverse DNS record, but you may want to verify this on your first attempt.

image

 

Verify your configuration with nslookup.

image

 

In this example, forward lookup is working with nslookup rhel6-01, but reverse lookup isn’t working with nslookup 192.168.2.70. This is because either there was no reverse lookup zone configured in the domain, or the Create associated pointer (PTR) record option was not selected. In this case, a reverse DNS lookup zone did not exist.

 

Create reverse lookup record

If you don’t have a reverse lookup zone in your DNS, you’ll need to create one in DNS Manager. Select the defaults in the wizard, entering your network Id – in this case, 192.168.2.x.

image

 

Now a PTR record can be created for the Linux host. One way to do this is to open the A record created in the previous step (in the forward lookup zone) and select Update associated pointer (PTR) record.

image

 

Testing reverse DNS lookup now returns the Linux host name, and now you’re ready to move onto the next step.

image

 

Create a Resource Pool

In a typical production environment, there will be a resource pool dedicated to monitoring cross-platform systems. This resource pool should contain at least two management servers for high-availability. This example adds management servers OM12-MS02 and OM12-MS03 to the new Cross-Platform Monitoring Resource Pool.

Navigate to Administration > Resource Pools in the Operations console to create a new resource pool.

image

 

Add the management servers.

image

 

Install Certificates on Resource Pool Members

Each management server in the cross-platform monitoring resource pool needs to have it’s certificate installed on each other management server. This is task is performed by running the scxcertconfig tool on each management server; first exporting the certificate from each management server, then importing each other management server’s certificate.

Login to each management server in your cross-platform resource pool, open a command prompt as administrator, navigate to %Program Files%\System Center\Operations Manager 2012\Server folder, and execute the following command:

scxcertconfig.exe -export <hostname.cert>

In this example, <hostname.cert>is replaced with OM12-MS02 on the first management server.

After exporting the certificate on each management server, import the certificate(s) from each other management server in the same manner. in this example, if I’m logged into OM12-MS02, I would execute the following command:

scxcertconfig.exe -import OM12-MS03.cert

More information about this procedure can be read on Technet at http://technet.microsoft.com/en-us/library/hh287152.aspx.

 

Install Management Packs

By default, there are several Unix/Linux management packs installed in SCOM.

image

 

However, this is not enough to perform agent installation using the Discovery Wizard. You will need to import the version specific management packs before even attempting to install a cross-platform agent, because they contain definitions specifically designed to detect which version of the the installation package to install on the target host.

In this example, we minimally need to install the Linux and Red Hat Libraries, and the RHEL6 bundle. These management packs can be found on your installation media, in the Management Packs folder. You can also download most recent versions from the online management pack catalog.

Navigate to Administration > Management Packs in the Operations console to install the required management packs: Microsoft.Linux.RHEL.6.mpb, Microsoft.Linux.Library.mp, and Microsoft.Linux.RedHat.Library.mp.

image

 

At this point, all the prerequisites are satisfied for installing the agent on a Red Hat Enterprise Linux 6 operating system by using the Discovery Wizard. Unless you don’t want to actually monitor the Linux system, you can skip to the Discovery Wizard section.

 

Configuring Runas Accounts and Profiles

One last thing we need to do in order for object discovery and monitoring workflows can run on our Linux system. This isn’t necessarily required to install cross-platform agents, but obviously there wouldn’t be much value in it if this task isn’t completed.

Navigate to Administration > Run As Configuration > UNIX/Linux Accounts in the Operations console and create a new account.

image

 

Create monitoring account

1. On the Account Type screen, select Monitoring account.

2. On the General Properties screen, enter a display name.

image

 

3. Enter your Linux monitoring account credentials on the Account Credentials screen – do not use elevation with this account. If you use elevation with this account, monitoring workflows implementing Linux commands that do not support sudo elevation will fail.

image

 

4. Select More secure on the Distribution Security screen. Then click Create and Close.

 

Create elevated monitoring account

Create another account that will be used for elevated monitoring tasks that require sudo.

1. On the Account Type screen, select Monitoring account.

2. On the General Properties screen, enter a display name.

image

 

3. Enter your Linux monitoring account credentials on the Account Credentials screen – use elevation with this account.

image

 

4. Select More secure on the Distribution Security screen. Then click Create and Close.

 

Associate accounts with profiles

Now the accounts need to be associated with the appropriate Runas profiles.

1. Navigate to Administration > Run As Configuration > Profiles and open the properties of the UNIX/Linux Action Account.

2. Click Add on the Run As Accounts screen, and select the regular monitoring account (not the privileged account). Keep the default – All targeted objects – and click Ok.

3. After clicking Save, the wizard gives an opportunity to securely distribute the account. Click on your account to step into the distribution wizard.

image

 

4. Clicking Add on the Distribution tab launches the Computer Search screen. Select Search by resource pool name option, then click Search. Select your cross-platform resource pool in the Available items box. Click Add, then Ok.

image

 

5. Click Ok, then Close the Run As Profile Wizard.

 

Perform the same steps for the UNIX/Linux Privileged Account profile, using the privileged account you setup earlier.

 

Discovery Wizard

Navigate to Administration > Device Management in the Operations console to start the discovery wizard, and UNIX/Linux computers.

image

 

On the Discovery Criteria screen, Add the Linux host information and click Set credentials. Keep Discovery type set to All computers.

image

 

On the Credentials Settings screen, select User name and password and enter the account information that was created previously. Select This account does not have privileged access on the drop down list. Then click Ok.

image

 

The Credentials section on the Discovery Criteria screen should look like this.

image

 

Click Save, select the Cross-Platform Monitoring Resource Pool, and click Discover.

image

 

The host should be discovered and you can continue by ticking the checkbox and clicking Manage. If you scroll to the right in this window, you can also see that operating system, version, and architecture information was discovered.

image

 

After clicking manage, the wizard goes through an installation, validation, and certificate signing process. Given that all goes well, a typical installation should take 30-60 seconds for a single host. Within minutes you should see the new Linux agent appear in the monitor space, with objects being discovered and monitored.

image

 

 

Here are some useful resources to check in run into problems.

Unable to add the domain to the subject

Troubleshooting UNIX/Linux Agent Discovery in System Center 2012 – Operations Manager

Scenario #1 – “There were no computers that met the specified discovery criteria”

Scenario #2 – “SSH discovery failed.” with “unspecified problem”

Scenario #3 – “Did not find a matching supported agent”

Scenario #4 – “New computer shows as Platform: Unknown and Version: Unknown”

Sudoers Manual

Output list of scripts contained in a management pack

I was perusing my library of powershell scripts today and stumbled across this neat little gem. I do not recall writing it, so probably cannot take the credit. But I wanted to post it here in case anyone else can make use of it.

If the original author is here, please raise your hand!

$xml = New-Object “System.Xml.XmlDocument”
$xml.Load(“path to management pack xml”)
$scripts = $xml.SelectNodes(“descendant::ScriptName”)
$scripts | group “#text” | select name | sort name

Query SQL Server using Powershell

Just provide the required parameters to query your SQL Server.

$dbConnection: FQDN[,Port]

$dbName: database name

Param([string]$dbConnection, [string]$dbName)

function QuerySqlDatabase
{
    # Setup the SQL connection
    $connection = new-object system.data.sqlclient.sqlconnection("Data Source=$dbConnection;Initial Catalog=$dbName;Integrated Security=SSPI;");

    # Define TSQL
    [string]$query =
                "
                your SQL statement here
                "
   
    # Create the dataset and catch errors
    Try {
        $execute = new-object system.data.sqlclient.sqldataadapter ($query, $connection)
        $dataset = new-object system.data.dataset
        $execute.Fill($dataset) | Out-Null #speeds processing
     }
    Catch {
        [System.Exception]
        write-host "Exception while attempting to connect to SQL Server"
        $connectionDetails = $error[0]
        Write-Host "Exception details: $connectionDetails"
        Write-Host "Exiting"
        Exit
    }
    # Create an empty collection
    [System.Collections.ArrayList]$Rows = @()
   
    # Add each row in dataset to the collection
    foreach ($row in $dataset.Tables[0]) {
        $Rows.Add($row) | Out-Null #speeds processing
    }
   
    # Pass collection to main script.
    return $Rows
   
}

QuerySqlDatabase $dbConnection $dbName

Starting a Fragment Library using Visual Studio Authoring Extensions

Started using Visual Studio Authoring Extensions recently for all MP authoring, and leaving the Authoring Console behind since it is no longer supported.

I still can’t get used to the templates that are included with VSAE, so I like to create empty management pack fragments. I do this in order to create a library of fragments that I can easily add to management packs in the future.

In this example, I will demonstrate how to start a fragment library by adding a powershell write action that will be used in a scheduled powershell rule.

Create a new Operations Manager 2012 add-on project and add a couple folders to the solution: modules and rules. Then create a new item in the modules folder.

image

Add a new empty management pack fragment.

image

Give it a logical name, like Powershell Write Action, and enter code similar to the following.

 

<ManagementPackFragment SchemaVersion=”2.0″ xmlns:xsd=”http://www.w3.org/2001/XMLSchema”>
<TypeDefinitions>
<ModuleTypes>
<WriteActionModuleType ID=”YourMp.YourPowershellWriteAction” Accessibility=”Internal” Batching=”true”>
<Configuration>
<xsd:element minOccurs=”1″ name=”TimeoutSeconds” type=”xsd:integer” />
<xsd:element minOccurs=”1″ name=”YourScriptParamter1″ type=”xsd:string” />
<xsd:element minOccurs=”1″ name=”YourScriptParamter2″ type=”xsd:string” />
</Configuration>
<OverrideableParameters>
<OverrideableParameter ID=”TimeoutSeconds” Selector=”$Config/TimeoutSeconds$” ParameterType=”int” />
<OverrideableParameter ID=”YourScriptParamter1″ Selector=”$Config/YourScriptParamter1$” ParameterType=”string” />
<OverrideableParameter ID=”YourScriptParamter2″ Selector=”$Config/YourScriptParamter2$” ParameterType=”string” />
</OverrideableParameters>
<ModuleImplementation>
<Composite>
<MemberModules>
<WriteAction ID=”WA” TypeID=”Windows!Microsoft.Windows.PowerShellWriteAction”>
<ScriptName>ScriptName.ps1</ScriptName>
<ScriptBody>
<![CDATA[
Param(
[string]$YourScriptParamter1,
[string]$YourScriptParamter2
)
Do stuff
]]>
</ScriptBody>
<Parameters>
<Parameter>
<Name>YourScriptParamter1</Name>
<Value>$Config/YourScriptParamter1$</Value>
</Parameter>
<Parameter>
<Name>YourScriptParamter2</Name>
<Value>$Config/YourScriptParamter2$</Value>
</Parameter>
</Parameters>
<TimeoutSeconds>$Config/TimeoutSeconds$</TimeoutSeconds>
</WriteAction>
</MemberModules>
<Composition>
<Node ID=”WA” />
</Composition>
</Composite>
</ModuleImplementation>
<InputType>System!System.BaseData</InputType>
</WriteActionModuleType>
</ModuleTypes>
</TypeDefinitions>
</ManagementPackFragment>

Of course, add you script in the script body section. This is an example that will pass in two parameters to the script.

Now add another new empty management pack fragment to the rules folder, and enter code similar to the following.

<ManagementPackFragment SchemaVersion=”2.0″ xmlns:xsd=”http://www.w3.org/2001/XMLSchema”>
<Monitoring>
<Rules>
<Rule ID=”YourMp.YourPowershellRule” Enabled=”true” Target=”YourMp.YourClass” ConfirmDelivery=”true” Remotable=”true” Priority=”Normal” DiscardLevel=”100″>
<Category>Operations</Category>
<DataSources>
<DataSource ID=”Schedule” TypeID=”System!System.SimpleScheduler”>
<IntervalSeconds>600</IntervalSeconds>
<SyncTime />
</DataSource>
</DataSources>
<WriteActions>
<WriteAction ID=”WA” TypeID=”YourMp.YourPowershellWriteAction”>
<TimeoutSeconds>300</TimeoutSeconds>
<YourScriptParamter1>first parameter to pass into script</YourScriptParamter1>
<YourScriptParamter2>second parameter to pass into script</YourScriptParamter2>
</WriteAction>
</WriteActions>
</Rule>
</Rules>
</Monitoring>
</ManagementPackFragment>

This rule uses a scheduler as a data source and your new powershell write action module. Remember to fill in the script paramter elements after the script body – this is how the parameter are actually passed into the script.

If you continue saving new fragments to this library project, you can easily add these to your management pack projects in the future by adding an existing item and navigating to your library project.

This is your starter fragment library.

image

Detected malicious verification code when verifying element

Quick post here. Hopefully you find this before spending an hour on it like I did. It ended up being a typo in a write action type id.

Basically, it simply came down to changing this:

<WriteAction ID=”WA” TypeID=”MyMp.MyWriteActin”>

To this:

<WriteAction ID=”WA” TypeID=”MyMp.MyWriteAction”>

Lesson learned – be careful when authoring XML directly and use intellisense when it’s available.

 

Below is the full error I receive by the Visual Studio Authoring Extensions.

Error    72    Detected malicious verification code when verifying element of type Microsoft.EnterpriseManagement.Configuration.ManagementPackRule with inner exception: System.Collections.Generic.KeyNotFoundException: The given key was not present in the dictionary.
at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
at Microsoft.EnterpriseManagement.Configuration.ManagementPackRule.VerifyDataTypes(Dictionary`2 moduletypes)
at Microsoft.EnterpriseManagement.Configuration.ManagementPackRule.Verify(VerificationContext context)
at Microsoft.EnterpriseManagement.Configuration.Verification.VerificationEngine.VerifyCollectionItems(Object context)   (Path = DR.DiscoveryAutomation)    Management Pack Display Name.mpx    11    10    Discovery Automation