Category Archives: Cross-Platform

Unable to add the domain to the subject

Quick break/fix post here, because I was unable to find the solution to this subject anywhere in the community blogs or in KB articles.

I ran into this error while attempting to perform a manual installation of the System Center 2012 SP1 Operations Manager agent on a Linux (Ubuntu 10.04) server.

Full context of this error is as follows:

jonathan@ubuntu-01:/etc/opt/microsoft/scx$ sudo dpkg -i /home/jonathan/scx-1.4.0-906.universald.1.x64.deb (Reading database ... 44245 files and directories currently installed.) Preparing to replace scx 1.4.0.906 (using .../scx-1.4.0-906.universald.1.x64.deb) ... * Shutting down Microsoft SCX CIM Server: [fail] invoke-rc.d: initscript scx-cimd, action "stop" failed. Unpacking replacement scx ... Setting up scx (1.4.0.906) ... Checking existence of /lib64/libssl.so.0.9.8k and /lib64/libcrypto.so.0.9.8k ... Checking existence of /lib64/libssl.so.0.9.8 and /lib64/libcrypto.so.0.9.8 ... Found /lib64/libssl.so.0.9.8 and /lib64/libcrypto.so.0.9.8 ... Generating certificate with hostname="ubuntu-01", domainname="scomskills.com." WARNING! Could not read 256 bytes of random data from /dev/random. Will revert to less secure /dev/urandom. See the security guide for how to regenerate certificates at a later time when more random data might be available. Error generating SSL certificate: 'Unable to add the domain to the subject.' dpkg: error processing scx (--install): subprocess installed post-installation script returned error exit status 3 Processing triggers for ureadahead ... Errors were encountered while processing: scx

Long story short, the problem was resolved by modifying the /etc/resolv.conf file. Specifically, removing the trailing “dot” at the end of scomskills.com..

Here is the full context after modifying that file, which resulted in a successful installation:

jonathan@ubuntu-01:/etc/opt/microsoft/scx/ssl$ sudo nano /etc/resolv.conf jonathan@ubuntu-01:/etc/opt/microsoft/scx/ssl$ sudo rm /etc/opt/microsoft/scx//ssl/scx-key.pem jonathan@ubuntu-01:/etc/opt/microsoft/scx/ssl$ sudo dpkg -i /home/jonathan/scx-1.4.0-906.universald.1.x64.deb (Reading database ... 44245 files and directories currently installed.) Preparing to replace scx 1.4.0.906 (using .../scx-1.4.0-906.universald.1.x64.deb) ... * Shutting down Microsoft SCX CIM Server: [fail] invoke-rc.d: initscript scx-cimd, action "stop" failed. Unpacking replacement scx ... Setting up scx (1.4.0.906) ... Checking existence of /lib64/libssl.so.0.9.8k and /lib64/libcrypto.so.0.9.8k ... Checking existence of /lib64/libssl.so.0.9.8 and /lib64/libcrypto.so.0.9.8 ... Found /lib64/libssl.so.0.9.8 and /lib64/libcrypto.so.0.9.8 ... Generating certificate with hostname="ubuntu-01", domainname="scomskills.com" WARNING! Could not read 256 bytes of random data from /dev/random. Will revert to less secure /dev/urandom. See the security guide for how to regenerate certificates at a later time when more random data might be available. * Starting Microsoft SCX CIM Server: [ OK ] Processing triggers for ureadahead ...

Notice that I first removed the scx-key.pem file that was generated by the failed install, and then ran the installer package again – don’t actually know if this was necessary, but I thought it might be best to clean it up. As you can see, the final result is a signing of the certificate and the SCX CIM Server started successfully.

A little more background to the problem (if your interested):

The sequence of events that (I believe) led to this situation was the fact that I initially had the Linux server directly connected to an internet accessible access point, and the Ubuntu box was also configured to receive its network settings via DHCP. For some reason, DHCP added an extra “dot” in the resolv.conf file domain lines, and this apparently was an invalid configuration in the certificate signing process.

Hope this helps someone out there in a similar situation.

Dual-Home Unix/Linux Agent (multi-home)

I ran into this request recently, but couldn’t find any official documentation on this. I’m sure this information is out there somewhere, but figured it’s a 2 minute post so here it is.

Turns out it’s actually quite easy, as long as you have your certificates lined up.

You’ll need to have all SCX certificates from every management server installed on every other management server that participates in a cross-platform resource pool. In other words, a complete certificate mesh between all multi-homed management groups!

Simply login to a management server that is a member of the cross-platform resource pool in the original management group, open a command prompt as administrator, navigate to %Program Files%\System Center\Operations Manager 2012\Server folder, and execute the following command:

scxcertconfig.exe -export xplat.cert

 

 

Next, copy that exported cert file to the same folder on each management server in the cross-platform resource pool in the second management group. Then import it by running the following command:

scxcertconfig.exe -import xplat.cert

 

Now you can run discovery/installation. The news is, no certificate signing or agent installation needs to happen, so it’s a very quick operation going forward.

Install SCOM Agent on Red Hat Enterprise Linux 6 (linux agent installation)

This is a step-by-step article on installing the SCOM agent on a RHEL6 system, both from an SCOM and Linux administrator perspective. Everything in this demonstration assumes a new installation of RHEL6 with no additional configuration. It is also assumed there are no firewalls between the SCOM management servers and the RHEL system. Windows Firewall is not turned on, and the RHEL system is running with default IPTABLES configuration.

Disclaimer: I do not claim to be an experienced Linux admin. This is intended to familiarize SCOM administrators with the entire process. There may be better practices or methods for configuring Linux – comments are  welcome.

If you do not have an SCOM environment or a RHEL system in your lab, you can download evaluation versions from the links below (these links may be outdated at the time you are reading this).

SCOM 2012 SP1 Evaluation

RHEL 6 Evaluation (download the rhel-server-6.3-x86_64-dvd ISO)

Installing RHEL6 on Hyper-V step-by-step

Here are the tasks that will be addressed in this walk-through:

   1: Create a Linux user account for the OpsMgr service

   2: Sudoers file configuration

   3: Configure iptables (Linux firewall)

   4: Verify DNS – forward and reverse lookup

   5: Create a cross-platform monitoring resource pool

   6: Install certificates on resource pool members

   7: Install required management packs

   8: Runas accounts and profiles

   9: Discovery wizard

 

Configuring Linux

Before attempting to discover and install the SCOM agent, an account needs to be created on the Linux system and the sudoers file must be configured to allow the account to execute certain commands. Most out-of-box monitoring activities are performed using low privileges on the Linux system, but some activities do require elevation using sudo.

In this example, SCOM will leverage a single Linux account to perform all activities: opsmgrsvc.

 

Create the opsmgrsvc account

There are several free tools you can use to establish a SSH session to a Linux system. The most common is Putty, which can be downloaded at http://www.putty.org/.

Launch Putty and connect to the Linux host by entering the host name or IP address of  the RHEL system and clicking Open.

image

 

Login with an account that has privileges to add user accounts and modify the sudoers file, or login with root. Create the opsmgrsvc account using the following command:

useradd opsmgrsvc

Next, set the password for the account using the following command:

passwd opsmgrsvc

You will be prompted to enter the password after issuing the command.

Configure sudoers file

Next you will configure the sudoers file, giving the opsmgrsvc account the ability to elevate and execute privileged commands without requiring a password (this is a SCOM requirement).

While still logged into the Linux SSH session as root, open the sudoers file editor by typing visudo and then:

1. Move down the file until you find the line Defaults requiretty, and enter Defaults:opsmgrsvc !requiretty just after it. This setting allows the service account to login without requiring a TTY session.

image

 

2. Move down the file until you find the line root ALL=(ALL) ALL, and enter opsmgrsvc ALL=(root) NOPASSWD: ALL just after it. This setting allows the service account to elevate without requiring a password.

image

 

3. Hit the ESC key to finish editing. To save and close the sudoers file, enter :wq. If you made a mistake in editing the file, you can close without saving your work by entering :q!.

 

NOTE!!! Your Linux administrator will likely want more granular control over which commands the SCOM service account can execute. Below are example sudoers file entries which would enable the opsmgrsvc account to only execute the command necessary for agent installation, maintenance tasks, and baseline monitoring.

More information about sudoers file requirements and examples for other cross-platform systems can be found at http://social.technet.microsoft.com/wiki/contents/articles/7375.configuring-sudo-elevation-for-unix-and-linux-monitoring-with-system-center-2012-operations-manager.aspx.

NOTE: Copy/Paste the following – there is no word wrap, so some of it may float off the screen!

#----------------------------------------------------------------------------------- 
#User configuration for Operations Manager agent: opsmgrsvc

#General requirements for OpsMgr service account
Defaults:opsmgrsvc !requiretty

#Agent discovery, installation, certificate signing, and maintenance tasks
opsmgrsvc ALL=(root) NOPASSWD: /bin/sh -c cp /tmp/scx-opsmgrsvc/scx.pem /etc/opt/microsoft/scx/ssl/scx.pem; rm -rf /tmp/scx-opsmgrsvc; /opt/microsoft/scx/bin/tools/scxadmin -restart
opsmgrsvc ALL=(root) NOPASSWD: /bin/sh -c sh /tmp/scx-opsmgrsvc/GetOSVersion.sh; EC=$?; rm -rf /tmp/scx-opsmgrsvc; exit $EC
opsmgrsvc ALL=(root) NOPASSWD: /bin/sh -c cat /etc/opt/microsoft/scx/ssl/scx.pem
opsmgrsvc ALL=(root) NOPASSWD: /bin/sh -c rpm -e scx
opsmgrsvc ALL=(root) NOPASSWD: /bin/sh -c /bin/rpm -F --force /tmp/scx-opsmgrsvc/scx-1.[0-9].[0-9]-[0-9][0-9][0-9].rhel.[0-9].x[6-8][4-6].rpm; EC=$?; cd /tmp; rpm -rf /tmp/scx-opsmgrsvc; exit $EC
opsmgrsvc ALL=(root) NOPASSWD: /bin/sh -c /bin/rpm -U --force /tmp/scx-opsmgrsvc/scx-1.[0-9].[0-9]-[0-9][0-9][0-9].rhel.[0-9].x[6-8][4-6].rpm; EC=$?; cd /tmp; rpm -rf /tmp/scx-opsmgrsvc; exit $EC

#Log file monitoring
opsmgrsvc ALL=(root) NOPASSWD: /opt/microsoft/scx/bin/scxlogfilereader -p

#Custom shell command monitoring example. Replace <shell command> with the correct command string
#opsmgrsvc ALL=(root) NOPASSWD: /bin/bash -c <shell command>

#Daemon diagnostic and restart recovery tasks (example using cron)
#opsmgrsvc ALL=(root) NOPASSWD: /bin/sh -c ps -ef | grep cron | grep -v grep
#opsmgrsvc ALL=(root) NOPASSWD: /usr/sbin/cron &

#Add sudo logging
Defaults logfile=/var/log/sudolog

#End user configuration for Operations Manager agent
#-----------------------------------------------------------------------------------
 
Configure Linux Firewall (iptables)

The firewall is enabled in a default installation of RHEL6. It comes preconfigured to allow SSH connections, ICMP on local subnet, and very basic network service functionality (these protocol ports may need to be manually configured on other distributions). To see the firewall configuration, run the following command in your SSH session:

iptables --list

image

SCOM discovery and installation uses SSH to copy installation files to the target computer and execute shell scripts, such as GetOSVersion.sh, but before this can happen SCOM needs to connect to the SCX CIM provider on the remote host using WinRM. This occurs over port 1270, which is not open in the default iptables file.

To allow WinRM to do it’s job, add the iptables rule with the following command (this is one line):

iptables -I INPUT 1 -p tcp -s 192.168.2.0/24 --dport 1270 -m state --state NEW,ESTABLISHED -j ACCEPT

Save iptables configuration with the following command:

/sbin/service iptables save

Take a look at the ruleset to make sure it saved correctly.

iptables --list

image

Important: You will need to save your iptables configuration and configure the network interface to restore iptables rules at startup. Otherwise, those rules will disappear the next time the computer restarts. Check your distribution for saving/restoring iptables.

 

iptables command explained

-I INPUT 1 : insert new rule at the beginning of the ruleset.

-p tcp : protocol to match.

-s 192.168.2.0/24 : accept requests from specified host (or hosts). Note: this must be changed for your environment or can be removed from the command to accept from anywhere.

--dport : destination port to match on the host.

-m : match specific connection states.

--state NEW, ESTABLISHED : connection states to match.

-j ACCEPT : accept connection and stop processing the ruleset.

 

DNS Configuration

If you host is not resolvable in DNS – forward and reverse – then read this section. Otherwise, you may skip to the next section about creating a resource pool.

Forward lookup zones are required for SCOM in general, and reverse DNS lookup zones are minimally required for a management server to establish communication with UNIX/Linux agents. connections with the management group. Some DNS systems do not automatically create DNS records, so in this example we will walk thru creating these records and zones in Windows DNS Manager.

 

Create forward lookup record

To register the Linux host in DNS, open DNS Manager and create a new A record in the forward lookup zone of your domain. If you have a reverse DNS lookup zone configured in your domain, select Create associated point (PTR) record. This should automatically create the reverse DNS record, but you may want to verify this on your first attempt.

image

 

Verify your configuration with nslookup.

image

 

In this example, forward lookup is working with nslookup rhel6-01, but reverse lookup isn’t working with nslookup 192.168.2.70. This is because either there was no reverse lookup zone configured in the domain, or the Create associated pointer (PTR) record option was not selected. In this case, a reverse DNS lookup zone did not exist.

 

Create reverse lookup record

If you don’t have a reverse lookup zone in your DNS, you’ll need to create one in DNS Manager. Select the defaults in the wizard, entering your network Id – in this case, 192.168.2.x.

image

 

Now a PTR record can be created for the Linux host. One way to do this is to open the A record created in the previous step (in the forward lookup zone) and select Update associated pointer (PTR) record.

image

 

Testing reverse DNS lookup now returns the Linux host name, and now you’re ready to move onto the next step.

image

 

Create a Resource Pool

In a typical production environment, there will be a resource pool dedicated to monitoring cross-platform systems. This resource pool should contain at least two management servers for high-availability. This example adds management servers OM12-MS02 and OM12-MS03 to the new Cross-Platform Monitoring Resource Pool.

Navigate to Administration > Resource Pools in the Operations console to create a new resource pool.

image

 

Add the management servers.

image

 

Install Certificates on Resource Pool Members

Each management server in the cross-platform monitoring resource pool needs to have it’s certificate installed on each other management server. This is task is performed by running the scxcertconfig tool on each management server; first exporting the certificate from each management server, then importing each other management server’s certificate.

Login to each management server in your cross-platform resource pool, open a command prompt as administrator, navigate to %Program Files%\System Center\Operations Manager 2012\Server folder, and execute the following command:

scxcertconfig.exe -export <hostname.cert>

In this example, <hostname.cert>is replaced with OM12-MS02 on the first management server.

After exporting the certificate on each management server, import the certificate(s) from each other management server in the same manner. in this example, if I’m logged into OM12-MS02, I would execute the following command:

scxcertconfig.exe -import OM12-MS03.cert

More information about this procedure can be read on Technet at http://technet.microsoft.com/en-us/library/hh287152.aspx.

 

Install Management Packs

By default, there are several Unix/Linux management packs installed in SCOM.

image

 

However, this is not enough to perform agent installation using the Discovery Wizard. You will need to import the version specific management packs before even attempting to install a cross-platform agent, because they contain definitions specifically designed to detect which version of the the installation package to install on the target host.

In this example, we minimally need to install the Linux and Red Hat Libraries, and the RHEL6 bundle. These management packs can be found on your installation media, in the Management Packs folder. You can also download most recent versions from the online management pack catalog.

Navigate to Administration > Management Packs in the Operations console to install the required management packs: Microsoft.Linux.RHEL.6.mpb, Microsoft.Linux.Library.mp, and Microsoft.Linux.RedHat.Library.mp.

image

 

At this point, all the prerequisites are satisfied for installing the agent on a Red Hat Enterprise Linux 6 operating system by using the Discovery Wizard. Unless you don’t want to actually monitor the Linux system, you can skip to the Discovery Wizard section.

 

Configuring Runas Accounts and Profiles

One last thing we need to do in order for object discovery and monitoring workflows can run on our Linux system. This isn’t necessarily required to install cross-platform agents, but obviously there wouldn’t be much value in it if this task isn’t completed.

Navigate to Administration > Run As Configuration > UNIX/Linux Accounts in the Operations console and create a new account.

image

 

Create monitoring account

1. On the Account Type screen, select Monitoring account.

2. On the General Properties screen, enter a display name.

image

 

3. Enter your Linux monitoring account credentials on the Account Credentials screen – do not use elevation with this account. If you use elevation with this account, monitoring workflows implementing Linux commands that do not support sudo elevation will fail.

image

 

4. Select More secure on the Distribution Security screen. Then click Create and Close.

 

Create elevated monitoring account

Create another account that will be used for elevated monitoring tasks that require sudo.

1. On the Account Type screen, select Monitoring account.

2. On the General Properties screen, enter a display name.

image

 

3. Enter your Linux monitoring account credentials on the Account Credentials screen – use elevation with this account.

image

 

4. Select More secure on the Distribution Security screen. Then click Create and Close.

 

Associate accounts with profiles

Now the accounts need to be associated with the appropriate Runas profiles.

1. Navigate to Administration > Run As Configuration > Profiles and open the properties of the UNIX/Linux Action Account.

2. Click Add on the Run As Accounts screen, and select the regular monitoring account (not the privileged account). Keep the default – All targeted objects – and click Ok.

3. After clicking Save, the wizard gives an opportunity to securely distribute the account. Click on your account to step into the distribution wizard.

image

 

4. Clicking Add on the Distribution tab launches the Computer Search screen. Select Search by resource pool name option, then click Search. Select your cross-platform resource pool in the Available items box. Click Add, then Ok.

image

 

5. Click Ok, then Close the Run As Profile Wizard.

 

Perform the same steps for the UNIX/Linux Privileged Account profile, using the privileged account you setup earlier.

 

Discovery Wizard

Navigate to Administration > Device Management in the Operations console to start the discovery wizard, and UNIX/Linux computers.

image

 

On the Discovery Criteria screen, Add the Linux host information and click Set credentials. Keep Discovery type set to All computers.

image

 

On the Credentials Settings screen, select User name and password and enter the account information that was created previously. Select This account does not have privileged access on the drop down list. Then click Ok.

image

 

The Credentials section on the Discovery Criteria screen should look like this.

image

 

Click Save, select the Cross-Platform Monitoring Resource Pool, and click Discover.

image

 

The host should be discovered and you can continue by ticking the checkbox and clicking Manage. If you scroll to the right in this window, you can also see that operating system, version, and architecture information was discovered.

image

 

After clicking manage, the wizard goes through an installation, validation, and certificate signing process. Given that all goes well, a typical installation should take 30-60 seconds for a single host. Within minutes you should see the new Linux agent appear in the monitor space, with objects being discovered and monitored.

image

 

 

Here are some useful resources to check in run into problems.

Unable to add the domain to the subject

Troubleshooting UNIX/Linux Agent Discovery in System Center 2012 – Operations Manager

Scenario #1 – “There were no computers that met the specified discovery criteria”

Scenario #2 – “SSH discovery failed.” with “unspecified problem”

Scenario #3 – “Did not find a matching supported agent”

Scenario #4 – “New computer shows as Platform: Unknown and Version: Unknown”

Sudoers Manual