Before beginning a Cloudera Manager upgrade, do the following:

1. Review the CDH 5 and Cloudera Manager 5 Requirements and Supported Versions for the new versions you are upgrading to.
2. Read the Cloudera Manager 5 Release Notes.
3. Read the Cloudera Security Bulletins.
4. Note the following:
   • Cloudera Management Service TLS/SSL configuration

     If you have enabled TLS security for the Cloudera Manager Admin Console, as of Cloudera Manager 5.1, Cloudera Management Service roles communicate with Cloudera Manager using TLS, and fail to start until TLS/SSL properties have been configured.

   • Navigator

     If you have enabled auditing with Cloudera Navigator, during the upgrade to Cloudera Manager 5, auditing is suspended and is only restarted when you restart the roles of audited services. You will be instructed to stop some services in a later step.

5. If you have previously installed Kafka 1.2, and are upgrading from Cloudera Manager 5.4 or lower, remove the Kafka CSD:
   1. Determine the location of the CSD directory:
      1. Select Administration > Settings.
      2. Click the Custom Service Descriptors category.
      3. Retrieve the directory from the Local Descriptor Repository Path property.
   2. Delete the Kafka CSD from the directory.
6. Review package (RPM) dependencies. A Cloudera Manager upgrade might introduce new package dependencies. If your organization has restrictions or requires prior approval for installation of packages, see the list of Package Dependencies before upgrading Cloudera Manager.

If your Cloudera Manager hosts use an unsupported version of the JDK, you must upgrade the hosts to a supported version of the JDK before upgrading Cloudera Manager. If you plan to upgrade CDH, you must also upgrade the JDK on all cluster hosts.

If you have enabled TLS/SSL, you must reinstall CA certificates to your truststores after upgrading the JDK. See Recommended Keystore and Truststore Configuration.

If the Cloudera Manager host does not have access to the internet, or you install a version lower than the latest version of Cloudera Manager, configure access to the Cloudera Manager software from either the Cloudera public repository or a local package repository that you create.

See Creating and Using a Package Repository for Cloudera Manager.

Cloudera Manager upgrades Cloudera Navigator as part of the Cloudera Manager upgrade process. If you are upgrading from Cloudera Navigator 2.6 or lower, follow the steps in this section to prepare the Cloudera Navigator data management component for upgrade and then continue with Step 7: Upgrade the Cloudera Manager Server. If you are upgrading from Cloudera Navigator 2.7 or higher, skip this section and continue with Step 7: Upgrade the Cloudera Manager Server.

1. If your cluster is running the embedded PostgreSQL database, stop all services that are using the embedded database. These can include:
   • Hive service and all services such as Impala and Hue that use the Hive metastore
   • Oozie
   • Sentry
2. If your cluster is running the Cloudera Navigator data management component and the following services are enabled for auditing, stop the following roles. (You also can elect to skip this step and leave these services running, but some audits by Cloudera Navigator may not occur during the Cloudera Manager upgrade process.)
   • HDFS - NameNode
   • HBase - Master and RegionServers
   • Hive - HiveServer2
   • Hue - Beeswax Server
     Note: Stopping these roles renders any service depending on these roles unavailable. For the HDFS - NameNode role, most of the services in the cluster are unavailable until the upgrade is finished.
   To determine which services are enabled for auditing:
   1. Go to the Home page in Cloudera Manager.
   2. Click Configuration > Navigator Settings.
   3. Type "Enable Audit" in the search box. The Enable Audit Collection property displays the services for which Cloudera Navigator auditing is enabled.
3. Stop the Cloudera Management Service:
   1. Select Clusters > Cloudera Management Service.
   2. Select Actions > Stop.
4. Stop Cloudera Manager Server, Database, and Agent:
   1. Use the Cloudera Manager Admin Console to stop any running commands. These include user commands and commands Cloudera Manager automatically triggers in response to a state change or a schedule. You can either wait for commands to complete, or stop any running commands. For more information on viewing and stopping running commands, see Viewing Running and Recent Commands.
        Important: If you do not stop all commands, the Cloudera Manager Server fails to start after upgrade.
   2. On the host running the Cloudera Manager Server, stop the Cloudera Manager Server:

      sudo service cloudera-scm-server stop

   3. If you are using the embedded PostgreSQL database with Cloudera Manager, stop the database on the host where the database runs, usually the Cloudera Manager Server host:

      sudo service cloudera-scm-server-db stop

        Important:

      If you are not running the embedded database service and you attempt to stop it, you receive a message indicating that the service cannot be found. If instead you get a message that the shutdown failed, the embedded database is still running, probably because services are connected to the Hive metastore. If the database shutdown fails due to connected services, issue the following command:

      • RHEL-compatible 7 and higher:

        sudo service cloudera-scm-server-db next_stop_fast
        sudo service cloudera-scm-server-db stop

      • All other Linux distributions:

        sudo service cloudera-scm-server-db fast_stop

   4. If the Cloudera Manager host is also running the Cloudera Manager Agent, stop the Cloudera Manager Agent:

      sudo service cloudera-scm-agent stop

5. Back up the following directories on the Cloudera Manager server host:
   • /etc/cloudera-scm-server
   • /etc/cloudera-scm-agent
6. Establish access to the Cloudera Manager Server packages. You can either upgrade from the Cloudera repository at https://archive.cloudera.com/cm5/, or you can create your own package repository, as described in Creating and Using a Package Repository for Cloudera Manager. You must create your own repository if Cloudera Manager does not have Internet access or you want to upgrade to a version of Cloudera Manager lower than the latest version.

   To upgrade using the Cloudera repository:

   1. Back up the current Cloudera Manager repo file, located in one of the following directories:

      RHEL

      /etc/yum.repos.d/

      SLES

      /etc/zypp/repos.d/

      Ubuntu or Debian

      /etc/apt/sources.list.d/

   2. Download the Cloudera .repo file for your distribution by starting at https://archive.cloudera.com/cm5/ and navigating to the directory that matches your operating system.

      • For Red Hat or CentOS 6, go to the appropriate release directory, for example, https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/. In that directory, find the repo file that contains information including the repository base URL and GPG key. The contents of the cloudera-manager.repo are similar to the following:

        [cloudera-manager]
        # Packages for Cloudera Manager, Version 5, on RHEL or CentOS 6 x86_64
        name=Cloudera Manager
        baseurl=https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5/
        gpgkey = https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/RPM-GPG-KEY-cloudera
        gpgcheck = 1

      • For Ubuntu or Debian systems, go to the appropriate release directory, for example, https://archive.cloudera.com/cm4/debian/wheezy/amd64/cm. The repo file, in this case, cloudera.list, is similar to the following:

        # Packages for Cloudera Manager, Version 5, on Debian 7.0 x86_64
        deb https://archive.cloudera.com/cm5/debian/wheezy/amd64/cm wheezy-cm5 contrib
        deb-src https://archive.cloudera.com/cm5/debian/wheezy/amd64/cm wheezy-cm5 contrib

      For example, you can use the following command to download the .repo file for Cloudera Manager version 5 and RHEL version 6:

      $ wget https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/cloudera-manager.repo

   3. Do one of the following on the Cloudera Manager Server host:
      • Upgrade to the most recent version of Cloudera Manager:
        Copy the cloudera-manager.repo file to the configuration location for the package management software for your system:

        RHEL

        Copy cloudera-manager.repo to /etc/yum.repos.d/

        SLES

        Copy cloudera-manager.repo to /etc/zypp/repos.d/

        Ubuntu or Debian

        Copy cloudera.list to /etc/apt/sources.list.d/

      • Upgrade to an specific version of Cloudera Manager:

        RHEL-compatible or SLES

        1. Edit the cloudera-manager.repo file to change the baseurl to point to the version of Cloudera Manager you want to download. For example, to install Cloudera Manager version 5.0.1, change:

           baseurl=https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5/

           to:

           baseurl=https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.0.1/.

        2. Save the edited file:
           • For RHEL or CentOS, save it in /etc/yum.repos.d/.
           • For SLES, save it in /etc/zypp/repos.d.

        Ubuntu or Debian

        1. Download the Cloudera Manager list file (cloudera.list) using the links provided at Cloudera Manager Version and Download Information. For example, for Ubuntu 10.04 (lucid), this file is located at

           https://archive.cloudera.com/cm5/ubuntu/lucid/amd64/cm/cloudera.list.

        2. Edit the cloudera.list file to change the second-to-last element to specify the version of Cloudera Manager you want to install. For example, with Ubuntu lucid, if you want to install Cloudera Manager version 5.0.1, change:

           deb https://archive.cloudera.com/cm5/ubuntu/lucid/amd64/cm lucid-cm5 contrib

           to:

           deb https://archive.cloudera.com/cm5/ubuntu/lucid/amd64/cm lucid-cm5.0.1 contrib.

        3. Save the edited file in the /etc/apt/sources.list.d/ directory.

   4. Run the following command to clean the cache directories and upgrade the software:

      RHEL

      sudo yum clean all
      sudo yum upgrade cloudera-manager-server cloudera-manager-daemons cloudera-manager-agent

        Note:

      • yum clean all cleans yum cache directories, ensuring that you download and install the latest versions of the packages.
      • If your system is not up to date, any underlying system components must be upgraded before yum update can run. yum indicates which components must be upgraded.
      • If the Cloudera Manager instance you are upgrading uses the embedded PostgreSQL database, add cloudera-manager-server-db-2 to the list of packages in the yum upgrade command. The embedded PostgreSQL database should not be used in production environments.

      SLES

      sudo zypper clean --all
      sudo zypper up -r https://archive.cloudera.com/cm5/sles/11/x86_64/cm/5/

      To upgrade from your own repository:

      sudo zypper clean --all
      sudo zypper rr cm
      sudo zypper ar -t rpm-md http://myhost.example.com/path_to_cm_repo/cm
      sudo zypper up -r http://myhost.example.com/path_to_cm_repo

      Ubuntu or Debian

      The following commands clean cached repository information and update Cloudera Manager components:

      sudo apt-get clean
      sudo apt-get update
      sudo apt-get dist-upgrade
      sudo apt-get install cloudera-manager-server cloudera-manager-daemons cloudera-manager-agent

      If the Cloudera Manager instance you are upgrading uses the embedded PostgreSQL database, add cloudera-manager-server-db-2 to the list of packages in the apt-get install command. The embedded PostgreSQL database should not be used in production environments.

      During this process, you might be prompted about your configuration file version:

      Configuration file `/etc/cloudera-scm-agent/config.ini'
      ==> Modified (by you or by a script) since installation.
      ==> Package distributor has shipped an updated version.
      What would you like to do about it ? Your options are:
      Y or I : install the package maintainer's version
      N or O : keep your currently-installed version
      D : show the differences between the versions
      Z : start a shell to examine the situation
      The default action is to keep your current version.

      You will receive a similar prompt for /etc/cloudera-scm-server/db.properties. Answer N to both prompts.

7. If you customized the /etc/cloudera-scm-agent/config.ini file, your customized file is renamed with the extension .rpmsave or .dpkg-old. Merge any customizations into the /etc/cloudera-scm-agent/config.ini file that is installed by the package manager.
8. On the Cloudera Manager Server host, verify that you now have the following packages, corresponding to the version of Cloudera Manager you installed, by running the following command:

   RPM-based distributions

   $ rpm -qa 'cloudera-manager-*'
   cloudera-manager-server-5.14.0-0.cm5140.p0.s2.el6.x86_64
   cloudera-manager-agent-5.14.0-0.cm5140.p0.s2.el6.x86_64
   cloudera-manager-daemons-5.14.0-0.cm5140.p0.s2.el6.x86_64

   Ubuntu or Debian

   ~# dpkg-query -l 'cloudera-manager-*'
   Desired=Unknown/Install/Remove/Purge/Hold
   | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
   |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
   ||/ Name                   Version                Description
   +++-======================-======================-============================================================
   ii  cloudera-manager-agent 5.14.0-0.cm5140.p0.s2~sq The Cloudera Manager Agent
   ii  cloudera-manager-daemo 5.14.0-0.cm5140.p0.s2~sq Provides daemons for monitoring Hadoop and related tools.
   ii  cloudera-manager-serve 5.14.0-0.cm5140.p0.s2~sq The Cloudera Manager Server

     Note: You might also see an entry for the cloudera-manager-server-db-2 if you are using the embedded PostgreSQL database, and additional packages for plug-ins, depending on what was previously installed on the server host. If the cloudera-manager-server-db-2 package is installed, and you do not plan to use the embedded database, you can remove this package.
9. Start Cloudera Manager Server. On the Cloudera Manager Server host (the host on which you installed the cloudera-manager-server package), do the following:
   1. If you are using the embedded PostgreSQL database for Cloudera Manager, start the database. If your installation uses other databases, Cloudera Manager reconnects with them after start up.

      sudo service cloudera-scm-server-db start

   2. Start the Cloudera Manager Server:

      sudo service cloudera-scm-server start

      You should see the following:

      Starting cloudera-scm-server:                              [  OK  ]

10. Log in to the Cloudera Manager Admin Console. It can take several minutes for Cloudera Manager Server to start, and the console is unavailable until the server startup is complete.

    The Upgrade Wizard displays.

11. Upgrade the Cloudera Manager Agent using Cloudera Manager or by manually upgrading the packages:

    Cloudera Manager upgrades Agent software

    When Cloudera Manager upgrades the Cloudera Manager agent, Cloudera Manager handles the upgrade and cleanup, and optionally upgrades the JDK.

    1. Select Yes, I would like to upgrade the Cloudera Manager Agent packages now and click Continue.
    2. Select the release of the Cloudera Manager Agent to install. Normally, this is the Matched Release for this Cloudera Manager Server. However, if you used a custom repository (instead of archive.cloudera.com) for the Cloudera Manager server, select Custom Repository and provide the required information. The custom repository location must contain the matched Agent version. See Creating and Using a Package Repository for Cloudera Manager. Custom repositories are required when your cluster cannot access the Internet, or when you need to upgrade to a version other that the most current version of Cloudera Manager.
    3. Click Continue. The JDK Installation Options page displays.
       • If you want Cloudera Manager to install JDK 1.7 on all cluster hosts, select Install Oracle Java SE Development Kit (JDK) .
       • If local laws permit you to deploy unlimited strength encryption, and you are running a secure cluster, select the Install Java Unlimited Strength Encryption Policy Files checkbox.
    4. Click Continue.
    5. Specify credentials and initiate Agent installation:
       • Select root or enter the username for an account that has password-less sudo permission.
       • Select an authentication method:
         • If you choose password authentication, enter and confirm the password.
         • If you choose public-key authentication, provide a passphrase and path to the required key files.
       • You can specify an alternate SSH port. The default value is 22.
       • You can specify the maximum number of host installations to run at once. The default value is 10.
    6. Click Continue.

       The Cloudera Manager Agent packages and, if selected, the JDK are installed.

    7. Click Continue.

       The Host Inspector runs to inspect your managed hosts for correct versions and configurations. If problems occur, you can make changes and then rerun the inspector.

       When you are satisfied with the inspection results, click Continue.

    Manually upgrade Agent software

    To manually upgrade the Cloudera Manager Agent software, you use package commands to clean up old versions, download the new version, and upgrade the software.

    To manually upgrade the Cloudera Manager agent:

    1. On all cluster hosts except the Cloudera Manager Server host, stop the Agent:

       sudo service cloudera-scm-agent stop

    2. Select No, I would like to skip the agent upgrade now and click Continue.
    3. Copy the repo file as described in step cm_ag_ug_cm5.html#concept_gs4_bsg_xw_unique_1__li_o12_rfs_2n_unique_1.
    4. Run the following commands on all hosts except the Cloudera Manager Server host:

       RHEL

       sudo yum clean all
       sudo yum upgrade cloudera-manager-daemons cloudera-manager-agent

         Note:

       • yum clean all cleans yum cache directories, ensuring that you download and install the latest versions of the packages.
       • If your system is not up to date, any underlying system components must be upgraded before yum update can run. yum indicates which components must be upgraded.
       • If the Cloudera Manager instance you are upgrading uses the embedded PostgreSQL database, add cloudera-manager-server-db-2 to the list of packages in the yum upgrade command. Do not use the embedded PostgreSQL database in production environments.

       SLES

       sudo zypper clean --all
       sudo zypper up -r https://archive.cloudera.com/cm5/sles/11/x86_64/cm/5/

       To upgrade from your own repository:

       sudo zypper clean --all
       sudo zypper rr cm
       sudo zypper ar -t rpm-md http://myhost.example.com/path_to_cm_repo/cm
       sudo zypper up -r http://myhost.example.com/path_to_cm_repo

       Ubuntu or Debian

       Use the following commands to clean cached repository information and update Cloudera Manager components:

       sudo apt-get clean
       sudo apt-get update
       sudo apt-get dist-upgrade
       sudo apt-get install cloudera-manager-agent cloudera-manager-daemons

       If the Cloudera Manager instance you are upgrading uses the embedded PostgreSQL database, add cloudera-manager-server-db-2 to the list of packages in the apt-get install command. Do not use the embedded PostgreSQL database in production environments.

       During this process, you might be prompted about your configuration file version:

       Configuration file '/etc/cloudera-scm-agent/config.ini'
       ==> Modified (by you or by a script) since installation.
       ==> Package distributor has shipped an updated version.
       What would you like to do about it ? Your options are:
       Y or I : install the package maintainer's version
       N or O : keep your currently-installed version
       D : show the differences between the versions
       Z : start a shell to examine the situation
       The default action is to keep your current version.

       You will receive a similar prompt for /etc/cloudera-scm-server/db.properties. Answer N to both prompts.

    5. If you customized the /etc/cloudera-scm-agent/config.ini file, your customized file is renamed with the extension .rpmsave or .dpkg-old. Merge any customizations into the /etc/cloudera-scm-agent/config.ini file that is installed by the package manager.
    6. On all cluster hosts, start the Agent:

       sudo service cloudera-scm-agent start

    7. Click Continue. The Host Inspector inspects your managed hosts for correct versions and configurations. If problems occur, you can make changes and then rerun the inspector. When you are satisfied with the inspection results, click Continue.

12. Click Finish.
13. If you are upgrading from Cloudera Manager 5.0 and are using an external database for Cloudera Navigator, the Database Setup page displays. Configure these database settings:
    1. Enter the database host, database type, database name, username, and password for the database.
    2. Click Test Connection to confirm that Cloudera Manager can communicate with the Cloudera Navigator database using the information you supplied. If the test succeeds in all cases, click Continue; otherwise, check and correct the information you have provided for the database and then try the test again. (For some servers, if you are using the embedded database, you will see a message saying the database will be created in a later step.)
14. The Review Changes page displays. Review the configuration changes to be applied and click Continue. The Upgrade wizard displays a dialog box allowing you to choose whether to restart the Cloudera Management Service.
15. Click Continue.

    If you keep the default selection, the Upgrade wizard restarts the Cloudera Management Service.

16. Click Finish.

    The Home page displays.

    All services (except for any services you stopped) should now be running.

17. If, as part of this upgrade, you stopped some selected services and roles, restart the following roles:
    • HDFS - NameNode
    • HBase - Master and RegionServers
    • Hive - HiveServer2
    • Hue - Beeswax Server
18. If you upgraded the JDK, do the following:
    1. If the Cloudera Manager Server host is also running a Cloudera Manager Agent, restart the Cloudera Manager Server:

       sudo service cloudera-scm-server restart

       If the Cloudera Manager Server does not start, see Troubleshooting Installation and Upgrade Problems.

    2. Restart all services:
       1. On the Home > Status tab, click next to the cluster name and select Restart.
       2. In the confirmation dialog box that displays, click Restart.
19. If Cloudera Manager reports stale configurations after the upgrade, you may need to restart the cluster services and redeploy the client configurations. If you are also upgrading CDH, this step is not required. Stale configurations can occur after a Cloudera Manager upgrade when a default configuration value has changed, often required to fix a serious problem. Configuration changes that result in Cloudera Manager reporting stale configurations are described the Cloudera Manager release notes.
    1. On the Home > Status tab, click next to the cluster name and select Restart.
    2. In the confirmation dialog box, click Restart.
    3. On the Home > Status tab, click next to the cluster name and select Deploy Client Configuration.
    4. In the confirmation dialog box, click Deploy Client Configuration.
20. If upgrading from Navigator 2.6 or lower:
    1. Start and log into the Cloudera Navigator data management component UI. The Upgrading Navigator page displays. Depending on the amount of data in the Navigator Metadata Server storage directory, the upgrade process can take three to four hours or longer.
21. When the upgrade is complete, click Continue. The Cloudera Navigator landing page is displayed.
22. If you are upgrading from Cloudera Manager 5.5.0 or lower to Cloudera Manager 5.5.0 or higher, hard restart the agent on all hosts to update and restart the supervisord process:

    RHEL 7 and higher:

    sudo service cloudera-scm-agent next_stop_hard
    sudo service cloudera-scm-agent restart

    Other Linux Distributions:

    sudo service cloudera-scm-agent hard_restart

You can upgrade other Cloudera Navigator components at any time. You do not have to perform these upgrades when upgrading Cloudera Manager or CDH.

See Upgrading Cloudera Navigator Components.

To upgrade CDH, see Upgrading CDH and Managed Services Using Cloudera Manager.