Description

This article describes how to scan and/or repair based on a custom group that has been created in Patch and Compliance Manager

Resolution

In Patch and Compliance Manager, there is the option to create a custom group.

To create a new group

1. Open the Security and Compliance tool group on the Core Server or Remote Console
2. Go to the Patch and Compliance tool.
3. Expand the "Groups" node in the left-hand pane.
4. Right-click on either "My Groups" or "Public Groups" and select "New".
   

Once you have created a new group, you can drag and drop definitions to the group as you desire. Then, in the Distribution and Patch Settings, you can set the job to scan for GROUP as opposed to TYPES, and point it to your custom group.  Once you have scanned against this, you can then REPAIR the entire custom group at once by right-clicking the group in Patch and Compliance Manager and selecting repair.

To create a repair task open Security and Compliance Manager and follow the steps below

1. Right-click the custom group desired
2. Choose "Repair".
3. Set up the desired options (i.e. scheduled task, stage and repair, policy, etc...)
4. Click "OK" to create the task
5. Now drag the machines or query desired into the repair task created (note; the number of all devices will not reflect the results of the query until the task is started.)
6. You can also right-click the scheduled task and choose properties to adjust the behavior and scheduling of the task.

Subject

Error: 'Core could not find a file' when running vulscan on clients.

Description When running a vulscan on the client side, you see an error message "Warning: Core could not find a file" relating to one of the following files:

• Defaults.txt
• agentbehavior_N.xml (where N is a number)
• av_behavior_N.xml
• cv_behavior_N.xml

Cause

1. The file being looked for has become corrupt.
2. The file being looked for is not in the Agentbehaviors directory.

Resolution

Check the vulscan.log file to see which file is missing. Then take the following steps:

**Defaults.txt**

This file should be located in the Agentbehaviors directory on the core. If it is not, you may find a backup under the Landesk\ManagementSuite\LDLogon\Spyware folder. If you do, COPY this file to the LDLOGON\AgentBehaviors directory. If the file does not exist anywhere, you will need to email the customer a copy from your core server or get one from another core in their environment.

**AgentBehavior**

This file is used for determining the behavior of the client when scanning and repairing. Check the AgentBehaviors folder to see if a filename exists that is matching.

-If the file does exist, try running 'vulscan /reset' on the client and run a new scan to see is this fixes the local file.

-If the file does NOT exist, try recreating the behavior file. To do this, go into Agent Configuration, select 'Security and Patch Scan' and at the bottom of the right hand pane, look to see what "Scan and Repair Settings" name is specified. Then click 'Configure'. From the 'Configure Scan and Repair Settings' window that comes up, select the behavior that was listed in the Agent Configuration and click 'edit'. Once in the behavior, change ANY value and click 'Ok' , and then 'Use Selected'. This will take you back to the Agent Configuration. Click 'Save' to resave the agent. If this works, we should now have an Agent Behavior that matches the one we are looking for. Run the vulscan again and see if it works.

-The new agentbehavior still fails, create a NEW set of scan and repair settings and push a Scheduled Update or a Settings Change task.

**AV or CV behavior**

These files should not keep the LANDesk agent from running a vulscan. If these files cause the error, simply create a new AV Behavior and a new Custom Variable Behavior, and push these out via a Change Settings Task. This is done via the Schedule a Task--> Change Settings button in Security and Patch Manager.

*Note on AV Behavior: If you no longer have LANDesk Antivirus installed and the avbehavior_X.xml is missing and the client is looking for the avbehavior vulscan will return an error in the log and the GUI but will complete sucessfully. To fix this error, go into the database, open the AgentBehavior table and delete the row that has the avbehavior listed. Then resave your agent this will recreate the default behavior in the directory and then patch manager will complete with no errors.

NOTE: In some cases, the entire AgentBehaviors folder may be gone. If this is the case, recreate the folder and try the above steps for Agent Behavior replacement. If that does not recreate the files, you will need to create new Scan and Repair Settings and push them via a Change Settings task.

Overview

When selecting a device and select "Security and Patch Information", one of the expected tabs "All Detected" is not displaying any information of detections and scans.

Problem

When looking for the information in "Security and Patch information" the results return back blank or have filtered results for a specific type of vulnerability outside of what is expected. When running a Vulscan it is shown to have detected vulnerabilities but is not showing in the Security and Patch information

Expected	Found

Cause

This is caused by the "Type" selection at the top of the window being set to a different vulnerability typing then what is expected.

Solution

Simply change the drop down from the current selection to the type of vulnerability of your choice.

This option is defaulted to what is chosen in the patch and compliance window "Type" filter. Whichever option is selected in this list is what the type will be when Security and Patch opened.

Overview

Ivanti Endpoint Manager’s Patch and Compliance tool now welcomes our Next Generation patch engine. This new architecture enables us to continue optimizing well into the future and is only applicable to the Windows environment. As a preliminary feature, we’re providing the ability to opt-in, allowing for a more controlled introduction of all Next Generation content into your environment. The new patch engine is currently available in the 2017.3 product version.

^Updated We are now offering this feature in ALL supported versions of the product.

Definition Downloads

In the definition download utility, a new definition type exists under Windows | Vulnerabilities | Next Gen Microsoft Windows Vulnerabilities (beta).

This option is not on by default and when selected, all associated Next Gen binaries/vulnerabilities definitions will be downloaded to the core. The binaries (about 30 MB) will be contained in Managementsuite \ Ldlogon \ Timber directory and the definition grouping will be based on your configuration and download filters. Upon definition download, the following can be expected:

Definition Download	Managementsuite \ Ldlogon \ Timber

The Managementsuite \ Ldlogon \ Timber  \ Content folder will contain a WindowsPatchData.zip file and associated Delta zip files. The WindowsPatchData.zip file contains all vulnerability detection rulesand the Delta zip files contain the differences. This content, along with the remaining Next Gen binaries, will be downloaded to the endpoint upon scanning against Next Gen content. The main WindowsPatchData.zip file will only be downloaded once, Deltas are downloaded to the Core if there are differences that aren't in the WindowsPatchData.zip file. Once the endpoint has the main zip file, it will only retrieve the Delta zip files when scanning against Next Gen content.

Upon definition download completion of Next Gen Microsoft Windows Vulnerabilities (beta), filtering for this definition type can be done by using the filter string "Next Gen". Every next-gen definition has the filter string hardcoded in the Summary column.

To isolate these definitions, a custom patch group can be created to house these definitions. If you elect to do so, a manual transfer has to take place. To further isolate which devices scan against this custom group, an alternate Distribution and Patch agent setting can be configured to scan against this group. More information on how to configure this is outlined in How to Scan and /or Repair against a custom group and  How to use Custom Groups to repair groups of computers.

Content Changes

Every Next Gen definition will contain a pre-defined fixed script for Detection and Remediation. The pre-defined detection script will evaluate Registry, File and Script logic to determine if a device is vulnerable to a definition. The detection details have been included at the beginning of the script content. The Files and Registry Settings section will be blank for all Next Gen content.

Sample Next Gen definition (Detection Logic)	Sample Next Gen definition (Repair Logic)

Distribution and Patch Agent Setting

^Updated The "Enable security scan debug trace log" UI feature is only available in 2017.3 and newer product versions. To enable debug trace logs for versions 9.6 - 2017.1 run the following cmd locally on the endpoint or distribute a script to the desired device:

 

vulscan /enableDpdTrace=true /showui (the showui switch is optional).

 

This will generate additional logging in the Programdata\Landesk\DebugLog folder consisting of the following (2) files:

• PatchManifestSyncSDK.log
• PatchScanSDKDpdTrace.log

To enhance the log level for all Next Gen content definitions, the following addition has been made to the Distribution and Patch agent settings:

This feature is only intended for troubleshooting purposes and should not be on in your default agent setting. When troubleshooting a Next Gen content issue, please create an alternate Distribution and Patch agent setting, enable this feature and assign this setting to the device during troubleshooting only.

Diagnostic Tool

^Updated The "Get debug logs and zip (patch)" feature is only available in 2017.3 and newer product versions.

To retrieve logging remotely access the Diagnostic tool and select the Logs | Client option to view client-side logs. An additional option "Get debug logs and zip (patch)" is present for debug logging for all Next Gen definitions. This will only function if the Distribution and Patch agent setting has Enable security scan debug trace log selected.

How Does Scanning and Remediation Work

If the endpoints are on a supported version of the product, the agent does not need to be updated immediately to take advantage of the enhanced patch engine. All devices on an unsupported product version will need to be upgraded. Upon initiation of the vulnerability scanner, the self-update feature will update the necessary vulscan files to ensure compatibility between the files on the client and the latest files on your 2017.3 core. For more on the Self Update feature please reference About Patch Manager Self Update. These binaries must be updated in order for the Next Gen binaries to work with vulscan.exe.

Scanning:

A security scan works the same as before for all current content. Whenever the scanner encounters a definition with Next Gen content it will launch the fixed script contained within the definition and perform the following actions:

 

1. Check for definition scan results in memory.
2. If this is the first Next Gen definition encountered in the current security scan, no scan results will be found on the client and the following will occur:
   1. The client will check if it needs to download any Next Gen binary files from the core (ldlogon/timber) and transfer them to the LDCLient\Timber directory:
      1. The detection rules “WindowsPatchData.zip” file (about 14MB) is updated on the content servers every time new content is added and will be download to the client. If WindowsPatchData.zip already exists on the client, the smaller delta files will be used to update this file to the current version.
      2. Additional Next Gen binary files will be downloaded if the current versions do not already exist on the client:
   2. The Next Gen COM object is then registered on the client to allow Vulscan.exe to interact with the Next Gen scan engine
   3. The Next Gen scan engine then scans for ALL vulnerabilities and stores the scan results “in-memory”. A FULL scan of all vul_defs is only completed if this is the first Next Gen definition encountered in the current security scan.
   4. The script then references the in-memory scan results to determine if the current definition is vulnerable.

 

The security scan continues scanning as usual.  For any remaining Next Gen content definitions in the current security scan, the detection script will return the result of the specified definition from the in-memory scan results.

The Next Gen scanner runs (maximum of once per vulscan instance) it checks for everything and stores that information in memory, but that information is only used by Next Gen definitions. Legacy definitions work the same way they always have.

 

Remediation:

 

1. Patch files (Default location - Ldlogon/Patch) uses the existing download mechanism "lddwnld.dll" to transfer the patch files to the sdmcache directory on the client.
2. The pre-defined remediation script calls the Next Gen SDK with a GUID, Language and a unique file name that’s used for patching.

• A temporary “package” is created during the repair and contained on the client (Programdata\Landesk\Timber\Pkgs) which is used by the Next Gen patch SDK.

 

Logging

The vulscan.log file will continue to serve as the primary log for content detection and remediation, however, several additional logs have been introduced to provide further insight on the activity of the Next Gen content.

• vulscan.log (Programdata\Landesk\Log folder)
• PatchManifestSyncSDK.log (Programdata\Landesk\DebugLog folder)
• PatchScanSDK.log (Programdata\Landesk\DebugLog folder, only created when debug trace logging is disabled)
• PatchScanSDKDpdTrace.log (Programdata\Landesk\DebugLog folder, only created when debug trace logging is enabled)
• STDeploy.log (Programdata\Landesk\Log folder, but only created when repairing)
• TimberDeployEvents.log (Programdata\Landesk\Log folder, only created when repairing)

This article demonstrates how to troubleshoot High CPU usage from the W3WP process that is tied to the LDAppVulnerability application pool.

This article assumes that the user is running Server 2012 R2 which includes IIS 8.5.

General IIS troubleshooting information from Microsoft: Troubleshoot : The Official Microsoft IIS Site

Basic process for troubleshooting the CPU usage issue is the following:

1. Review the running W3WP processes.    (See What is the W3WP process?)
2. Review the IIS logs and look for what is causing the volume of traffic and for errors. (See Troubleshooting IIS logs)
3. Check the WSVulnerability.core.dll log for methods being called and their related information.  Information about the traffic going through the WSVulnerabilityCore web application.
4. Check the database performance.

Processes using the WSVulnerabilityCore web application

WSVulnerabilityCore is responsible for much of the information traveling from the managed client to the core server especially during the Patch and Compliance scan process, and then written to or retrieved from the database.   Among these are:

• Getting hash information for files
• Retrieving patch install and uninstall information
• Receiving DeviceID to retrieve computer information from the database
• Retrieving results of vulscan scan and repair
• Retrieving Antivirus information such as licensing, pattern file dates, quarantine, and infection information, etc.
• Supplying File Reputation information.

What is the W3WP process?

W3WP.EXE processes are windows worker process that run Web Applications for IIS.  Each of these processes is responsible for handling requests sent to a Web Server for a specific application pool.

Several different W3WP processes can be found in Task Manager.  By default, there will be at least two.  One servicing the LDAppMain application pool, and one servicing the LDAppVulnerability application pool.  The LDAPPVulnerability application pool services the WSVulnerabilityCore web application.

This can be seen by looking at the Advanced Settings for the WSVulnerabilityCore application in IIS.

(Click for full size)

In order to identify what the WP3WP process is assigned to, you will need to add the "Command Line" column to the Task Manager columns.

As an alternative, you can open an Administrative Command Prompt, switch into the %windir%\System32\inetsrv folder (cd %windir%\System32\inetsrv) and run appcmd list wp. This will show the process identifier (PID) of the w3wp.exe process in quotes. You can match that PID with the PID available in Task Manager.

If more than one W3WP process exists tied to LDAppVulnerability, it means that more than one worker process has been assigned to the LDAppVulnerability pool in IIS.

Many performance issues can be related to an incorrectly configured application pool in IIS.  The default settings as installed by Ivanti EPM are the recommended settings.  Outdated information exists that recommends adding worker processes, adjusting timeouts, etc.  This is not recommended.

The following screenshot demonstrates the default settings, click for full size:

(If the server is a NUMA capable server, additional options will exist. These should be left at default as well)

Troubleshooting IIS

Two types of IIS logs exist:

• W3SVC1: These logs show all IIS traffic.  These are by default  stored in C:\inetpub\logs\LogFiles\W3SVC1.  Particular attention should be paid to looking at the requests that are coming in:
  • Are they repetitive in excess?
  • Do they contain error status codes?  (403, 500, etc)

Troubleshooting HTTP 400 Errors in IIS : The Official Microsoft IIS Site

• HTTPERROR: The following list identifies the kinds of errors that the HTTP API logs:   (Further information: https://support.microsoft.com/en-us/kb/820729
  • Responses to clients The HTTP API sends an error response to a client, for example, a 400 error that is caused by a parse error in the last received request. After the HTTP API sends the error response, it closes the connection.
  • Connection time-outs The HTTP API times out a connection. If a request is pending when the connection times out, the request is used to provide more information about the connection in the error log.
  • Orphaned requests A user-mode process stops unexpectedly while there are still queued requests that are routed to that process. The HTTP API logs the orphaned requests in the error log.

Troubleshooting Failed Requests Using Tracing in IIS 8.5 : The Official Microsoft IIS Site

IIS log files store the date and time using GMT (Greenwich Mean Time).  You will need to convert that to the local time of the Core Server in order to correctly interpret when the log file entries occurred. 

How to troubleshoot IIS using Log Parser Studio from Microsoft

Using the Debug Diagnostics Tool to troubleshoot High CPU Usage by a process in IIS

Microsoft has the following article that details using their tool to troubleshoot high CPU usage.

Troubleshooting: Debug Diagnostics Tool to troubleshoot high CPU usage by a process in IIS

Microsoft Article for troubleshooting high CPU in an Application Pool

Additional article for troubleshooting High CPU in an IIS application pool:

Troubleshooting High CPU in an IIS 7.x Application Pool : The Official Microsoft IIS Site

Troubleshooting SQL Server Performance Issues

Database maintenance plans should be set up, regular indexing of the database should help keep performance up to par.

Evaluate SQL Performance by reviewing this Community Article: How to create a perfmon trace to analyze Ivanti EPM database performance

According to database performance, the following registry key can be modified to increase or decrease the number of threads that are available from the Web Service to the database:

Increasing the Number of Web Process to Database Threads

You can increase the number of threads the web service uses to talk to the database. This is by default set to "10". Be careful when setting this number, as you can overwhelm the database with web service requests. This number can be set to 40 or 50, however after doing so, monitor to make sure we are not overrunning the database.

The number of threads can be changed by adding the following registry key:

HKLM\Software\LANDesk\ManagementSuite\PatchManagement

       

Add a string value of "WebServiceMaxThreads) set to "40" or so. (This number may need to be adjusted)

(Caution: Be careful when modifying this registry key, and document changes to it)

The current value can be seen by looking in the \LANDESK\ManagementSuite\log\WSVulnerabilityCore.dll log.

The following line shows this information:

"----------WSVulnerabilityCore initializing with semaphore count: {0}----------“. 

Verify schedules for Inventory Scans, Patch and Compliance Scans, Software Distribution Jobs, etc.

If a lot of scheduled activities are scheduled to run at the same time, this can overwhelm the core server or the database resources.

Periods of excessive traffic can be pinpointed by gathering IIS logs and running the "Requests per hour" query in Log Parser Studio.

This very much can reflect scheduled activities.

How to troubleshoot IIS using Log Parser Studio from Microsoft

• Issue
• Cause
• Resolution
  • Step 1
  • Step 2
  • Step 3
  • Step 4

Issue

Despite a valid Ivanti Antivirus subscription, Patch and Compliance Manager is not able to download the newest antivirus definition.

Instead, the Patch and Compliance Manager displays the following error:

Verifying access to site US West Coast (https://patch.landesk.com)

Verification complete

Processing Ivanti Antivirus Updates

Checking for updates...

Invalid XML file 951_updates.xml. There is an error in XML document (2, 2).

No updates found.

Completed updating definitions

  

Rebooting the core server or switching to another update source site doesn't help.  Additionally, the vaminer.log shows the following entries:

Cause

This issue appears because one or more antivirus configuration files got corrupted on the core server.

Resolution

Step 1

Close the Ivanti EPM Console

Step 2

On the core server, open Windows explorer and locate the following files:

C:\Program Files\LANDesk\ManagementSuite\ldlogon\antivirus8\Mac\basesEP\LdBasesInfo.xml

C:\Program Files\LANDesk\ManagementSuite\ldlogon\antivirus8\Mac\pre.basesEP\LdBasesInfo.xml

C:\Program Files\LANDesk\ManagementSuite\ldlogon\antivirus8\Win\bases8\LdBasesInfo.xml

C:\Program Files\LANDesk\ManagementSuite\ldlogon\antivirus8\Win\basesEP\LdBasesInfo.xml

C:\Program Files\LANDesk\ManagementSuite\ldlogon\antivirus8\Win\pre.basesEP\LdBasesInfo.xml

Step 3

Rename each file - for instance, you can add the ".old" extension to each filename.

Step 4

Launch the console - this will recreate the configuration files renamed in step 2.

Try again to download the newest Ivanti Antivirus definition.

This should fix the issue.

Issue

You have some Windows 2012 servers managed by Ivanti EPM.

After running the Patch and Compliance Manager, you notice that very few patches are detected for these client machines.

However, if you run Windows Update manually on one of these servers, a lot of missing patches are detected.

Cause

This issue can have several roots.

Here are the steps to troubleshoot this specific issue.

Resolution

Patch Manager doesn't automatically install all the patches that show up in Windows Update.  This topic is covered in more details in this article :

Issue: Microsoft Hotfixes aren't included by default in Ivanti EPM Security and Patch Manager

To find out which patches are included by default in the Patch and Compliance, please see the Ivanti EPM Security Bulletin:

Security and Patch Bulletins

1. Make sure that your core server has a valid Patch Management subscription.
   
   
2. Check also that the Patch Management subscription matches the current version of your core server (for instance 9.5 or 9.6) and is not shown as "expired".
   
   
3. Make sure that the "Windows vulnerabilities" option is checked in the Patch Manager configuration window and that no error is displayed while downloading the latest definitions
   
   
4. Ensure that "Update 1" is installed on the Windows 2012 server-managed clients.

Otherwise, the most recent patches for Windows 2012 servers will not be detected by the Ivanti EPM Patch Manager.

Note: Windows 7 requires SP1 in order to install IE11.

Description

With Microsoft's recent End Of Support for versions of IE older than 11, its important to get up to date in order to receive technical support and security updates for workstations.

This document covers the procedures and expectations for installing Internet Explorer 11 using Ivanti EPM Patch and Compliance Manager.

Identifying Workstations that need to upgrade to Internet Explorer 11

We will be using a Query to identify what workstations currently have a version of Internet Explorer less than 11 installed.

To get the most accurate results, we'll want to identify all versions of Internet Explorer 11 that are currently installed on workstations, and exclude any device running those versions. This query will vary between networks, but the following screenshot should give a rough idea:

Be sure to Include Computer > Browser > Version in the result columns so the different versions can be seen in the results.

Now that we have our list of devices that need to upgrade, we know what devices we can apply to the upgrade process.

Creating a Custom Group to include Internet Explorer 11 Patch and Prerequisites

For the sake of Simplicity, we'll want to gather all necessary patches for the Internet Explorer 11 upgrade to one area. To do this, create a custom group and name it something related to Internet Explorer 11.

Search the "Scan" folder for patch ID IE11, and drag it to the new Custom Group just created. A prompt will open asking if the 9 prerequisites should be included.

These prerequisites are needed in order for Internet Explorer 11 to be detected and installed on the machine. Click Yes.

Another prompt may be generated warning that not all patches are downloaded. Be sure to download all associated patches for these 10 IDs as the job will fail otherwise.

Now that we have all necessary patches grouped together, its time to apply them to machines using either Manual Installation or Autofix.

Using Autofix to apply Internet Explorer 11

Assuming that Autofix is not enabled globally, you will need to useutilize scopes for the Internet Explorer 11 autofix upgrade. To create the appropriate scope, simply browse the Network View. right-click Scopes and select New Scope from Query.

Select the Query created above and click OK. This will create a new Scope with the same name as the Query created earlier.

Highlight all 10 of the patches in the Custom Group created above, right click and select Autofix> Autofix Settings.

Select the checkbox next to the Scope created earlier and click OK. 

The devices that qualify for the scope created will now upgrade to Internet Explorer 11 as they scan for definitions.

Apply Internet Explorer 11 manually using Scheduled Repair jobs

The manual installation of Internet Explorer 11 will be using theQuery and Custom Group created earlier in this document.

To begin, highlight all patches in theCustom Groupand selectRepair. This will create a scheduled task window.

There are a number of ways to assign devices to this scheduled task. The first being to associate the Query created earlier to the scheduled task.

This is done by selecting Target Devices > Targeted Queries > *Query Name.* This will add all devices associated with the Query to apply to this scheduled task. Furthermore, the devices will fall out of the query as the patch is applied via policy.

A different method would be to stagger the devices within the Query and add them in increments to the scheduled task.

Click Save. A new task will be added to the Scheduled Task Window.

If the task was associated with a Query, Right-Click the task and select Start Now> All. Otherwise, devices will need to be added to the task before starting.

What to Expect

1. When using Autofix to upgrade Internet Explorer 11, the device will automatically fall out of the Query created as they upgrade. By association, they will fall out of the scope. There's no need to worry about devices running the upgrade more than once.

2. This upgrade will often require 1 or more Reboots. Plan accordingly with Reboot behavior. In this lab, a Windows 7 x64 install with Internet Explorer 8 was used. A total of 3 Reboots was required to fully upgrade.

3. The install script for Internet Explorer 11 includes a /norestart command. This means the final reboot needed will not immediately prompt the user.

4. Internet Explorer does NOT need to be closed before the patch is applied. It can be done silently in the background, and the install will complete on the next reboot.

Issue

The problem typically occurs when the core is trying to patch a newly imaged machine. The error message is "String not found: PatchAction.39"

The reason for this error is because the Core & Client machine are on different versions of Ivanti EPM.

Resolution

The solution for this error is to patch both the core and client to the same version.

Ivanti recommends that you update to the latest publicly available component patch.

This is available in the Ivantiupdates category within the Security and Patch Manager tool in the Ivanti EPM console.

Purpose

This article will provide a step by step in creating recurring Patch and Compliance Content Download Tasks.

Step by Step:

1. Navigate to Tools>Security and Compliance>Patch and Compliance on your Ivanti EPM Console.

2. Open the Download Updates window from within Patch and Compliance.

3. Select the applicable Content for recurring downloads (this is configurable and dependent upon the needs of your environment; selections chosen are for examples).

4. Select "Schedule Downloads

5. Name your Task and select OK.

6. Utilize the Start Later: Date; Time and Repeat functions to automate your downloads.

7. Select 'Save' and your task will populate within your Scheduled Tasks window.

Problem:

When performing “Download Updates” in the console, error message “Waiting for file lock” appears.

Cause:

This can happen when more than one instance of downloading updates is running simultaneously. For example, if you try to manually download patches while a scheduled patch download task is already running.

Solution:

• Kill all running VAMiner.exe processes.
• Log off the console and log in again.
• Reboot core server.

If none of the above works, try executing the following query against your Ivanti EPM database to remove the lock.

SELECT * FROM PatchSettings WHERE Name LIKE '%LOCK.UpdVulnLock%'

DELETE FROM PatchSettings WHERE Name LIKE '%LOCK.UpdVulnLock%'

This document illustrates the files, registry, settings and other information necessary to effectively troubleshoot a vulnerability scan.

In addition, this document walks through the steps that a basic Patch and Compliance scan (otherwise known as a vulnerability scan) takes.

This article will not describe every single step that the Vulnerability Scanner takes, but those steps where a failure can occur.

For the purposes of this document a simple scan is performed by running the following at the client command line:

vulscan /scan=0 /showui

This command tells the vulnerability scanner to scan Windows vulnerabilities (type 0) and to show a user interface.

The name "LDMS2016" and "LDMS2016_v###" will be seen throughout this document.   This refers to the Core Server name of "LDMS2016" which is the name of the core server that the author had when creating the document.

Settings

The settings that control how the vulnerability scanner will behave are stored in the Distribution and Patch Settings within the Agent Settings tool.

These settings control behaviors such as user input options, Cloud Services Appliances patch options, scanner CPU utilization, etc.

These settings are stored in the Ivanti EPM Database in the AgentSettings table and physically on the core server in the

\Program Files\LANDESK\ManagementSuite\ldlogon\AgentBehaviors folder.  The Distribution and Patch Settings are stored in the AgentBehavior_(Corename)_v###.xml file within this folder.

                                                                            Example

Product Licensing

The categories available to scan and repair are controlled by the Product license that has been purchased and activated on the core server.

The following graphic shows the categories available within the Download Updates function within the Patch and Compliance tool for those with a license for all categories.

Click for full size

Registry Keys

Core

HLKM\Software\LANDESK\ManagementSuite\PatchManagement\WebServiceMaxThreads

This key does not exist by default and should only be created with an understanding of how this key works and the full ramifications of creating this key and changing the default value.  This changes the number of default threads

This key is documented here: https://community.landesk.com/docs/DOC-36027#jive_content_id_Increasing_the_Number_of_Web_Process_to_Database_Threads

Client

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\landesk\managementsuite\WinClient\Vulscan

This registry key contains the following information:

Note: The populated "Data' entries are provided as an example.  Yours will differ.

Gathering information for Ivanti Support

The vulnerability scan log files are located in the C:\ProgramData\LANDESK\log folder.

When in doubt just .ZIP up the entire folder and send it.

Otherwise, the following logs should be gathered:

• vulscan*.log
• statusdlg*.log

It is very useful to turn on Xtrace with the following enabled from the registry key prior to duplicating the problem and gathering logs:

From HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\landesk\managementsuite\LogOptions:

How To: Enable XTrace Diagnostic Logging for the Ivanti Core and Clients

Tips and Tricks

Vulscan can be used as a shortcut to open various folders.  The following should be run on the client command line:

Vulscan e - Open the folder where Vulscan resides

Vulscan c - Open the LDClient folder,

Vulscan log - Open the ProgramData\LANDESK\Log folder

Issue: Cannot open vulscan logs folder from the command line using "vulscan e"

Ivanti Patch and Compliance (vulnerability) Scan Process Flow

It is important to note that the following must all be able to take place:  Client contact to core through IIS and several web services.  Core contact to Database Server  Core Contact to client  Correct permissions on core ManagementSuite\Incoming directory and \ManagementSuite\LDLOGON\VulnerabilityData and VulscanResults folders.

Note that issues can come and go during a vulscan.  This would indicate intermittent issues.  Most of the time this occurs when the server or database has connectivity issues or are too overwhelmed to respond to requests.

• About Java Patches
• Java patches within Ivanti Patch and Compliance Manager content
• Download, rename and place patch in the patch location
• Custom Variables Tab within the Java definitions
  • Force Mode
  • Disable Java Update
  • JRE Installation
  • Remove Old Version
  • Expiry Date of JRE

About Java Patches

Java patches are available for the Java Runtime Environment (JRE) or Java Software Development Kit (JDK).

JDK is a super-set of JRE, and contains everything that is in JRE, plus tools such as the compilers and debuggers necessary for developing applets and applications. JRE provides the libraries, the Java Virtual Machine (JVM), and other components to run applets and applications written in the Java programming language.

Java patches within Ivanti Patch and Compliance Manager content

• JREJDKv#U###_Manual   (JRE and JDK installation that requires a manual download)
• JREv#U###_Upgrade   (JRE upgrade that can be downloaded automatically)
• JREv#U###   (JRE installation that can be downloaded automatically)
• JDKvU###_Manual   (JDK installation that requires a manual download)
• JDKv#U###_Manual_Upgrade   (JDK Upgrade that requires a manual download)

Some examples:

JREJDKv7U111_Manual - Manual update for updating both JRE AND JDK Version 7 to the 111 update.

As you can see the name specifies if it is a JRE or JDK update or both.   It then says what version the patch is for, and then the patch number.

Some simply upgrade the existing installation to the newer version.  These are denoted by the word "Upgrade" in the title.

Others perform an upgrade by uninstalling the old version and installing the new version.

                                                               Example List

Download, rename and place patch in the patch location

The following website typically is the location to download Java SE JDK and JRE patches:

Java SE - Downloads | Oracle Technology Network | Oracle

The patches that end in _Manual require the following

1. The user to accept a license agreement and/or require a user to log in.
2. Download the patch.
3. Rename it as required (this is listed in the description of the patch, which you can get to by right-clicking the vulnerability, selecting properties, and going to the second tab).
   
   Example Description
   

   Java™ SE Runtime Environment 8 Update 102 (JRE 8u102)
   Java SE 8u102 includes important security fixes. Oracle strongly recommends that all Java SE 8 users upgrade to this release.
   Please download the install packages manually from http://www.oracle.com/technetwork/java/javase/downloads/index.html
   Oracle re-releases Java SE 8u102, so please rename patch file as below list, then put them in the "patch" folder.
   jre-8u102-windows-i586.exe
   jre-8u102-windows-x64.exe

4. Place it into the patch directory that will be used to deploy the patch.
5. Run "Download Updates" again to update the Downloaded status of the patch within Patch Manager.

Custom Variables Tab within the Java definitions

Within the Java definitions there is a Custom Variables tab.  A custom variable is a configurable setting used to extend the usability of the definition.

There are typically up to 5 options that can be adjusted

Force Mode

The "No" option is the default value. If the Java application or browsers are running the repair task will fail and an error will be reported and you will be prompted to close the Java application and/or browsers.

If you chose "Yes", you will proceed to install the Java update with following actions, the installation process will terminate your Java application and/or close any open browsers.

Disable Java Update

The "Yes" option is the default value. When the default value of "Yes" is selected, the Java auto-update feature is disabled. If "No" is selected, the Java auto-update feature will be enabled.

If you intend to manage Java Updates solely through Patch Manager this option should be set to "No".  The Java auto-update feature is Java periodically checking to see if there are new updates and auto-applying them itself.

JRE Installation

The patch-in-place mode implies that when a version of the JRE exists on a machine, any updates belonging to the same JRE family will be done in place, meaning, the existing JRE will be patched with changes. A JRE is installed in patch-in-place mode by default. When a JRE is installed in static mode, it will not be updated in place by later versions. A later version from the same JRE family will be installed in a separate directory. This mode ensures that vendors, who require a specific version of the JRE for their product, can be certain that the JRE will not be overwritten by a later version.

For more information: Patch-in-Place and Static JRE Installation

Remove Old Version

The "No" option is the default value. When the default value of "No" is selected, Old version Java will be skipped. If "Yes" is selected, the old version Java will be removed.  This is in case you need to have multiple versions of Java installed to maintain backward compatibility.

Expiry Date of JRE

The expiration date is calculated to end after the scheduled release of the next Critical Patch Update. After this date, Java will provide additional warnings and reminders to update to the newer version.

The "No" option is the default value. When the default value of "No" is selected, the Expiry Date for JRE feature is disabled. If "Yes" is selected, the Expiry Date for JRE feature will be enabled.

Description

The vulscan.exe are accumulating. vulscan.exe is adding one each day, there are couple of vulscan.exe process since last computer reboot.

After the security scan is over, the window won't close and there is no any expected 1 minute automatically close message.

Resolution

This is because customer is using the option Require end user input before closing. This can cause multiple vulscan.exe processes running, and no no any expected 1 minute automatically close message.

Using the option Close after timeout can fix this issue. This setting will allow the vulscan process stop automatically after the security scan instead of manually close the security scan page.

Description

Repair task fails with result "One or more definitions in repair request have not yet been scanned."

Return code 467

Cause

Vulnerability being repaired is not in Scan folder. Patch and Compliance Manager will always scan for a patch before installing it to verify that it is needed.

Resolution

Identify which vulnerability is not in the Scan folder.

Once the vulnerability is identified, open the properties of the vulnerability and navigate to the Scan tab.

Set the global scan status to "Scan."

Re-run the repair task.

Problem

When accessing the Properties of some patch definitions, you aren't able to see any patch files available under General tab>Detection rules. Below Detection Rules is an empty box.

While a normal patch definition should look like this one:

Cause

Those problematic definitions are probably incomplete or corrupted.

Solution / Workaround

1. Delete the problematic patch definitions.
   
2. Download them again.

Issue

When accessing the Dashboards in Patch and Compliance and check the "Patch Installed" dashboard, it shows no data at all, even though you have recently run a repair task which had successfully installed patches to the client machines.

Cause

The historical information gathered only included definitions published less than 90 days ago by default, any previously released definition will not be reported on.

Solution / Workaround

1. Go to Patch and Compliance > Create a task > Gather historical information.

2. Set the value of the "Build report data for definitions published less than" to a longer period than the release date of the patch you wanted to report on. 

E.g. The patch I have installed with Patch Manager was released on 2016/7/12, thus I have set this value to "250" days. ( Today's date is 2017/2/6)

3. Select "Save and gather now", wait for the job to finish. 

4. Now data has been imported and is shown on the dashboard.

Ivanti Agent May Continually Prompt to Reboot

• Ivanti Agent May Continually Prompt to Reboot
  • Detecting a Pending Reboot
    • Pending File Rename Operations
    • Vulscan Reboot Registry Key
  • When is Vulscan Allowed to Reboot?
  • Common Causes
    • "Stuck" registry key
    • Continuation
    • User Error

Ivanti EPM uses the vulnerability scanner process (vulscan.exe) to evaluate and conduct necessary reboots.  Vulscan considers your agent settings and references Windows API and two specific registry keys to determine whether or not a Windows device needs a reboot. Having the agent installed on your devices allows them to present a reboot GUI to your end users if a reboot is pending and if your Agent Settings configuration allows.  Depending on your agent settings options and the tasks you are deploying, vulscan may cause the device to prompt for reboots multiple times.

Under some circumstances multiple reboot prompts may appear in short order, even after allowing a reboot.  Distribution and Patch agent settings, along with Reboot settings, are very flexible and can be configured in many different ways.  Sometimes the combination of options chosen causes unexpected behavior.  This document provides an overview of the applicable systems and settings, and discusses some common reasons and configurations that may result in multiple reboots or reboot prompts.

Detecting a Pending Reboot

It is normal expected behavior for vulscan to detect if a reboot is pending.  It does so during every scan job by evaluating the presence of the below registry keys:

Pending File Rename Operations

[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\Session Manager]

"PendingFileRenameOperations"

The vulscan log will contain this text:

Pending file rename data is present.  Reboot is needed.

Vulscan Reboot Registry Key

[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\landesk\managementsuite\WinClient\VulscanReboot ]

The vulscan log will contain this text:

Vulscan reboot key exists. Reboot is needed.

When is Vulscan Allowed to Reboot?

Vulscan detects when reboots are pending during any job but will only process a reboot when all of the below conditions are met:

1. One of the above registry keys indicates a reboot is pending
2. Vulscan attempted to install patches (a scan-only job will not trigger reboot processing)
3. The applied agent settings give vulscan permission to reboot the device (vulscan will still process any end user prompts, honor automatic reboot conditions, and honor the automatic reboot window (if configured)).

Common Causes

"Stuck" registry key

The keys mentioned previously are set by the OS or by vulscan.  They are volatile and should be removed when the device reboots.  Under some circumstances (for example if either key is manually set) they aren't removed upon reboot.  This causes vulscan to detect a pending reboot again the next time it runs, although as mentioned previously detecting that a reboot is pending is not enough by itself to cause a reboot to happen.

If you determine that either key is stuck, it's usually sufficient to manually delete the key.  Afterward, it should be volatile when the OS or vulscan next set the key.

You may also find at times that faulty applications constantly re-set the Pending File Rename Operations key.  The key identifies the files in question that are pending and you can use this info to identify the misbehaving application.  You may need to work with the application vendor's support team to identify and solve why it keeps setting reboot keys. 

Continuation

Your distribution and patch settings contain a 'continuation' option:

Continuation allows vulscan to automatically continue a repair task after the computer reboots or other necessary prerequisites are met.  A common situation that triggers continuation is when patching must halt due to a pending reboot, or when some patches fail due to a necessary reboot.  After the reboot occurs, vulscan can automatically continue and may repair additional patches.  If vulscan attempts an install, even if it's due to continuation, it will evaluate your reboot settings and if allowed, may reboot or prompt the user to reboot.  By default Continuation is allowed up to 5 additional repairs.

User Error

It is unfortunately true that many end users don't understand technology as well as we'd like, and sometimes misunderstand or misinterpret what they see and experience.  Our support teams have encountered many of these situations.  One common example is when an end user gets the reboot prompt and chooses to defer.  Sometimes the user will log off or lock their computer, or go home for the day, or otherwise forget that they deferred the reboot.  The prompt later appears again according to agent settings, and the end user assumes that they are being prompted to reboot again, when in reality they never completed the first reboot.

Beyond educating the user there isn't a lot that can be done to fix this.  It is helpful as a first step to determine if\when the computer rebooted and which process was responsible for it.  You can most easily see this by opening event viewer and filtering the Windows Application logs by event ID 6006 and 6008:

In some cases you may find that the computer did not reboot, or did not reboot due to any Ivanti process.

Overview

Ivanti Endpoint Manager's Patch and Compliance vulnerability scanner (vulscan.exe), will attempt to download an additional  30MBs (approx.) of data from the core server (source) when scanning against Next Generation definitions. This file download size can stress environments not configured to accommodate this additional data and could cause latency on the network. To minimize the bandwidth constraints, Ivanti's download functionality is designed to retrieve content from (3) locations; Peer, Preferred Server, Source. This document outlines how stage the Next Generation binaries which allow your existing endpoints to leverage Ivanti's download capabilities allowing for more efficient file transfers throughout your environment.

Peer Staging

In order for the Next Generation Binaries to exist on the core server, a download of the Next Gen Microsoft Windows Vulnerabilities (beta) definitions must be completed from the Download Updates interface. The required binaries for scanning against Next Generation definitions will reside in ManagementSuite\Ldlogon\Timber.

To control the scanning of all Next Generation definitions, transfer the "Next Gen" definitions to a custom patch group, assign the custom group to an alternate Distribution and Patch agent setting and assign the setting to (1) device per subnet. Once (1) device on the subnet has the necessary binaries to scan against Next Generation definitions, the use of Peer to Peer downloading can be leveraged when the remaining devices need the next generation binaries.

The steps to do so are as follows:

Step 1: Create a custom patch group.

This is accomplished by right-clicking and selecting "New Group" in Patch and Compliance | Groups | My custom groups. Name the group as desired; in the below example the custom group name is "Next Gen".

Step 2: Isolate the Next Gen definitions and add them to the custom patch group.

To do this, perform a search in Patch and Compliance | All Types for keyword "Next Gen". Once filtered, highlight (ctrl+a) these definitions and move them to your custom patch group. To move you can drag and drop the highlighted definitions to the group or right-click copy and paste.

Step 3: Create an alternate Distribution and Patch agent setting

This can be done by right-clicking "New" after navigating to Agent Settings | My Agent Settings | Distribution and Patch

Step 4: Assign the custom group to an alternate Distribution and Patch agent setting

Right-click on the alternate Distribution and Patch agent settings and select properties. From this view select Patch-only settings | Scan Options. In the "Scan for" section select Group and associate your custom group for scanning and save. This will restrict the vulnerability scanner's view to the definitions residing in this group. Any device assigned this setting will only be able to scan definitions from this custom group.

Step 5: Assign the alternate Distribution and Patch agent settings to a device

This new setting configured to scan against only Next Gen definitions has to be assigned to (1) device per subnet. This will allow the vulnerability scanner to download the next generation binaries to the endpoint. Once these binaries reside on (1) device in the subnet, all remaining devices will be able to pull the files from a peer negating the need to traverse the network to the source.

To do this select "Create a task" and choose "Change Settings" under Agent Settings.

This will present you with a Patch and Compliance - Change Settings task interface. Choose the alternate agent setting previously configured to scan against "Next Gen" definitions and select save. Upon saving, a change settings task will be created for you to target the desired subnets throughout your environment.

Preferred Server Staging

The same approach taken with Peer can be done with a Preferred Server on the same subnet as your endpoints. You are less likely to have (1) Preferred Server per subnet so it is recommended to use the functionality available through Content Replication to transfer for the Next Generation binaries from the source to a preferred server.

The following documentation outlines How to use Ivanti EPM Content Replication

References

About Distribution and Patch Bandwidth Throttling (Advanced)

How to troubleshoot Download Failures in Software Distribution (Advanced) 

This article describes the content verification feature within Ivanti Patch and Compliance Manager

Content verification can be enabled to cause the Ivanti EPM Core server to add in a hash checking feature when downloading content from the Ivanti EPM Patch Content servers.

The content verification feature applies to the content only, it does not apply to individual patch files themselves.   The patch file hash information is contained within the definition information and is verified as part of the patch installation process.

Content verification is only available for the following content types:

• Microsoft Windows Vulnerabilities
• Microsoft Windows Security Threats
• LANDesk Updates

Note: When content verification is enabled, but content types other than the types mentioned above are downloaded (Apple Macintosh definitions, for example), errors may be thrown.

Example of errors for content types that do not support Content Verification:

Even though an error is thrown, the content is still downloaded correctly.

Content verification can be enabled within the Download Updates tool under the Content Tab:

Verify definition signatures/hashes before downloading

NOTE: When checked, any definitions that do not have a valid SHA256 hash will not be downloaded. Also, any lists of definitions that do not have a valid signature will not be processed. The download progress form will show any download failures due to invalid/missing signatures or hashes.