Troubleshooting 4.3.1 Insufficient System Resources Error for External Mail Flow on Exchange 2013

EdgeTransport.exe.config QueueDatabasePathI’ve recently been contracting out to an educational organisation assisting with their BAU work as well as helping them modernising some of their processes and server environment. I was recently given a trouble ticket where I had to troubleshoot slow external mail flow. They have an ancient spam filtering appliance coupled with an Exchange 2013 deployment. My first look was onto the appliance, after checking out the inbound mail queue I could quickly see e-mails sitting there with the following response code;

452 4.3.1 Insufficient system resources

The above response code, which indicates a temporary failure generally means that Exchange is running out of resources and you’ll have the Exchange back-pressure issue where the Transport service will reject message submission because there is not enough free disk space (which is by default 10%). From the information I had their databases sat on another disk with at least 50% free space. Going through the even log I could find no trace of this. Their Mailbox Database servers were also Transport servers so they performed messaging queuing that meant I had another database that should be taken into consideration, which is the Message Queue database. By default it sits in the C:\Program Files\Microsoft\Exchange Server\ directory and is not easily configured to be moved like a standard mailbox database using PowerShell.

As I was unsure of their environment, I quickly navigated to EdgeTransport.exe.config file located under the Bin directory and found the value of QueueDatabasePath pointing to their C drive which had only around 5% free space. I then quickly cleared out some log files that are not required and wrote up a PowerShell script that will run nightly until more space is provisioned on those servers.

Moral of the story is always give your Exchange servers plenty of free space to prevent such issues.

Finding the location of a device using an IP or MAC Address in HP intelligent Management Centre IMC (like you could in ProCurve Manager)

Find a device using IMCI was out at a client recently helping to clean up their iMC implementation and make some recommendations regarding their network setup. I was asked if you could look up Switch and Port number location of an end user device like you could in ProCurve Manager (PCM). Sure you can, the feature has been around since HP Intelligent Management Centre 5.1 and to find it you simply need to go to;

Resources > Terminal Access > Real-Time Location.

You then just need to type in either an IP Address or MAC Address of the device and click OK. Results usually come back almost instantaneous (unlike PCM).  There isn’t any special configurations needed so long as iMC is talking to your switches.

PowerShell Script to Install Updates Offline in a WIM image using DISM

WSUS Offline Downloader in actionI’ve been helping out a customer build a new MDT deployment environment and move away from Ghost and the 90’s. As they are not going to be implementing Systems Center Configuration Manager and SUP to automatically maintain their images offline any time soon, we need a way to keep their image up to date with updates, without having to re-build it every time.

I knew you could already do offline servicing with DISM but wanted to make it nice and easy for them. I’m using WSUS Offline Update to download all of the updates in one shot, you could also use WUD but their lists haven’t been updated for a while.  I copied the zip and extracted it to their deployment server and downloaded all of the updates for Windows 7 x64 SP1 and saved them all to a single updates folder. I then built up the below PowerShell script to offline service their image and apply the updates downloaded.

$UpdatesPath = "E:\Updates\*"
$MountPath = "E:\MDTDeploymentShare\Operating Systems\W7X64SP1\Mount"
$WimFile = "E:\MDTDeploymentShare\Operating Systems\W7X64SP1\REFW7X64.wim"

DISM /Mount-Wim /WimFile:$WimFile /index:1 /Mountdir:$MountPath
$UpdateArray = Get-Item $UpdatesPath
ForEach ($Updates in $UpdateArray)
{
DISM /image:$MountPath /Add-Package /Packagepath:$Updates
Start-Sleep –s 5
}
Write-Host "Updates Applied to WIM"
DISM /Unmount-Wim /Mountdir:$MountPath /commit
DISM /Cleanup-Wim

If you have 100+ updates this process can take a while so sit back and drink a coffee while you run the script. Hope that helps.

Activating Windows 8.1 clients on a KMS Host running Windows Server 2008 R2

I was recently helping out a friend who had a KMS server activating their Windows 7 machines but when they started moving their fleet to Windows 8.1 they saw they couldn’t install the KMS Host product key for Windows 8.1 recieving the following error:

0xc004f015: The Software Licensing Service reported that the license is not installed. 
SL_E_PRODUCT_SKU_NOT_INSTALLED

To allow Server 2008 R2 to activate newer versions of windows you need to install KB 2885698. The update adds support for Windows 8.1 and Windows Server 2012 R2 clients to Windows Server 2008, Windows 7, Windows Server 2008 R2, Windows 8, and Windows Server 2012 KMS hosts (see Windows Server 2012 R2 & Windows 8.1 KMS Service Activation).

Once you install the update you will need to restart the server (so do it out of hours if it’s on a DC or something critical). Once you’re back up and running, you simply need to issue the install product key command with your new KMS Host key.

slmgr.vbs /ipk YOUR-PROD-UCTK-EYHE-RE33

And your clients will begin activating once you hit the 25 client minimum. The key will continue to activate lower versions of Windows and if you install a Server key, it will do both Server and Client editions.

How to Rename a SharePoint 2013 Content database name hosted on SQL Server

Dismounting and Renaming a SharePoint DatabaseSo I’ve been toying with SharePoint 2013 in my lab at home, doing the install and configuration at the moment, setting up my first Site after the initial configuration but noticed that I made a typo in the Database name.  So how do I go about fixing or renaming that database in SharePoint and in SQL Server.

First thing first is we need to know the name of the Content database, in my case it is easy, because we only have one and I can easily see the spelling mistake I made. But if you need to work out the name of the Database, you can look in Application Management > Manage Content Databases and click on the Site Collection you are after; alternatively you can use SharePoint’s management shell and issue the following two cmdlet’s;

Get-SPWebApplication | Select-Object Url, ContentDatabases | Format-List

What sites are using this content database;

Get-SPContentDatabase | Where-Object {$_.Name -eq "SharePint_ContosoContent"} | Select-Object Sites | Format-List

So now we know the name of our content database, we need dismount it before we can rename it in SQL Server. To dismount use the following cmdlet;

Dismount-SPContentDatabase SharePint_ContosoContent

Now the database is ready to rename.  Open up SQL Management Studio and connect to your Database Server hosting SharePoint.  Now usually when you go to rename a database you will get an error message stating that the database is in use.  To get around that we turn our database into Single User mode, rename it and then set it back to multi user mode with the following SQL snippet.

USER Master
-- Set to Single User to prevent unable to lock out
ALTER DATABASE SharePint_ContosoContentDB SET SINGLE_USER WITH ROLLBACK IMMEDIATE
-- Rename Database
ALTER DATABASE SharePint_ContosoContentDB MODIFY NAME = [SharePoint_ContosoContentDB]
-- Set back to multi user for production use
ALTER DATABASE SharePoint_ContosoContentDB SET MULTI_USER

Now once we have renamed the database on our SQL Server, it is time to mount it back onto SharePoint, we do that by issuing the following PowerShell on our SharePoint server

Mount-SPContentDatabase SharePoint_ContosoContentDB -WebApplication http://intranet.contoso.com/

And there you have it, renaming a SharePoint content database if you make a typo like me.

How to spoof or mimic a hardware or MAC Address of another device on an interface on a FortiGate

I was recently out at a client and they were in the process of getting another link installed, which happened to be Cable internet which they were using for redundancy.  We put their cable modem in Bridge mode but the connection wasn’t coming up after connecting everything up.  Most cable providers authenticate with certificates and the hardware address of the router so this made me thinking that we needed to make their provider think the FortiGate was their original modem.  So firing up an SSH session I logged in and issued the following commands;

FGT40C # config system interface
FGT40C (interface) # edit wan2
FGT40C (wan2) # set macaddr xx:xx:xx:xx:xx:xx
FGT40C (interface) # end

Now you will want to replace the interface, in this case wan2 with your interface and the mac address with the hardware address you want to spoof or mimic.

I gave the cable modem a power cycle for good measure and the connection came up.

How to reset the Remote Desktop Server Licensing Grace Period on Windows Server 2012 with Remote Desktop Services

So we recently started looking into Terminal Services and RemoteFX to power some of our admin users and move them off to thin clients instead of full blown desktops.  As a trial I begun setting up RDS on one of our Dev machines.  After going through the motions of enabling the Remote Desktop Features and setting up RemoteFX on a Virtual Machine for testing, I found that I couldn’t login via RDP to that machine.  Going back to the RDS host I found the Licensing popup that informed me that the 128 day trial license had expired.  Since this was a test I didn’t want to go using keys to activate or setup a licensing server (purely a PoC for us in IT at this stage).

To reset the grace period there is a registry key that we need to delete. As always when editing the registry, take a backup of the key/s you’re modifying. Navigate to the following location

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server\RCM\GracePeriod

Now there should be a binary key value (like in the screenshot), you first need to have full access permissions to the folder (or take ownership) and then simply delete that value and reboot the server.  Once back up your RDS Licensing should be back at the start of a 128 Day grace period.  I wouldn’t do this if you have already obtained licensing and should definitely not be used in a production environment.

The Network Policy Server Service (NPS or IAS) Fails to Start on Windows SBS Server 2008 with an unspecified error (0x80004005)

vss registry keySo I was recently helping out a client clean up their Small Business Server environment, one thing they weren’t using was SharePoint, so we decided to remove it from their SBS 2008 server. The removal went smoothly, simply uninstalling SharePoint Services via Add/Remove control panel did the trick. But after a restart we found that clients were no longer connecting to their Wireless network, so I went did some troubleshooting and found that the NPS service hadn’t started since boot-up.

I then tried starting it up but got an error after trying to start the service with an unspecified error 0x80004005 which was very helpful. After some more digging around and Google-Fu I found that this had to do with permissions on a registry entry for VSS. The following key needed to be updated from the value 1 to 0.

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\VssAccessControl

Make sure that the setting for NT AUTHORITY\NETWORK SERVICE is set to 1. If this is not and set to 0, change it to 1.  Once you have changed the value you can restart or terminate any IAShost.exe that are running via task manager and then start up the NPS service.

This issue has been documented by Microsoft to occur if you remove SharePoint from SBS Server 2008, so just a heads up if you do and rely on NPS for authentication for things like Remote Access or Wireless.

How to log into ASDM for a Cisco ASA when you get the Unable to launch device manager error

I like to keep my client’s devices up to date to prevent any issues from arising and have access to the devices latest features. Unfortunately not everyone is happy to pay that little extra to have an on-going support / maintenance agreement.  I was recently out at a customer who had an ASA 5505 which had been running for years but needed some new NAT translations added in.  So I fired up my laptop and loaded up ASDM Launcher, after typing in the credentials I got Unable to launch device manager on ASA-5505 error.  One thing I like to do when this happens is to check why, click on the Java icon in the login window and you will be prompted with the Java console, I saw the following;

javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: Java couldn't trust Server
    at sun.security.ssl.Alerts.getSSLException(Unknown Source)
    at sun.security.ssl.SSLSocketImpl.fatal(Unknown Source)
    at sun.security.ssl.Handshaker.fatalSE(Unknown Source)
    at sun.security.ssl.Handshaker.fatalSE(Unknown Source)
    at sun.security.ssl.ClientHandshaker.serverCertificate(Unknown Source)
    at sun.security.ssl.ClientHandshaker.processMessage(Unknown Source)

java_securityThe main thing sticking out to me was Java couldn’t trust Server to fix this I needed to load up the Java control panel once open navigate to the Security tab > Edit Site List… > Add > Enter the URL of ASDM.  In my case that was https://192.168.1.254/ once I added that, click Ok and Apply and close out of your browser / ASDM Launcher.  Now if you try to log in again it should let you through into ASDM and allow you to make the changes you need.

Another method to resolve this error is to export the certificate of the device to your computer (use firefox and navigate to the appliance web page and export the SSL certificate) then import it into Java using the control panel method above but hit Manage Certificates…

If you’re running a new version of ASDM, this doesn’t happen, also some say downgrading Java to 1.6 will remove the issue also.  Hope that helps someone out.

Backing up your EVA SAN (4400 or P63x0) Configuration with HP StorageWorks Storage System Scripting Utility (SSSU)

SSSUJust a short one today.  I was assisting in the change from a stand-alone Hyper-V environment to a clustered environment, the customer had an HP EVA P6300 SAN which was going to be used for the Clustered Shared Volumes.  As a precaution before beginning work on the EVA, I loaded up SSSU which was installed on one of their servers and took a backup of their current configuration of the EVA.  The backup file would come in handy if something were to happen and can assist HP Support in restoring your array if something were to happen.

When you load up SSSU it will firstly ask you to login with a user and password with rights to the Storage Array, once in, we need to select a Storage System and then finally issue a configuration backup to the local workstation.

select system
capture configuration filename

Replace filename with something like c:\SAN_backup.txt the backup is a plaintext file telling the Storage Array how it has been configured with LUNs and Hosts.