Author: John

Solving the FIM (Forefront Identity Manager 2010 R2) FIMService start timeout (Portal) and getting it to Start

John Leave a comment

We were recently making changes to our FIM environment where our Forefront Identity Manager boxes required restarts.  With FIM we’re always making changes in our Development kit before moving into production (which is something everyone should try do).  We quickly found that we couldn’t get back into the FIM portal and taking a quick look at the services management console we could see the FIM Service as stopped.  We had already set it to delayed start in the beginning of the setup as we found it had a much more reliable rate of starting up in our particular environment.

After some Google-fu and digging through event logs seeing entries such as simply The service did not respond to the start or control request in a timely fashion. You may also get Error 1920. Service ‘Forefront Identity Manager Service’ (FIMService) failed to start. Verify that you have sufficient privileges to start system services. Or A timeout was reached (30000 milliseconds) while waiting for the Forefront Identity Manager Service to connect. Basically, one of the main reasons for this service not starting is around .Net verifying the Authenticode signatures for the FIM service.  To try and mitigate the service timeouts we can increase how long the OS is going to wait before issuing an error by adding the following registry key onto the FIM box.

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control
Name: ServicesPipeTimeout
Type: REG_DWORD
Value (decimal): 60000

You could also go and disable the .Net Authenticode check by following the instructions at http://social.technet.microsoft.com/wiki/contents/articles/13946.fim-troubleshooting-fim-service-start-up-timeout.aspx.

Connection closed gracefully error when sending bulk or large quantity of e-mails in an Exchange 2007/2010 environment

John Leave a comment

I was recently helping out an old work colleague who were having issues with their CRM software and sending bulk emails through their Exchange 2010 server. After around ten minutes they would receive an error message with connection closed gracefully.  They would then have to restart their mail out and need to monitor it for this issue to ensure it didn’t get stuck every ten minutes with this error.

After poking around their receive connectors on their hub transport server I noticed that the particular connector (for their internal applications) they were using had a connection time out of 10 minutes, which would result in us receiving the connection closed gracefully error from our end user application.  The fix for this is to simply increase the ConnectionTimeout value for our Recieve Connector to anything reasonable, for us it is 3 hours. We would do by issuing the following Exchange PowerShell command:

Set-ReceiveConnector "Internal Connector - Synergetic" -ConnectionTimeout 03:00:00 -ConnectionInactivityTimeout 01:00:00

You will want to make sure that your receive connector is protected (i.e. is only set to allow internal networks) as this basically allows anything to stay connected for up to 3 hours so it could be abused.

Checking the performance of your Windows Server 2012 Hyper-V Server with Performance Monitor and PAL

John Leave a comment

Windows Server 2012 brings some great improvements to Virtualization.  We’re currently running it in production and it works wonders, especially with the new Hyper-V Replica feature which is great and free way of implementing DR.  So now you’ve got that cluster running, how can we tell if it is performing well.   I recently stumbled upon PAL or Performance Analysis of Logs Tool, which is freely available from http://pal.codeplex.com/.  The tool comes with a Performance Monitor Template file that we can use to easily record the performance of a Hyper-V 2012 Server and get some visibility into our Servers.

pal_wizard

When it comes to Performance things we would want to look at would be Disk Performance, Memory and CPU but understanding what effect these have on Hyper-V isn’t always straightforward, thankfully PAL helps us being making it easy and straightforward.  So first off go and download it from codplex and install it onto your workstation on desktop PC (not on your Hyper-V Host).  Once installed, open it up.  We need to get our Perfmon template so go into the Threshold File tab and select “Microsoft Windows Server 2012 Hyper-V” from the Threshold file title, then choose Export to Perfmon template file. Get this onto your Hyper-V Host and open up Performance Monitor on the host we want to watch.

Open up Performance Monitor on our Hyper-V Host and open up Data Collector Sets, right click on User Defined and select New Data Collector Set.  We can call it Hyper-V Performance Counter Set or what every you wish, keep Create from a template (Recommended) selected.  Click next and then browse, selecting the  XML Template file that we exporter earlier from PAL.  Once you finish the wizard it will show up under the User Defined Data Collector Sets. You can now open up the properties and set it to run from 30 minutes to 1 hour depending on what you’re after.  It is best to run this during peak load times (so business hours).

avg_responseNow that you’ve collected the data you want to browse to it and copy the .blg file to your workstation (or where ever you installed PAL). Back to PAL, select the log file under the Counter Log tab, then move onto the Questions Tab where you need to specify things such as processors, total RAM, drive configuration to help specify thresholds for your performance report. Navigate to the other tabs if you wish to modify anything else and finally execute the report, this can be time consuming depending on how long you’ve captured performance data for (it took me around 20 minutes for a 1 hour capture). Once it has completed generating the report it opens up in your web browser.  Key things to look at are disk latency (as per the graph with RED being critical), memory and CPU issues.

That is a free and easy way to check on the health and performance of your new Hyper-V 2012 Cluster.  You can also use PAL for a Windows Server 2008 R2 Hyper-V Cluster and other services such as Exchange or SQL Server, you just need to select the right template.

How do you stop Server Manager from loading up at start-up on Windows Server 2012

John 5 Comments

For quite a few of our servers, we would rather not have the server manager boot up every time we login. The quick and easy way to stop it from appearing when you log in is to go into Server Manager, click on the Manage menu item, then go to Server Manager Properties. Once there, simply tick on Do not start Server Manager automatically at logon. That will prevent it from starting up every time.

 

Changing the recovery mode doesn’t shrink an SQL Database log file, how to shrink logs manually.

John Leave a comment

So I found out recently that one of our servers was running out of space.  It’s our AV server so I was like what the hell, why is it running out.  Turns out it had an instance of SQL Server on there as a quarantine and configuration database.  The Virtual Machine was being backed up but not the database itself therefore no log back ups and log truncates after that.

To check exactly how much space the logs are taking up you can run the following SQL cmd:

SELECT * FROM <database>.sys.sysfiles

Or you could just as easily right-click the database and check file sizes from there.

To fix this I simply changed the recovery mode from FULL to Simple for the databases, but without a backup of the databases themselves the logs wouldn’t truncate.  Doing a backup from the right-click menu won’t truncate them either.  Since I wasn’t really worried about backing up the database itself I could just force SQL Server to truncate the logs.  I ran the following command to shrink the log file:

DBCC SHRINKFILE('<database_log>')

With <database_log> being the name of the database log file you want to shrink.  That solves that problem.

Fixing KDC Authentication Problems when upgrading your domain and forest functional level from 2003 to 2008 R2

John 11 Comments

We recently upgraded our Domain and Forest Functional Level from 2003 to 2008 R2, after a day or so I started having problems connecting to a number of 2008 R2 Hyper-V Virtual Machines. When attempting to connect I would receive the following error:
An Authentication Error Has Occurred. The Encryption Type Requested Is not supported by the KDC
At around the same time we also had one of our Exchange 2010 Transport Servers stop servicing clients, when I attempted to open the Exchange management console on the local server console ended with a HTTP server error status 500 and “Kerberos” authentication failed. So I decided to take a look through the event viewer to see what was up.

As part of Exchange there is an Active Directory Topology Service which will scan your environment for Active Directory Servers every 15 minutes or so, all of the exchange services rely on this service (if you ever have to restart all exchange services, simply restart the AD Topology Service). In the application event log I noticed the following error message:
Process MSEXCHANGEADTOPOLOGYSERVICE.EXE (PID=xxxx). Topology discovery failed, error 0×80040952 (LDAP_LOCAL_ERROR (Client-side internal error or bad LDAP message))….
There were also issues with the Exchange STORE service with the following two events:
Process STORE.EXE (PID=xxxx). All Global Catalog Servers in forest DC=xxx,DC=xx,DC=xx are not responding.
Process STORE.EXE (PID=xxxx). All Domain Controller Servers in use are not responding
The rather simple resolution to all this trouble is simply to restart the KERBEROS DISTRIBUTION KEY or KDC service on all Domain controllers. While simply restarting the Service will solve the problem, you’re probably better off just doing a proper restart after upgrading your functional levels, only from 2003 to 2008 / 2008 R2.

FortiAnalyzer shows xx of xx database tables need to be upgraded

John Leave a comment

So I was recently presented with a message after a firmware upgrade that a number of database tables needed to be upgraded.  I searched the Fortinet knowledge base but found nothing. After a discussion with a Support Rep from Fortinet we quickly found the solution and being that it is a very common issue I thought I would post the steps on here for the general public.

Log onto your Analyzer and follow the below instructions.

– System – > Config -> SQL Database set the location to “Disabled”
– run on the CLI “execute sql-local remove-db” and confirm
– On the CLI run “execute reset sqllog transfer”
– SQL Database reactivate

Depending on how large your logs are the remove-db command make take several minutes.

Update WSUS 3.0 SP2 to support Windows 8 and Windows Server 2012 Clients

John Leave a comment

Just a quick one today.  Microsoft have released an update for those running WSUS 3.0 SP2 which allows you  to provide updates to clients running Windows 8 and Server 2012.  The update is available at this Knowledge Base Article for both 32 and 64 bit environments.

Also, no word yet on when they will be releasing a patch for the IE flaw (see here and here), but should be available over the next few days.

Admin SVC Error when trying to install SharePoint solutions (WSP) and how to fix it

John 1 Comment

Short and sweet post about deploying SharePoint solutions.  You usually do this by invoking Install-SpSolution under the SharePoint PowerShell, but sometimes you’ll get the following error:

Install-SPSolution: Admin SVC must be running in order to 
create deployment timer job

All you need to do is to go to the Services section in Control Panel\System and Security\Administrative Tools and look for the service called SharePoint 2010 Administration and start it. You might also find that the service is on manual start-up, you can optionally change it to automatic to make sure that the service is always available and ready. Simple isn’t it!

What is new with Hyper-V in Windows Server 2012

John Leave a comment

I’ve been reading a lot lately about what people have been saying about Microsoft’s latest go at a hypervisor and many say that now with Hyper-V 2012, Microsoft is catching up to VMware in the enterprise virtualization arena (one example from ZDnet).  So what is all the fuss about…really.  I’ll identify some of the key points of what Hyper-V 2012 is bringing to the virtualization world. To summarise:

  • 32 Virtual CPUs and 512GB to a Virtual Machine
  • VHDX File Format for Virtual Hard Disks (16TB of Storage)
  • Native NIC teaming
  • 64 Node Clustering
  • Cluster-Aware updating
  • Hyper-V Replica (Replication of Virtual Machines)

Two of my favourite additions though are the Cluster-Aware Updating and Hyper-V Replica features.
Cluster-Aware Updating helps to manage down time by maintaining availability during windows update time, so you pretty much schedule when you want the updates to occur and let the cluster take care of the rest, automatically moving Virtual Machines,  maintaining services and availability and then moving them back once the server has restarted and updates are completed. Hyper-V Replica, which performs asynchronous replication of VMs to a replica site (either stand alone or cluster).

The last point has been my biggest gripe with using Hyper-V, how do we achieve DR with a relative low-cost.  We already have all the hardware but the licensing and software costs to perform replication were astronomical.  Now our DR solution is quite straight forward and cost-effective thanks to Server 2012.

This post is by no means exhaustive, but simply the features that stand out to me, and the reasons as to why we are planning to move to Windows Server 2012 and Hyper-V 2012.