I was recently helping out an old work colleague who were having issues with their CRM software and sending bulk emails through their Exchange 2010 server. After around ten minutes they would receive an error message with connection closed gracefully. They would then have to restart their mail out and need to monitor it for this issue to ensure it didn’t get stuck every ten minutes with this error.
After poking around their receive connectors on their hub transport server I noticed that the particular connector (for their internal applications) they were using had a connection time out of 10 minutes, which would result in us receiving the connection closed gracefully error from our end user application. The fix for this is to simply increase the ConnectionTimeout value for our Recieve Connector to anything reasonable, for us it is 3 hours. We would do by issuing the following Exchange PowerShell command:
Set-ReceiveConnector "Internal Connector - Synergetic" -ConnectionTimeout 03:00:00 -ConnectionInactivityTimeout 01:00:00
You will want to make sure that your receive connector is protected (i.e. is only set to allow internal networks) as this basically allows anything to stay connected for up to 3 hours so it could be abused.
Windows Server 2012 brings some great improvements to Virtualization. We’re currently running it in production and it works wonders, especially with the new Hyper-V Replica feature which is great and free way of implementing DR. So now you’ve got that cluster running, how can we tell if it is performing well. I recently stumbled upon PAL or Performance Analysis of Logs Tool, which is freely available from http://pal.codeplex.com/. The tool comes with a Performance Monitor Template file that we can use to easily record the performance of a Hyper-V 2012 Server and get some visibility into our Servers.
When it comes to Performance things we would want to look at would be Disk Performance, Memory and CPU but understanding what effect these have on Hyper-V isn’t always straightforward, thankfully PAL helps us being making it easy and straightforward. So first off go and download it from codplex and install it onto your workstation on desktop PC (not on your Hyper-V Host). Once installed, open it up. We need to get our Perfmon template so go into the Threshold File tab and select ”Microsoft Windows Server 2012 Hyper-V” from the Threshold file title, then choose Export to Perfmon template file. Get this onto your Hyper-V Host and open up Performance Monitor on the host we want to watch.
Open up Performance Monitor on our Hyper-V Host and open up Data Collector Sets, right click on User Defined and select New Data Collector Set. We can call it Hyper-V Performance Counter Set or what every you wish, keep Create from a template (Recommended) selected. Click next and then browse, selecting the XML Template file that we exporter earlier from PAL. Once you finish the wizard it will show up under the User Defined Data Collector Sets. You can now open up the properties and set it to run from 30 minutes to 1 hour depending on what you’re after. It is best to run this during peak load times (so business hours).
Now that you’ve collected the data you want to browse to it and copy the .blg file to your workstation (or where ever you installed PAL). Back to PAL, select the log file under the Counter Log tab, then move onto the Questions Tab where you need to specify things such as processors, total RAM, drive configuration to help specify thresholds for your performance report. Navigate to the other tabs if you wish to modify anything else and finally execute the report, this can be time consuming depending on how long you’ve captured performance data for (it took me around 20 minutes for a 1 hour capture). Once it has completed generating the report it opens up in your web browser. Key things to look at are disk latency (as per the graph with RED being critical), memory and CPU issues.
That is a free and easy way to check on the health and performance of your new Hyper-V 2012 Cluster. You can also use PAL for a Windows Server 2008 R2 Hyper-V Cluster and other services such as Exchange or SQL Server, you just need to select the right template.
For quite a few of our servers, we would rather not have the server manager boot up every time we login. The quick and easy way to stop it from appearing when you log in is to go into Server Manager, click on the Manage menu item, then go to Server Manager Properties. Once there, simply tick on Do not start Server Manager automatically at logon. That will prevent it from starting up every time.
So I found out recently that one of our servers was running out of space. It’s our AV server so I was like what the hell, why is it running out. Turns out it had an instance of SQL Server on there as a quarantine and configuration database. The Virtual Machine was being backed up but not the database itself therefore no log back ups and log truncates after that.
To check exactly how much space the logs are taking up you can run the following SQL cmd:
SELECT * FROM <database>.sys.sysfiles
Or you could just as easily right-click the database and check file sizes from there.
To fix this I simply changed the recovery mode from FULL to Simple for the databases, but without a backup of the databases themselves the logs wouldn’t truncate. Doing a backup from the right-click menu won’t truncate them either. Since I wasn’t really worried about backing up the database itself I could just force SQL Server to truncate the logs. I ran the following command to shrink the log file:
With <database_log> being the name of the database log file you want to shrink. That solves that problem.
We recently upgraded our Domain and Forest Functional Level from 2003 to 2008 R2, after a day or so I started having problems connecting to a number of 2008 R2 Hyper-V Virtual Machines. When attempting to connect I would receive the following error:
An Authentication Error Has Occurred. The Encryption Type Requested Is not supported by the KDC
At around the same time we also had one of our Exchange 2010 Transport Servers stop servicing clients, when I attempted to open the Exchange management console on the local server console ended with a HTTP server error status 500 and “Kerberos” authentication failed. So I decided to take a look through the event viewer to see what was up.
As part of Exchange there is an Active Directory Topology Service which will scan your environment for Active Directory Servers every 15 minutes or so, all of the exchange services rely on this service (if you ever have to restart all exchange services, simply restart the AD Topology Service). In the application event log I noticed the following error message:
Process MSEXCHANGEADTOPOLOGYSERVICE.EXE (PID=xxxx). Topology discovery failed, error 0×80040952 (LDAP_LOCAL_ERROR (Client-side internal error or bad LDAP message))….
There were also issues with the Exchange STORE service with the following two events:
Process STORE.EXE (PID=xxxx). All Global Catalog Servers in forest DC=xxx,DC=xx,DC=xx are not responding.
Process STORE.EXE (PID=xxxx). All Domain Controller Servers in use are not responding
The rather simple resolution to all this trouble is simply to restart the KERBEROS DISTRIBUTION KEY or KDC service on all Domain controllers. While simply restarting the Service will solve the problem, you’re probably better off just doing a proper restart after upgrading your functional levels, only from 2003 to 2008 / 2008 R2.
So I was recently presented with a message after a firmware upgrade that a number of database tables needed to be upgraded. I searched the Fortinet knowledge base but found nothing. After a discussion with a Support Rep from Fortinet we quickly found the solution and being that it is a very common issue I thought I would post the steps on here for the general public.
Log onto your Analyzer and follow the below instructions.
- System – > Config -> SQL Database set the location to “Disabled”
- run on the CLI “execute sql-local remove-db” and confirm
- On the CLI run “execute reset sqllog transfer”
- SQL Database reactivate
Depending on how large your logs are the remove-db command make take several minutes.
Just a quick one today. Microsoft have released an update for those running WSUS 3.0 SP2 which allows you to provide updates to clients running Windows 8 and Server 2012. The update is available at this Knowledge Base Article for both 32 and 64 bit environments.
Also, no word yet on when they will be releasing a patch for the IE flaw (see here and here), but should be available over the next few days.
Short and sweet post about deploying SharePoint solutions. You usually do this by invoking Install-SpSolution under the SharePoint PowerShell, but sometimes you’ll get the following error:
Install-SPSolution: Admin SVC must be running in order to
create deployment timer job
All you need to do is to go to the Services section in Control Panel\System and Security\Administrative Tools and look for the service called SharePoint 2010 Administration and start it. You might also find that the service is on manual start-up, you can optionally change it to automatic to make sure that the service is always available and ready. Simple isn’t it!
I’ve been reading a lot lately about what people have been saying about Microsoft’s latest go at a hypervisor and many say that now with Hyper-V 2012, Microsoft is catching up to VMware in the enterprise virtualization arena (one example from ZDnet). So what is all the fuss about…really. I’ll identify some of the key points of what Hyper-V 2012 is bringing to the virtualization world. To summarise:
- 32 Virtual CPUs and 512GB to a Virtual Machine
- VHDX File Format for Virtual Hard Disks (16TB of Storage)
- Native NIC teaming
- 64 Node Clustering
- Cluster-Aware updating
- Hyper-V Replica (Replication of Virtual Machines)
Two of my favourite additions though are the Cluster-Aware Updating and Hyper-V Replica features.
Cluster-Aware Updating helps to manage down time by maintaining availability during windows update time, so you pretty much schedule when you want the updates to occur and let the cluster take care of the rest, automatically moving Virtual Machines, maintaining services and availability and then moving them back once the server has restarted and updates are completed. Hyper-V Replica, which performs asynchronous replication of VMs to a replica site (either stand alone or cluster).
The last point has been my biggest gripe with using Hyper-V, how do we achieve DR with a relative low-cost. We already have all the hardware but the licensing and software costs to perform replication were astronomical. Now our DR solution is quite straight forward and cost-effective thanks to Server 2012.
This post is by no means exhaustive, but simply the features that stand out to me, and the reasons as to why we are planning to move to Windows Server 2012 and Hyper-V 2012.
If you’re in an enterprise environment, chances are that you have a KMS server running (usually either Windows 7 or Server 2008 R2). So now that Windows 8 and Server 2012 has hit VLSC for SA customers you’ll most likely want to begin testing and performing pilot deployments but we need to get our Key Management Server to accept these new clients. If you have tried to activate your KMS keys with existing KMS hosts you will receive the following error message:
Error: 0xC004F050 The Software Licensing Service reported that the product
key is invalid.
Thankfully for us, Microsoft has released an update that will upgrade Windows 7 or Server 2008 R2 KMS Hosts to support Windows 8 and Windows Server 2012, you can view the KB article by clicking here (KB2691586). You will need to request the hotfix from the KB. Once downloaded, open up an elevated command prompt and execute the update (which acts as a windows update package). When it finishes installing you will need to restart your server.
Now that our server is back up, we need to replace the KMS Host key with one for 2012 or Windows 8. Open an elevated command prompt and enter slmgr.vbs /upk which will show us Uninstalled product key successfully when complete. Now we install our new key with slmgr.vbs /ipk product-key-here
You should now be presented with a product activated successfully window. You can also run
slmgr.vbs /dlv and under description you should see VOLUME_KMS_2012. Now you can start to activate your new Windows 8 and Server 2012 clients.