11 sierpnia 2011

Why SharePoint 2010 parallel content database upgrade can make your life difficult?

Parallel means faster upgrade

I've been faced with the upgrade of ~200 content databases (~2TB) from MOSS 2007 to SharePoint 2010. Content databases will be hosted on four different SP2010 farms, and the method chosen for the content upgrade was attach-database. This method is using Mount-SPContentDatabase command to attach and upgrade particular database. If upgrade fails for some reason you can restart it by using Upgrade-SPContentDatabase (it worked pretty well for me). Single database can take a significant time to upgrade (e.g. 50GB database took 4h), but the migration time depends on the number of factors (e.g. hardware, network latency, SharePoint data characteristics, disk space available).

The first attempt to tackle the task was by doing sequential upgrade (I just forgot that I can do it in a parallel way, good for me :)). After fixing problems with the content described here and here, I got to the point where I was able to upgrade all content databases without any failures (single error causes upgrade being marked as failed). These were the happy times.

However, content migration was taking to much time, so as being proactive I picked parallel upgrade approach. My inner voices were sceptic, however reading Microsoft documentation made me more confident.

You can attach and upgrade multiple databases at a time to speed up the upgrade process overall. The maximum number of parallel upgrades depends on your hardware. ....  Faster upgrade times for your overall environment.

I didn't get lucky enough

Hurray...., I said to myself. I've opended four poweshell sessions on the single front-end server and run my batch files full of Mount-SPContentDatabase calls. I was upgrading four content database at the same time. To my surprise, I've received a bunch of errors during this upgrade:

Error Type Error Message

Feature upgrade failure

Feature upgrade incomplete for Feature 'Fields' (Id: 'ca7bd552-10b1-4563-85b9-5ed1d39c962a') in Site '[SiteURLHere]'. Exception: A transport-level error has occurred when sending the request to the server. (provider: TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.)
Database schema upgrade failure Action of Microsoft.SharePoint.Upgrade.SPContentDatabaseSequence failed. Exception: A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - The specified network name is no longer available.)
Upgrade failure This upgrade session has been stopped. Possible causes include the process being terminated abruptly or the OS has rebooted. Please restart the upgrade again.
Upgrade completed successfully Content database is being shown on Check upgrade status page (http://CentralAdminUrl/_admin/UpgradeStatus.aspx) as migrated, however it's not. This  can be verified on the  Review database status page (http://CentralAdminUrl/_admin/DatabaseStatus.aspx).

When accessing SharePoint sites  from not-upgraded content database  through API  you can get the following error:

Microsoft.SharePoint.Upgrade.SPUpgradeCompatibilityException: There is a compatibility range mismatch between the Web server and database "Content database name", and connections to the data have been blocked to due to this incompatibility. This can happen when a content database has not been upgraded to be within the compatibility range of the Web server, or if the database has been upgraded to a higher level than the web server. The Web server and the database must be upgraded to the same version and build level to return to compatibility range

So, does parallel upgrade sucks?

I was unpleasantly surprised to see random TCP stack errors, database schema upgrade failures appearing on random content databases. I suspected that my fellow "parallel upgrade" was causing all the hassle. I've restored content databases that have experienced failures to the MOSS 2007 version. I rerun upgrade sequentially (only one content database being upgraded on the SharePoint farm) and all errors where gone.

I'm not saying "parallel upgrade" is the ultimate evil. It just need to be handled carefully. Performance of the SQL Server IO subsystem, network latency and memory pressure on front-end, back-end servers needs to be monitored. And if need, hardware or system settings will have to be adjusted.

Or you can just stick to the sequential upgrade to be on the safe side during the "migration weekend". At least this will not increase the risk getting yourself into trouble you won't have time to solve.

Hope this helps.

Brak komentarzy: