This is excellent news..! Reposted from the Microsoft SharePoint Product Group Team Blog.

Today we are announcing two related changes to the way we describe data storage in SharePoint. First, by taking advantage of performance and reliability improvements in SP1 and by defining specific requirements for large data storage in SharePoint, Microsoft is able to increase the supported limits for data storage in SharePoint.

Additionally, we are announcing that the SQL Server FILESTREAM RBS provider is now supported for use with SharePoint so that lower cost iSCSI connected NAS disk can be used. This post outlines the new data storage support limits and guidelines for scaling to those limits and it defines RBS including the new FILESTREAM RBS provider.

The SharePoint Content Database Data Size Limit

With the release of SharePoint 2010 SP1 and some new guidance we are changing the supported data size limits for SharePoint content databases. Prior to SP1 the content database limit was 200 GB for collaboration and 1 TB for document archive. The content database size includes both metadata and BLOBs regardless of where the BLOBs are located and use of RBS does not bypass or increase these limits.

The new guidance for supported content database size details outlines specific guidance for SharePoint administrators as the data size grows. If this new guidance is followed SharePoint can support up to 4 TB of data in all usage scenarios and has no imposed size limit for document archive scenarios. The details are in the TechNet document SharePoint Server 2010 capacity management: Software boundaries and limits and the primary changes are:

1. For a SharePoint content database up to 200 GB there are no special requirements and this limit is included for consistency.
2. For a SharePoint content database up to 4 TB you need to additionally plan for the following two requirements:

   • Requires disk sub-system performance of 0.25 IOPS per GB, 2 IOPS per GB is recommended for optimal performance.

   • Requires the customer to have plans for high availability, disaster recovery, future capacity, and performance testing.

   • And you need to review additional considerations in the TechNet Boundaries and Limits article.

3. For a SharePoint content database over 4TB specifically for a Document Archive scenario you are required to additionally plan for the following:
   • SharePoint sites must be based on Document Center or Records Center site templates and must be an archive scenario where less than 5% of content is actively read from each month and less than 1% of content is actively written to.
   • Do not use alerts, workflows, link fix-ups, or item level security on any SharePoint objects in the content database. Note: document archive content databases can be the recipient of documents as a result of Content Routing workflow.
4. Other specific limits changes being made at the same time:
   • A new limit of 60million items in any one SharePoint content database
   • The specific 5 TB limit per SQL Server instance has been removed. Instead you should work with a SQL Server professional to plan for database storage.

Please review the full TechNet Article SharePoint Server 2010 capacity management: Software boundaries and limits document. We have also published a guide on SharePoint 2010 scalability here: http://go.microsoft.com/fwlink/?LinkId=223599. In the near future we will publish a test report of large scale testing that supports these new size limits.

The Value of Remote Blob Store with SharePoint

RBS (Remote Blob Store) is a set of standardized APIs that allow storage/retrieval of BLOBs (binary large object data) outside of your main SQL database where a dedicated BLOB store is desirable. RBS uses a provider model for plugging in any dedicated BLOB store that implements the RBS APIs. RBS was introduced in SharePoint 2010 and providers can be installed into SharePoint and are used to store BLOBs. Documents in SharePoint document libraries are BLOBs and with RBS they can be stored remote to the SQL Server database. This commonly means the BLOBs are stored on the same machine as SQL Server though they may be on a network connected SQL Server machine.

Above are two diagrams showing common architectures for SharePoint using RBS. Both show the RBS Client Provider which is installed on the SharePoint Web Front End. The left diagram shows the generic RBS implementation where a third party has implemented RBS to access their storage. The right diagram shows the SQL Server FILESTREAM RBS provider which stores blobs in the Windows file system.

By storing BLOBs outside of the SQL Server database there can be certain advantages such as:

· RBS enables SharePoint Foundation 2010 running on SQL Express to store more data than the SQL Express limit of 4 GB. In SQL Express 2008 R2 this limit was increased to 10 GB.

· Some operations can be performance optimized with average blob sizes over 1Mb. This result is from tests with the SQL RBS Provider. Ref: http://msdn.microsoft.com/en-us/library/cc949109(SQL.100).aspx

· There could be storage optimizations with potential disk space and disk cost savings from differential backups or tiered storage.

· We have completed testing on the SQL RBS FILESTREAM provider which can enable iSCSI connected storage for RBS use. Using iSCSI allows for the use of lower cost NAS storage.

· Other potential data optimizations may be developed by ISV’s using the supported public RBS APIs and SharePoint APIs.

There are a few things to be careful with when implementing RBS:

· Backup strategy must be carefully considered. Both document metadata and document BLOBs must be backed up at exactly the same point in time. This means any third party backup solution needs to be capable of restoring both the SQL database used by SharePoint and the BLOBs used by SharePoint as a set where no variance occurs which would have the database reference BLOBs that are not available from the same backup.

· RBS is most likely to be used for document archive scenarios where documents are written and not updated. BLOBs in RBS are never updated once they are written; instead a new BLOB is created for any update. BLOBs are immutable, old BLOBs are garbage collected later. You can read more about RBS garbage collection in this article: http://technet.microsoft.com/en-us/library/ff628583.aspx

· RBS providers are required to return the first byte of data in a request in 20ms. This applies for all requests between SharePoint and the RBS provider storage layer.

· The SharePoint database is not intended to be read from or written to except by SharePoint. RBS providers don’t have separate access to the data. This includes direct access to blobs. Ref: http://support.microsoft.com/kb/841057/en-us

· Performance may decrease for smaller BLOB sizes when using RBS. This is also shown in the “FILESTREAM Storage in SQL Server 2008” article referenced above.

· There are many RBS providers available and customers should evaluate them for suitability for their implementations.

Additional Documentation from Microsoft on RBS in SharePoint

TechNet Documentation RBS Links:

· Plan for RBS (SharePoint Server 2010) [http://technet.microsoft.com/en-us/library/ff628583.aspx]

· Overview of RBS (SharePoint Server 2010) [http://technet.microsoft.com/en-us/library/ee748649.aspx]

· Maintain RBS (SharePoint Server 2010) [http://technet.microsoft.com/en-us/library/ff943565.aspx]

Q & A

· Q: How come you couldn’t provide these increased data limits when SharePoint 2010 launched?

· A: We have learned more about how customers implement document archive solutions on SharePoint in the past 12 months. Now by providing specific guidance around data size scaling and focusing supportability around those we can have an increased data size limit for SharePoint and avoid having a data size limit for the document archive scenario.

· Q: What is the new data size limit for document archives on SharePoint

· A: There isn’t a data size limit, though the new guidance factors for building supportable large scale systems must be followed. If the additional factors are not properly addressed then the lower supportability limit applies.

· Q: What if I really need more than 4 TB on a SharePoint farm and it isn’t document archive?

· A: You should use a scale out topology. This involves having multiple content databases in a single farm and spread sites out amongst them. Each content database can grow to 4 TB by following the guidance.

· Q: What if I had incorrectly assumed the 200 GB limit could be avoided by moving BLOBs to a Remote Blob Storage provider thereby reducing the amount of SQL Server stored data for SharePoint?

· A: We recommend that you upgrade to SharePoint 2010 SP1 and follow the new guidance for the total size you have. Consult the company you purchased your RBS provider from to ensure they are tested with SharePoint 2010 SP1. If you have a deployment that falls outside of the new and old limits we recommend you to contact Microsoft Support and request a supportability review. This is a paid support review and the support engineer will be able to tell you if your current implementation can be supported or if changes to reduce the data per content database are recommended.

· Q: Since NAS is supported, does the SQL Server RBS FILESTREAM provider allow use of a network share to store BLOBs on?

· A: No, NAS must be connected using iSCSI and appear as a local drive on the SQL Server machine.

· Q: Will the content database size limit or the 20mS TTFB limit be enforced in the software?

· A: No. These are support limits that we recommend customers stay within for best performance and in order to get the best support from Microsoft. They are not hard boundaries that are measured by the SharePoint software.

· Q: Where was the old 200 GB limit detailed on TechNet?

· A: It was listed on the SharePoint Capacity Planning Boundaries and Limits page on TechNet. Whilst RBS and BLOBs were not previously specifically called out, the limit of 200 GB was clearly stated for a SharePoint Content Database which includes metadata and BLOBs. This article has been updated for the new limits and to list RBS to be more explicit and to avoid any future misinterpretation.

· Q: Can a large document archive have multiple SharePoint sites collections?

· A: Yes. However our guidance is that if you have a site collection over 100 GB, it should be the only site collection in a content database.

· Q: Can a large document archive have multiple document libraries?

· A: Yes. You can have multiple document libraries with different permissions set.

· Q: Is SharePoint 2010 SP1 required to take advantage of these new content database limits?

· A: No. The limits apply to SharePoint 2010 regardless of whether SP1 is applied. However due to improvements in SharePoint 2010 SP1 you are strongly encouraged to install it.

Thanks

Kris Wagner