Searching Database Content with Fast Search for SharePoint 2010 – Part 1

Searching Database Content with Fast Search for SharePoint 2010 – Part 1

The search story is much better in 2010 for SharePoint. There’s a new Search Engine available deploy with SharePoint Server 2010 – FAST Search for SharePoint 2010 (FS4SP). I really could have used FAST Search for SharePoint in my past Enterprise SharePoint Search projects. It produces significantly better relevant results than the MOSS 2007 search engine. It has many more options available for tweaking the search UI and search engine. It also has better options available for developers developing and extending custom search solutions.

Why is this quality search engine now available to deploy with SharePoint? Microsoft acquired FAST Search & Transfer and its FAST ESP product for $US 1.2 Billion in early 2008. This is a common Microsoft approach at getting ahead in a market, acquire an innovative company with a quality product. The same happened with MCMS 2002 (the forerunner to SharePoint WCM) which was based on a product gained from the NCompass Labs acquisition.

Microsoft Announces Offer to Acquire Fast Search & Transfer.

Microsoft Completes Tender Offer for FAST Search & Transfer

Artistitic re-enactment of the acquisition.

Since the FAST acquisition, Microsoft has had the time to bake in a version of FAST ESP into the next version of SharePoint, and it’s time to make use of this awesome new enhancement to the platform.

Options for Searching Database Content with FAST Search for SharePoint 2010

So you want to search custom SQL Server databases using FAST?

If you have never indexed anything other than SharePoint Sites with FS4SP, I recommend first checking out this page on TechNet:

Plan and design for content collection (FAST Search Server 2010 for SharePoint)

It gives a great summary of the options available to you, and links to further TechNet pages containing step-by-step details on connecting FAST to File Shares, Exchange, People Profiles, Web sites, Lotus Notes and more.

This page on TechNet lists two options for using FS4SP to crawl and search databases:

Business Data Catalog-based indexing connectors

• Use if the preferred configuration method is using the Microsoft SharePoint Designer 2010.
• Use when you want to use time stamp based change detection for incremental database crawls.
• Use when the preferred operation method is using the Microsoft SharePoint Server 2010 Central Administration.
• Use when you want to enable crawling based on the change log. This can be achieved by directly modifying the connector model file and creating a stored procedure in the database.

FAST Search database connector

• Use when the preferred configuration method is using SQL queries.
• Use when you want advanced data joining operation options through SQL queries.
• Use when you want to use advanced incremental update features. FAST Search database connector uses checksum based change detection for incremental crawls if there is no update information available. The connector also supports time stamp based change detection and change detection based on update and delete flags.

That’s right, you’ve got not just one, but two approaches that you can use with FAST Search for SharePoint to make this possible. Why are there two ways to do this?

The preferred method of integrating database content and LOB systems with SharePoint 2010 is using the BCS. You can also use the BCS to crawl databases to use in SharePoint 2010 Search and FS4SP. It also provides extra goodies for use in SharePoint such as external lists. It also provides a nicer user interface to get the job done.

The second option available to use is a port of the FAST ESP JDBC database connector. FAST ESP had to have a way to connect to databases before a port of FAST ESP was implemented in SharePoint, it had the JDBC connector. In comparison to the BCS, the JDBC connector doesn’t have as nice a gui to setup the connection and crawling. It also requires installation of the 32 bit Java JRE, That’s right folks, you need to have a JVM on your FAST server to get the JDBC connector to work! A few other ‘FAST Search specific’ connectors include the web connector and Lotus Notes connector.

More on these 'FAST Search Specific' connectors:

Plan to deploy FAST Search specific connectors (FAST Search Server 2010 for SharePoint)

Parts 2 of this series details steps to get FS4SP to Index a SQL Server 2008 products database using the FAST Search JDBC Database Connector.

Part 3 of this series detail steps to get FS4SP to Index a SQL Server 2008 products database using the BCS Connector.

Part 4 Comparison of the BCS and FAST Search database connector. [stay tuned...]

Working on FAST Search for SharePoint 2010 - Watch this space

I've been busy on a number of FAST Search for SharePoint 2010 (FS4SP) projects recently and I'm planning on blogging about some of my experiences in the comming weeks and months, so stay tuned folks.

Perth SharePoint User Group Webcasts!

We've been recording our recent Perth SharePoint User Group meeting presentations.

Jeremy Thake is encoding and uploading these to the Internet for the world to see what we're getting up to in our user group here in Perth, Western Australia.

Upgrading from MOSS 2007 to SharePoint Server 2010 Sezai Komur

http://www.sharepointdevwiki.com/display/SPPodCasts/2010/05/14/SPWebCast+014+-+SharePoint+2010+Upgrade+at+Perth+SharePoint+User+Group

Build SharePoint 2010 Solutions with Visual Studio 2010 Michael Hanes

http://www.sharepointdevwiki.com/display/SPPodCasts/2010/05/28/SPWebCast+017+-+Build+SharePoint+2010+Solutions+with+Visual+Studio+2010

Annual Leave Form - InfoPath 2010 BCS SPD 2010 Workflow and SP2010 Matt Menezes

http://www.sharepointdevwiki.com/display/SPPodCasts/2010/05/28/SPWebCast+016+-+Annual+Leave+Form+-+InfoPath+2010+BCS+SPD+2010+Workflow+and+SP2010

SharePoint 2007 and HP TRIM 7 Integration Overview Jeremy Thake

http://www.sharepointdevwiki.com/display/SPPodCasts/2010/04/17/SPWebCast+011+-+SharePoint+2007+and+HP+TRIM+7+Integration+Overview

You can view a variety of other web casts on the SharePointDevWiki here: http://www.sharepointdevwiki.com/display/SPPodCasts/Home

My New Job at CSG

I've just started a new job as a SharePoint Solutions Architect working at CSG http://www.csg.com.au

CSG is a large Australian IT company and I'm looking forward to implementing some big SharePoint Server 2010 projects here.

I'll also try and focus more time on blogging and community stuff which I've been neglecting in recent months.

Wish me luck!

Perth SharePoint User Group - September Presentation

I presented at the Perth SharePoint User Group September meeting.

The PowerPoint for the presentation can be downloaded here.

It was a blast! thanks to everyone who attended and the questions about SPDocGen, Database Management, Creating Explicit Managed Paths, Database Autogrowth settings and SharePoint Object Model Disposal.

SPDocGen - SharePoint Documentation Generator - Full Source Code Uploaded

I've done some minor tweaks to SPDocGen, added some console output and a little error handling.

I've also uploaded the full source code for the solution, so feel free to modify it as you need.

SPDocGen - SharePoint Documentation Generator

Here's some ideas I have for further improving and building on the solution:

1. Implement command line parameters to let users specify what is output.
2. Create a winforms UI to allow users to specify what is output, and specify the XML, XSLT and DOC filenames.
3. Document even more Farm level information...
4. SPDocGen currently produces Farm Level documentation, the next step is to produce detailed Site Collection level documentation with detailed reporting of Site Collection specific info such as Content Types, Site Columns, Lists etc. etc.

Post a comment here if you're interested in being involved and collaborating on this further.

SPDocGen - SharePoint Documentation Generator

I've just created this project on CodePlex http://spdocgen.codeplex.com/

SPDocGen aka. SharePoint Documentation Generator is a console app that generates WSS 3.0 and MOSS 2007 farm documentation or reports using the SharePoint object model api. It produces an XML file, which is then transformed into a .doc file using XSLT and WordML.

I started this project to save myself time, because I've been tasked with documenting SharePoint Deployments quite a few times, and its usually a time consuming process of clicking through Central Admin, running batch commands of STSADM or using SharePoint Manager then copy and pasting the info into a word document. Service Pack 2 also has a preupgradecheck command that can produce some documentation for you. It might also be worth having a look at the SharePoint Admin Toolkit, SPSReport, SPDiag or MOSSRAP

Hope it saves everyone some documentation time so you can get back to spending your time on more worthwhile tasks like SharePoint dev or SharePoint admin.

Get SPDocGen

SharePoint 2010 Sneak Peek

Microsoft have released some sneak peek info on SharePoint 2010!

Go check out the movies and info here: http://sharepoint.microsoft.com/2010/Sneak_Peek

Microsoft plan to release a lot of info at the SharePoint conference in Vegas in October, http://www.mssharepointconference.com it's nice to get this sneak peek info while we wait for the conference.

Can't Crawl Sites, Can't Access Sites on Server 401.1 Access Denied

I hit this nasty problem recently.

I could access central admin fine on http://servername:port but when trying to crawl sites locally, the crawler failed on sites with a custom host header, returning access denied messages. That's strange I thought, so launched up IE on the server and tried to access the sites, get prompted to login 3 times then boom access denied 401.1. These same sites work just fine when accessed on other client machines in the network.

Something definetly aint right here!!!

Luckily Todd Klindt saved the day for me, he blogs this issue nicely here. Further to that Ishai Sagi also has a blog post on it. So have a read of those awesome blog posts and then check out http://support.microsoft.com/kb/896861 for a resolution, the cause is stated in the hotfix as:

"A loopback check security feature that is designed to help prevent reflection attacks on your computer. Therefore, authentication fails if the FQDN or the custom host header that you use does not match the local computer name."

The resolution involes a little bit of regedit.

Phew disaster averted!

Perth SharePoint User Group Lead

I am the new Lead of the Perth SharePoint User Group !

http://www.sharepointusers.org.au/Perth/Lists/Announcements/DispForm.aspx?ID=23

Rick Rosato of Connected Systems resurrected the Perth SharePoint User Group in early 2007 and has done a great job of running the user group since then, we’ve had some great sessions in the last two years and many people have benefitted from the user group. Rick has decided to hand over the presidency to me so he can focus on other things, and I gladly accept the position and all the responsibility that it entails – cheers Rick !

Recent Meetings & Presentations

Late last month, on the 25th March Jeremy Thake of Readify presented an excellent talk on unit testing in SharePoint, check out the webcast here.

For this months meeting, on the 21st April, the infamous Paul Culmsee of Seven Sigma Business Solutions graced the Perth SharePoint User Group as a speaker for the very first time, to present a talk on "Wicked Problems and SharePoint - Rethinking the Approach" you can view the presentation slides on Paul's blog.


If you are based in Perth and working with SharePoint, you've got to come along to our meetings!

Check out the user group website regularly for up and coming events. If you are keen on sharing your SharePoint adventures with others, get in contact with me, as we need speakers! We'll slot you in as a presenter so you can share your experiences with the community.

10 New Themes for SharePoint!

Noticed this post on Randy Drisgills blog regarding 10 new awesome themes recently released by Microsoft
http://blog.drisgill.com/2009/03/microsoft-released-10-new-sharepoint.html

They look great - best themes for SharePoint I have ever laid my eyes on. I also agree with Randy that these should have been released in Solution package(s). Still the format they are released in are better than those extra Master pages released by Microsoft a year or two ago.

Download the 10 new themes here:
http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=0a87658f-20b8-4dcc-ad7a-09ad22641f3a

Update
Daniel Brown - SharePoint MVP based out of Adelaide, South Australia has "solutionised" the themes. Check them out here - http://www.danielbrown.id.au/Lists/Posts/Post.aspx?ID=256

Go and download SPDisposeCheck !

If you are into WSS 3.0 and MOSS 2007 custom development go and download the SharePoint Dispose Checker Tool (aka SPDisposeCheck) -
http://code.msdn.microsoft.com/SPDisposeCheck

Announced by Paul Andrew at Microsoft -
http://blogs.msdn.com/pandrew/archive/2009/01/29/spdisposecheck-v1-3-1-is-released.aspx

Proper disposal of SharePoint objects in custom code is very important. If you are causing memory leaks you will create memory pressure problems in production Farms, which can result in serious side affects and system downtime.

Announcing SharePoint Site Configurator

http://www.codeplex.com/spsiteconfigurator

One controversial topic of late is - should you develop with custom site definitions? A lot of people argue no - they are too complex and will cause upgrade problems into the future.

The majority of SharePoint developers should be familiar with the SharePoint Solutions and Features framework for developing and deploying customisations to SharePoint. A smaller number of developers work with and are familiar with custom site definitions - they are complex and difficult to master. This can cause problems with resourcing as some SharePoint developers will not be productive having to sort out and work with another developers site definition.

To complicate things further - when developing with custom site definitions one can go for the miniminalist approach where most work is implemented in Features, and these Features are simply referenced in the site definition for activation purposes, or you can go ahead and build the work straight into the site definitions. Another problem is developers use the OOTB WSS and MOSS site definitions as examples, and many of those implement functionality in the site definitions themselves instead of in Features.

So it's great to see MCS release the SharePoint Site Configurator on codeplex which will hopefully help to do away with complex custom site definitions.

Isolator for SharePoint !

If you are a big fan of TDD and you develop on SharePoint then you need to check out Typemock Isolator for SharePoint - develop and test SharePoint code without a SharePoint server - how cool is that!!!

Head over to Andrew Woodward's blog - http://www.21apps.com for some more TDD goodness - specifically these two posts that link to whitepapers and code for download.
http://www.21apps.com/agile/beginners-guide-to-test-driven-web-part-development/
http://www.21apps.com/agile/unit-testing-sharepoint-getting-into-the-object-model/

Typemock are offering their new product for unit testing SharePoint called Isolator For SharePoint, for a special introduction price. it is the only tool that allows you to unit test SharePoint without a SharePoint server. To learn more click here.
The first 50 bloggers who blog this text in their blog and tell us about it, will get a Full Isolator license, Free. for rules and info click here.

Anouncing SPDisposeCheck for SharePoint Developers

http://blogs.msdn.com/sharepoint/archive/2008/11/12/announcing-spdisposecheck-tool-for-sharepoint-developers.aspx

So what is SPDisposeCheck

"SPDisposeCheck, will open your custom compiled assemblies recursively and validate them against the Microsoft published guidance. The output from the tool will contain messages that may indicate the SPSite and SPWeb Dispose() methods guidance is are not being followed in the customers source code. While these messages need expert evaluation in order to determine if the software is not performing properly, in some cases just running the tool on your custom code can lead you to simple fixes that improve the quality and performance of custom code on SharePoint. "

"This tool is planned for release during the coming North American Winter. Customers who are currently experiencing difficulties with memory management in their custom applications should review the guidance listed above. Customers who are currently experiencing difficulties with Microsoft Office SharePoint Server 2007 should contact their regular Microsoft Customer Support Services contact, or refer to http://support.microsoft.com/."



Disposing of SharePoint objects correctly is a big problem in custom developed code, if your SharePoint implementation is constantly "Dieing" ie, after regular use the memory and cpu max out and you need to reboot your server to get things going again then its highly likely you are running custom SharePoint Object Model code that is not correctly disposing of objects causing memory leaks. The rate of memory leakage can be significant.

I have since this ugly problem rear its head in a number of projects I have been involved in. So yeah when you finally do fixup the problem and dispose of objects correctly you will notice the improvement of system performance and your server should run as if it has a simple vanilla install of SharePoint.

Another major problem with this is that.... These Dispose() rules are COMPLEX. The biggest problem faced is the difference in coding patterns depending on where your custom code is running.

ie. If your SharePoint OM code is running in a Web Part or Custom Control then you have access to the SPContext.Current object which is your starting point to get to your SPSite, SPWeb, SPList, SPListItem, SPFile etc. etc. its important to note that objects accessed from the current context should not be disposed of. The SPContext.Current.Site and its SPWebs that hang of it are singletons and should not be disposed as SharePoint makes use of these and will need to possibly use these objects after your custom code executes. That means the using(SPWeb =...) pattern will dispose of objects you are not meant to, you will get an error.

BUT - if you are running the code in a console app or 'out-of-context' then you should always dispose the objects as you are the one creating them.

Confused? Complicated? Yeh it is but it is important to understand the issues if you are writing custom SharePoint Object Model code.

SharePoint MVP Blogs OPML

If you're big on RSS and use it to keep track of the latest SharePoint news out there, then here's a link to the SharePoint MVP Blogs OPML.

http://services.newsgator.com/sharepoint/opml


My friend Jeremy Thake also has a nice collection of SharePoint links on his Diigo that he seems to update quite often. Here's the RSS feed for Jeremy's SharePoint Diigo links - http://www.diigo.com/rss/user/Jthake/sharepoint

Patterns & Practises SharePoint Guidance and Ramp-Up Program SharePoint for Developers

Microsoft Patterns and Practices - SharePoint Guidance

http://msdn.microsoft.com/en-us/library/dd203468.aspx

Guidance at a Glance
This guidance helps architects and developers build SharePoint intranet applications. The guidance contains a reference implementation (RI) that demonstrates solutions to common architectural, development, and lifecycle management challenges.



Ramp-Up !

(http://www.myrampup.com/) has just launched a brand-new learning track: SharePoint for Developers, Part I today.

http://msdn.microsoft.com/en-us/rampup/default.aspx
http://msdn.microsoft.com/en-us/rampup/dd221355.aspx

Free Open Source CMS / WCM on WSS 3.0 - CompleteSharePoint.NET

My old buddy Tommy Segoro recently released CompleteSharePoint.NET a free, open source CMS / WCM system that sits on top of WSS 3.0
http://www.codeplex.com/completesharepoint
http://www.completesharepoint.net/Home/Pages/Default.aspx

Tommy is a great buddy of mine and we've worked on some cool projects together in the past - Nice work Tommy! I miss working with Tommy because he's probably one of the fastest developers I have ever worked with.

Enabling the Word 2007 Developer Tab in the Ribbon to Always Show the Document Information Panel (DIP) When Opening a Document

Last Friday I went to a special Perth SharePoint User Group presentation hosted by William Cornwill of Microsoft. The presentation was excellent and provided a nice overview on Custom Content Types, Templates and Document Information Panel interaction between SharePoint and Word 2007. Awesome presentation!

One question that came up in the user group presentation that a number of people wanted answered was –

"How do I always force the Document Information Panel to show on opening a document"

I had some trouble with this a few weeks ago until I figured out that you can add the Developer Toolbar to Word 2007 which provides the facility to configure quite a lot in the document.

I came across this excellent blog post by Andrew Coates also another Australian Microsofty like William Cornwill – and just like William, Andrew presented at the Perth SharePoint UG a few months ago.

So to enabled the Developer Toolbar in Word following the steps in Andrew's post :

Enabling the Word 2007 Developer Tab

http://blogs.msdn.com/acoat/archive/2007/02/28/enabling-the-word-2007-developer-tab.aspx

You basically go into Word 2007 options to enable the Developer tab in the Ribbon – fairly simple but it took me AGES to figure this out until I came across Andrew's blog.

After enabling the Developer Tab follow these steps to always force the Document Information Panel to show, click on the Developer Tab and then click on the last icon – Document Panel.

And then tick the Always show Document Information Panel on document on open and initial save checkbox to always show the DIP.

Its as simple as that! If you tick that checkbox and save the word document – and then use this word document as a Content Type template in a SharePoint Document Library – then every time a user creates a new document or opens a document they will be presented with the DIP showing.

UPDATE: You need to set this in the Content Type's settings page, there's a link to Document Information Panel Settings.

See - http://stephenmuller.com/2008/07/17/using-the-dip-and-quick-parts-with-office-pt-1/

The developer tab has a bunch of other useful goodies you can play with one of which is adding various content controls such as Picture content controls.

Another important function of the developer tab is Formatting quick parts. Have you ever wanted to change the date format of a Date Quick Part? Add a Quick Part to the page that is based on a Date and you get it formatted as dd/MM/yyyy.

With the quick part selected switch to the Developer tab and click the Properties button in the Controls section of the ribbon

You then have the following options available for configuring this Quick Part (Content Control).

Click Okay – and you now have applied a format to your Date Quick Part ! Other options available such as the Content cannot be edited forces users to always updated the DIP instead of the word document. Ie. By default users can type into quick parts/content controls in a word doc and changes are reflected in the DIP – with this option set users must edited this content in the DIP.

Another fine blog post on Andrew's blog that helped me recently is –

Linking Word 2007 Content Controls to Custom XML

http://blogs.msdn.com/acoat/archive/2007/03/01/linking-word-2007-content-controls-to-custom-xml.aspx

Andrew gives a nice overview of how to make use of the Word 2007 Content Control Toolkit to manipulate custom XML stored in a word document, the tool allows you to add ANY custom XML to a document and then bind it to a Content Control.

Finally in summary I came across the following EXCELLENT resource if you are implementing this sort of advanced document management in SharePoint –

Using Office Open XML Formats to Support Electronic Health Records Portability and Health Industry Standards

http://msdn.microsoft.com/en-us/library/bb879915.aspx

By the Ted Pattison Group – if those guys are involved then you know this is going to be top class.

And the Visual Studio Solution for the article is downloadable here - http://www.microsoft.com/downloads/details.aspx?FamilyId=B36AEBDC-8217-4D64-BFD0-187E58B708BD&displaylang=en

Cloud Computing - Announcing Azure Services Platform and Microsoft SharePoint Services

Ray Ozzie opened the PDC 2008 by anouncing Windows Azure - Microsoft's big solution to Cloud Computing.

To learn more about Azure go to - http://www.azure.com/

http://blogs.msdn.com/sharepoint/archive/2008/10/27/pdc-2008-announcing-azure-services-platform-and-microsoft-sharepoint-services.aspx

http://www.readwriteweb.com/archives/windows_azure.php


http://www.readwriteweb.com/archives/microsoft_azure_redefine_os.php


One problem is the name, I think it sounds too much like Azureus - http://azureus.sourceforge.net/

Azure http://en.wikipedia.org/wiki/Sky_blue "Commonly it refers to a bright blue, resembling the sky on a bright, clear day"

"The term azure derives from the Persian لاژورد (lazhward), which was the name of a place in modern Turkestan known for its deposits of lapis lazuli (“stone of lazhward”). The word was adopted into Old French by the twelfth century.
From the French it was adopted into English as a near synonym for "blue". The first recorded use of azure as a color name in English was in 1374."

Other than the name, this is exciting stuff people and likely to be the way to do work in the not too distant future.