Recently a colleague of mine hit some trouble with using the SharePoint 2007 Search API and I was tasked with doing a bit of investigation into a work around.
What we were trying to do was create a custom search page that has a meta-data property on it that can be multi-selected. It has almost 200 options in it and the user can select a combination of these to filter the search by.
This worked fine with a few selected but somewhere around 10 selections the query started returned the error 'Malformed Query' as soon as it was executed.
The search was performed against a custom scope and custom managed properties. After a bunch of investigation it appears the issue is solved if you tick the "Allow this property to be used in scopes" option. After selecting that for the field/s that you need to have as conditions with 10+ options the query processes fine each time.
This sounds like a bug to me as we are not actually using the property for defining a scope, but the server we are running on is update to date with current patches and hotfixes so looks like one that is yet to be fixed, luckily the work around is easy :)
The below sample shows the query and code being used with a few field name changes.
Query was like the below but with lots more OR'ed values for Field 1:
SELECT Field1, Field2, ModifiedBy, Created FROM portal..scope() WHERE \"SCOPE\" = 'Custom Scope' AND (ContentType = 'ContentType1' OR ContentType = 'ContentType2') AND (Field1 = 'test' OR Field1 = 'test2' )
FullTextSqlQuery sqlQuery = new FullTextSqlQuery(site);
sqlQuery.ResultTypes = ResultType.RelevantResults;
sqlQuery.QueryText = sql;
sqlQuery.TrimDuplicates = false;
sqlQuery.StartRow = startRow;
sqlQuery.RowLimit = rowLimit;
ResultTableCollection results = sqlQuery.Execute();