.NET FOLLOWER

    I was asked to transfer a few SharePoint 2007 lists with their contents into a SharePoint 2010 application. Obvious steps were create the lists’ templates (.stp files) in SP 2007 and redeploy them in SP 2010. Having successfully created and uploaded the templates into the List Template Gallery (_catalogs/lt) of a 2010 Site Collection, I got the following error whenever I tried to create a new list instance based on any of the uploaded templates:

Microsoft SharePoint Foundation version 3 templates are not supported 
in this version of the product.

Fortunately, a straightforward solution was found quite quickly.

.stp File Content

A .stp file is just a .cab archive (similar to a .wsp) and, after changing the file’s extension to cab, it can be opened and viewed with any popular archiver (WinRar, 7 zip and so on).

In the .cab you’ll see at least one file, manifest.xml (frequently, there is nothing but the one). The manifest.xml contains the target list’s schema and content (if the template was created with the appropriate option). The first lines of the manifest.xml look like the following:

<?xml version="1.0" encoding="UTF-8" ?>
<ListTemplate WebUrl="http://myapplication/mysitecollection">
    <Details>
        <TemplateDescription></TemplateDescription>
        <TemplateTitle>MyList</TemplateTitle>
        <ProductVersion>3</ProductVersion>
        <Language>1033</Language>
        <TemplateID>20</TemplateID>
        <Configuration>0</Configuration>
        <FeatureId>{00BFEA71-DE22-43C2-A848-C05706800101}</FeatureId>
        <TemplateType>100</TemplateType>
        <BaseType>0</BaseType>
    </Details>
...
</ListTemplate>

Pay attention to the <ProductVersion> element containing the value 3. So, the solution mentioned above is to modify the file so that its <ProductVersion> would contain 4. Having done that, we need to recreate the .cab file with the altered manifest.xml and change the output file’s extension back to stp.

Recreate .cab

The most tricky part in the solution is to repack file or group of files into .cab as the most popular archivers don’t allow that. So, we have to do it by ourselves using the Microsoft’s makecab.exe utility usually located at C:\Windows\System32. If your template contains only manifest.xml, use the command line like the following to repack it into .cab:

makecab.exe C:\ExtractedStpFiles\manifest.xml c:\RepackedStpFiles\MyList.cab

The first argument is a path to the file to be wrapped into .cab. Here the altered manifest.xml is assumed to be in the c:\ExtractedStpFiles folder. The second argument is a path to the output archive file. Here the destination folder is c:\RepackedStpFiles.

To avoid the step with changing the output file’s extension from cab to stp, you can modify the above command line to

makecab.exe C:\ExtractedStpFiles\manifest.xml c:\RepackedStpFiles\MyList.stp

If the list template contains more than one file inside, first of all, we need to prepare a directive file (.ddf file) containing instructions for makecab.exe how to compress and package the files. Each instruction starts with “.”, comment starts with “;”. For our task such file may look as follows (let’s name it MyList.ddf and place in C:\ExtractedStpFiles):

.OPTION EXPLICIT
 
.Set CabinetNameTemplate=MyList.stp ; Name of the output .cab file, in our case we can specify the file's extension as .stp
.Set Cabinet=on
.Set CompressionType=MSZIP ; All files will be compressed

.Set DiskDirectoryTemplate=CDROM ; All compressed files in the output .cab will be in a single directory
.Set DiskDirectory1=c:\RepackedStpFiles ; The output .cab file will be placed in this directory

; the next lines specify the files to be included into the output .cab
"c:\ExtractedStpFiles\manifest.xml"
"c:\ExtractedStpFiles\10000000.000"

Ok, now we need to run the following command line to repackage required files:

makecab.exe /f "C:\ExtractedStpFiles\MyList.ddf"

The /f key indicates that to create .cab file, the makecab.exe should use the directive file pointed next.

Converting Step-by-step

So, the straightforward solution to make a 2007 list template compatible with SharePoint 2010 includes the following steps:

1. Change the extension of the original 2007 .stp file to cab;
2. Extract files with any archiver into a folder (for example, c:\ExtractedStpFiles);
3. Open the manifest.xml and set the value of <ProductVersion> element to 4;
4. Repackage the modified manifest.xml and other files (if any) into .stp (generally .cab) file. Use for that either the command line like

   makecab.exe C:\ExtractedStpFiles\manifest.xml c:\RepackedStpFiles\MyList.stp
   

   when the list template in question contains only the manifest.xml, or the directive file (for example, C:\ExtractedStpFiles\MyList.ddf)

   .OPTION EXPLICIT
    
   .Set CabinetNameTemplate=MyList.stp ; Name of the output .cab file, in our case we can specify the file's extension as .stp
   .Set Cabinet=on
   .Set CompressionType=MSZIP ; All files will be compressed
   
   .Set DiskDirectoryTemplate=CDROM ; All compressed files in the output .cab will be in a single directory
   .Set DiskDirectory1=c:\RepackedStpFiles ; The output .cab file will be placed in this directory
   
   ; the next lines specify the files to be included into the output .cab
   "c:\ExtractedStpFiles\manifest.xml"
   "c:\ExtractedStpFiles\10000000.000"
   

   along with the command line

   makecab.exe /f "C:\ExtractedStpFiles\MyList.ddf"
   

   when there are more than one files in the list template.

After that the output .stp file is ready for uploading into the List Template Gallery (_catalogs/lt) of the target Site Collection.

Summary

Be aware that this solution is not a best practice and it may not work in some cases.

If you have a lot of .stp files to migrate you can automate this process using the PowerShell scripts posted here and here.

    After migration of a SharePoint 2007 application to SharePoint 2010 we encountered an unhandled exception occurring arbitrarily when a list item is opening for editing. The exception looks as follows:

Exception Details: System.ArgumentNullException: Value cannot be null.
Parameter name: s

Source Error: 
An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below. 

Stack Trace: 

[ArgumentNullException: Value cannot be null. Parameter name: s]
   System.IO.StringReader..ctor(String s) +10151478
   Microsoft.SharePoint.Publishing.Internal.WorkflowUtilities.FlattenXmlToHashtable(String strXml) +117
   Microsoft.SharePoint.Publishing.Internal.WorkflowUtilities.DoesWorkflowCancelWhenItemEdited(String associationXml) +12
   Microsoft.SharePoint.Publishing.WebControls.ConsoleDataSource.EnsurePageNotInLockingWorkflowIfInEditMode() +207
   Microsoft.SharePoint.Publishing.WebControls.ConsoleDataSource.LoadDataSource() +199
   Microsoft.SharePoint.Publishing.WebControls.ConsoleDataSource.OnLoad(EventArgs e) +98
   Microsoft.SharePoint.Publishing.WebControls.XmlConsoleDataSource.OnLoad(EventArgs e) +201
   Microsoft.SharePoint.Publishing.WebControls.PublishingSiteActionsMenuCustomizer.OnLoad(EventArgs e) +186
   System.Web.UI.Control.LoadRecursive() +66
   System.Web.UI.Control.LoadRecursive() +191
   System.Web.UI.Control.LoadRecursive() +191
   System.Web.UI.Control.LoadRecursive() +191
   System.Web.UI.Control.LoadRecursive() +191
   System.Web.UI.Control.LoadRecursive() +191
   System.Web.UI.Control.LoadRecursive() +191
   System.Web.UI.Control.LoadRecursive() +191
   System.Web.UI.Control.LoadRecursive() +191
   System.Web.UI.Control.LoadRecursive() +191
   System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +2428

Having analyzed the stack trace we found out that the issue is, evidently, caused by workflow infrastructure. In the SharePoint application we have both handmade custom workflows and the ones that were made in SharePoint Designer (so called SPD Workflows). It’s interesting that the exception was rising for both workflow types. After some investigation using .Net Reflector, we discovered that the AssociationData xml-element is a cause of our troubles. The AssociationData is an element of the Workflow Definition Schema and specifies any custom data to pass to the workflow association form. Additionally, we use the association data when starting workflow on list items through the code. The workflow infrastructure, in SharePoint 2010, supposes that the AssociationData element is a valid xml string. So, if it’s empty or contains any data without xml-tags, the exception is thrown as opposite to SP 2007.

The solution consists of two steps. Firstly, for every Workflow Definition in your project you need to set <AssociationData> so that it contains a valid xml. Regardless of whether association data is actually used or not, <AssociationData> has to be presented in Workflow Definition and contain at least a fake valid xml. For example, I set dummy <Data />. In case you deploy a custom workflow through a SharePoint Feature, the Workflow Definition is usually located in Elements.xml file. If <AssociationData> element doesn’t exist within <Workflow> tag, add it at the same level as <MetaData>. The resultant workflow definition should look like:

<?xml version="1.0" encoding="utf-8" ?> 
<Elements xmlns="http://schemas.microsoft.com/sharepoint/">
  <Workflow ...>
    ...
    <AssociationData>
      <Data />
    </AssociationData>
    <MetaData>
      ...
    </MetaData>
    ...
  </Workflow>
</Elements>

The described first step of the solution affects only new applications created after the alterations have been applied to Workflow Definitions. Even if a SharePoint Feature containing a custom workflow is reactivated, changes are only applied to workflow instances that start after the workflow association is modified. But for live applications created before or migrated from previous version of SharePoint (like in my case), the first step is useless. Additionally, you may have the SPD workflow which doesn’t include usual Workflow Definition at all. So, the second step is a necessary and sufficient part of the solution.

The second step is get all SPWorkflowAssociation objects and adjust their AssociationData properties programmatically. I’ve developed a few methods allowing to achieve this goal. See the listing below:

using System;
using System.Xml;
using Microsoft.SharePoint;
using Microsoft.SharePoint.Workflow;

...

/// <summary>
/// Adjusts AssociationData properties of all Workflow Associations related to the passed Web, its child Lists and Content Types
/// </summary>
public static void AdjustAllWorkflowAssociations(SPWeb web)
{
    AdjustWebWorkflowAssociation(web);

    for (int i = 0; i < web.ContentTypes.Count; i ++)
        AdjustContentTypeWorkflowAssociation(web.ContentTypes[i]);

    for (int i = 0; i < web.Lists.Count; i ++)
    {
        AdjustListWorkflowAssociation(web.Lists[i]);
        for (int j = 0; j < web.Lists[i].ContentTypes.Count; j ++)
            AdjustContentTypeWorkflowAssociation(web.Lists[i].ContentTypes[j]);
    }
}

/// <summary>
/// Adjusts AssociationData properties of all Workflow Associations related to the passed List
/// </summary>
public static void AdjustListWorkflowAssociation(SPList list)
{
    AdjustAssociationData(list.WorkflowAssociations);
}

/// <summary>
/// Adjusts AssociationData properties of all Workflow Associations related to the passed Web
/// </summary>
public static void AdjustWebWorkflowAssociation(SPWeb web)
{
    AdjustAssociationData(web.WorkflowAssociations);
}

/// <summary>
/// Adjusts AssociationData properties of all Workflow Associations related to the passed Content Type
/// </summary>
public static void AdjustContentTypeWorkflowAssociation(SPContentType contentType)
{
    AdjustAssociationData(contentType.WorkflowAssociations);
}

/// <summary>
/// Adjusts AssociationData properties of all Workflow Associations in the passed collection
/// </summary>
public static void AdjustAssociationData(SPWorkflowAssociationCollection collection)
{
    for (int i = 0; i < collection.Count; i ++)
        AdjustAssociationData(collection[i], collection);
}

/// <summary>
/// Sets AssociationData property if it's not valid
/// </summary>
public static void AdjustAssociationData(SPWorkflowAssociation workflowAssociation, SPWorkflowAssociationCollection collection)
{
    if (!IsValidXml(workflowAssociation.AssociationData))
    {
        string newValue = string.IsNullOrEmpty(workflowAssociation.AssociationData)
                                ? "<Data />"
                                : string.Format("<Data>{0}</Data>", workflowAssociation.AssociationData);
        workflowAssociation.AssociationData = newValue;
        collection.Update(workflowAssociation);
    }
}

/// <summary>
/// Checks if the passed string is a valid xml
/// </summary>
public static bool IsValidXml(string str)
{
    if (!string.IsNullOrEmpty(str))
    {
        try
        {
            XmlDocument xmlDoc = new XmlDocument();
            xmlDoc.LoadXml(str);
            return true;
        }
        catch {}
    }
    return false;
}

The basic method in the set is AdjustAssociationData, which examines AssociationData property of a passed SPWorkflowAssociation object and then applies a valid xml-value to the property if it’s necessary. As we know, workflows can be associated with webs, lists or content types. The AdjustWebWorkflowAssociation, AdjustListWorkflowAssociation and AdjustContentTypeWorkflowAssociation methods adjust AssociationData of a passed SPWeb, SPList or SPContentType object respectively. Finally, the AdjustAllWorkflowAssociations tries to adjust all available workflow associations that can be reached through a passed SPWeb instance.

Note that I everywhere used the for-statement instead of foreach, because otherwise the exception “Collection was modified; enumeration operation may not execute” will be thrown.

Below is the piece of code from the simple console application, which I used against our problem SharePoint application:

static void Main(string[] args)
{
    try
    {
        AppArguments arguments = new AppArguments(args);

        if(string.IsNullOrEmpty(arguments.Url))
        {
            Console.WriteLine("Invalid application URL!");
            return;
        }

        using (SPSite spSite = new SPSite(arguments.Url))
            using (SPWeb spWeb = spSite.OpenWeb())
                AdjustAllWorkflowAssociations(spWeb);

    }
    catch (Exception ex)
    {
        Console.WriteLine(ex.Message);
    }
}

Where the AppArguments is a class derived from InputArguments. InputArguments is described here – Simple Command Line Arguments Parser.

public class AppArguments : InputArguments
{
    public string Url
    {
        get { return GetValue("url"); }
    }

    public AppArguments(string[] args) : base(args)
    {
    }
}

As for our SharePoint application, it has only workflows associated with lists. So, after executing of the console app described above, the exception in question is gone. I hope it’ll help somebody as well.

The full version of the console app you can download here (Visual Studio 2010 solution). To execute it use the command line with parameters as follows:

AdjustAssociationData.exe -url "http://yourservername/yourapppath"

    As you probably know, in SharePoint 2010 Business Data Connectivity replaced Business Data Catalog of SharePoint 2007. Some changes affects how Business Data Columns are presented in a list’s schema. In SP 2007 a declaration of a Business Data Column in a schema.xml may look like the following:

<Field Type="BusinessData" DisplayName="Product"
Required="FALSE" ID="{bc203358-6113-470f-9b08-f6100cc034f2}"
StaticName="Product" BaseRenderingType="Text" Name="Product"
SystemInstance="ExternalProductDB_Instance" Entity="Products"
BdcField="Name" Profile="" HasActions="False"
RelatedField="Products_ID"
RelatedFieldBDCField="" RelatedFieldWssStaticName="Products_ID"

SecondaryFieldBdcNames="Price:Producer"
SecondaryFieldWssNames="Product_x003a__x0020_Price:Product_x003a__x0020_Producer"
SecondaryFieldsWssStaticNames="Product_x003a__x0020_Price:Product_x003a__x0020_Producer" />

In contrast, in SP 2010 it looks like

<Field Type="BusinessData" DisplayName="Product"
Required="FALSE" ID="{bc203358-6113-470f-9b08-f6100cc034f2}"
StaticName="Product" BaseRenderingType="Text" Name="Product"
SystemInstance="ExternalProductDB_Instance" Entity="Products"
BdcField="Name" Profile="" HasActions="False"
RelatedField="Products_ID"
RelatedFieldBDCField="" RelatedFieldWssStaticName="Products_ID"

SecondaryFieldBdcNames="6%209%20Price%20Producer%204"
SecondaryFieldWssNames="27%2030%20Product%5Fx003a%5F%5Fx0020%5FPrice%20Product%5Fx003a%5F%5Fx0020%5FProducer%206"
SecondaryFieldsWssStaticNames="27%2030%20Product%5Fx003a%5F%5Fx0020%5FPrice%20Product%5Fx003a%5F%5Fx0020%5FProducer%206" />

Undoubtedly, in SP 2010 the secondary fields became practically unreadable. Indeed, the format of secondary fields‘ presentation is revised. Moreover some kind of URL encoding are applied to them. Let’s examine how these secondary fields could look before the URL encoding is applied:

<Field
...
SecondaryFieldBdcNames="6 9 Price Producer 4"
SecondaryFieldWssNames="27 30 Product_x003a__x0020_Price Product_x003a__x0020_Producer 6"
SecondaryFieldsWssStaticNames="27 30 Product_x003a__x0020_Price Product_x003a__x0020_Producer 6" />

Now it’s pretty easy to figure out the new format. Take a look at the SecondaryFieldBdcNames attribute. It contains names of two secondary bdc fields: ‘Price’ and ‘Producer’. 6 is the length of the ‘Price’ name + 1 for a space character right after the name. 9 is the length of the ‘Procuder’ name + 1 for a space character after the name. 4 is the length of the sub-string ‘6 9 ‘ (including spaces), which contains the lengths of the fields’ names. See a picture below:

Note that the SecondaryFieldBdcNames, SecondaryFieldWssNames and SecondaryFieldsWssStaticNames have the same format.

We have a lot of code interacting with Business Data Columns, thus we were interested in means allowing easily to decode, encode and parse Secondary Fields attributes. In the Microsoft.SharePoint.dll, there is the internal BdcClientUtil class containing the basic methods to work with Secondary Fields:

internal class BdcClientUtil
{
    ...
    string[] SplitStrings(string combinedEncoded);
    string   CombineStrings(string[] strings);
    ...
}

So, using .Net Reflector I’ve extracted these methods along with several others auxiliary ones and put them into the helper-class called SecondaryFieldNamesHelper. All internal methods and properties were honestly stolen from Microsoft.SharePoint.dll, the public ones were added by me and described below:

• string Encode(string[] secondaryFieldNames) – accepts an array of field names and returns the string formatted and encoded according to the SharePoint 2010 requirements;
• string[] Decode(string str) – accepts an encoded string, decodes it and returns a resultant array of field names;
• bool IsEncodedString(string str) – checks whether a passed string is encoded;
• string ConvertToSP2010(string str) – converts a SP 2007 colon-separated string of secondary fields into another one formatted and encoded according to the SharePoint 2010 requirements;

Below is the source code of the SecondaryFieldNamesHelper:

using System;
using System.Collections;
using System.Text;
using System.Globalization;
using System.IO;
using System.Web.UI;

namespace Helpers
{
    public static class SecondaryFieldNamesHelper
    {
        #region fields & properties
        private static string[] s_crgstrUrlHexValue = new string[] 
        { 
            "%00", "%01", "%02", "%03", "%04", "%05", "%06", "%07", "%08", "%09", "%0A", "%0B", "%0C", "%0D", "%0E", "%0F", 
            "%10", "%11", "%12", "%13", "%14", "%15", "%16", "%17", "%18", "%19", "%1A", "%1B", "%1C", "%1D", "%1E", "%1F", 
            "%20", "%21", "%22", "%23", "%24", "%25", "%26", "%27", "%28", "%29", "%2A", "%2B", "%2C", "%2D", "%2E", "%2F", 
            "%30", "%31", "%32", "%33", "%34", "%35", "%36", "%37", "%38", "%39", "%3A", "%3B", "%3C", "%3D", "%3E", "%3F", 
            "%40", "%41", "%42", "%43", "%44", "%45", "%46", "%47", "%48", "%49", "%4A", "%4B", "%4C", "%4D", "%4E", "%4F", 
            "%50", "%51", "%52", "%53", "%54", "%55", "%56", "%57", "%58", "%59", "%5A", "%5B", "%5C", "%5D", "%5E", "%5F", 
            "%60", "%61", "%62", "%63", "%64", "%65", "%66", "%67", "%68", "%69", "%6A", "%6B", "%6C", "%6D", "%6E", "%6F", 
            "%70", "%71", "%72", "%73", "%74", "%75", "%76", "%77", "%78", "%79", "%7A", "%7B", "%7C", "%7D", "%7E", "%7F", 
            "%80", "%81", "%82", "%83", "%84", "%85", "%86", "%87", "%88", "%89", "%8A", "%8B", "%8C", "%8D", "%8E", "%8F", 
            "%90", "%91", "%92", "%93", "%94", "%95", "%96", "%97", "%98", "%99", "%9A", "%9B", "%9C", "%9D", "%9E", "%9F", 
            "%A0", "%A1", "%A2", "%A3", "%A4", "%A5", "%A6", "%A7", "%A8", "%A9", "%AA", "%AB", "%AC", "%AD", "%AE", "%AF", 
            "%B0", "%B1", "%B2", "%B3", "%B4", "%B5", "%B6", "%B7", "%B8", "%B9", "%BA", "%BB", "%BC", "%BD", "%BE", "%BF", 
            "%C0", "%C1", "%C2", "%C3", "%C4", "%C5", "%C6", "%C7", "%C8", "%C9", "%CA", "%CB", "%CC", "%CD", "%CE", "%CF", 
            "%D0", "%D1", "%D2", "%D3", "%D4", "%D5", "%D6", "%D7", "%D8", "%D9", "%DA", "%DB", "%DC", "%DD", "%DE", "%DF", 
            "%E0", "%E1", "%E2", "%E3", "%E4", "%E5", "%E6", "%E7", "%E8", "%E9", "%EA", "%EB", "%EC", "%ED", "%EE", "%EF", 
            "%F0", "%F1", "%F2", "%F3", "%F4", "%F5", "%F6", "%F7", "%F8", "%F9", "%FA", "%FB", "%FC", "%FD", "%FE", "%FF"
        };
        #endregion

        #region public methods
        public static bool IsEncodedString(string str)
        {
            if (string.IsNullOrEmpty(str))
                return false;

            bool res = true;
            try
            {
                string[] splittedString = SplitStrings(str);
            }
            catch
            {
                res = false;
            }
            return res;
        }

        public static string Encode(string[] secondaryFieldNames)
        {
            return CombineStrings(secondaryFieldNames);
        }

        public static string[] Decode(string str)
        {
            if(string.IsNullOrEmpty(str))
                return new string[0];
            return SplitStrings(str);
        }

        public static string ConvertToSP2010(string str)
        {
            if (IsEncodedString(str))
                return str;

            string[] fieldNames = str.Split(new string[] { ":" }, StringSplitOptions.RemoveEmptyEntries);
            string encodedVal = CombineStrings(fieldNames);
            return encodedVal;
        }
        #endregion

        #region internal methods
        private static string[] SplitStrings(string combinedEncoded)
        {
            string[] array = null;
            ArrayList list = new ArrayList();
            if ("0" == combinedEncoded)
                return new string[0];
            try
            {
                string str = UrlKeyValueDecode(combinedEncoded);
                string[] strArray2 = str.Split(new char[] { ' ' }, StringSplitOptions.None);
                int result = 0;
                if ((strArray2 == null) || !int.TryParse(strArray2[strArray2.Length - 1], NumberStyles.Integer, 

CultureInfo.InvariantCulture, out result))
                    throw new ArgumentException(string.Empty, "combinedEncoded");
                int num2 = str.LastIndexOf(' ');
                string str2 = str.Substring(result, num2 - result);
                int length = str2.Length;
                int index = 0;
                int startIndex = 0;
                while (startIndex < length)
                {
                    string s = strArray2[index];
                    int num6 = 1;
                    if ((s != null) && (s.Length == 0))
                        list.Add(null);
                    else
                    {
                        if (!int.TryParse(s, NumberStyles.Integer, CultureInfo.InvariantCulture, out num6))
                            throw new ArgumentException(string.Empty, "combinedEncoded");
                        list.Add(str2.Substring(startIndex, num6 - 1));
                    }
                    startIndex += num6;
                    index++;
                }
                array = new string[list.Count];
                list.CopyTo(array);
            }
            catch (Exception exception)
            {
                throw new ArgumentException(string.Empty, "combinedEncoded", exception);
            }
            return array;
        }

        private static string UrlKeyValueDecode(string keyOrValueToDecode)
        {
            if (string.IsNullOrEmpty(keyOrValueToDecode))
                return keyOrValueToDecode;
            return UrlDecodeHelper(keyOrValueToDecode, keyOrValueToDecode.Length, true);
        }

        private static string UrlDecodeHelper(string stringToDecode, int length, bool decodePlus)
        {
            if ((stringToDecode == null) || (stringToDecode.Length == 0))
                return stringToDecode;
            StringBuilder builder = new StringBuilder(length);
            byte[] bytes = null;
            int nIndex = 0;
            while (nIndex < length)
            {
                char ch = stringToDecode[nIndex];
                if (ch < ' ')
                    nIndex++;
                else
                {
                    if (decodePlus && (ch == '+'))
                    {
                        builder.Append(" ");
                        nIndex++;
                        continue;
                    }
                    if (IsHexEscapedChar(stringToDecode, nIndex, length))
                    {
                        if (bytes == null)
                            bytes = new byte[(length - nIndex) / 3];
                        int count = 0;
                        do
                        {
                            int num3 = (FromHexNoCheck(stringToDecode[nIndex + 1]) * 0x10) + FromHexNoCheck(stringToDecode[nIndex + 

2]);
                            bytes[count++] = (byte)num3;
                            nIndex += 3;
                        }
                        while (IsHexEscapedChar(stringToDecode, nIndex, length));
                        builder.Append(Encoding.UTF8.GetChars(bytes, 0, count));
                        continue;
                    }
                    builder.Append(ch);
                    nIndex++;
                }
            }
            if (length < stringToDecode.Length)
                builder.Append(stringToDecode.Substring(length));
            return builder.ToString();
        }

        private static bool IsHexEscapedChar(string str, int nIndex, int nPathLength)
        {
            if ((((nIndex + 2) >= nPathLength) || (str[nIndex] != '%')) || (!IsHexDigit(str[nIndex + 1]) || !IsHexDigit(str[nIndex + 

2])))
                return false;
            if (str[nIndex + 1] == '0')
                return (str[nIndex + 2] != '0');
            return true;
        }

        private static bool IsHexDigit(char digit)
        {
            if ((('0' > digit) || (digit > '9')) && (('a' > digit) || (digit > 'f')))
                return (('A' <= digit) && (digit <= 'F'));
            return true;
        }

        private static int FromHexNoCheck(char digit)
        {
            if (digit <= '9')
                return (digit - '0');
            if (digit <= 'F')
                return ((digit - 'A') + 10);
            return ((digit - 'a') + 10);
        }

        private static string CombineStrings(string[] strings)
        {
            StringBuilder builder = new StringBuilder();
            int index = 0;
            for (int i = 0; i < strings.Length; i++)
            {
                string str = strings[i];
                string str2 = ((str != null) ? ((str.Length + 1)).ToString(CultureInfo.InvariantCulture) : string.Empty) + ' ';
                builder.Insert(index, str2);
                index += str2.Length;
                builder.Append(str + ' ');
            }
            builder.Append(index.ToString(CultureInfo.InvariantCulture));
            return UrlKeyValueEncode(builder.ToString());
        }

        private static string UrlKeyValueEncode(string keyOrValueToEncode)
        {
            if ((keyOrValueToEncode == null) || (keyOrValueToEncode.Length == 0))
                return keyOrValueToEncode;
            StringBuilder sb = new StringBuilder(0xff);
            HtmlTextWriter output = new HtmlTextWriter(new StringWriter(sb, CultureInfo.InvariantCulture));
            UrlKeyValueEncode(keyOrValueToEncode, output);
            return sb.ToString();
        }

        private static void UrlKeyValueEncode(string keyOrValueToEncode, TextWriter output)
        {
            if (((keyOrValueToEncode != null) && (keyOrValueToEncode.Length != 0)) && (output != null))
            {
                bool fUsedNextChar = false;
                int startIndex = 0;
                int length = 0;
                int num3 = keyOrValueToEncode.Length;
                for (int i = 0; i < num3; i++)
                {
                    char ch = keyOrValueToEncode[i];
                    if (((('0' <= ch) && (ch <= '9')) || (('a' <= ch) && (ch <= 'z'))) || (('A' <= ch) && (ch <= 'Z')))
                        length++;
                    else
                    {
                        if (length > 0)
                        {
                            output.Write(keyOrValueToEncode.Substring(startIndex, length));
                            length = 0;
                        }
                        UrlEncodeUnicodeChar(output, keyOrValueToEncode[i], (i < (num3 - 1)) ? keyOrValueToEncode[i + 1] : '\0', out 

fUsedNextChar);
                        if (fUsedNextChar)
                            i++;
                        startIndex = i + 1;
                    }
                }
                if ((startIndex < num3) && (output != null))
                    output.Write(keyOrValueToEncode.Substring(startIndex));
            }
        }

        private static void UrlEncodeUnicodeChar(TextWriter output, char ch, char chNext, out bool fUsedNextChar)
        {
            bool fInvalidUnicode = false;
            UrlEncodeUnicodeChar(output, ch, chNext, ref fInvalidUnicode, out fUsedNextChar);
        }

        private static void UrlEncodeUnicodeChar(TextWriter output, char ch, char chNext, ref bool fInvalidUnicode, out bool 

fUsedNextChar)
        {
            int num = 0xc0;
            int num2 = 0xe0;
            int num3 = 240;
            int num4 = 0x80;
            int num5 = 0xd800;
            int num6 = 0xfc00;
            int num7 = 0x10000;
            fUsedNextChar = false;
            int index = ch;
            if (index <= 0x7f)
                output.Write(s_crgstrUrlHexValue[index]);
            else
            {
                int num8;
                if (index <= 0x7ff)
                {
                    num8 = num | (index >> 6);
                    output.Write(s_crgstrUrlHexValue[num8]);
                    num8 = num4 | (index & 0x3f);
                    output.Write(s_crgstrUrlHexValue[num8]);
                }
                else if ((index & num6) != num5)
                {
                    num8 = num2 | (index >> 12);
                    output.Write(s_crgstrUrlHexValue[num8]);
                    num8 = num4 | ((index & 0xfc0) >> 6);
                    output.Write(s_crgstrUrlHexValue[num8]);
                    num8 = num4 | (index & 0x3f);
                    output.Write(s_crgstrUrlHexValue[num8]);
                }
                else if (chNext != '\0')
                {
                    index = (index & 0x3ff) << 10;
                    fUsedNextChar = true;
                    index |= chNext & 'Ͽ';
                    index += num7;
                    num8 = num3 | (index >> 0x12);
                    output.Write(s_crgstrUrlHexValue[num8]);
                    num8 = num4 | ((index & 0x3f000) >> 12);
                    output.Write(s_crgstrUrlHexValue[num8]);
                    num8 = num4 | ((index & 0xfc0) >> 6);
                    output.Write(s_crgstrUrlHexValue[num8]);
                    num8 = num4 | (index & 0x3f);
                    output.Write(s_crgstrUrlHexValue[num8]);
                }
                else
                    fInvalidUnicode = true;
            }
        }
        #endregion
    }
}

The SecondaryFieldNamesHelper can be used as shown below:

SPBusinessDataField bdcField = ...

string secondaryFieldWssNamesProperty = bdcField.GetProperty("SecondaryFieldWssNames");
string[] secondaryWssFieldNames = SecondaryFieldNamesHelper.Decode(property);

string secondaryFieldBdcNamesProperty = bdcField.GetProperty("SecondaryFieldBdcNames");
string[] secondaryFieldBdcNames = SecondaryFieldNamesHelper.Decode(secondaryFieldBdcNamesProperty);

string sp2010WssStaticNames = 
   SecondaryFieldNamesHelper.ConvertToSP2010("Product_x003a__x0020_Price:Product_x003a__x0020_Producer");

As a .cs file the SecondaryFieldNamesHelper class is available here.

    After in-place upgrading one of our SharePoint applications, we had been faced with the fact that some content types comprised duplicated fields. In other words, within the several <ContentType> sections of a list’s schema, we could find the pair <FieldRef> nodes with identical identifiers and names. Schematically, it looked like the following:

<?xml version="1.0" encoding="utf-8"?>
<List Name="SomeList" Title="Some List" BaseType="0" Url="Lists/SomeList" 
           Type="100" Id="a5bba3b3-5b1d-4186-ada7-bbd82b17f76d" ...>
  <MetaData>
    <Views>
       ...
    </Views>

    <Fields>
       ...
    </Fields>

    <ContentTypes>
       ...
       <ContentType ID="0x0100078C8A39971A4532AB9C5EB6DCB388A3" Name="SomeContentType" ...>
        <FieldRefs>
           ...
           <FieldRef Name="SomeField" ID="{b986cd1a-8bd0-4072-93af-5c48571bbf56}" />
           ...
           <FieldRef Name="SomeField2" ID="{6d245d53-63ef-4650-b676-6e4ee66dcda5}" />
           ...           
           <FieldRef Name="SomeField2" ID="{6d245d53-63ef-4650-b676-6e4ee66dcda5}" />
           ...   
           <FieldRef Name="SomeField" ID="{b986cd1a-8bd0-4072-93af-5c48571bbf56}" />
           ...
        </FieldRefs>
       ...    
    </ContentTypes>

    <Forms>
       ...
    </Forms>
   </MetaData>
</List>

The reason of such duplication still isn’t clear for me, but I’ve figured out how to get rid of it 🙂 Below is a simple method for deleting the excess fields from a passed content type:

protected static void RemoveDuplicatedFields(SPContentType spContentType)
{
    bool duplicationFound = false;

    // identify how many times every field encounter
    Dictionary<Guid, int> tmpDir = new Dictionary<Guid, int>();
    foreach (SPFieldLink spFieldLink in spContentType.FieldLinks)
        if (!tmpDir.ContainsKey(spFieldLink.Id))
            tmpDir.Add(spFieldLink.Id, 1);
        else
        {
            tmpDir[spFieldLink.Id]++;
            duplicationFound = true;
        }
   

    if (duplicationFound)
        // remove all excess mentions of fields
        foreach (KeyValuePair<Guid, int> keyValuePair in tmpDir)
        {
            int removeIterationCount = keyValuePair.Value - 1;
            for (int i = 0; i < removeIterationCount; i++)
                spContentType.FieldLinks.Delete(keyValuePair.Key);
        }
}

    Trying to import a legacy Application Definition File of SharePoint 2007 into Business Data Connectivity Service of SharePoint 2010, you apparently got at least one of the errors shown below:

• Application definition import failed. The following error occurred: The root element of a valid Metadata package must be ‘Model’ in namespace ‘http://schemas.microsoft.com/windows/2007/BusinessDataCatalog’. The root in the given package is ‘LobSystem’ in namespace ‘http://schemas.microsoft.com/office/2006/03/BusinessDataCatalog’. Error was encountered at or just before Line: ‘2’ and Position: ‘2’;
• Application definition import failed. The following error occurred: BDC Model does not correctly match the schema. The required attribute ‘Namespace’ is missing. Error was encountered at or just before Line: ’20’ and Position: ’10’;
• Application definition import failed. The following error occurred: ReturnTypeDescriptor of MethodInstance with Name ‘ProductSpecificFinderInstance’ on Entity (External Content Type) with Name ‘Product’ in Namespace ‘ExternalProductDB’ should not be a Collection TypeDescriptor for MethodInstances of Type ‘SpecificFinder’. Parameter name: rawValues.ReturnTypeDescriptorId Error was encountered at or just before Line: ‘171’ and Position: ’18’;
• and so on

As it’s known, in SharePoint 2010, Business Data Catalog (BDC) was replaced with Business Data Connectivity with the same abbreviation. One of the changed things is the format of xml-based Application Definition Files. If you make an in-place upgrade of a live SharePoint 2007 application to SharePoint 2010, bdc metadata will be automatically upgraded as well and will become usable with the Business Data Connectivity. But if the in-place upgrade isn’t an option for you, you can upgrade your xml-based Application Definition Files manually. The manual algorithm step by step is described here – How to: Manually Upgrade Business Data Catalog Application Definitions to Business Data Connectivity Models.

For one of our applications we settled on the manual upgrade of its metadata files. But If I call myself a programmer, I have to try to automate the algorithm, especially taking into account 20+ files required to upgrade. So, I’ve developed a simple application for alteration of the legacy xml-based Application Definition Files to make them compatible with SharePoint 2010. However I’d like to notice that the given converter doesn’t follow entirely the procedure described by Microsoft, but performs only steps allowing our particular metadata files to be successfully imported into the Business Data Connectivity Service. For example, our files don’t comprise actions and associations, thus the application does nothing at all with <Action> and <Association> elements. So, consider this converter as a start point of developing the new one satisfying your own conditions and requirements.

Below I enumerated the necessary and sufficient changes to be applied to our particular metadata files so that it enables us to make them compatible with SharePoint 2010. Exactly these very steps and a few less important I’ve implemented in the converter.

• the root element in the Application Definition File must be a <Model>;
• the <Model> must contain <LobSystems> element, which in turn must wrap the former root node – <LobSystem>;
• the <LobSystem> element mustn’t contain the Version-attribute;
• the <Entity> element must contain the new attributes – Namespace and Version;
• the <Identifier> element mustn’t contain an assembly name in its TypeName-attribute; For example, TypeName=”System.String, mscorlib” has to turn into TypeName=”System.String”;
• the <MethodInstance> element with Type-attribute value of SpecificFinder should include the Default-attribute with value of true;
• if the <TypeDescriptor> element has the IsCollection attribute set to true, the MethodInstance return TypeDescriptor should be updated to be an element of the collection. In practice, that means the ReturnTypeDescriptorPath-attribute with an appropriate value should be added to <MethodInstance> element, and the obsolete ReturnTypeDescriptorLevel and ReturnTypeDescriptorName attributes should be deleted;

Note that when modifying a <Entity> element, the values of the Namespace and Version attributes are copied respectively from the values of Name and Version attributes of the <LobSystem> element, which wraps the <Entity> element. The same approach is used while the in-place upgrade takes place.

After the changes are applied, and if they are sufficient for your metadata, the result files can be imported into Business Data Connectivity Service. During the import process, you may get the warnings. Consider fixing them in the future, but at the present stage you can simply ignore them. The most popular warnings are listed below:

• This Model contains LobSystem (External System) of Type ‘WebService’ which is deprecated and may be removed in future releases.
• The MethodInstance of type ‘Finder’ with Name ‘FindProducts’ does not have a Limit Filter.
• The TypeDescriptor ‘From’ on Parameter ‘GetProductsFiltered’ on Method ‘GetItems’ of Entity (External Content Type) ‘Product’ with Namespace ‘ExternalProductDB’ has a DateTime value, but there is no Interpretation describing what the External System expects and returns as the DateTimeKind. The DateTime value is assumed to be UTC.

Note: I made the converter-application fix the warnings regarding the DateTime type and UTC, so they won’t bother you.

The converter is very straightforward to use. Using the button ‘+’, simply add to the left section the files to be upgraded. Then click the button ‘>>’, and you’ll get the upgraded ones in the right section. Double click on file name opens an overview form to browse the input or result xml. Physically, the result files are located in c:\output folder. The application doesn’t use any SharePoint-related libraries.



You can download the application from this page or by using the direct link. The Visual Studio 2010 solution and appropriate executable file are in the archive.