Archive

Posts Tagged ‘C#’

WebLoading Library: Downloading data from RESTful sources

July 23rd, 2012 No comments

    I often run into necessity to load data from sources published in the Internet. In most cases the sources are RESTful web-services, thus the data they emit is in xml or json formats. More and more web-services providers prefer spreading their data in an easier-to-use REST formats as a simpler alternative to SOAP and WSDL based web-services.

WebLoading Library Overview

For downloading data via the Http protocol I implemented a set of generic classes, let’s call it WebLoading Library. Almost every class in the library is a loader, which operates at a certain level of abstraction. The library allows to easily derive a new class from one of the exposed loaders, defining a type of the data to be loaded and the type of the ready-to-use data to be returned out of the loader. See the class diagram below.

Class Diagram of WebLoading Library

I single out such basic steps of getting data as raw data loading, verification and conversion into a ready-to-use view. All loaders exposes the LoadData method aimed at performing these steps.The listing below demonstrates the LoadData‘s base implementation in the BaseLoader class. BaseLoader is a root base class for all loaders in the WebLoading Library hierarchy.

BaseLoader Source

using System;

namespace WebLoading
{
    /// <summary>
    /// Abstracts the data loading from a specified source. Provides such basic steps as data loading, verification and interpretation.
    /// </summary>
    /// <typeparam name="TRawData">A type of data to be loaded</typeparam>
    /// <typeparam name="TParsedData">A type of the analyzed data to be returned</typeparam>
    public abstract class BaseLoader<TRawData, TParsedData>
        where TRawData    : class
        where TParsedData : class
    {
        /// <summary>
        /// Loads data from a specified source, verifies it and returns its ready-to-use view.
        /// </summary>
        /// <returns>The ready-to-use view of the loaded data</returns>
        public virtual TParsedData LoadData()
        {
            TRawData rawData = LoadRawData();

            if (!IsRawDataValid(rawData))
                return default(TParsedData);
            
            TParsedData parsedData = ParseData(rawData);            

            return parsedData;
        }

        /// <summary>
        /// Loads data from a specified source. Has to be overridden in a derived class
        /// </summary>
        /// <returns>The loaded raw data</returns>
        protected abstract TRawData LoadRawData();

        /// <summary>
        /// Verifies the response retrieved from the specified source
        /// </summary>
        /// <param name="rawData">The raw data to be verified</param>
        /// <returns>True if the response is correct. Otherwise, False</returns>
        protected virtual bool IsRawDataValid(TRawData rawData)
        {
            return true;
        }       

        /// <summary>
        /// Analyzes the raw data and returns its ready-to-use view. Has to be overridden in a derived class
        /// </summary>
        /// <param name="rawData">The raw data</param>
        /// <returns>The ready-to-use view of the data</returns>
        protected abstract TParsedData ParseData(TRawData rawData);
    }
}

BaseLoader abstracts the loading from any source. Unlike BaseLoader, its direct descendant, HttpLoader, implies loading from a source supporting the Http protocol. See the listing of the HttpLoader class below.

HttpLoader Source

using System;
using System.Net;
using System.IO;

namespace WebLoading
{
    /// <summary>    
    /// Abstracts the data loading from a specified source using Http. Provides such basic steps as data loading, verification and interpretation.
    /// </summary>
    /// <typeparam name="TRawData">A type of data to be loaded</typeparam>
    /// <typeparam name="TParsedData">A type of the analyzed data to be returned</typeparam>
    public abstract class HttpLoader<TRawData, TParsedData> : BaseLoader<TRawData, TParsedData>
        where TRawData    : class
        where TParsedData : class
    {
        /// <summary>
        /// An initial url to load data from
        /// </summary>
        protected readonly string _requestUrl;

        /// <summary>
        /// Initializes a new instance of the HttpLoader class
        /// </summary>
        /// <param name="requestUrl">An initial url to load data from</param>
        public HttpLoader(string requestUrl)
        {
            _requestUrl = requestUrl;
        }

        /// <summary>
        /// Initializes a new instance of the HttpLoader class
        /// </summary>
        public HttpLoader()
        {            
        }

        /// <summary>       
        /// Loads data from a specified source using Http, returns a ready-to-analyze data view
        /// </summary>
        /// <returns>A ready-to-analyze view of the data</returns>
        protected override TRawData LoadRawData()
        {
            HttpWebRequest httpWebRequest = PrepareRequest();

            TRawData rawData;
            using (HttpWebResponse httpWebResponse = (HttpWebResponse)httpWebRequest.GetResponse())
                using (Stream responseStream = httpWebResponse.GetResponseStream())
                    rawData = ReadResponseStream(responseStream);

            return rawData;
        }

        /// <summary>
        /// Creates and initializes a Http request object
        /// </summary>
        /// <returns>A Http request object</returns>
        protected virtual HttpWebRequest PrepareRequest()
        {
            string url = PrepareUrl();

            HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create(url);
            httpWebRequest.UserAgent      = @"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)";
            return httpWebRequest;
        }

        /// <summary>
        /// Prepares and returns an url the data should be loaded from. If the parameterless constructor is used, the method has to be overridden in a derived class
        /// </summary>
        /// <returns>An url to load the data from</returns>
        protected virtual string PrepareUrl()
        {
            return _requestUrl;
        }

        /// <summary>
        /// Reads from a stream and returns the received data in a ready-to-analyze view. Has to be overridden in a derived class
        /// </summary>
        /// <param name="stream">A data stream</param>
        /// <returns>The ready-to-analyze view of the data</returns>
        protected abstract TRawData ReadResponseStream(Stream stream);        
    }
}

The next level in the hierarchy comprises such loaders as ImageLoader, XmlLoader and JsonLoader. I’ll dwell upon the last two a bit later. As regards the first one, ImageLoader is a quite simple class derived from HttpLoader that downloads an image from a specified source via Http. Being initially intended for dealing with REST formats, the loaders family can be easily extended to any data formats. So, the ImageLoader is a good example to that. The class is shown below:

ImageLoader Source and How to use

using System;
using System.Drawing;
using System.Drawing.Imaging;
using System.IO;

namespace WebLoading
{
    /// <summary>
    /// Downloads an image from a specified source using Http
    /// </summary>
    public class ImageLoader : HttpLoader<Image, Image>
    {
        /// <summary>
        /// Initializes a new instance of the ImageLoader class
        /// </summary>
        /// <param name="url">An url to load image from</param>
        public ImageLoader(string url) : base(url)
        {
        }

        /// <summary>
        /// Initializes a new instance of the ImageLoader class. Intended for use in a derived class
        /// </summary>
        protected ImageLoader()
        {
        }

        /// <summary>
        /// Reads an image from a stream
        /// </summary>
        /// <param name="stream">A data stream</param>
        /// <returns>An Image object</returns>
        protected override Image ReadResponseStream(Stream stream)
        {
            Image image = Image.FromStream(stream);
            return image;
        }

        /// <summary>
        /// Does nothing. Just returns a loaded image.
        /// </summary>
        /// <param name="rawData">The loaded image</param>
        /// <returns>The loaded image with no changes</returns>
        protected override Image ParseData(Image rawData)
        {
            return rawData;
        }
    }
}

//...
// "how to use" sample
ImageLoader imageLoader = new ImageLoader("http://dotnetfollower.com/wordpress/wp-content/uploads/2012/06/All_Site_Content_Links.png");
Image img = imageLoader.LoadData();

WebLoading Library Overview: XmlLoader

XmlLoader is an abstract class derived from HttpLoader and intended for xml data downloading. The downloaded raw xml is presented as a XDocument object. Create a descendent of the XmlLoader class to verify and turn the XDocument into any another desired view-object. The XmlLoader looks as follows:

XmlLoader Source

using System;
using System.IO;
using System.Xml.Linq;

namespace WebLoading
{
    /// <summary>    
    /// Abstracts the xml data loading from a specified source using Http. Provides such basic steps as data loading, verification and interpretation.
    /// </summary>
    /// <typeparam name="TParsedData">A type of the analyzed data to be returned</typeparam>
    public abstract class XmlLoader<TParsedData> : HttpLoader<XDocument, TParsedData> where TParsedData : class
    {
        /// <summary>
        /// Initializes a new instance of the XmlLoader class
        /// </summary>
        /// <param name="url">An initial url to load data from</param>
        public XmlLoader(string url) : base(url)
        {
        }

        /// <summary>
        /// Initializes a new instance of the XmlLoader class
        /// </summary>
        public XmlLoader()
        {
        }

        /// <summary>        
        /// Reads from a stream and returns the received xml data
        /// </summary>
        /// <param name="stream">A data stream</param>
        /// <returns>The received xml data</returns>
        protected override XDocument ReadResponseStream(Stream stream)
        {
            XDocument xDoc = XDocument.Load(stream);
            return xDoc;
        }
    }
}

As a simple example, below is a class-descendent demonstrating how to get the expected maximum temperatures for the next 7 days. The data source is http://graphical.weather.gov, which provides with xml data like the following:

Xml Data Sample

<?xml version="1.0"?>
<dwml version="1.0" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://graphical.weather.gov/xml/DWMLgen/schema/DWML.xsd">
  <head>
    <product srsName="WGS 1984" concise-name="time-series" operational-mode="official">
      <title>NOAA's National Weather Service Forecast Data</title>
      <field>meteorological</field>
      <category>forecast</category>
      <creation-date refresh-frequency="PT1H">2012-07-13T02:13:29Z</creation-date>
    </product>
    <source>
      <more-information>http://graphical.weather.gov/xml/</more-information>
      <production-center>Meteorological Development Laboratory<sub-center>Product Generation Branch</sub-center></production-center>
      <disclaimer>http://www.nws.noaa.gov/disclaimer.html</disclaimer>
      <credit>http://www.weather.gov/</credit>
      <credit-logo>http://www.weather.gov/images/xml_logo.gif</credit-logo>
      <feedback>http://www.weather.gov/feedback.php</feedback>
    </source>
  </head>
  <data>
    <location>
      <location-key>point1</location-key>
      <point latitude="28.50" longitude="-81.37"/>
    </location>
    <moreWeatherInformation applicable-location="point1">http://forecast.weather.gov/MapClick.php?textField1=28.50&amp;textField2=-81.37</moreWeatherInformation>
    <time-layout time-coordinate="local" summarization="none">
      <layout-key>k-p24h-n7-1</layout-key>
      <start-valid-time>2012-07-13T08:00:00-04:00</start-valid-time>
      <end-valid-time>2012-07-13T20:00:00-04:00</end-valid-time>
      <start-valid-time>2012-07-14T08:00:00-04:00</start-valid-time>
      <end-valid-time>2012-07-14T20:00:00-04:00</end-valid-time>
      <start-valid-time>2012-07-15T08:00:00-04:00</start-valid-time>
      <end-valid-time>2012-07-15T20:00:00-04:00</end-valid-time>
      <start-valid-time>2012-07-16T08:00:00-04:00</start-valid-time>
      <end-valid-time>2012-07-16T20:00:00-04:00</end-valid-time>
      <start-valid-time>2012-07-17T08:00:00-04:00</start-valid-time>
      <end-valid-time>2012-07-17T20:00:00-04:00</end-valid-time>
      <start-valid-time>2012-07-18T08:00:00-04:00</start-valid-time>
      <end-valid-time>2012-07-18T20:00:00-04:00</end-valid-time>
      <start-valid-time>2012-07-19T08:00:00-04:00</start-valid-time>
      <end-valid-time>2012-07-19T20:00:00-04:00</end-valid-time>
    </time-layout>
    <parameters applicable-location="point1">
      <temperature type="maximum" units="Fahrenheit" time-layout="k-p24h-n7-1">
        <name>Daily Maximum Temperature</name>
        <value>92</value>
        <value>93</value>
        <value>93</value>
        <value>94</value>
        <value>92</value>
        <value>94</value>
        <value>93</value>
      </temperature>
    </parameters>
  </data>
</dwml>

The GraphicalWeatherLoader class accepts a zip code of the region you need to get forecast for. See the next listing:

GraphicalWeatherLoader Source and How to use

using System;
using System.Collections.Generic;
using System.Xml.Linq;
using System.Xml.XPath;
using WebLoading;

namespace TestApp
{
    public class GraphicalWeatherLoader : XmlLoader<List<GraphicalWeatherLoader.DailyMaximumTemperature>>
    {
        public class DailyMaximumTemperature
        {
            public DateTime Day         { get; set; }
            public string   Temperature { get; set; }
        }

        protected static string _urlTemplate = "http://graphical.weather.gov/xml/SOAP_server/ndfdXMLclient.php?whichClient=NDFDgenMultiZipCode&lat=&lon=&listLatLon=&lat1=&lon1=&lat2=&lon2=&resolutionSub=&listLat1=&listLon1=&listLat2=&listLon2=&resolutionList=&endPoint1Lat=&endPoint1Lon=&endPoint2Lat=&endPoint2Lon=&listEndPoint1Lat=&listEndPoint1Lon=&listEndPoint2Lat=&listEndPoint2Lon=&zipCodeList={0}&listZipCodeList=&centerPointLat=&centerPointLon=&distanceLat=&distanceLon=&resolutionSquare=&listCenterPointLat=&listCenterPointLon=&listDistanceLat=&listDistanceLon=&listResolutionSquare=&citiesLevel=&listCitiesLevel=&sector=&gmlListLatLon=&featureType=&requestedTime=&startTime=&endTime=&compType=&propertyName=&product=time-series&begin=2004-01-01T00%3A00%3A00&end=2016-06-27T00%3A00%3A00&Unit=e&maxt=maxt&Submit=Submit";

        public GraphicalWeatherLoader(string zipCode) : base(string.Format(_urlTemplate, zipCode))
        {
        }

        protected override List<GraphicalWeatherLoader.DailyMaximumTemperature> ParseData(XDocument rawData)
        {
            List<DailyMaximumTemperature> res = new List<DailyMaximumTemperature>();

            XElement dataElement = rawData.XPathSelectElement("dwml/data");

            IEnumerator<XElement> dates        = dataElement.XPathSelectElements("time-layout/start-valid-time").GetEnumerator();
            IEnumerator<XElement> temperatures = dataElement.XPathSelectElements("parameters/temperature/value").GetEnumerator();
            
            while(dates.MoveNext() && temperatures.MoveNext())
            {
                DailyMaximumTemperature dailyMaximumTemperature = new DailyMaximumTemperature();
                dailyMaximumTemperature.Day         = DateTime.Parse(dates.Current.Value);
                dailyMaximumTemperature.Temperature = temperatures.Current.Value;
                res.Add(dailyMaximumTemperature);
            }            

            return res;
        }

        protected override bool IsRawDataValid(XDocument rawData)
        {
            string data = rawData.ToString();
            if(!string.IsNullOrWhiteSpace(data))
                return !data.Contains("SOAP ERROR");
            return false;
        }
    }
}

//...
// "how to use" sample
GraphicalWeatherLoader graphicalWeatherLoader = new GraphicalWeatherLoader("32801");
List<GraphicalWeatherLoader.DailyMaximumTemperature> temperatures = graphicalWeatherLoader.LoadData();
if (temperatures != null)
    foreach (GraphicalWeatherLoader.DailyMaximumTemperature temperature in temperatures)
        Console.WriteLine(string.Format("{0} - {1} Fahrenheit", temperature.Day.ToShortDateString(), temperature.Temperature));

WebLoading Library Overview: JsonLoader

Like XmlLoader, the JsonLoader class is abstract and derived from HttpLoader too. It’s dedicated for downloading json data. The downloaded json is processed by the JsonParser borrowed from the fastJSON project by Mehdi Gholam. Undoubtedly it’s the fastest json parser I’ve ever met. So, many thanks to Mehdi. The parsed json data is presented as a Dictionary<string, object>, which, besides simple objects, may contain arrays and other dictionaries nested on different levels. To have an ability to verify and transform the dictionary into something different, just create a descendent of the JsonLoader. Below is the listing of the JsonLoader:

JsonLoader Source

using System;
using System.Collections;
using System.Collections.Generic;
using System.IO;
using fastJSON;

namespace WebLoading
{
    /// <summary>    
    /// Abstracts the json data loading from a specified source using Http. Provides such basic steps as data loading, verification and interpretation.
    /// </summary>
    /// <typeparam name="TParsedData">A type of the analyzed data to be returned</typeparam>
    public abstract class JsonLoader<TParsedData> : HttpLoader<Dictionary<string, object>, TParsedData> where TParsedData : class
    {
        /// <summary>
        /// Initializes a new instance of the JsonLoader class
        /// </summary>
        /// <param name="url">An initial url to load data from</param>
        public JsonLoader(string url) : base(url)
        {
        }

        /// <summary>
        /// Initializes a new instance of the JsonLoader class
        /// </summary>
        public JsonLoader()
        {
        }

        /// <summary>        
        /// Reads json data from a stream and represents it as a tree of key-value collections
        /// </summary>
        /// <param name="stream">A data stream</param>
        /// <returns>A root key-value collection</returns>
        protected override Dictionary<string, object> ReadResponseStream(Stream stream)
        {
            StreamReader reader = new StreamReader(stream);
            string jsonText = reader.ReadToEnd();

            JsonParser parser = new JsonParser(jsonText, false);
            return parser.Decode() as Dictionary<string, object>;
        }

        /// <summary>
        /// Obtains object from a tree of key-value collections using a xpath-like path (for example, "set/items/item")
        /// </summary>
        /// <param name="dictionary">A root key-value collection</param>
        /// <param name="path">A xpath-like path</param>
        /// <returns>A found object</returns>
        protected static object SelectObject(Dictionary<string, object> dictionary, string path)
        {
            List<object> res = SelectObjects(dictionary, path, true, true);
            return res.Count > 0 ? res[0] : null;
        }

        /// <summary>
        /// Obtains object from a tree of key-value collections using a xpath-like path (for example, "set/items/item")
        /// </summary>
        /// <param name="dictionary">A root key-value collection</param>
        /// <param name="path">A xpath-like path</param>
        /// <param name="returnNull">Determines if nulls should be returned or skipped</param>
        /// <returns>A found object</returns>
        protected static List<object> SelectObjects(Dictionary<string, object> dictionary, string path, bool returnNull)
        {
            return SelectObjects(dictionary, path, returnNull, false);
        }

        /// <summary>
        /// Obtains object from a tree of key-value collections using a xpath-like path (for example, "set/items/item")
        /// </summary>
        /// <param name="dictionary">A root key-value collection</param>
        /// <param name="path">A xpath-like path</param>
        /// <param name="returnNull">Determines if nulls should be returned or skipped</param>
        /// <param name="onlyFirstFound">Determines if the search should be stopped after the first found object</param>
        /// <returns>A found object</returns>
        private static List<object> SelectObjects(Dictionary<string, object> dictionary, string path, bool returnNull, bool onlyFirstFound)
        {
            List<object> res = new List<object>();

            string[] pathPatterns = path.Split(new string[] { "/" }, StringSplitOptions.RemoveEmptyEntries);

            List<Dictionary<string, object>> dicts = new List<Dictionary<string, object>>() { dictionary };
            for (int i = 0; i < pathPatterns.Length; i++)
            {
                string pattern = pathPatterns[i];

                Dictionary<string, object>[] dictsToProcess = dicts.ToArray();
                dicts.Clear();

                foreach (Dictionary<string, object> currentDictionary in dictsToProcess)
                {
                    if (currentDictionary.ContainsKey(pattern))
                    {
                        object obj = currentDictionary[pattern];

                        if (i == pathPatterns.Length - 1) // is it last pattern
                        {
                            if (obj != null || returnNull)
                                res.Add(obj);
                        }
                        else // move forward
                        {
                            if (obj is Dictionary<string, object>)
                                dicts.Add(obj as Dictionary<string, object>);
                            else
                                if (obj is IEnumerable)
                                    foreach (object childObj in obj as IEnumerable)
                                        if (childObj is Dictionary<string, object>)
                                            dicts.Add(childObj as Dictionary<string, object>);
                        }
                    }

                    if (onlyFirstFound && res.Count > 0)
                        break;
                }

                if (dicts.Count == 0)
                    break;
            }

            return res;
        }
    }
}

Pay attention to the SelectObjects methods that obtain an specified object(s) from a tree of arrays and dictionaries using a xpath-like path (for example, “set/items/item”).

Let’s take a look at yet another example, this time it’s a descendent of JsonLoader. The derived class is quite pointless and just gets names of some capital-cities. The http://api.geonames.org service returns the json data in the following form:

Json Data Sample

{
   "geonames":[
      {
         "fcodeName":"capital of a political entity",
         "toponymName":"Mexico City",
         "countrycode":"MX",
         "fcl":"P",
         "fclName":"city, village,...",
         "name":"Mexiko-Stadt",
         "wikipedia":"",
         "lng":-99.12766456604,
         "fcode":"PPLC",
         "geonameId":3530597,
         "lat":19.428472427036,
         "population":12294193
      },
      {
         "fcodeName":"capital of a political entity",
         "toponymName":"Manila",
         "countrycode":"PH",
         "fcl":"P",
         "fclName":"city, village,...",
         "name":"Manila",
         "wikipedia":"",
         "lng":120.9822,
         "fcode":"PPLC",
         "geonameId":1701668,
         "lat":14.6042,
         "population":10444527
      },
      {
         "fcodeName":"capital of a political entity",
         "toponymName":"Dhaka",
         "countrycode":"BD",
         "fcl":"P",
         "fclName":"city, village,...",
         "name":"Dhaka",
         "wikipedia":"",
         "lng":90.40743827819824,
         "fcode":"PPLC",
         "geonameId":1185241,
         "lat":23.710395616597037,
         "population":10356500
      }
   ]
}

The next listing is the GeoNamesCitiesLoader class:

GeoNamesCitiesLoader Source and How to use

using System;
using System.Collections;
using System.Collections.Generic;
using WebLoading;

namespace TestApp
{
    public class GeoNamesCitiesLoader : JsonLoader<List<GeoNamesCitiesLoader.City>>
    {
        public class City
        {
            public string  FCodeName   { get; set; }
            public string  ToponymName { get; set; }
            public string  CountryCode { get; set; }
            public string  Name        { get; set; }
            public string  Wikipedia   { get; set; }
            public decimal Population  { get; set; }
            public double  Longitude   { get; set; }
            public double  Latitude    { get; set; }            
        }

        protected override bool IsRawDataValid(Dictionary<string, object> rawData)
        {
            return base.IsRawDataValid(rawData);
        }

        protected override string PrepareUrl()
        {
            return "http://api.geonames.org/citiesJSON?north=44.1&south=-9.9&east=-22.4&west=55.2&lang=de&username=demo";
        }

        protected override List<City> ParseData(Dictionary<string, object> rawData)
        {
            List<City> res = new List<City>();

            IEnumerable arrayOfDictionaries = SelectObject(rawData, "geonames") as IEnumerable;
            foreach (Dictionary<string, object> dict in arrayOfDictionaries)
                res.Add(CreateCity(dict));

            return res;
        }
       
        private City CreateCity(Dictionary<string, object> metadata)
        {
            City res = new City();

            foreach (KeyValuePair<string, object> pair in metadata)
            {
                switch (pair.Key.ToLower())
                {
                    case "fcodename":   res.FCodeName   = Convert.ToString(pair.Value);  break;
                    case "toponymname": res.ToponymName = Convert.ToString(pair.Value);  break;
                    case "countrycode": res.CountryCode = Convert.ToString(pair.Value);  break;
                    case "name":        res.Name        = Convert.ToString(pair.Value);  break;
                    case "wikipedia":   res.Wikipedia   = Convert.ToString(pair.Value);  break;
                    case "lng":         res.Longitude   = Convert.ToDouble(pair.Value);  break;
                    case "lat":         res.Latitude    = Convert.ToDouble(pair.Value);  break;
                    case "population":  res.Population  = Convert.ToDecimal(pair.Value); break;                    
                }
            }            

            return res;
        }
    }
}

//...
// "how to use" sample
GeoNamesCitiesLoader geoNamesCitiesLoader = new GeoNamesCitiesLoader();
List<GeoNamesCitiesLoader.City> cities = geoNamesCitiesLoader.LoadData();
if (cities != null)
    foreach (GeoNamesCitiesLoader.City city in cities)
        Console.WriteLine(string.Format("{0} [Population: {1}]", city.ToponymName, city.Population));

The source code and examples are available to download here.

Categories: C#, JSON, XML Tags: , ,

SharePoint: Working with BDC Secondary Fields

April 17th, 2012 No comments

    As you probably know, in SharePoint 2010 Business Data Connectivity replaced Business Data Catalog of SharePoint 2007. Some changes affects how Business Data Columns are presented in a list’s schema. In SP 2007 a declaration of a Business Data Column in a schema.xml may look like the following:

<Field Type="BusinessData" DisplayName="Product"
Required="FALSE" ID="{bc203358-6113-470f-9b08-f6100cc034f2}"
StaticName="Product" BaseRenderingType="Text" Name="Product"
SystemInstance="ExternalProductDB_Instance" Entity="Products"
BdcField="Name" Profile="" HasActions="False"
RelatedField="Products_ID"
RelatedFieldBDCField="" RelatedFieldWssStaticName="Products_ID"

SecondaryFieldBdcNames="Price:Producer"
SecondaryFieldWssNames="Product_x003a__x0020_Price:Product_x003a__x0020_Producer"
SecondaryFieldsWssStaticNames="Product_x003a__x0020_Price:Product_x003a__x0020_Producer" />

In contrast, in SP 2010 it looks like

<Field Type="BusinessData" DisplayName="Product"
Required="FALSE" ID="{bc203358-6113-470f-9b08-f6100cc034f2}"
StaticName="Product" BaseRenderingType="Text" Name="Product"
SystemInstance="ExternalProductDB_Instance" Entity="Products"
BdcField="Name" Profile="" HasActions="False"
RelatedField="Products_ID"
RelatedFieldBDCField="" RelatedFieldWssStaticName="Products_ID"

SecondaryFieldBdcNames="6%209%20Price%20Producer%204"
SecondaryFieldWssNames="27%2030%20Product%5Fx003a%5F%5Fx0020%5FPrice%20Product%5Fx003a%5F%5Fx0020%5FProducer%206"
SecondaryFieldsWssStaticNames="27%2030%20Product%5Fx003a%5F%5Fx0020%5FPrice%20Product%5Fx003a%5F%5Fx0020%5FProducer%206" />

Undoubtedly, in SP 2010 the secondary fields became practically unreadable. Indeed, the format of secondary fields‘ presentation is revised. Moreover some kind of URL encoding are applied to them. Let’s examine how these secondary fields could look before the URL encoding is applied:

<Field
...
SecondaryFieldBdcNames="6 9 Price Producer 4"
SecondaryFieldWssNames="27 30 Product_x003a__x0020_Price Product_x003a__x0020_Producer 6"
SecondaryFieldsWssStaticNames="27 30 Product_x003a__x0020_Price Product_x003a__x0020_Producer 6" />

Now it’s pretty easy to figure out the new format. Take a look at the SecondaryFieldBdcNames attribute. It contains names of two secondary bdc fields: ‘Price’ and ‘Producer’. 6 is the length of the ‘Price’ name + 1 for a space character right after the name. 9 is the length of the ‘Procuder’ name + 1 for a space character after the name. 4 is the length of the sub-string ‘6 9 ‘ (including spaces), which contains the lengths of the fields’ names. See a picture below:

Format of Secondary Fields

Note that the SecondaryFieldBdcNames, SecondaryFieldWssNames and SecondaryFieldsWssStaticNames have the same format.

We have a lot of code interacting with Business Data Columns, thus we were interested in means allowing easily to decode, encode and parse Secondary Fields attributes. In the Microsoft.SharePoint.dll, there is the internal BdcClientUtil class containing the basic methods to work with Secondary Fields:

internal class BdcClientUtil
{
    ...
    string[] SplitStrings(string combinedEncoded);
    string   CombineStrings(string[] strings);
    ...
}

So, using .Net Reflector I’ve extracted these methods along with several others auxiliary ones and put them into the helper-class called SecondaryFieldNamesHelper. All internal methods and properties were honestly stolen from Microsoft.SharePoint.dll, the public ones were added by me and described below:

  • string Encode(string[] secondaryFieldNames) – accepts an array of field names and returns the string formatted and encoded according to the SharePoint 2010 requirements;
  • string[] Decode(string str) – accepts an encoded string, decodes it and returns a resultant array of field names;
  • bool IsEncodedString(string str) – checks whether a passed string is encoded;
  • string ConvertToSP2010(string str) – converts a SP 2007 colon-separated string of secondary fields into another one formatted and encoded according to the SharePoint 2010 requirements;

Below is the source code of the SecondaryFieldNamesHelper:

SecondaryFieldNamesHelper Sources

using System;
using System.Collections;
using System.Text;
using System.Globalization;
using System.IO;
using System.Web.UI;

namespace Helpers
{
    public static class SecondaryFieldNamesHelper
    {
        #region fields & properties
        private static string[] s_crgstrUrlHexValue = new string[] 
        { 
            "%00", "%01", "%02", "%03", "%04", "%05", "%06", "%07", "%08", "%09", "%0A", "%0B", "%0C", "%0D", "%0E", "%0F", 
            "%10", "%11", "%12", "%13", "%14", "%15", "%16", "%17", "%18", "%19", "%1A", "%1B", "%1C", "%1D", "%1E", "%1F", 
            "%20", "%21", "%22", "%23", "%24", "%25", "%26", "%27", "%28", "%29", "%2A", "%2B", "%2C", "%2D", "%2E", "%2F", 
            "%30", "%31", "%32", "%33", "%34", "%35", "%36", "%37", "%38", "%39", "%3A", "%3B", "%3C", "%3D", "%3E", "%3F", 
            "%40", "%41", "%42", "%43", "%44", "%45", "%46", "%47", "%48", "%49", "%4A", "%4B", "%4C", "%4D", "%4E", "%4F", 
            "%50", "%51", "%52", "%53", "%54", "%55", "%56", "%57", "%58", "%59", "%5A", "%5B", "%5C", "%5D", "%5E", "%5F", 
            "%60", "%61", "%62", "%63", "%64", "%65", "%66", "%67", "%68", "%69", "%6A", "%6B", "%6C", "%6D", "%6E", "%6F", 
            "%70", "%71", "%72", "%73", "%74", "%75", "%76", "%77", "%78", "%79", "%7A", "%7B", "%7C", "%7D", "%7E", "%7F", 
            "%80", "%81", "%82", "%83", "%84", "%85", "%86", "%87", "%88", "%89", "%8A", "%8B", "%8C", "%8D", "%8E", "%8F", 
            "%90", "%91", "%92", "%93", "%94", "%95", "%96", "%97", "%98", "%99", "%9A", "%9B", "%9C", "%9D", "%9E", "%9F", 
            "%A0", "%A1", "%A2", "%A3", "%A4", "%A5", "%A6", "%A7", "%A8", "%A9", "%AA", "%AB", "%AC", "%AD", "%AE", "%AF", 
            "%B0", "%B1", "%B2", "%B3", "%B4", "%B5", "%B6", "%B7", "%B8", "%B9", "%BA", "%BB", "%BC", "%BD", "%BE", "%BF", 
            "%C0", "%C1", "%C2", "%C3", "%C4", "%C5", "%C6", "%C7", "%C8", "%C9", "%CA", "%CB", "%CC", "%CD", "%CE", "%CF", 
            "%D0", "%D1", "%D2", "%D3", "%D4", "%D5", "%D6", "%D7", "%D8", "%D9", "%DA", "%DB", "%DC", "%DD", "%DE", "%DF", 
            "%E0", "%E1", "%E2", "%E3", "%E4", "%E5", "%E6", "%E7", "%E8", "%E9", "%EA", "%EB", "%EC", "%ED", "%EE", "%EF", 
            "%F0", "%F1", "%F2", "%F3", "%F4", "%F5", "%F6", "%F7", "%F8", "%F9", "%FA", "%FB", "%FC", "%FD", "%FE", "%FF"
        };
        #endregion

        #region public methods
        public static bool IsEncodedString(string str)
        {
            if (string.IsNullOrEmpty(str))
                return false;

            bool res = true;
            try
            {
                string[] splittedString = SplitStrings(str);
            }
            catch
            {
                res = false;
            }
            return res;
        }

        public static string Encode(string[] secondaryFieldNames)
        {
            return CombineStrings(secondaryFieldNames);
        }

        public static string[] Decode(string str)
        {
            if(string.IsNullOrEmpty(str))
                return new string[0];
            return SplitStrings(str);
        }

        public static string ConvertToSP2010(string str)
        {
            if (IsEncodedString(str))
                return str;

            string[] fieldNames = str.Split(new string[] { ":" }, StringSplitOptions.RemoveEmptyEntries);
            string encodedVal = CombineStrings(fieldNames);
            return encodedVal;
        }
        #endregion

        #region internal methods
        private static string[] SplitStrings(string combinedEncoded)
        {
            string[] array = null;
            ArrayList list = new ArrayList();
            if ("0" == combinedEncoded)
                return new string[0];
            try
            {
                string str = UrlKeyValueDecode(combinedEncoded);
                string[] strArray2 = str.Split(new char[] { ' ' }, StringSplitOptions.None);
                int result = 0;
                if ((strArray2 == null) || !int.TryParse(strArray2[strArray2.Length - 1], NumberStyles.Integer, 

CultureInfo.InvariantCulture, out result))
                    throw new ArgumentException(string.Empty, "combinedEncoded");
                int num2 = str.LastIndexOf(' ');
                string str2 = str.Substring(result, num2 - result);
                int length = str2.Length;
                int index = 0;
                int startIndex = 0;
                while (startIndex < length)
                {
                    string s = strArray2[index];
                    int num6 = 1;
                    if ((s != null) && (s.Length == 0))
                        list.Add(null);
                    else
                    {
                        if (!int.TryParse(s, NumberStyles.Integer, CultureInfo.InvariantCulture, out num6))
                            throw new ArgumentException(string.Empty, "combinedEncoded");
                        list.Add(str2.Substring(startIndex, num6 - 1));
                    }
                    startIndex += num6;
                    index++;
                }
                array = new string[list.Count];
                list.CopyTo(array);
            }
            catch (Exception exception)
            {
                throw new ArgumentException(string.Empty, "combinedEncoded", exception);
            }
            return array;
        }

        private static string UrlKeyValueDecode(string keyOrValueToDecode)
        {
            if (string.IsNullOrEmpty(keyOrValueToDecode))
                return keyOrValueToDecode;
            return UrlDecodeHelper(keyOrValueToDecode, keyOrValueToDecode.Length, true);
        }

        private static string UrlDecodeHelper(string stringToDecode, int length, bool decodePlus)
        {
            if ((stringToDecode == null) || (stringToDecode.Length == 0))
                return stringToDecode;
            StringBuilder builder = new StringBuilder(length);
            byte[] bytes = null;
            int nIndex = 0;
            while (nIndex < length)
            {
                char ch = stringToDecode[nIndex];
                if (ch < ' ')
                    nIndex++;
                else
                {
                    if (decodePlus && (ch == '+'))
                    {
                        builder.Append(" ");
                        nIndex++;
                        continue;
                    }
                    if (IsHexEscapedChar(stringToDecode, nIndex, length))
                    {
                        if (bytes == null)
                            bytes = new byte[(length - nIndex) / 3];
                        int count = 0;
                        do
                        {
                            int num3 = (FromHexNoCheck(stringToDecode[nIndex + 1]) * 0x10) + FromHexNoCheck(stringToDecode[nIndex + 

2]);
                            bytes[count++] = (byte)num3;
                            nIndex += 3;
                        }
                        while (IsHexEscapedChar(stringToDecode, nIndex, length));
                        builder.Append(Encoding.UTF8.GetChars(bytes, 0, count));
                        continue;
                    }
                    builder.Append(ch);
                    nIndex++;
                }
            }
            if (length < stringToDecode.Length)
                builder.Append(stringToDecode.Substring(length));
            return builder.ToString();
        }

        private static bool IsHexEscapedChar(string str, int nIndex, int nPathLength)
        {
            if ((((nIndex + 2) >= nPathLength) || (str[nIndex] != '%')) || (!IsHexDigit(str[nIndex + 1]) || !IsHexDigit(str[nIndex + 

2])))
                return false;
            if (str[nIndex + 1] == '0')
                return (str[nIndex + 2] != '0');
            return true;
        }

        private static bool IsHexDigit(char digit)
        {
            if ((('0' > digit) || (digit > '9')) && (('a' > digit) || (digit > 'f')))
                return (('A' <= digit) && (digit <= 'F'));
            return true;
        }

        private static int FromHexNoCheck(char digit)
        {
            if (digit <= '9')
                return (digit - '0');
            if (digit <= 'F')
                return ((digit - 'A') + 10);
            return ((digit - 'a') + 10);
        }

        private static string CombineStrings(string[] strings)
        {
            StringBuilder builder = new StringBuilder();
            int index = 0;
            for (int i = 0; i < strings.Length; i++)
            {
                string str = strings[i];
                string str2 = ((str != null) ? ((str.Length + 1)).ToString(CultureInfo.InvariantCulture) : string.Empty) + ' ';
                builder.Insert(index, str2);
                index += str2.Length;
                builder.Append(str + ' ');
            }
            builder.Append(index.ToString(CultureInfo.InvariantCulture));
            return UrlKeyValueEncode(builder.ToString());
        }

        private static string UrlKeyValueEncode(string keyOrValueToEncode)
        {
            if ((keyOrValueToEncode == null) || (keyOrValueToEncode.Length == 0))
                return keyOrValueToEncode;
            StringBuilder sb = new StringBuilder(0xff);
            HtmlTextWriter output = new HtmlTextWriter(new StringWriter(sb, CultureInfo.InvariantCulture));
            UrlKeyValueEncode(keyOrValueToEncode, output);
            return sb.ToString();
        }

        private static void UrlKeyValueEncode(string keyOrValueToEncode, TextWriter output)
        {
            if (((keyOrValueToEncode != null) && (keyOrValueToEncode.Length != 0)) && (output != null))
            {
                bool fUsedNextChar = false;
                int startIndex = 0;
                int length = 0;
                int num3 = keyOrValueToEncode.Length;
                for (int i = 0; i < num3; i++)
                {
                    char ch = keyOrValueToEncode[i];
                    if (((('0' <= ch) && (ch <= '9')) || (('a' <= ch) && (ch <= 'z'))) || (('A' <= ch) && (ch <= 'Z')))
                        length++;
                    else
                    {
                        if (length > 0)
                        {
                            output.Write(keyOrValueToEncode.Substring(startIndex, length));
                            length = 0;
                        }
                        UrlEncodeUnicodeChar(output, keyOrValueToEncode[i], (i < (num3 - 1)) ? keyOrValueToEncode[i + 1] : '\0', out 

fUsedNextChar);
                        if (fUsedNextChar)
                            i++;
                        startIndex = i + 1;
                    }
                }
                if ((startIndex < num3) && (output != null))
                    output.Write(keyOrValueToEncode.Substring(startIndex));
            }
        }

        private static void UrlEncodeUnicodeChar(TextWriter output, char ch, char chNext, out bool fUsedNextChar)
        {
            bool fInvalidUnicode = false;
            UrlEncodeUnicodeChar(output, ch, chNext, ref fInvalidUnicode, out fUsedNextChar);
        }

        private static void UrlEncodeUnicodeChar(TextWriter output, char ch, char chNext, ref bool fInvalidUnicode, out bool 

fUsedNextChar)
        {
            int num = 0xc0;
            int num2 = 0xe0;
            int num3 = 240;
            int num4 = 0x80;
            int num5 = 0xd800;
            int num6 = 0xfc00;
            int num7 = 0x10000;
            fUsedNextChar = false;
            int index = ch;
            if (index <= 0x7f)
                output.Write(s_crgstrUrlHexValue[index]);
            else
            {
                int num8;
                if (index <= 0x7ff)
                {
                    num8 = num | (index >> 6);
                    output.Write(s_crgstrUrlHexValue[num8]);
                    num8 = num4 | (index & 0x3f);
                    output.Write(s_crgstrUrlHexValue[num8]);
                }
                else if ((index & num6) != num5)
                {
                    num8 = num2 | (index >> 12);
                    output.Write(s_crgstrUrlHexValue[num8]);
                    num8 = num4 | ((index & 0xfc0) >> 6);
                    output.Write(s_crgstrUrlHexValue[num8]);
                    num8 = num4 | (index & 0x3f);
                    output.Write(s_crgstrUrlHexValue[num8]);
                }
                else if (chNext != '\0')
                {
                    index = (index & 0x3ff) << 10;
                    fUsedNextChar = true;
                    index |= chNext & 'Ͽ';
                    index += num7;
                    num8 = num3 | (index >> 0x12);
                    output.Write(s_crgstrUrlHexValue[num8]);
                    num8 = num4 | ((index & 0x3f000) >> 12);
                    output.Write(s_crgstrUrlHexValue[num8]);
                    num8 = num4 | ((index & 0xfc0) >> 6);
                    output.Write(s_crgstrUrlHexValue[num8]);
                    num8 = num4 | (index & 0x3f);
                    output.Write(s_crgstrUrlHexValue[num8]);
                }
                else
                    fInvalidUnicode = true;
            }
        }
        #endregion
    }
}

The SecondaryFieldNamesHelper can be used as shown below:

SPBusinessDataField bdcField = ...

string secondaryFieldWssNamesProperty = bdcField.GetProperty("SecondaryFieldWssNames");
string[] secondaryWssFieldNames = SecondaryFieldNamesHelper.Decode(property);

string secondaryFieldBdcNamesProperty = bdcField.GetProperty("SecondaryFieldBdcNames");
string[] secondaryFieldBdcNames = SecondaryFieldNamesHelper.Decode(secondaryFieldBdcNamesProperty);

string sp2010WssStaticNames = 
   SecondaryFieldNamesHelper.ConvertToSP2010("Product_x003a__x0020_Price:Product_x003a__x0020_Producer");

As a .cs file the SecondaryFieldNamesHelper class is available here.

C#: Simple Command Line Arguments Parser

March 22nd, 2012 2 comments

    Working with SharePoint applications I very often develop small console applications for adjusting lists, list items, content types and so on. To pass some parameters into those utilities I employ the command line arguments. Then inside applications I deal with an array of arguments – string[] args:

class Program
{
    static void Main(string[] args)
    {
        // get passed parameters from args
    }
}

I usually use the following notation: -paramName paramValue. Here a parameter name with the leading minus sign (‘-‘) is followed by a parameter value. If the presence of value isn’t assumed, only -paramName is used. For example, the following command line

-url "http://dotnetfollower.com" -useElevatedPrivileges

directs an utility to “process the web site http://dotnetfollower.com and use the elevated objects for that”.

To simplify parameters fetching from the args array I’ve implemented an auxiliary class – InputArguments, which parses the arguments and fills a dictionary out with proper key-value pairs. The dictionary allows accessing a parameter value in the way like this: InputArguments[“-url”] or InputArguments[“url”]. The source code of the InputArguments class is listed below:

using System;
using System.Collections.Generic;

namespace Common
{
    public class InputArguments
    {
        #region fields & properties
        public const string DEFAULT_KEY_LEADING_PATTERN = "-";

        protected Dictionary<string, string> _parsedArguments   = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
        protected readonly string            _keyLeadingPattern;

        public string this [string key]
        {
            get { return GetValue(key); }
            set
            {
                if (key != null)
                    _parsedArguments[key] = value;
            }
        }
        public string KeyLeadingPattern
        {
            get { return _keyLeadingPattern; }
        }
        #endregion

        #region public methods
        public InputArguments(string[] args, string keyLeadingPattern)
        {
            _keyLeadingPattern = !string.IsNullOrEmpty(keyLeadingPattern) ? keyLeadingPattern : DEFAULT_KEY_LEADING_PATTERN;

            if (args != null && args.Length > 0)
                Parse(args);
        }
        public InputArguments(string[] args) : this(args, null)
        {
        }

        public bool Contains(string key)
        {
            string adjustedKey;
            return ContainsKey(key, out adjustedKey);
        }

        public virtual string GetPeeledKey(string key)
        {
            return IsKey(key) ? key.Substring(_keyLeadingPattern.Length) : key;
        }
        public virtual string GetDecoratedKey(string key)
        {
            return !IsKey(key) ? (_keyLeadingPattern + key) : key;
        }
        public virtual bool IsKey(string str)
        {
            return str.StartsWith(_keyLeadingPattern);
        }
        #endregion

        #region internal methods
        protected virtual void Parse(string[] args)
        {
            for (int i = 0; i < args.Length; i ++)
            {
                if(args[i] == null) continue;
                
                string key = null;
                string val = null;

                if(IsKey(args[i])) 
                {
                    key = args[i];

                    if(i + 1 < args.Length && !IsKey(args[i + 1]))
                    {
                        val = args[i + 1];
                        i ++;
                    }
                }
                else
                    val = args[i];

                // adjustment
                if (key == null)
                {
                    key = val;
                    val = null;
                }
                _parsedArguments[key] = val;
            }
        }

        protected virtual string GetValue(string key)
        {
            string adjustedKey;
            if(ContainsKey(key, out adjustedKey))
                return _parsedArguments[adjustedKey];

            return null;
        }

        protected virtual bool ContainsKey(string key, out string adjustedKey)
        {
            adjustedKey = key;

            if (_parsedArguments.ContainsKey(key))
                return true;

            if (IsKey(key))
            {
                string peeledKey = GetPeeledKey(key);
                if(_parsedArguments.ContainsKey(peeledKey))
                {
                    adjustedKey = peeledKey;
                    return true;
                }
                return false;
            }

            string decoratedKey = GetDecoratedKey(key);
            if(_parsedArguments.ContainsKey(decoratedKey))
            {
                adjustedKey = decoratedKey;
                return true;
            }
            return false;
        }
        #endregion
    }
}

As you can see, the InputArguments stores the parsed parameters and theirs values in the _parsedArguments dictionary, which is accessible through the special indexer. Pay attention to the ContainsKey methods, which attempts different variations (with or without leading symbols) of parameter name in case the passed name isn’t presented among the stored arguments. So, passing the

-url "http://dotnetfollower.com" -useElevatedPrivileges

arguments into an application, the InputArguments can be used as follows:

static void Main(string[] args)
{
    InputArguments arguments = new InputArguments(args);

    Console.WriteLine(arguments["-url"]);
    if (arguments.Contains("useElevatedPrivileges"))
        Console.WriteLine("useElevatedPrivileges is set");
}

The above mentioned command line arguments will be turned into the following key-value pairs:

{["-url", "http://dotnetfollower.com"]}
{["-useElevatedPrivileges", null]}

Below is the more complex command line borrowed from the WSPBuilder and slightly modified to have orphaned values without a foregoing parameter name:

"some orphaned value" -ExpandTypes false -BuildSafeControls true  
-WSPName mySPSolution.wsp  -Outputpath "C:\WSPDeployment\myApp" 
-SolutionId d403bb18-c5f2-4b43-9d55-12b256a6295a 
-SolutionPath "C:\WSPDeployment\myApp" -TraceLevel Verbose 
-DLLReferencePath "C:\WSPDeployment\ReferencedAssemblies"

This arguments will be turned into the following:

{["some orphaned value", null]}
{["-ExpandTypes", "false"]}
{["-BuildSafeControls", "true"]}
{["-WSPName", "mySPSolution.wsp"]}
{["-Outputpath", "C:\WSPDeployment\myApp"]}
{["-SolutionId", "d403bb18-c5f2-4b43-9d55-12b256a6295a"]}
{["-SolutionPath", "C:\WSPDeployment\myApp"]}
{["-TraceLevel", "Verbose"]}
{["-DLLReferencePath", "C:\WSPDeployment\ReferencedAssemblies"]}

Frankly speaking, I don’t use the InputArguments class directly. For every utility I create a derived class to provide an easy access to the values of parameters specific for that particular application. For example, the derived class employing the -url and -useElevatedPrivileges may look like the following:

public class UtilityArguments : InputArguments
{
    public bool UseElevatedPrivileges
    {
        get { return GetBoolValue("-useElevatedPrivileges"); }
    }

    public string Url
    {
        get { return GetValue("url"); }
    }

    public UtilityArguments(string[] args) : base(args)
    {
    }

    protected bool GetBoolValue(string key)
    {
        string adjustedKey;
        if (ContainsKey(key, out adjustedKey))
        {
            bool res;
            bool.TryParse(_parsedArguments[adjustedKey], out res);
            return res;
        }
        return false;
    }
}

It can be used as follows:

static void Main(string[] args)
{
    UtilityArguments arguments = new UtilityArguments(args);

    Console.WriteLine("Url: " + arguments.Url);
    Console.WriteLine("UseElevatedPrivileges: " + arguments.UseElevatedPrivileges);
}

If you prefer using another sign or group of signs before parameter name in command line, use the proper constructor of the InputArguments class

static void Main(string[] args)
{
    InputArguments arguments = new InputArguments(args, "/");
    // ...
}

or

public class MyArguments : InputArguments
{
    // ...
    public MyArguments(string[] args) : base(args, "--")
    {
    }
    // ...
}

Categories: C# Tags: ,

SharePoint: Manually Upgrade Business Data Catalog Application Definitions to Business Data Connectivity Models

March 13th, 2012 No comments

    Trying to import a legacy Application Definition File of SharePoint 2007 into Business Data Connectivity Service of SharePoint 2010, you apparently got at least one of the errors shown below:

  • Application definition import failed. The following error occurred: The root element of a valid Metadata package must be ‘Model’ in namespace ‘http://schemas.microsoft.com/windows/2007/BusinessDataCatalog’. The root in the given package is ‘LobSystem’ in namespace ‘http://schemas.microsoft.com/office/2006/03/BusinessDataCatalog’. Error was encountered at or just before Line: ‘2’ and Position: ‘2’;
  • Application definition import failed. The following error occurred: BDC Model does not correctly match the schema. The required attribute ‘Namespace’ is missing. Error was encountered at or just before Line: ’20’ and Position: ’10’;
  • Application definition import failed. The following error occurred: ReturnTypeDescriptor of MethodInstance with Name ‘ProductSpecificFinderInstance’ on Entity (External Content Type) with Name ‘Product’ in Namespace ‘ExternalProductDB’ should not be a Collection TypeDescriptor for MethodInstances of Type ‘SpecificFinder’. Parameter name: rawValues.ReturnTypeDescriptorId Error was encountered at or just before Line: ‘171’ and Position: ’18’;
  • and so on

As it’s known, in SharePoint 2010, Business Data Catalog (BDC) was replaced with Business Data Connectivity with the same abbreviation. One of the changed things is the format of xml-based Application Definition Files. If you make an in-place upgrade of a live SharePoint 2007 application to SharePoint 2010, bdc metadata will be automatically upgraded as well and will become usable with the Business Data Connectivity. But if the in-place upgrade isn’t an option for you, you can upgrade your xml-based Application Definition Files manually. The manual algorithm step by step is described here – How to: Manually Upgrade Business Data Catalog Application Definitions to Business Data Connectivity Models.

For one of our applications we settled on the manual upgrade of its metadata files. But If I call myself a programmer, I have to try to automate the algorithm, especially taking into account 20+ files required to upgrade. So, I’ve developed a simple application for alteration of the legacy xml-based Application Definition Files to make them compatible with SharePoint 2010. However I’d like to notice that the given converter doesn’t follow entirely the procedure described by Microsoft, but performs only steps allowing our particular metadata files to be successfully imported into the Business Data Connectivity Service. For example, our files don’t comprise actions and associations, thus the application does nothing at all with <Action> and <Association> elements. So, consider this converter as a start point of developing the new one satisfying your own conditions and requirements.

Below I enumerated the necessary and sufficient changes to be applied to our particular metadata files so that it enables us to make them compatible with SharePoint 2010. Exactly these very steps and a few less important I’ve implemented in the converter.

  • the root element in the Application Definition File must be a <Model>;
  • the <Model> must contain <LobSystems> element, which in turn must wrap the former root node – <LobSystem>;
  • the <LobSystem> element mustn’t contain the Version-attribute;
  • the <Entity> element must contain the new attributes – Namespace and Version;
  • the <Identifier> element mustn’t contain an assembly name in its TypeName-attribute; For example, TypeName=”System.String, mscorlib” has to turn into TypeName=”System.String”;
  • the <MethodInstance> element with Type-attribute value of SpecificFinder should include the Default-attribute with value of true;
  • if the <TypeDescriptor> element has the IsCollection attribute set to true, the MethodInstance return TypeDescriptor should be updated to be an element of the collection. In practice, that means the ReturnTypeDescriptorPath-attribute with an appropriate value should be added to <MethodInstance> element, and the obsolete ReturnTypeDescriptorLevel and ReturnTypeDescriptorName attributes should be deleted;

Note that when modifying a <Entity> element, the values of the Namespace and Version attributes are copied respectively from the values of Name and Version attributes of the <LobSystem> element, which wraps the <Entity> element. The same approach is used while the in-place upgrade takes place.

After the changes are applied, and if they are sufficient for your metadata, the result files can be imported into Business Data Connectivity Service. During the import process, you may get the warnings. Consider fixing them in the future, but at the present stage you can simply ignore them. The most popular warnings are listed below:

  • This Model contains LobSystem (External System) of Type ‘WebService’ which is deprecated and may be removed in future releases.
  • The MethodInstance of type ‘Finder’ with Name ‘FindProducts’ does not have a Limit Filter.
  • The TypeDescriptor ‘From’ on Parameter ‘GetProductsFiltered’ on Method ‘GetItems’ of Entity (External Content Type) ‘Product’ with Namespace ‘ExternalProductDB’ has a DateTime value, but there is no Interpretation describing what the External System expects and returns as the DateTimeKind. The DateTime value is assumed to be UTC.
  • Note: I made the converter-application fix the warnings regarding the DateTime type and UTC, so they won’t bother you.

    The converter is very straightforward to use. Using the button ‘+’, simply add to the left section the files to be upgraded. Then click the button ‘>>’, and you’ll get the upgraded ones in the right section. Double click on file name opens an overview form to browse the input or result xml. Physically, the result files are located in c:\output folder. The application doesn’t use any SharePoint-related libraries.

    Upgrade 2007 BDC model to SP2010

    You can download the application from this page or by using the direct link. The Visual Studio 2010 solution and appropriate executable file are in the archive.

SharePoint: Migration of custom upload page derived from UploadPage to SP 2010

March 5th, 2012 No comments

    In our SharePoint 2007 application, there was a custom upload page for a document library. The custom upload page was derived from the Microsoft.SharePoint.ApplicationPages.UploadPage defined in the Microsoft.SharePoint.ApplicationPages.dll. Within application web pages, all links to the upload page were direct, while the page itself was located in the _layouts folder. After migration to the SharePoint 2010, I’ve found out that every request to the custom upload page is transferred to the Uploadex.aspx (located in the _layouts folder as well): the URL in browser corresponds to our upload page, but the content corresponds to the Uploadex.aspx. After a short investigation by means of .Net Reflector I found the reason in the UploadPage base class defined in Microsoft.SharePoint.ApplicationPages.dll, Version=14.0.0.0. Let’s take a look at the OnPreInit method of the UploadPage:

protected override void OnPreInit(EventArgs e)
{
    if (!this.customPage)
    {
        string customUploadPage = base.Web.CustomUploadPage;
        if (!string.IsNullOrEmpty(customUploadPage))
        {
            try
            {
                base.Server.Transfer(customUploadPage);
            }
            catch (Exception)
            {
            }
        }
    }
    base.OnPreInit(e);
}

*Note: this code was added in SharePoint 2010 and wasn’t presented in SharePoint 2007

As we can see, the SharePoint 2010 itself allows to define the CustomUploadPage for a website. If it’s defined, the UploadPage automatically transfers every request to it. Apparently, after migration the CustomUploadPage was somehow set to _layouts/Uploadex.aspx. Another interesting moment here is the customPage boolean field, which acts as marker indicating whether the current page is the custom upload page or not.

Probably, the acceptable solution for our issue would be to set the CustomUploadPage property of the SPWeb object to the URL of our own custom upload page. But, firstly, the problem here is that our upload page derived from the UploadPage doesn’t know that it’s a custom one as the customPage field is set to false by default. Thus, without any code modification, we will have a kind of loop here, because our custom page due to the UploadPage base class will be transferring the request to itself infinitely. Secondly, I don’t want to rely on the CustomUploadPage property, which can be unexpectedly changed. I just want when I click on the direct link to our upload page, this page would be opened without any sudden redirections and transfers.

So, the best solution is to set the customPage field to true within the constructor of our custom upload page. No transfer happens in this case, and we don’t need to deal with the CustomUploadPage at all. In our case It looks like:

public class MyUploadPage : UploadPage
{
    // ...
    public MyUploadPage()
    {
        customPage = true;
    }
    // ...
}

That’s all!