Archive

Author Archive

Dynamics CRM: Awaiting for Bulk Delete completion

October 29th, 2013 No comments

    In my previous post I described my method for awaiting of async job completion. I have a similar one for awaiting Bulk Delete operation completion as well. Dealing with Bulk Delete we have to keep in mind the following: initiating the operation through the code, the GUID returned in the BulkDeleteResponse is not the id of the Bulk Delete operation, but the id of some async job; the target Bulk Delete operation will be created a bit later and will be associated with the async job; The code below illustrates the submitting of Bulk Delete request:

...
OrganizationServiceProxy proxy   = ...;
QueryExpression[]        queries = ...;

var bulkDeleteRequest = new BulkDeleteRequest
{
	JobName               = "Bulk Delete " + DateTime.Now,
	QuerySet              = queries,
	StartDateTime         = DateTime.Now.AddDays(-2), /* -2 is to avoid the problem with 
                                                          local and utc times */
	ToRecipients          = new Guid[] {},
	CCRecipients          = new Guid[] {},
	SendEmailNotification = false,
	RecurrencePattern     = String.Empty
};        

var bulkDeleteResponse = (BulkDeleteResponse)proxy.Execute(bulkDeleteRequest);

// id of the job which will be associated with the Bulk Delete operation
Guid jobId = bulkDeleteResponse.JobId;

So, having id of the async job that will be shortly associated with the Bulk Delete operation, the awaiting process supposes two steps:

  • wait until the Bulk Delete operation has been created;
  • wait until the Bulk Delete operation has completed;

For both steps the RetrieveMultiple is repeatedly called against the bulkdeleteoperation records. Firstly, we try to find a record, the asyncoperationid attribute of which equals the async job id received before. Once the record has been found, it means the Bulk Delete operation has been created. Secondly, then we wait for the Bulk Delete operation completion.

The steps mentioned above have been implemented in the WaitForBulkDeleteCompletion method shown below. Like the WaitForAsyncJobCompletion, the WaitForBulkDeleteCompletion allows adjusting number of retries and sleep interval. It also analyses operation’s current state (and status) and provides actual progress on every iteration of the awaiting process.

public static BulkDeleteState WaitForBulkDeleteCompletion(OrganizationServiceProxy proxy, 
														BulkDeleteAwaiting asyncJobAwaiting)
{
	var state = new BulkDeleteState();

	#if WRITE_TRACE_INFO
	WriteInfo(string.Format(@"Waiting for completion of the bulk 
			delete associated with the Async Job {0}...", asyncJobAwaiting.AsyncJobId));
	#endif

	// Query for bulk delete operation and check for status.
	var bulkQuery = new QueryByAttribute(BulkDeleteOperation.EntityLogicalName)
		{
			ColumnSet = new ColumnSet(true)
		};
	bulkQuery.Attributes.Add("asyncoperationid");
	bulkQuery.Values.Add(asyncJobAwaiting.AsyncJobId);

	state.AsyncJobId        = asyncJobAwaiting.AsyncJobId;
	state.CurrentRetryCount = asyncJobAwaiting.Params.RetryCount;
	state.CurrentTimeout    = asyncJobAwaiting.Params.StartTimeout;

	do
	{
		Thread.Sleep(state.CurrentTimeout);

		EntityCollection entityCollection = proxy.RetrieveMultiple(bulkQuery);

		#if WRITE_TRACE_INFO
		WriteInfo(string.Format("Found {0} entities ...", entityCollection.Entities.Count));
		#endif

		if (entityCollection.Entities.Count > 0)
		{
			// Grab the one bulk operation that has been created.
			var createdBulkDeleteOperation = (BulkDeleteOperation)entityCollection.Entities[0];

			state.BulkDeleteOperationId = createdBulkDeleteOperation.BulkDeleteOperationId;
			state.CurrentState          = createdBulkDeleteOperation.StateCode;
			state.CurrentStatus         = createdBulkDeleteOperation.StatusCode != null ? 
					(bulkdeleteoperation_statuscode)createdBulkDeleteOperation.StatusCode.Value : 
					(bulkdeleteoperation_statuscode?) null;
			state.SuccessCount          = createdBulkDeleteOperation.SuccessCount;
			state.FailureCount          = createdBulkDeleteOperation.FailureCount;
		}
		
		#if WRITE_TRACE_INFO
		if (state.IsCreated)
		{
			if (state.IsSuspended)
				WriteInfo(string.Format("Bulk Delete (Id: {0}) is suspended", state.BulkDeleteOperationId));
			if (state.IsPaused)
				WriteInfo(string.Format("Bulk Delete (Id: {0}) is paused", state.BulkDeleteOperationId));
			if (state.CurrentState != null)
				WriteInfo(string.Format("Bulk Delete (Id: {0}) state is {1} {2}", asyncJobAwaiting.AsyncJobId, state.CurrentState, (state.CurrentStatus != null ? "(Status: " + state.CurrentStatus.Value + ")" : string.Empty)));

			WriteInfo(string.Format("{0} records were successfully deleted", state.SuccessCount ?? 0));
			WriteInfo(string.Format("{0} records failed to be deleted", state.FailureCount ?? 0));
		}
		else
			WriteInfo(string.Format("Bulk Delete is still not created. Associated Async Job Id:{0})", asyncJobAwaiting.AsyncJobId));
		#endif

		asyncJobAwaiting.FireOnProgress(state);

		if (asyncJobAwaiting.Params.TimeoutStep != null && 
		    asyncJobAwaiting.Params.EndTimeout != null && 
		    state.CurrentTimeout < asyncJobAwaiting.Params.EndTimeout)
				state.CurrentTimeout += asyncJobAwaiting.Params.TimeoutStep.Value;

		if (asyncJobAwaiting.Params.WaitForever)
			continue;

		if (state.IsSuspended && asyncJobAwaiting.Params.DoNotCountIfSuspended)
			continue;
		if (state.IsPaused && asyncJobAwaiting.Params.DoNotCountIfPaused)
			continue;
		if(!state.IsCreated && asyncJobAwaiting.Params.DoNotCountIfNotCreated)
			continue;

		state.CurrentRetryCount--;

	} while (!state.IsCompleted && state.CurrentRetryCount > 0);

	return state;
}

Classes involved in the method are derived from the same base classes as those used for WaitForAsyncJobCompletion. The diagram below demonstrates dependencies between the base classes and the ones specific for Bulk Delete operations.

Bulk Delete Waiting Class Diagram

The source code of the base classes AsyncOperationState, AsyncOperationAwaitingParams and AsyncOperationAwaiting are shown in the post – Awaiting for async job completion. The rest of used classes are listed below:

/// <summary>
/// Represents the awaiting parameters for Bulk Delete operations
/// </summary>
public class BulkDeleteAwaitingParams : AsyncOperationAwaitingParams
{
	private bool _doNotCountIfNotCreated = true;

	/// <summary>
	/// Indicates if an iteration counts when the target Bulk Delete operation 
	/// hasn't been created yet
	/// </summary>
	public bool DoNotCountIfNotCreated 
	{
		get { return _doNotCountIfNotCreated; }
		set { _doNotCountIfNotCreated = value; }
	}
}

In comparison with the AsyncOperationAwaitingParams, I added one more property specifying whether retries are counted while Bulk Delete operation hasn’t been created yet.

/// <summary>
/// Represents parameters for tracking of a Bulk Delete operation
/// </summary>
public class BulkDeleteAwaiting : AsyncOperationAwaiting<BulkDeleteState, 
													BulkDeleteAwaitingParams>
{
	/// <summary>
	/// Id of the async job that will be associated with the target Bulk Delete operation
	/// </summary>
	public Guid AsyncJobId
	{
		get { return _asyncOperationId; }
	}

	public BulkDeleteAwaiting(Guid asyncJobId, BulkDeleteAwaitingParams asyncJobAwaitingParams):
						base(asyncJobId, asyncJobAwaitingParams)
	{
	}
}

Note that the AsyncJobId here represents an auxiliary async job that is supposed to be associated with the target Bulk Delete operation.

/// <summary>
/// Represents current state of a Bulk Delete operation
/// </summary>
[Serializable]
public class BulkDeleteState : AsyncOperationState<BulkDeleteOperationState?, 
												bulkdeleteoperation_statuscode?>
{
	/// <summary>
	/// Id of the async job associated with the target Bulk Delete operation
	/// </summary>
	public Guid  AsyncJobId            { get; set; }

	/// <summary>
	/// Id of the target Bulk Delete operation. Null when the operation isn't created.
	/// </summary>
	public Guid? BulkDeleteOperationId { get; set; }

	/// <summary>
	/// Current number of successfully deleted records
	/// </summary>
	public int?  SuccessCount          { get; set; }

	/// <summary>
	/// Current number of problem records
	/// </summary>
	public int?  FailureCount          { get; set; }

	/// <summary>
	/// Indicates if the Bulk Delete is suspended
	/// </summary>
	public override bool IsSuspended
	{
		get { return CurrentState != null && 
					CurrentState.Value == BulkDeleteOperationState.Suspended; }
	}
	/// <summary>
	/// Indicates if the Bulk Delete is paused
	/// </summary>
	public override bool IsPaused
	{
		get { return CurrentState != null && 
					CurrentState.Value == BulkDeleteOperationState.Locked && 
					CurrentStatus != null && 
					CurrentStatus.Value == bulkdeleteoperation_statuscode.Pausing; }
	}
	/// <summary>
	/// Indicates if the Bulk Delete is completed
	/// </summary>
	public override bool IsCompleted
	{
		get { return CurrentState != null && 
					CurrentState.Value == BulkDeleteOperationState.Completed; }
	}
	/// <summary>
	/// Indicates if the number of retries have run out while the Bulk Delete is 
	/// still uncompleted
	/// </summary>
	public override bool IsTimedOut
	{
		get { return !IsCompleted && CurrentRetryCount <= 0; }
	}
	/// <summary>
	/// Indicates if the Bulk Delete is failed
	/// </summary>
	public override bool IsFailed
	{
		get { return CurrentStatus != null && 
					CurrentStatus.Value == bulkdeleteoperation_statuscode.Failed; }
	}
	/// <summary>
	/// Indicates if the Bulk Delete is created
	/// </summary>
	public bool IsCreated
	{
		get { return BulkDeleteOperationId != null && 
					BulkDeleteOperationId.Value != Guid.Empty; }
	}
}

The class represents a current state of the Bulk Delete operation being awaited. The current state on each iteration is supplied by the WaitForBulkDeleteCompletion through the OnProgress event available in the passed BulkDeleteAwaiting instance. In the end the WaitForBulkDeleteCompletion method returns the final state. The bulkdeleteoperation_statuscode enum is defined in the OptionSets.cs at sdk\samplecode\cs\helpercode.

All the source code can be downloaded here – ConnectToCrm4.zip

Related posts:

Dynamics CRM: Awaiting for async job completion

September 27th, 2013 No comments

    Having initiated an async job through the code, it’s often needed to wait until the job is completed. A very simple method doing that could be found in SDK samples provided by Microsoft. The WaitForAsyncJobCompletion method is contained at SDK\SampleCode\CS\DataManagement\DataImport\BulkImportHelper.cs and used for awaiting of completion of async jobs participating in a bulk data import:

public static void WaitForAsyncJobCompletion(OrganizationServiceProxy serviceProxy, 
                                                            Guid asyncJobId)
{
	ColumnSet cs = new ColumnSet("statecode", "statuscode");
	AsyncOperation asyncjob = 
             (AsyncOperation)serviceProxy.Retrieve("asyncoperation", asyncJobId, cs);

	int retryCount = 100;

	while (asyncjob.StateCode.Value != AsyncOperationState.Completed && retryCount > 0)
	{
		asyncjob = (AsyncOperation)serviceProxy.Retrieve("asyncoperation", asyncJobId, cs);
		System.Threading.Thread.Sleep(2000);
		retryCount--;
		Console.WriteLine("Async operation state is " + asyncjob.StateCode.Value.ToString());
	}

	Console.WriteLine("Async job is " + asyncjob.StateCode.Value.ToString() + 
		" with status " + ((asyncoperation_statuscode)asyncjob.StatusCode.Value).ToString());
}

This method is acceptable, but, as for me, it has some disadvantages. First of all, it’s better to have adjustable retryCount and sleep interval, so that to make them more suitable for each particular job. For example, if we know that an operation takes a very long time, we would extend the sleep interval and decrease max number of retries to minimize CPU time spent on checking of the job status. We could even make the waiting more adaptive and extend the sleep interval gradually: the more idle retries we have, the bigger the sleep interval gets. Secondly, the WaitForAsyncJobCompletion doesn’t analyze thoroughly the current state (and status) of the job. The job could be in Suspended or Paused state, in both cases, I think, it’s better not to count such retry as the job isn’t running at that moment. In addition, I would like to get some intermediate state on each iteration of the while-statement as it would allow tracing the job state changing in the time.

Based on the above weaknesses and wishes I’ve implemented my own WaitForAsyncJobCompletion method listed below:

#define WRITE_TRACE_INFO

using System;
using System.Threading;
using Microsoft.Xrm.Sdk.Client;
using Microsoft.Xrm.Sdk.Query;
...
public static AsyncJobState WaitForAsyncJobCompletion(OrganizationServiceProxy proxy, 
                   AsyncJobAwaiting asyncJobAwaiting)
{
	var state = new AsyncJobState();

	#if WRITE_TRACE_INFO
	WriteInfo(string.Format("Waiting for async job (Id: {0}) completion ...", 
                   asyncJobAwaiting.AsyncJobId));
	#endif

	var columnSet = new ColumnSet("statecode", "statuscode");

	state.AsyncJobId        = asyncJobAwaiting.AsyncJobId;
	state.CurrentRetryCount = asyncJobAwaiting.Params.RetryCount;
	state.CurrentTimeout    = asyncJobAwaiting.Params.StartTimeout;

	do
	{
		Thread.Sleep(state.CurrentTimeout);

		var asyncJob = 
      (AsyncOperation)proxy.Retrieve("asyncoperation", asyncJobAwaiting.AsyncJobId, columnSet);

		state.CurrentState  = asyncJob.StateCode;
		state.CurrentStatus = asyncJob.StatusCode != null ? 
          (asyncoperation_statuscode)asyncJob.StatusCode.Value : (asyncoperation_statuscode?)null;
		
		#if WRITE_TRACE_INFO
		if (state.IsSuspended)
			WriteInfo(string.Format("Async job (Id: {0}) is suspended", asyncJobAwaiting.AsyncJobId));
		if (state.IsPaused)
			WriteInfo(string.Format("Async job (Id: {0}) is paused", asyncJobAwaiting.AsyncJobId));
		if (asyncJob.StateCode != null)
			WriteInfo(string.Format("Async job (Id: {0}) state is {1} {2}", asyncJobAwaiting.AsyncJobId, asyncJob.StateCode.Value, (asyncJob.StatusCode != null ? "(Status: " + (asyncoperation_statuscode)asyncJob.StatusCode.Value + ")" : string.Empty)));
		#endif

		asyncJobAwaiting.FireOnProgress(state);

		if (asyncJobAwaiting.Params.TimeoutStep != null && 
            asyncJobAwaiting.Params.EndTimeout != null && 
            state.CurrentTimeout < asyncJobAwaiting.Params.EndTimeout)
			  state.CurrentTimeout += asyncJobAwaiting.Params.TimeoutStep.Value;

		if (asyncJobAwaiting.Params.WaitForever)
			continue;
		if (state.IsSuspended && asyncJobAwaiting.Params.DoNotCountIfSuspended)
			continue;
		if (state.IsPaused && asyncJobAwaiting.Params.DoNotCountIfPaused)
			continue;

		state.CurrentRetryCount--;

	} while (!state.IsCompleted && state.CurrentRetryCount > 0);

	return state;
}

public static void WriteInfo(string message)
{
	Console.WriteLine(message);
}

Note that the first thing I do in the while-statement is the sleeping for the current timeout value as I think It’s pointless to examine an async job for completion right after it has been created.

As you can see, this implementation is still quite simple, however it depends on a few classes referred to as parameters of waiting and current state of the target job. The diagram below depicts these assistant classes and their dependencies. If you wonder why I use inheritance and produced so many classes for a simple async job awaiting I can just say in excuse of this that the base classes are employed for keeping track of other operations as well (e.g. Bulk Delete operations).

Async Job Awaiting Class Diagram

The source code of the classes along with some explanations are shown below.

/// <summary>
/// Represents the awaiting parameters
/// </summary>
public class AsyncOperationAwaitingParams
{
	private const int DefaultTimeout    = 2000;
	private const int DefaultRetryCount = 100;

	private int  _startTimeout          = DefaultTimeout;
	private int  _retryCount            = DefaultRetryCount;
	private bool _doNotCountIfSuspended = true;
	private bool _doNotCountIfPaused    = true;

	/// <summary>
	/// If set, specifies the value to be added to the sleep timeout on each iteration
	/// </summary>
	public int? TimeoutStep { get; set; }
	/// <summary>
	/// If set, specifies the top limit of the sleep timeout
	/// </summary>
	public int? EndTimeout  { get; set; }

	/// <summary>
	/// Indicates if we need to wait for the operation completeness eternally
	/// </summary>
	public bool WaitForever { get; set; }

	/// <summary>
	/// Specifies the initial and maximal number of iterations
	/// </summary>
	public int RetryCount
	{
		get { return _retryCount; }
		set { _retryCount = value; }
	}
	/// <summary>
	/// Specifies the initial sleep timeout
	/// </summary>
	public int StartTimeout
	{
		get { return _startTimeout; }
		set { _startTimeout = value; }
	}
	/// <summary>
	/// Indicates if an iteration counts when the operation is suspended
	/// </summary>
	public bool DoNotCountIfSuspended
	{
		get { return _doNotCountIfSuspended; }
		set { _doNotCountIfSuspended = value; }
	}
	/// <summary>
	/// Indicates if an iteration counts when the operation is paused
	/// </summary>
	public bool DoNotCountIfPaused
	{
		get { return _doNotCountIfPaused; }
		set { _doNotCountIfPaused = value; }
	}
}

I added the WaitForever option just in case. However, I recommend avoiding the use of it as it’s better to play with RetryCount and sleep timeouts (EndTimeout, TimeoutStep, StartTimeout) and find the most suitable values for them for each specific type of operations.

/// <summary>
/// Comprises the awaiting parameters and id of the operation to be tracked. 
/// Provides the OnProgress event supposed to be fired on each iteration of awaiting process.
/// </summary>
/// <typeparam name="TState">Represents a type of the operation state</typeparam>
/// <typeparam name="TParams">Represents a type of the awaiting parameters</typeparam>
public abstract class AsyncOperationAwaiting<TState, TParams>
{
	protected readonly Guid    _asyncOperationId;
	protected readonly TParams _asyncOperationAwaitingParams;

	/// <summary>
	/// Represents the awaiting parameters
	/// </summary>
	public TParams Params
	{
		get { return _asyncOperationAwaitingParams; }
	}

	/// <summary>
	/// Event to fire and get operation state on each iteration
	/// </summary>
	public event EventHandler<TState> OnProgress;

	public void FireOnProgress(TState state)
	{
		if (OnProgress != null)
			OnProgress(this, state);
	}

	protected AsyncOperationAwaiting(Guid asyncOperationId, TParams asyncOperationAwaitingParams)
	{
		_asyncOperationId             = asyncOperationId;
		_asyncOperationAwaitingParams = asyncOperationAwaitingParams;
	}
}

The descendants of the AsyncOperationAwaiting are supposed to supply parameters for more specific operations like async jobs, Bulk Deletes and so on.

/// <summary>
/// Represents parameters for tracking of an async job
/// </summary>
public class AsyncJobAwaiting : 
               AsyncOperationAwaiting<AsyncJobState, AsyncOperationAwaitingParams>
{
	/// <summary>
	/// Id of the async job
	/// </summary>
	public Guid AsyncJobId
	{
		get { return _asyncOperationId; }
	}
	
	public AsyncJobAwaiting(Guid asyncJobId, AsyncOperationAwaitingParams asyncJobAwaitingParams) : 
        base(asyncJobId, asyncJobAwaitingParams)
	{
	}
}

The AsyncJobAwaiting supplies the awaiting parameters for async jobs. An instance of the AsyncJobAwaiting class is passed to the WaitForAsyncJobCompletion method.

/// <summary>
/// Represents current state of an operation
/// </summary>
/// <typeparam name="TState">Represents a type of the operation state</typeparam>
/// <typeparam name="TStatus">Represents a type of the operation status</typeparam>
[Serializable]
public abstract class AsyncOperationState<TState, TStatus>
{
	/// <summary>
	/// Indicates how many retries remain
	/// </summary>
	public int  CurrentRetryCount    { get; set; }
	/// <summary>
	/// Indicates sleep timeout on the last iteration
	/// </summary>
	public int  CurrentTimeout       { get; set; }

	/// <summary>
	/// Represents current operation state
	/// </summary>
	public TState  CurrentState      { get; set; }
	/// <summary>
	/// Represents current operation status
	/// </summary>
	public TStatus CurrentStatus     { get; set; }

	/// <summary>
	/// Indicates if the operations is suspended
	/// </summary>
	public abstract bool IsSuspended { get; }
	/// <summary>
	/// Indicates if the operations is paused
	/// </summary>
	public abstract bool IsPaused    { get; }
	/// <summary>
	/// Indicates if the operations is completed
	/// </summary>
	public abstract bool IsCompleted { get; }
	/// <summary>
	/// Indicates if the number of retries have run out while the operation is still uncompleted
	/// </summary>
	public abstract bool IsTimedOut  { get; }
	/// <summary>
	/// Indicates if the operations is failed
	/// </summary>
	public abstract bool IsFailed    { get; }
}

The descendants of the AsyncOperationState are supposed to be states for more specific operations like async jobs, Bulk Deletes and so on. The class is marked as Serializable, it gives us an ability to persist the state somewhere like logs, databases etc.

/// <summary>
/// Represents current state of an async job
/// </summary>
[Serializable]
public class AsyncJobState : 
     AsyncOperationState<AsyncOperationState?, asyncoperation_statuscode?>
{
	/// <summary>
	/// Id of the async job
	/// </summary>
	public Guid AsyncJobId { get; set; }

	/// <summary>
	/// Indicates if the async job is suspended
	/// </summary>
	public override bool IsSuspended
	{
		get { return CurrentState != null && 
                     CurrentState.Value == AsyncOperationState.Suspended; }
	}
	/// <summary>
	/// Indicates if the async job is paused
	/// </summary>
	public override bool IsPaused
	{
		get { return CurrentState != null && CurrentState.Value == AsyncOperationState.Locked && 
             CurrentStatus != null && CurrentStatus.Value == asyncoperation_statuscode.Pausing; }
	}
	/// <summary>
	/// Indicates if the async job is completed
	/// </summary>
	public override bool IsCompleted
	{
		get { return CurrentState != null && 
                     CurrentState.Value == AsyncOperationState.Completed; }
	}
	/// <summary>
	/// Indicates if the number of retries have run out while the async job is still uncompleted
	/// </summary>
	public override bool IsTimedOut
	{
		get { return !IsCompleted && CurrentRetryCount <= 0; }
	}
	/// <summary>
	/// Indicates if the async job is failed
	/// </summary>
	public override bool IsFailed
	{
		get { return CurrentStatus != null && 
                     CurrentStatus.Value == asyncoperation_statuscode.Failed; }
	}
}

The class is a current state of the async job being tracked. The WaitForAsyncJobCompletion method returns the final state and provides the current state on each iteration through the OnProgress event available in the passed AsyncJobAwaiting instance. The asyncoperation_statuscode is defined in the OptionSets.cs that could be found at sdk\samplecode\cs\helpercode.

All the source code can be downloaded here – ConnectToCrm3.zip

Related posts:

Dynamics CRM: Import of several files to one CRM Entity at once = An item with the same key has already been added

September 10th, 2013 No comments

    Importing data to CRM programmatically (so-called Bulk Import), we create an Import instance and then create one or more ImportFile instances associated with the Import and upload the files. Each ImportFile instance also has to be associated with an ImportMap. The subsequent Parse, Transform and ImportRecords operations are executed for all of the ImportFiles grouped under the Import. The size of a file that could be uploaded to CRM is limited. So, when I needed to import a huge number of records to a CRM Entity, I decided to split the file with the records into several ones satisfying the limitation. When the files of smaller size had been created and associated with one Import and one ImportMap, the data parsing was requested. Unfortunately, the records weren’t imported as the Import failed. I found the following exception thrown on the CRM server side:

Unhandled Exception: System.ArgumentException: 
An item with the same key has already been added.
  at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
  at Microsoft.Crm.Asynchronous.ImportHelperData.InitializeImportHelperData(Guid importId, 
            Boolean initializeDictionaries, IOrganizationService crmService, 
            Guid organizationId, Int32 languageCode)
  at Microsoft.Crm.Asynchronous.ImportHelperData..ctor(Guid importId, 
            Boolean initializeDictionaries, IOrganizationService crmService, 
            Guid organizationId, Int32 languageCode)
  at Microsoft.Crm.Asynchronous.ImportOperationParse.ExecuteImportOperation(
            Guid organizationId, Guid importId, Int32 operationType)
  at Microsoft.Crm.Asynchronous.ImportOperation.InternalExecute(AsyncEvent asyncEvent)

Having armed with .Net Reflector, I found the exact place where the exception had occurred. It’s the InitializeImportHelperData method of the ImportHelperData class defined in Microsoft.Crm.Asynchronous.DataManagement assembly. A bit simplified code of the InitializeImportHelperData is listed below:

private void InitializeImportHelperData(Guid importId, bool initializeDictionaries, 
                     IOrganizationService crmService, Guid organizationId, int languageCode)
{
    this._crmService = crmService;
    this._importId = importId;
    ...
    RetrieveRequest request = new RetrieveRequest {
        ColumnSet = new ColumnSet(true),
        Target = new EntityReference("import", importId)
    };
    RetrieveResponse response = (RetrieveResponse) this._crmService.Execute(request);
    Entity entity = response.Entity;
    ...
    EntityCollection importFileCollection = this.RetrieveImportFiles(this._importId);
    this.ProcessImportFilesForImportEntityMappings(ref importFileCollection);
    this._importFileIdToImportFileObjectDictionary = new Dictionary<Guid, Entity>();
    if (initializeDictionaries)
    {
        this._sourceEntityToImportFileIdDictionary = new Dictionary<string, Guid>();
        this._importFileIdToImportFileHelperDataDictionary = 
                              new Dictionary<Guid, ImportFileHelperData>();
    }
    foreach (Entity entity2 in importFileCollection.Entities)
    {
       Guid id = entity2.Id;
       this._importFileIdToImportFileObjectDictionary.Add(id, entity2);
       if (initializeDictionaries)
       {
          if (entity2.Attributes.Contains("sourceentityname") && 
                                       entity2["sourceentityname"] != null)
 /***/     this._sourceEntityToImportFileIdDictionary.Add(
              DataManagementHelper.GetStringFromDynamicEntity(entity2, "sourceentityname"), id);
          this._importFileIdToImportFileHelperDataDictionary.Add(id, null);
       }
    }
}

The exception is thrown in the highlighted line. This code gets the ImportFiles, goes through them and fills the dictionary with ImportFile Ids, using the SourceEntityName as a key. Obviously, if we have at least two ImportFiles with the same SourceEntityName (my case), we get the exception.

I was looking for a workaround and the first thought that came into my mind was to set different SourceEntityName for each ImportFile. However, SourceEntityName of ImportFile must coincide with the one specified in an associated ImportMap. I had one ImportMap for all of ImportFiles as they supply records for one CRM Entity. So, following this way, I would have to provide with as many almost identical ImportMaps as many ImportFiles are to be uploaded. The difference between the ImportMaps would be only in the SourceEntityName attribute. I discarded this idea as it’s extremely not optimal.

The approach I chose in the end is just to use an individual Import for every ImportFile. So, if you have a number of ImportFiles and each ImportFile is associated with a distinct ImportMap and supplies records for a distinct CRM Entity, you can combine them into one package (group under one Import instance). But if you are planning to import records from several ImportFiles to one and the same CRM Entity using one and the same ImportMap, use an individual package (separate Import instance) for every such file.

The fact that we are not able to import data from several files to a CRM Entity at once (i.e. using one Import package) is a very strange limitation for me. I hope this inconvenience will be fixed in upcoming CRM versions.

Dynamics CRM: Get current user information

September 5th, 2013 No comments

    One of the most frequently used operation with OrganizationServiceProxy is to get information about the current user. For simplicity such operation could be implemented as an extension to the OrganizationServiceProxy class. So, let’s put the following code somewhere in a project:

using Microsoft.Crm.Sdk.Messages;
using Microsoft.Xrm.Sdk.Client;

namespace dotNetFollower
{
    public static class OrganizationServiceProxyExtensions
    {
        public static WhoAmIResponse GetCurrentUser(this OrganizationServiceProxy proxy)
        {
            return (WhoAmIResponse)proxy.Execute(new WhoAmIRequest());
        }
    }
}

How to use the extension is shown below:

var serverConnection = new ServerConnectionEx();
serverConnection.DoInOrganizationServiceProxyContext(crmConfiguration, proxy =>
{
	Guid userId = proxy.GetCurrentUser().UserId;

	// do something else
});

The code above supposes that you use the ServerConnection helper class in your project. The ServerConnectionEx class, in turn, is derived from ServerConnection and described in the article Dynamics CRM: How to connect to CRM through the code.

More elegant way, as for me, is to add a lazy-loading property to a class derived from OrganizationServiceProxy. But in this case we have to make a lot of small changes within the ServerConnection class as the class creates and operates the instance of OrganizationServiceProxy. So, the derived class in this case looks like the following:

using System;
using System.ServiceModel.Description;
using Microsoft.Crm.Sdk.Messages;
using Microsoft.Xrm.Sdk;
using Microsoft.Xrm.Sdk.Client;

namespace dotNetFollower
{
  public class OrganizationServiceProxyEx : OrganizationServiceProxy
  {
    private WhoAmIResponse _whoAmIResponse;

    public WhoAmIResponse CurrentUser
    {
      get { return _whoAmIResponse ?? 
              (_whoAmIResponse = (WhoAmIResponse) Execute(new WhoAmIRequest())); }
    }

    public OrganizationServiceProxyEx(IServiceConfiguration<IOrganizationService> serviceConfiguration, 
                             ClientCredentials clientCredentials)
        : base(serviceConfiguration, clientCredentials)
    {            
    }
    public OrganizationServiceProxyEx(IServiceConfiguration<IOrganizationService> serviceConfiguration,
                                    SecurityTokenResponse securityTokenResponse)
        : base(serviceConfiguration, securityTokenResponse)
    {            
    }
    public OrganizationServiceProxyEx(IServiceManagement<IOrganizationService> serviceManagement,
                                    ClientCredentials clientCredentials)
        : base(serviceManagement, clientCredentials)
    {            
    }
    public OrganizationServiceProxyEx(IServiceManagement<IOrganizationService> serviceManagement,
                                    SecurityTokenResponse securityTokenResponse)
        : base(serviceManagement, securityTokenResponse)
    {            
    }
    public OrganizationServiceProxyEx(Uri uri, Uri homeRealmUri, ClientCredentials clientCredentials,
                                    ClientCredentials deviceCredentials)
        : base(uri, homeRealmUri, clientCredentials, deviceCredentials)
    {            
    }
  }
}

The next step is to open CrmServiceHelpers.cs and replace all mentions of OrganizationServiceProxy with OrganizationServiceProxyEx. Then do the same for ServerConnectionEx in case you use it. Or you can just take the demonstration project, where all required changes are applied. Download it here – ConnectToCrm2.zip.

Below is how to use the CurrentUser property exposed by the custom proxy class:

var serverConnection = new ServerConnectionEx();
serverConnection.DoInOrganizationServiceProxyContext(SampleCrmConfiguration, proxy =>
{
	Guid userId = proxy.CurrentUser.UserId;
	
	// do something else
});

Dynamics CRM: How to connect to CRM through the code

August 22nd, 2013 No comments

    Starting development for Microsoft Dynamics CRM 2011, the first challenge we face is how to connect to the CRM. Communication with Microsoft Dynamics CRM 2011 through the code mostly comes to interaction with two Web Services supplied by the CRM: so-called Discovery (IDiscoveryService) and Organization (IOrganizationService) Web Services. For a certain Microsoft Dynamics CRM installation the URL to the IDiscoveryService is unique. To be more precise, it’s unique for an on-premise installation while for Microsoft Dynamics CRM Online it might be one of a few variants depending on the world region. Read more about possible URLs to the Discovery Web Service here. In the same time, a single Microsoft Dynamics CRM installation can host multiple business organizations on multiple servers. Accessing data for a particular organization through the IOrganizationService, we have to use the URL specified for that organization.

Most of the time we work with data (entity records) and metadata (entities, attributes, relationships, etc.) of our organization(s), so the IOrganizationService is a primary Web Service to deal with. The IDiscoveryService, in turn, is generally used to get the right URL to an Organization Web Service. So, the initial connection process could be divided into two stages:

  • a client application grabs CRM‘s server name, protocol (http or https) and port (if applicable), and user credentials (login and password), then sends request to the IDiscoveryService. The IDiscoveryService Web Service determines the organizations that the user is a member of and sends the information about found organizations back to the client. Among other things the information includes actual URLs to Organization Web Services. The client application chooses one organization to interact with and fetches out the URL to the corresponding IOrganizationService;
  • the client application connects to IOrganizationService using the URL and manages data and metadata of the organization;

Often after the initial connection the received URL to IOrganizationService is stored somewhere to be used next times as it allows omitting the interaction with Discovery Web Service. However, we have to bear in mind that, in boundaries of a particular installation of Microsoft Dynamics CRM or Microsoft Dynamics CRM Online, the location of servers and organizations may change due to some datacenter management or load balancing. Therefore, we should use IDiscoveryService from time to time since it provides the actual URL to the IOrganizationService that serves our organization at a given moment. For example, we could try refreshing our connection string for IOrganizationService when we fail to access organization using the URL received previously.

To develop for Microsoft Dynamics CRM 2011, Microsoft provides with the Software Development Kit (SDK), which contains a good set of samples to start playing with CRM. After the SDK has been installed, the samples are usually available at SDK\SampleCode\CS. The most of the sample solutions connect to CRM by means of the special class called ServerConnection and defined in the SDK\SampleCode\CS\helpercode\CrmServiceHelpers.cs. The ServerConnection implements all typical steps necessary to connect to CRM: obtains user credentials, communicates with Discovery Web Service to get list of organizations accessible for user, allows user to pick an organization, creates an IOrganizationService proxy and refreshes the WCF connection security token from time to time. The code for connecting to CRM through the ServerConnection class usually looks like the following:

...
using Microsoft.Crm.Sdk.Samples;
using Microsoft.Xrm.Sdk.Client;
...
// Obtain the user logon credentials,  send request to Discovery Web Service, 
// pick out an organization and then get the target organization's Web address
ServerConnection serverConnect = new ServerConnection();
ServerConnection.Configuration config = serverConnect.GetServerConfiguration();

// Connect to the Organization service. 
using (OrganizationServiceProxy serviceProxy = ServerConnection.GetOrganizationProxy(config))
{
	// Enable early-bound type support.
	serviceProxy.EnableProxyTypes();

	// do something useful
}

The ServerConnection uses the DeviceIdManager class inside. The class defined in the SDK\SampleCode\CS\helpercode\DeviceIdManager.cs is intended to register a computing device with Microsoft account (Windows Live ID) and needed only for authentication in Microsoft Dynamics CRM Online. So, to use the ServerConnection we have to add both CrmServiceHelpers.cs and DeviceIdManager.cs files to a project.

The ServerConnection.Configuration class is to encapsulate all information needed for setting up a connection to Discovery Web Service. Based on the object of ServerConnection.Configuration, ServerConnection creates a proxy for IOrganizationService.

Note, however, since ServerConnection is designed to work within sample console-based applications, it has some “features” that are not applicable for other types of projects. For example, the following could be placed among such “features”: prompting user to type credentials in the console window; persisting CRM server connection information to a file C:\Users\<username>\AppData\Roaming\CrmServer\Credentials.xml for later reuse; storing user password in the Windows Credential Manager. Fortunately, all of the mentioned points might be overcome by creating a class inherited from the ServerConnection and by implementing our own mechanism of reading parameters required for establishing a connection to CRM. The custom class shown below exposes the methods that parse the xml string containing connection parameters and return a proper instance of ServerConnection.Configuration. That allows keeping parameters wherever we want, we are not bound to the file system or Windows Credential Manager anymore. That’s quite important if we plan to use the code in Windows Azure, for example.

using System;
using System.Linq;
using System.ServiceModel.Description;
using System.Xml.Linq;
using Microsoft.Crm.Sdk.Samples;
using Microsoft.Crm.Services.Utility;
using Microsoft.Xrm.Sdk.Client;
using Microsoft.Xrm.Sdk.Discovery;

namespace dotNetFollower
{
  public class ServerConnectionEx : ServerConnection
  {
    /// <summary>
    /// Parses xml string and builds a Configuration object with connection parameters
    /// </summary>
    public Configuration Parse(string xml)
    {
      // read xml and fetch out parameters
      XElement configurationsFromFile = XElement.Parse(xml);
      var xmlConfig = configurationsFromFile.Nodes().First() as XElement;

      var newConfig = new Configuration();
            
      var serverAddress = xmlConfig.Element("ServerAddress");
      if (serverAddress != null && !String.IsNullOrEmpty(serverAddress.Value))
        newConfig.ServerAddress = serverAddress.Value;

      var organizationName = xmlConfig.Element("OrganizationName");
      if (organizationName != null && !String.IsNullOrEmpty(organizationName.Value))
        newConfig.OrganizationName = organizationName.Value;

      var discoveryUri = xmlConfig.Element("DiscoveryUri");
      if (discoveryUri != null && !String.IsNullOrEmpty(discoveryUri.Value))
        newConfig.DiscoveryUri = new Uri(discoveryUri.Value);

      var organizationUri = xmlConfig.Element("OrganizationUri");
      if (organizationUri != null && !String.IsNullOrEmpty(organizationUri.Value))
        newConfig.OrganizationUri = new Uri(organizationUri.Value);

      var homeRealmUri = xmlConfig.Element("HomeRealmUri");
      if (homeRealmUri != null && !String.IsNullOrEmpty(homeRealmUri.Value))
        newConfig.HomeRealmUri = new Uri(homeRealmUri.Value);

      var vendpointType = xmlConfig.Element("EndpointType");
      if (vendpointType != null)
        newConfig.EndpointType = RetrieveAuthenticationType(vendpointType.Value);

      var xElement = xmlConfig.Element("Credentials");
      if (xElement != null && xElement.HasElements)
        newConfig.Credentials = ParseInCredentials(xmlConfig.Element("Credentials"), newConfig.EndpointType);

      if (newConfig.EndpointType == AuthenticationProviderType.LiveId)
        newConfig.DeviceCredentials = DeviceIdManager.LoadOrRegisterDevice();

      var userPrincipalName = xmlConfig.Element("UserPrincipalName");
      if (userPrincipalName != null && !String.IsNullOrWhiteSpace(userPrincipalName.Value))
        newConfig.UserPrincipalName = userPrincipalName.Value;

      // save new configuration in the config property as the property 
      // is used by original ServerConnection
      config = newConfig;

      // get OrganizationUri in case it is not set
      if (newConfig.OrganizationUri == null)
      {
        if(string.IsNullOrEmpty(newConfig.OrganizationName))
          throw new ApplicationException("At least one of OrganizationUri and OrganizationName must be spesified in the xml configuration.");
        GetOrganizationInfo();
      }

      return newConfig;
    }

    ...

    /// <summary>
    /// Gets Organization Web Service url by friendly or unique name of the organization
    /// </summary>
    protected void GetOrganizationInfo()
    {
      using (DiscoveryServiceProxy serviceProxy = GetDiscoveryProxy())
      {
        // Obtain organization information from the Discovery service. 
        if (serviceProxy != null)
        {
          // Obtain information about the organizations that the system user belongs to.
          OrganizationDetailCollection orgs = DiscoverOrganizations(serviceProxy);

          if(orgs.Count == 0)
            throw new ApplicationException("User does not belong to any organizations on the specified server");

          // look for a specified organization
          foreach (var organizationDetail in orgs)
          {
            if (string.Equals(organizationDetail.UniqueName, 
                               config.OrganizationName, StringComparison.OrdinalIgnoreCase) ||
                string.Equals(organizationDetail.FriendlyName, 
                               config.OrganizationName, StringComparison.OrdinalIgnoreCase))
            {
              config.OrganizationUri  = 
                       new Uri(organizationDetail.Endpoints[EndpointType.OrganizationService]);
              config.OrganizationName = organizationDetail.FriendlyName;
              return;
            }
          }

          throw new ApplicationException("Couldn't find the " + config.OrganizationName + "organization");
        }
                
        throw new ApplicationException("An invalid server name was specified.");
      }
    }

    /// <summary>
    /// Parses xml element to get user name and password
    /// </summary>
    protected static ClientCredentials ParseInCredentials(XElement credentials, 
                             AuthenticationProviderType endpointType)
    {
      ClientCredentials result = null;

      if (credentials.HasElements)
      {
        result = new ClientCredentials();

        switch (endpointType)
        {
          case AuthenticationProviderType.ActiveDirectory:
          {
            result.Windows.ClientCredential = new System.Net.NetworkCredential()
              {
                UserName = credentials.Element("UserName").Value,
                Domain   = credentials.Element("Domain").Value,
                Password = credentials.Element("Password").Value
              };
              break;
          }
          case AuthenticationProviderType.LiveId:
          case AuthenticationProviderType.Federation:
          case AuthenticationProviderType.OnlineFederation:
          {
            result.UserName.UserName = credentials.Element("UserName").Value;
            result.UserName.Password = credentials.Element("Password").Value;
            break;
          }
        }
      }

      return result;
    }
  }
  ...
}

In case the OrganizationUri isn’t set in the xml string the Parse method tries getting the URL from Discovery Web Service using the name specified in the OrganizationName node. The OrganizationName can contain friendly or unique name of the target organization.

Note we need to slightly modify the original ServerConnection class, we need to make protected such its members as RetrieveAuthenticationType (method), GetDiscoveryProxy (method) and config (field) so that they would be accessible from within the derived ServerConnectionEx class.

Below is an example of how the custom ServerConnectionEx could be used:

// the xml string below might be stored to and read from any data source
string crmConfiguration = @"<?xml version=""1.0"" encoding=""utf-8""?>
  <Configurations>
  <Configuration>
    <ServerAddress>MyCrmOnPremise</ServerAddress>
    <OrganizationName>My Company Friendly Name</OrganizationName>
    <DiscoveryUri>http://MyCrmOnPremise/XRMServices/2011/Discovery.svc</DiscoveryUri>
    <OrganizationUri>http://MyCrmOnPremise/MyCompanyFriendlyName/XRMServices/2011/Organization.svc</OrganizationUri>
    <HomeRealmUri />                      
    <Credentials>
      <UserName>dotNetFollower</UserName>
      <Domain>hq</Domain>
      <Password>11111</Password>
    </Credentials>
    <EndpointType>ActiveDirectory</EndpointType>
    <UserPrincipalName />                                        
  </Configuration>
</Configurations>";

// Obtain the target organization's Web address and client credentials from the xml string.
ServerConnectionEx serverConnection = new ServerConnectionEx();
ServerConnection.Configuration configuration = serverConnection.Parse(crmConfiguration);

// Connect to the Organization service. 
using (OrganizationServiceProxy serviceProxy = ServerConnection.GetOrganizationProxy(configuration))
{
	// This statement is required to enable early-bound type support.
	serviceProxy.EnableProxyTypes();

	// do something useful
}

In the example we have a hardcoded xml string with connection parameters, in the reality, however, the string could be stored in whatever data store.

To make the use of ServerConnectionEx more comfortable, let’s add the following two methods to ServerConnectionEx:

public class ServerConnectionEx : ServerConnection
{
	...
	public static void DoInOrganizationServiceProxyContext(Configuration configuration,
							Action<OrganizationServiceProxy> action)
	{
		using (var proxy = GetOrganizationProxy(configuration))
		{
			proxy.EnableProxyTypes();
			action(proxy);
		}
	}

	public void DoInOrganizationServiceProxyContext(string configXml, 
							Action<OrganizationServiceProxy> action)
	{
		Configuration configuration = Parse(configXml);
		DoInOrganizationServiceProxyContext(configuration, action);
	}
	...
}

Now to deal with the OrganizationServiceProxy we can use the code like the following:

string crmConfiguration = ... // the same xml-string as used above

var serverConnection = new ServerConnectionEx();
serverConnection.DoInOrganizationServiceProxyContext(crmConfiguration, proxy =>
{
	// do something
});

Note also that to use CrmServiceHelpers.cs and DeviceIdManager.cs you have to add such references to your project as System.Runtime.Serialization, System.Security, System.ServiceModel, System.DirectoryServices.AccountManagement and Microsoft.IdentityModel. The Microsoft.IdentityModel is a part of Windows Identity Foundation, so install it if it’s not already installed. The dll itself can be found at Program Files\Reference Assemblies\Microsoft\Windows Identity Foundation\v3.5 or in GAC. And, of course, you need to add references for Microsoft.Crm.Sdk.Proxy and Microsoft.Xrm.Sdk. You can find them in SDK\BIN.

A simple demonstration project you can download here – ConnectToCrm.zip. Among others files it contains the adapted CrmServiceHelpers.cs and DeviceIdManager.cs.