.NET FOLLOWER

    Importing data to CRM programmatically (so-called Bulk Import), we create an Import instance and then create one or more ImportFile instances associated with the Import and upload the files. Each ImportFile instance also has to be associated with an ImportMap. The subsequent Parse, Transform and ImportRecords operations are executed for all of the ImportFiles grouped under the Import. The size of a file that could be uploaded to CRM is limited. So, when I needed to import a huge number of records to a CRM Entity, I decided to split the file with the records into several ones satisfying the limitation. When the files of smaller size had been created and associated with one Import and one ImportMap, the data parsing was requested. Unfortunately, the records weren’t imported as the Import failed. I found the following exception thrown on the CRM server side:

Unhandled Exception: System.ArgumentException: 
An item with the same key has already been added.
  at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
  at Microsoft.Crm.Asynchronous.ImportHelperData.InitializeImportHelperData(Guid importId, 
            Boolean initializeDictionaries, IOrganizationService crmService, 
            Guid organizationId, Int32 languageCode)
  at Microsoft.Crm.Asynchronous.ImportHelperData..ctor(Guid importId, 
            Boolean initializeDictionaries, IOrganizationService crmService, 
            Guid organizationId, Int32 languageCode)
  at Microsoft.Crm.Asynchronous.ImportOperationParse.ExecuteImportOperation(
            Guid organizationId, Guid importId, Int32 operationType)
  at Microsoft.Crm.Asynchronous.ImportOperation.InternalExecute(AsyncEvent asyncEvent)

Having armed with .Net Reflector, I found the exact place where the exception had occurred. It’s the InitializeImportHelperData method of the ImportHelperData class defined in Microsoft.Crm.Asynchronous.DataManagement assembly. A bit simplified code of the InitializeImportHelperData is listed below:

private void InitializeImportHelperData(Guid importId, bool initializeDictionaries, 
                     IOrganizationService crmService, Guid organizationId, int languageCode)
{
    this._crmService = crmService;
    this._importId = importId;
    ...
    RetrieveRequest request = new RetrieveRequest {
        ColumnSet = new ColumnSet(true),
        Target = new EntityReference("import", importId)
    };
    RetrieveResponse response = (RetrieveResponse) this._crmService.Execute(request);
    Entity entity = response.Entity;
    ...
    EntityCollection importFileCollection = this.RetrieveImportFiles(this._importId);
    this.ProcessImportFilesForImportEntityMappings(ref importFileCollection);
    this._importFileIdToImportFileObjectDictionary = new Dictionary<Guid, Entity>();
    if (initializeDictionaries)
    {
        this._sourceEntityToImportFileIdDictionary = new Dictionary<string, Guid>();
        this._importFileIdToImportFileHelperDataDictionary = 
                              new Dictionary<Guid, ImportFileHelperData>();
    }
    foreach (Entity entity2 in importFileCollection.Entities)
    {
       Guid id = entity2.Id;
       this._importFileIdToImportFileObjectDictionary.Add(id, entity2);
       if (initializeDictionaries)
       {
          if (entity2.Attributes.Contains("sourceentityname") && 
                                       entity2["sourceentityname"] != null)
 /***/     this._sourceEntityToImportFileIdDictionary.Add(
              DataManagementHelper.GetStringFromDynamicEntity(entity2, "sourceentityname"), id);
          this._importFileIdToImportFileHelperDataDictionary.Add(id, null);
       }
    }
}

The exception is thrown in the highlighted line. This code gets the ImportFiles, goes through them and fills the dictionary with ImportFile Ids, using the SourceEntityName as a key. Obviously, if we have at least two ImportFiles with the same SourceEntityName (my case), we get the exception.

I was looking for a workaround and the first thought that came into my mind was to set different SourceEntityName for each ImportFile. However, SourceEntityName of ImportFile must coincide with the one specified in an associated ImportMap. I had one ImportMap for all of ImportFiles as they supply records for one CRM Entity. So, following this way, I would have to provide with as many almost identical ImportMaps as many ImportFiles are to be uploaded. The difference between the ImportMaps would be only in the SourceEntityName attribute. I discarded this idea as it’s extremely not optimal.

The approach I chose in the end is just to use an individual Import for every ImportFile. So, if you have a number of ImportFiles and each ImportFile is associated with a distinct ImportMap and supplies records for a distinct CRM Entity, you can combine them into one package (group under one Import instance). But if you are planning to import records from several ImportFiles to one and the same CRM Entity using one and the same ImportMap, use an individual package (separate Import instance) for every such file.

The fact that we are not able to import data from several files to a CRM Entity at once (i.e. using one Import package) is a very strange limitation for me. I hope this inconvenience will be fixed in upcoming CRM versions.