Detect Unicorn duplicate items during CI/CD build

Hello friends!

Do you use Unicorn often in your Sitecore projects? I bet you do, unless you prefer Sitecore Content Serialization.

If so — have you ever faced a situation when your source code serialized files contains duplicate item IDs which fails the content sync?

If yes — this article might get super handy for you.

This is how it usually looks like in Unicorn sync log:

https://sitecore.stackexchange.com/questions/22849/unicorn-duplicate-item-ids

Duplicate serialized item IDs were detected (c385a844-4b5a-437c-a746-9de832713749) - this usually indicates corruption in the serialization provider data.
Item 1: master:/sitecore/content/App/globals/Test Directory/Tests/#/xxxxx (c385a844-4b5a-437c-a746-9de832713749)
Item 1 ProviderId: 
ERROR: Consistency check failed - aborting loading. (Unicorn.Loader.ConsistencyException)

In many cases Unicorn sync happens in the end of the deployment — which makes it worse, as now you are wasting time fixing duplicates and triggering build again.

But no need to worry! Powershell comes to the rescue.

Why not run the Powershell script right before doing solution build and validate the serialized YML files?

It’s simple as running RegEx expression for all the files to find IDs, and checking the IDs against the dictionary for duplicates.

And this is how we can run it in Gitlab:

validateUnicorn:
 stage: build
 script:
 - . .\validate-unicorn.ps1
 rules:
 - if: '$CI_PIPELINE_SOURCE == "merge_request_event"

Note that we are validating Unicorn here on merge requests, which would save us time on release builds.

Now lets test run that!

OH NO!

This new merge request have 3 duplicate ids in our serialized items!

But at least now we know about this beforehand and can fix the merge request before merging it into the develop branch.

We can even find now the paths where the duplicates are located. How savvy is that?

Now lets fix that and run the CI/CD for our MR again!

We’ll delete the duplicate items (don’t forget to double check they are the right one in Sitecore!)

Commit the changes and voila!

Now our validation is passing and we can safely merge the changes into the source control.

Hope this small validation script would save time for you to avoid breaking your Unicorn sync.

Have a great day!