Symposium 2018 roundup links

In preparation for a Symposium roundup presentation for the Perth Sitecore User Group, I collated a bunch of links to blog post roundups, slide decks, session videos and even some Github repos of demo code.  There was way too much content to convey at a SUG presentation, but well worth sharing.  So….here they are in no particular order!  Of course, let me know if you’d like any links added.

Roundups:

Session blog posts, slides and repos:

9.1 (Not specifically Symposium related):

Advertisements

Active tests not showing

A/B Testing is great

Sitecore’s Experience Optimization content testing functionality is a powerful tool in the Sitecore XP arsenal.  When coupled with a solid engagement value strategy, marketers can get deep insights into what actually works (rather than what we think may work).

….when it works

Recently we hit an issue in a recently upgraded (8.2 -> 9.0.1) installation that would not display newly created tests correctly.  The process of creating the test all went smoothly, the test would actually be running…but it would not display in the “Active Tests” tab of Experience Optimization.  Drafts and Historical tests all displayed as expected.

Capture
I just created a new test….but no active tests 😦

Testing through our environments identified that things were working locally, but any upstream deployments all reproduced the issue.

All tests get indexed by your search provider upon creation.  As part of the upgrade we’d also switched search providers to Azure Search (locally we use SOLR) so thought I’d investigate the [sitecore-testing-index].  All looked ok ¯\_(ツ)_/¯ .  Documents were being added for each test we created with what seemed appropriate data. There was also a fairly cryptic INFO message in the logs when attempting to load the Active tests:

9300 02:31:53 INFO AzureSearch Query [sitecore_testing_index]: &search=This_Is_Equal_ConstNode_Return_Nothing

After hitting up colleagues and Sitecore community slack we were still at a loss, so raised a Sitecore support ticket.  After an open dialogue and many configs later, we had a solution!

The solution

This was a config issue with Azure search.  One of the Sitecore fields to be added to the [sitecore-testing-index] is a flag to whether the test is running (“__is running“).  The default configs fail to add this field to the index correctly.  Easily fixed…..it’s a one line change (isn’t it always?).

The Content Testing configs for 9.0.1 (and 9.0.2) provided in the Sitecore package downloads (for Azure AppService) you’ll see the following:

<field fieldName="__is_running" cloudFieldName="is_running____" boost="1f" type="System.Boolean" settingType="Sitecore.ContentSearch.Azure.CloudSearchFieldConfiguration, Sitecore.ContentSearch.Azure" />

This code needs to change to the following  (I recommend patching it to avoid any future deployment/upgrade issues):

<field fieldName="__is running" cloudFieldName="is_running_____b" boost="1f" type="System.Boolean" settingType="Sitecore.ContentSearch.Azure.CloudSearchFieldConfiguration, Sitecore.ContentSearch.Azure" />

The changes are subtle, but important! Firstly, it’s referencing the field name correctly (with a space). Secondly it is giving the index field name a _b suffix to further indicate it is a boolean.

After making this change and then rebuilding the [sitecore-testing-index] in the control panel, marketers and analytics teams rejoiced as they were able to view all of their glorious active tests.

 

Marketing Automation stored procedures and tables missing

Seen this message in the exception logs of your shiny new Azure app service instance of Sitecore 9?

Could not find stored procedure ‘xdb_ma_pool.AutomationPool_Stall’

ARRRRRGH.  Missing Stored Procedures?!?! Who’s been dropping stuff in my DB!  It’s newly deployed…how can this be! It worked on my dev machine! Never fear, there’s an explanation and easy fix.

No…you haven’t been dropping DB objects in your sleep.  This just appears to be an oversight in the Marketing Automation SQL dacpac in the Sitecore 9 Azure App Service web deploy packages.  In fact there are a few other objects missing too:

Stored Procedures
AutomationPool_Stall
ContactWorkerAffinity_ReleaseLease
ContactWorkerAffinity_TakeLease

Table
StalledAutomationPool

This has been confirmed by Sitecore support and they’ve published a KB  https://kb.sitecore.net/articles/065636 . Also the KB now contains a SQL script to create the missing objects.  Running this script on the Marketing Automation DB in your Azure environment should create the missing objects and resolve the issue.   This is confirmed as an issue on 9.0.1 and 9.0.2.

We’ll be adding this as a post-install script to all our ARM template deployments to avoid any manual steps in future deployments.

 

xDB index rebuild with Azure Search

As part of a Sitecore migration (8.2 to 9.0.1) we had a requirement to migrate a sizeable xDB implementation. The xDB data migration tool makes this super easy. It reads straight out of your existing mongoDB and pushes it to xConnect. As new contacts and interactions are pushed to the 9.0.1 xDB (in our case in Azure SQLDB) the indexer service will pick up the changes and update the appropriate index. Sweet!

So what happens if you want to change your search provider from Solr to Azure Search? It stands to reason that you should just be able to hook up Azure Search in xConnect in a similar way to the core role indexes (check out Azure Search on Sitecore 9) with the appropriate Configs and connection strings, then rebuild the search index. One problem, there are some serious caveats on the xDB index rebuild for Azure search. They are alluded to in the Rebuild the xDB Search index documentation (Update May 2018: The doco has been updated and is much clearer now!), but may need some further clarification. Important points:

  • A full xDB index rebuild is not supported on Azure Search as of 9.0.1. That means that if you want to rebuild the xDB search with historical data, it’s not supported out of the box. Please note though that the indexer will rebuild the index for everything that is still available in the change tracking log in the SQL db (by default this is a 5 day retention). But anything before that will not be included.
  • This may not be a deal-breaker for a migration as your data will be indexed as the xDB migration tool pulls everything across. Just plan your migration accordingly with the limitations in mind.
  • This will affect anyone looking to switch from Solr to Azure search on 9.0.1
  • There is probably some potential to code this yourself!
  • Sitecore support have it as a feature planned for 9.0.2, so may be best to hold tight for now and get support ootb.
  • I’m hoping the doco gets updated to be a bit more clear as a few in the community have been thrown by this.  (Update May 2018: it has been!)

Sitecore 9: Azure search and index all fields

Index all fields

Out of the box Sitecore 9 configs will index all fields in each of the basic indexes (core,master,web, etc..). This is a great strategy to ensure that everything you add to your templates gets indexed down the line. However this also leads to a couple problems:

  • Exceeding the 1000 field maximum on Azure Search indexes
  • Over indexing

1000 field limit

Seen this error pop up in your logs when using the Azure search provider?

ManagedPoolThread #3 09:59:00 ERROR [Index=sitecore_master_index] Commit failed
Exception: System.AggregateException
Message: One or more errors occurred.
Source: mscorlib
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Func`4 bodyWithLocal, Func`1 localInit, Action`1 localFinally)
at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](IEnumerable`1 source, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Action`3 bodyWithStateAndIndex, Func`4 bodyWithStateAndLocal, Func`5 bodyWithEverything, Func`1 localInit, Action`1 localFinally)
at System.Threading.Tasks.Parallel.ForEach[TSource](IEnumerable`1 source, Action`1 body)
at Sitecore.ContentSearch.Azure.Http.CompositeSearchService.PostDocuments(ICloudBatch batch)
at Sitecore.ContentSearch.Azure.CloudSearchUpdateContext.Commit()
Nested Exception
Exception: Sitecore.ContentSearch.Azure.Http.Exceptions.BadRequestException
Message: Error in the request URI, headers, or body
Source: Sitecore.ContentSearch.Azure
at Sitecore.ContentSearch.Azure.Http.SearchServiceClient.EnsureSuccessStatusCode(HttpResponseMessage response)
at Sitecore.ContentSearch.Azure.Http.SearchServiceClient.UpdateIndex(IndexDefinition indexDefinition)
at Sitecore.ContentSearch.Azure.Schema.SearchServiceSchemaSynchronizer.SyncRemoteService(IndexDefinition sourceIndexDefinition, IEnumerable`1 incomingFields)
at Sitecore.ContentSearch.Azure.Schema.SearchServiceSchemaSynchronizer.c__DisplayClass17_0.b__0()
at Sitecore.ContentSearch.Azure.Utils.Retryer.RetryPolicy.Execute(Action action)
at Sitecore.ContentSearch.Azure.Http.SearchService.PostDocumentsImpl(ICloudBatch batch)
at Sitecore.ContentSearch.Azure.Http.SearchService.PostDocuments(ICloudBatch batch)
at Sitecore.ContentSearch.Azure.Http.CompositeSearchService.c__DisplayClass15_0.b__0(ISearchService searchService)
at System.Threading.Tasks.Parallel.c__DisplayClass17_0`1.b__1()
at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
at System.Threading.Tasks.Task.c__DisplayClass176_0.b__0(Object )

Nested Exception
Exception: Sitecore.ContentSearch.Azure.Http.Exceptions.AzureSearchServiceRESTCallException
Message: {"error":{"code":"","message":"The request is invalid. Details: definition : Invalid index: The index contains 1033 field(s). An index can have at most 1000 fields.\r\n"}}

Uh oh, this doesn’t happen on Solr!

Azure search has a hard limit of 1000 fields per index 😢

This is confirmed by checking the limits and quotas for an S1 instance of Azure search service:

Azure search index limits
Source: https://docs.microsoft.com/en-us/azure/search/search-limits-quotas-capacity

An out if the box Sitecore 9.0.1 instance will create some indexes with more than 900 fields…so it works, but allows for very little head room when adding your own fields. As the last point indicates the default settings on the basic indexes index all fields, so any customisation (eg. adding a couple of fields to templates for items that gets indexed) pretty much leads to exceeding this limit. You’ll then start having a bad time when trying to add/modify items or rebuilding the index.

After confirmation from Sitecore support, the default index configs do explicitly include a list of required fields to support basic functionality. So you can change the indexAllFields flag in the default documentOptions to false to ensure all of your custom fields are not automatically added to the index. Patch in this change with a config patch like so:

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:set="http://www.sitecore.net/xmlconfig/set/" xmlns:search="http://www.sitecore.net/xmlconfig/search/" xmlns:role="http://www.sitecore.net/xmlconfig/role/" >
<sitecore role:require="Standalone or ContentManagement or ContentDelivery" search:require="Azure">
<contentSearch>
<indexConfigurations>
<defaultCloudIndexConfiguration type="Sitecore.ContentSearch.Azure.CloudIndexConfiguration, Sitecore.ContentSearch.Azure">
<documentOptions type="Sitecore.ContentSearch.Azure.CloudSearchDocumentBuilderOptions,Sitecore.ContentSearch.Azure">
<indexAllFields>false</indexAllFields>
</documentOptions>
</defaultCloudIndexConfiguration>
</indexConfigurations>
</contentSearch>
</sitecore>
</configuration>

Rebuilding your index should now complete successfully.

If your solution does need additional fields, be picky. Only index what is needed! Change the default indexAllFields flag so that basic functionality still works, then add a new custom index with the fields your solution requires.  Keep it sub 1000 fields!

This will keep your indexes lean and mean as your project grows. *Hint: and also fix the concern below!

Over indexing

While it’s handy to have everything right there in your search indexes, it can lead to decreases in performance as your content grows.  To keep this in check, only index what you need.

  • Only crawl parts of the tree you need to search
  • Only index the fields you need.
  • Use custom indexes to help keep it organised and performant

 

Azure Search on Sitecore 9

Hello, is it me you’re looking for?

Sitecore has implemented Azure Search and Solr search providers for 9+ XP installs.  They both have their pros and cons. But the best way to evaluate and compare them is by getting your hands dirty and breaking some things on your local machine.

Azure Search is Microsoft’s Paas offering that is supported by Sitecore and is the Out Of The Box search provider for Sitecore 9+ destined for PaaS environments. The Sitecore Quickstart ARM templates for all Cloud XP installs will include the appropriate service infrastructure and configs.

Solr is the recommended option for XP on prem deployments.  It is battle tested and fully featured, but has a difficult time fitting in with the Azure Paas offerings many organisations are investing in.

When it comes to your dev box, you’ll likely have installed an on prem package which uses Solr as the default search provider.  Using Azure search for your dev instance full time may be cost prohibitive as you need a Standard instance at minimum. But as an inquisitive dev, it’s easy to have a play with it locally to create and consume custom indexes or just to get a feel for the differences.   Note there are some limitations with using the Azure Search provider as outlined in the Sitecore Azure Search documentation.   There are workarounds for most of these, so really comes down to your implementation and being aware of these limitations when you’ve got the headphones on and in “code mode”.

Azure Search on your dev machine

If you’re game and have a few Azure credits up your sleeve (remember you get a fair whack with your MSDN subscription), then you can get up and running quickly on any 9+ install.

NB: You’ll need a working Sitecore 9+ install (that is using Solr) up and running. If you don’t already have one you can install via SIFLess, for fast installs that won’t take all night long.  If you have an existing customised install there are a number of considerations to keep in mind, which I’ll address in a follow up post.

Create the Azure Search Service

You’ll need to create a search service in the Azure portal.

  1. New Service > Azure Search Service
  2. Give it a URL, it just needs to be unique
  3. Select the appropriate subscription and resource group for your account
  4. Pick the location closest to you to minimise the latency on queries (NB: Azure search is not available in all regions and prices/offerings do vary!)
  5.  Selects a standard pricing tier (you’ll require 15 indexes minimum)
  6. Create it and wait a short while for the service to provision.
  7. Make a note of the search service URL (on the Overview blade) and API key (on the Keys blade) for the following steps.

Create a search service

Modify the configs

The Sitecore 9+ configs allow for an easy switch between providers, assuming you haven’t modified the OOTB configs:

Add a new connection string called “cloud.search”

Using the Search URL and API key from the Azure portal, add a connection string to your ConnectionStrings.config like the below.

<ConnectionStrings>
...
<add name="cloud.search" connectionString="serviceUrl=https://YOURSEARCHURL;apiVersion=2015-02-28-preview;apiKey=YOURAPIKEY"/>
...
</ConnectionStrings>
Change the app search definition app setting in your web.config

Sitecore 9 allows for roles based configuration so, switching providers is just a matter of changing the search definition in app settings.

<AppSettings>
...
<add key="search:define" value="Azure" />
...
</AppSettings>

Rebuild the indexes

Your instance should now be “talking” to your Azure search service, but none of the appropriate data will be populated in the indexes.  Rebuild all your indexes by:

  • Logging into your sitecore instance
  • Go to the Control Panel > Indexing Manager
  • Rebuild Indexes

That’s about it. You’re dancing on the ceiling with searches running in the cloud.  Do some tests.  Have a play, maybe even implement a custom index for your site search.  Just remember to delete your Azure search instance in the portal once you’re done and save those valuable credits!

Now I’m a Sitecore Developer

It’s official.  I have a certificate to prove it. A Sitecore Professional Developer.  For me this is the first step in a journey.  But why did I feel the need to get certified? Couldn’t I just y’know….do the dev work and still enjoy it?

Why should you get certified?

Certification is more than a piece of paper (although that makes you feel pretty good about yourself), it also allows you to:

  • Get recognized both inside and outside the Sitecore community for the time and effort you’ve put in.
  • You get access to Sitecore support.
  • You learn things the Sitecore way….from the ground up.

I thought I’d share my experiences throughout the training and certification process.

Background

My background is in .NET development. Designing, developing and managing teams building .NET apps for Web and Web API .  About 6 months ago, I joined a company that is building out a small Sitecore team internally. I had no Sitecore dev experience beyond watching a couple of youtube clips.  Given there was no way for developers to just grab a copy of Sitecore and have a crack, I have found this is a really common pathway.  None the less, at it’s core, Sitecore is a big ass .NET Web App.  So skills and experience as a .NET web developer are 100% transferable.  There is a really active sitecore community (this surprised me immensely) so if you need help getting started, hit up Sitecore Community Slack, Stack Exchange or the official Sitecore Community Forums.

(NB: Things have changed in the last couple of months!  For .NET devs intrigued by Sitecore and wanting to give it a spin, you can request a trial license at https://www.sitecore.net/getting-started/implementation/developing-on-sitecore)

How do you get certified as a “Sitecore Professional Developer”?

Although certification only really comes in one flavour for developers, there are a number of pathways.  Personally I was lucky enough that my employer sent me out for the 4 day “classroom” training, which also includes a voucher for the exam upon completion.  I believe the training can also be provided online, but can highly recommend the classroom training. The trainer was a really good resource to query given some of the quirks that Sitecore throws up every now and then.

The classroom sessions essentially consists of :

  • Hard copy “Student Lab Guide”
  • Projected slides (available on completion)
  • Some base code to model the labs off
  • and of course the knowledge of the trainer

Over the 4 days we worked through theory and examples presented by the trainer, followed by a lab putting the theory into practice.  Each lab was a step by step guide to achieve the task at hand (E.G. Setting up a Sitecore Instance with SIM or Configuring Personalization).  Often there were bumps in the road with the steps, however quick checks with the trainer usually pointed to user error 😉 .  Each chapter (which usually consisted of a few labs) ended with review questions which were all addressed by the trainer.

Exam Preparation

I booked the exam about a month after the training.  I’d say that was probably a bit too long as some of the memories were a little hazy and did need a bit of revision to get clarity.

I used the following to revise the content:

  • The training student lab guide.  I re-did the labs I was a bit sketchy on.  In my case this was surrounding search and personalization.  Do the optional labs if you skipped them in the course.  They deal with some of the gotchas Sitecore will throw your way.
  • The slides from training.  I made notes on questions that were raised in my mind as I flicked through these, then followed up on the answers in the training guide,  Sitecore doco.
  • The Student Lab Guide chapter review questions.  PROTIP: Revise and understand these.  The exam reflects the training course content. No, these are not the questions.  But if you understand the concepts asked in the review questions, you’ll do just fine!
  • Ask questions for anything you don’t understand.  I am lucky enough to have some experienced and knowledgeable colleagues.  These guys/gals and slack answered pretty much any “dumb question” I had.
  • Sitecore provide a study guide which outlines the breakdown of areas the exam focuses on and some example questions. It gives a great overview of what to expect.

Although not explicit “revision”, I was working on Sitecore for almost 6 months during this period.  A lot of the concepts I did already have a grasp of and continued to hone my skills over that time.  There is no substitute for experience, but certainly brushing up on the sections you aren’t familiar with is crucial.  Your instance won’t implement everything!  Some of the questions are very detailed, but often you’ll find a process of elimination will end up giving you a 50/50 chance!

Sitting the Exam

The exam can be taken online or in person at a Kryterion testing centre.  There was one close by, so I decided to just go to the testing centre.

To be honest the testing centre was the most disappointing part of the whole experience.  The testing space was small, noisy (next to the server room….but they supplied earplugs!), stuffy and hot.  Not ideal conditions to take a 2 hour exam in an Australian summer.  Lesson learned….go to a different test centre next time.  I’m not going to name names, but if you’re taking the exam in Perth, Australia….ping me a message first so I can tell you where to avoid!

The exam itself was in a pretty straight forward format.  70 multiple choice questions. No notes, no books, no references, no internet.  Just read the question, click an answer, then “next”….70 times.  The exam application was bit slow to move to the next question (hey I’m a dev….of course I’m going to look at perf!) but the “time remaining” clock did appear to take this into account. I relied heavily on the feature to flag answers for review, so that I could come back to them at the end.  You can review any question before you submit the exam for marking, but flagging them for review just lets you be a bit more organised in your approach.  I don’t want to comment on the exact content but suffice to say the majority of questions reflect the training courseware, however there are a few curly questions that were beyond what was covered in training. 2 hours was plenty of time to answer the questions, review the flagged items, then go through everything again for silly mistakes.

You get your results via email ~15 minutes after completion.  This is just a “You got x%”, whether you passed (you’ll need 70%+) and a PDF certificate.  I was pretty nervous about my result, but once I got the email I was pretty stoked with only a few questions wrong. Unfortunately I’ll never know which ones!

After the Exam

First things first. I printed the damn certificate and stuck it on my office desk pinboard.  I put in the effort and am proud of the result.  You should be too. Don’t undervalue that.  I’ve found it’s a great conversation starter to promote some of the great features that Sitecore has to offer that your users may not know about. There have been a few “Really, it can do THAT?” moments.

I learned a stack of things I wouldn’t have been exposed to had I not done the training & exam.  Sure, I’d get there through carrying out my daily duties, but feel I have a much better base understanding and exposure to some of the features I simply haven’t come across day to day.  In particular I hope to really push the envelope in personalization and leveraging the power of xDB.

Finally, I’d like to get further involved with the community.  In particular I’ve found slack to be really active with expertise ranging from total n00bs to MVPs.  I’m yet to find anyone not willing to help or point you in the right direction.  There are also lots of local user groups, unfortunately in my city meetups have dwindled.  Something I hope we can rectify in coming months!  If you’re looking to get involved jump on slack and join #sitecore-user-groups and ask what’s going on in your area. While you’re there, say G’day!