RFC 0027 - The Future of Search

[mta-umbraco]

Request for Comments: The Future of Search

Read the full RFC document here.

This RFC discusses adding a new search abstraction to Umbraco. We would love your feedback on the described feature.

How do I contribute?

Most importantly, we don’t want to miss anything, so everything goes in terms of clarifications, questions, suggestions, etc.

Please do the following things if you want to contribute:

1. Read the RFC document here
2. Please read and respect the RFC Code of Conduct
3. Come back here and add comments down below 👇

[Shazwazza]

Thanks @kjac + @bergmania for putting this together. I have some suggestions and questions:

Document structure

As this document is basically a design/spec, ideally it would have a section before the detailed design for Requirements that captures both functional and non-functional requirements that this spec needs to fulfill. My suggestion would be to organize the headings as:

• Summary
• Terminology
• Objectives (instead of Motivation)
  • Requirements
    • Functional - typically in user story format. (i.e. The indexing must explicitly exclude sensitive data)
    • Non-Functional - quantifiable requirements. (i.e. Index rebuilding performance should improve by XX%)
  • Out of Scope
• Detailed Design

This way the reader knows exactly what the expectations are for what the detailed design proposes. Further, terminology for requirements are: “Must” means mandatory/required. “May” means permitted. “Should” means recommended.

So far the only listed requirements are:

• The abstraction should be easy to use.
• It should be possible to implement with any search provider.

From what I can gather in the doc, the requirements (not limited to) are:

• The abstraction should be easy to use.
• It should be possible to implement with any search provider.
• Indexing and searching must support filtering, faceting, and full text search.
• Indexing and search should support Autocomplete/Suggestions functionality.
• Indexing must support both protected content and content variance.
• Indexed data must explicitly exclude sensitive data.
• Indexing must only be performed on servers with the Publisher role.
• Additional indexes should be possible to define (for example Umbraco Forms).
• Master data should not be stored within the indexes.
• The process of rebuilding an index must have a clear indexing start and end status that is transactional.
• The indexing API should allow for customizing the index document before it is indexed.
• Field boosting should be automatically applied for the individual document fields based on a set of rules.

There might be others that I've missed but IMO it would be good to call all of this out at the start of the document to make it clear what outcomes are expected.

Yet to be defined

There are several items in the document explaining what hasn't been defined yet. I think these should also be called out within a single section (maybe at the top of Detailed Design?)

• Index Extensibility - "also index these items" (etc).
• Search Extensibility - "We also envision an extension model for searching".
• Index rebuilding start/end transactional status.
• Field naming conventions (i.e. prefix).
• Configuration of the search and indexing abstraction.

Concerns

"Replacing the current Examine-based implementation"

In the Summary this sentence is telling folks that you are removing Examine which is not actually the case. I would suggest removing this part of the sentence since it is not what the intention of this document is.
 

Separating media and documents into separate indexes

Generally less indexes is better for both performance reasons but also for manageability and costing when using search-as-a-service providers since many providers will charge for more indexes or have index count limitations. Further, many search providers do not support cross index searching in a single search operation which means that a single search operation could not find both media and content in the same query which also means that scoring between content and media would not be possible because you'd have to execute 2 disparate queries.

I'm unsure what the real reason would be to separate these indexes? If there is a specific requirement that this is fulfilling it should be called out in the requirements section.

Field prefixes

Some search providers have strict limitations on how fields are named and this should strive to ensure that field names align with all search provider limitations. This should be trivial by simply not using special chars in the naming. For example, here's the limitation for Azure Search: First character must be a letter or number, No consecutive dashes or underscores.

I would suggest that one of the requirements should be: Index field names must be named with a convention that supports all search providers.

Questions

• My assumption for this abstraction is purely to support Umbraco specific operations such as: All back office search requirements, Basic/standard search functionality for the front-end? For example, I assume that the faceting requirement for this abstraction is because the back office will at some point use this functionality? Else, I think the abstraction should reduce its surface area as much as possible to only support what is absolutely mandatory for the back office.
• The section regarding "The index data format will include the following" which lists Texts, TextsH1/H2/etc... could be made more clear as to what is meant by "TextsH*" fields. Are "TextsH*" fields are specifically talking about 'headers'? or is this more abstract and talking about automatic boosting?

[liamlaverty]

Thanks for releasing this RFC.

Support for date/times

In the Out of scope section, dates & times are listed as excluded (though datetimeoffset is included elsewhere in the RFC). Please consider support for dates & times as early as possible. The ability to search around datetime ranges is a consistent requirement for medium-to-large Umbraco sites. The following scenario may be covered by the included datetimeoffset: I've worked on many sites which have a features like an Events or Our History listing page, where we need to know "what events will happen in Feb 2025"/"did happen in July 1990". I'd assume most agencies encounter requests for this genre of features on most builds. Without it as a core feature, the community will end up creating multiple different implementations, or our clients will end up using third-party websites for those features. Hopefully this feature can be achieved with datetimeoffset, but if not, it'd be worth giving higher priority to those out of scope date and time features.

Support for providers
What's the extent of "support" in the statement:

We will not support other search providers in the initial release.

I've read it like "[Umbraco] will not implement other search providers for the initial release", but site maintainers will be able to implement their own alternative providers with the initial release. Is that correct, or would this implementation work exclusively with Examine at the initial release?

[bielu]

I was one of people which done few approaches to introduce abstractions, so let me start on my comments the most important:

1. Persisting index data in the database this whole section doesnt make sense for any provider excluding of examine, so this should be feature of provider not of Umbraco abstraction...
2. Faceting is feature which is really hard to abstract, especially if we look into support facets from examine, they have specific way of implementing them by modifying indexes. Elastic search for example allows for faceting on any field without modification.
3. Variance with new abstractions, we should support indexing in one or multiple indexes, there is multiple reasons why different cultures should be indexed separately (starting from analyzers, ending up of geo load balancing).
4. I think Umbraco should drop Examine provider as default provider in examine for something not file based (such as Lifti or similar provider)
5. "Umbraco will be handing off either individual content items or collections of content for indexing, but for large sites it will eventually not be possible to hand off all content in a single operation. However, we still want to support zero downtime re-indexing if at all possible." zero downtime indexing should be only on some providers which support it, such as elastic search, trying implement it on top of examine means a lot storage usage and memory usage.

[bergmania]

Thank you all for the valuable feedback so far! #H5RY

@Shazwazza

Suggestions for document structure
Thank you for your recommendations. My concern is that listing elements in the document may lead to discrepancies as the document evolves through amendments.

Your concerns

1. “Replacing the current Examine-based implementation”

You’re absolutely right—this statement needs clarification. While it’s correct that the current Examine-based implementation will be replaced, it doesn’t imply that Examine will be entirely removed. At present, removing Examine is not part of the plan.

2. “Separating media and documents into separate indexes”

I understand your concern, particularly with Azure Search. From an abstraction perspective, however, we believe separating these types is appropriate. That said, custom implementations of the abstraction can consolidate this data into a single index, using a type field for filtering during searches.

3. Field prefixes

Thanks for highlighting Azure Search’s limitations in this area. Using universally safe prefixes is indeed wise. However, implementations can ultimately adapt field names to accommodate specific providers where necessary.

Regarding your questions:

1. “My assumption for this abstraction is purely to support Umbraco-specific operations…”

The objective is to deliver a robust abstraction that serves both the back office and websites. Features like faceting and basic filtering reflect this dual focus, as outlined in the RFC:

rfcs/cms/0027-the-future-of-search.md

Line 22 in 8e06e7f

This RFC proposes a new search and indexing abstraction for Umbraco, replacing the current Examine-based implementation.

2. “The section regarding ‘The index data format’…”

Your interpretation is correct. We will clarify this section to specify that the “TextsH*” fields represent a hierarchy akin to HTML headings versus regular text. The abstraction provides seven “buckets” for implementation-specific boosting with a defined order of importance.

---

@liamlaverty

Thank you for your input—it’s greatly appreciated.

1. Support for date/times

Initially, we considered including date/time support but scoped it out for the first version to simplify implementation. Most scenarios can still be addressed using DateTimeOffset, including your example. However, we recognize there may be cases requiring expanded type support and are open to revisiting this.

2. Support for providers

You understood correctly, though we will refine the RFC to make this clearer. Initially, HQ will provide and support a single implementation. However, we aim to foster community-driven contributions and hope packages for additional search providers will emerge. HQ plans to prototype a single alternative implementation to validate the abstraction’s viability.

---

@bielu
Thank you for your input

1. We disagree — search provider type does not change the fact that data preparation for search operations, particularly with nested structures like block grids/lists, is computationally intensive.

2. While faceting is inherently challenging to abstract, we believe including it is important to address the majority of common use cases. Some implementations may opt not to provide it, but the abstraction supports it.

3. Ultimately this will be up to the implementations as long as the abstraction have the information available to the implementation both at index and search time.

4. Your point is valid. Currently, Examine remains the best default for local development due to its feature set and compatibility.

5. The primary focus is ensuring the abstraction supports functionality that implementations like Elasticsearch may require.

[bielu]

@AndyButland pdf case is still valid in 2025... I am currently working on the project where it is required case.

[bjarnef]

First and foremost, just to ensure we've got the same baseline for this discussion: In the backoffice, searches are always "per type". Documents, media, members and all other searchable entity types are searched individually, and search results are not mixed.

@kjac has it been considered having a multi-index searcher, which currently is able to search in e.g. both ExternalIndex and a custom index. By default Umbraco doesn't use cross-index searching in backoffice IIRC, but perhaps any packages or custom implementation does?

[Shazwazza]

We will certainly need to wrap all consumption of the search abstraction in try/catch handling, as the concrete implementations might be targeting out of process and thus be prone to all kinds of potential exception states. That being said, it's an implementation decision if a NotSupportedException is the correct answer when attempting to use a feature that is not supported; the answer could also just be an empty list of facet results.

I'm with @bielu on this - please only create an abstraction for the absolute necessary things that Umbraco needs, and more specifically, needs 'now'. Creating an abstraction for things that Umbraco 'might' use will probably end up costly... which brings me back to my point about listing out all specific requirements explicitly at the top of the doc.

Relying on exceptions is not a great idea. Relying on empty results is also not good because to an end-user perspective things will look broken. You could go down the route of the old .NET Framework ASP.Net Identity APIs where it had many interfaces for many different things and then the implementation would or would not implement them, and/or it had the main interface expose IsXYZSupported {get;} properties to denote what the implementation supports or not - but I still think this goes down an unnecessary road.

By separating the indexes it means we:
Can make querying the index simpler when you only want to search documents or media. Whether this is via Umbraco's search abstraction or the provider's one, each query won't have to include a "type" is "document" filter.

From an end-user point of view they shouldn't need to think about this. Umbraco will take care of this to make things easier for the end-user so I don't think the argument of making it easier for Umbraco devs is a valid point.

Can be a bit more secure by default - in the sense that media isn't published, and creating a media item would make it immediately available in the website's search if it included media.

Not really, because as already mentioned the argument can be made that this is implementation specific and an implementation could quite easily combine media and content into a single index - as it is today. Then in which case, this RFC isn't talking about separate physical indexes per-se, it is talking about building APIs to separate how to index/query different types of items.

Can more easily avoid surfacing media results in a full site search that aren't appropriate (e.g. whilst a PDF file may be useful to list - but see below - an image file that happens to match the node name probably isn't).
Not preclude an implementation using a single index under the hood (via a type discriminator field).
But it means we can't offer a single search for documents and media. They could be individually searched of course, but a combined result with paging wouldn't be feasible.

PDFs are one thing but media info is another. ExamineX for example, will AI generate descriptions, tags, categories, etc... of images uploaded which can be useful to search on, for both content and media items. If a content item has a media picker and it picks an image of a sunset which is used as its header image but there is nothing concrete indexed as 'sunset' for the content item or its alt-text (which could also be auto-generated in the media item), then searching for 'sunset' for a content item will result in no results, but it could be possible if both the content and media items existed in the same index.

... What I'm eluding too, is separating content and media into different physical indexes seems to reduce flexibility and so far there doesn't seem to be a real concrete reason to do this apart from making the queries that Umbraco writes behind the scenes easier.

[kjac]

Hi everyone 👋

We've had a few talks on these subjects over the past weeks, and here's the outcome:

Regarding faceting

It might not be entirely clear to everyone that the search abstraction will be used to power both backoffice and frontend searches. I have amended the RFC to point this out explicitly.

Since faceting is an expected feature of frontend searches, we will be moving along with faceting as part of the abstraction.

Regarding content types

We don't want the search abstraction to concern itself with different types of content (documents, media, members).

Ideally, it should be possible to create custom indexes with anything, and to that end, a known set of content "sub-types" will be in the way.

We also perceive the need for cross scoring between document and media items as an edge case. Or to be blunt, media items will rarely have any place among documents in a site search.

To that end, we will move forward with separate indexes for documents and media, to make the search abstraction cater to the common scenario instead of the edge case.

Keep in mind that:

• The search implementation can still store documents and media (and members, for that matter) in the same physical index. It will then be responsible for separating items at query time.
• It should be possible to define a custom index containing both documents and media items for those edge cases where cross scoring is necessary.

Regarding graceful degradation

Lastly we've been talking about what happens when a search implementation is unable to fulfil the whole abstraction. At this point, the answer is kind of vague: We want to support graceful degradation, but the specifics are yet to be clarified.

I have added a section about graceful degradation to the RFC.

[bielu]

Regarding faceting
It might not be entirely clear to everyone that the search abstraction will be used to power both backoffice and frontend searches. I have amended the RFC to point this out explicitly.

Since faceting is an expected feature of frontend searches, we will be moving along with faceting as part of the abstraction.

That was clear from beginning, however i still dont think umbraco should deliver any complex search features in search abstractions. Faceting is complex feature and maintaining facets in search abstraction is a significant amount of work. By introducing facet abstractions you might limit potential search providers from being introduced, anyone who want use advanced feature imho should just native api, or implement own abstraction as then limitation are introduce based on project specification not by CMS.

[DanDiplo]

If there are to be separate "published" and "draft" document indexes, how would one search both together (eg. in the back office)? It wouldn't be enough to search both and combine the results, as this wouldn't account for proper ranking.

[bielu]

the draft document index will also contain the draft version of any published documents

Tbh it is not clear from RFC, word draft suggest it will store only draft version, so what happens if we publish content and there is no more draft? 🤔, based on what you said it will be still in index but word draft than not fits for the name :)

[AndyButland]

The intention here is that the "draft" index that will be used for the backoffice search includes "the most recent version" of the document. So for a published document with no draft, it will have the published version. For an edited document that has a draft and a published version, the draft will be the one indexed.

[mattbrailsford]

If we are planning ahead here, should we be thinking about documents having multiple drafts?

[bielu]

@AndyButland as i said then "draft" name is not suitable as it suggest drafts only, preview I guess would be less misleading.

[AndyButland]

I've added the clarification I posted above to the RFC itself.

[kjac]

Hi everyone. Thank you all for all your comments and your enthusiasm on this subject ❤️

This RFC has now entered the "final comment period" and will be closed (accepted) within the next couple of weeks 👍

[kjac]

We are closing this RFC as accepted today.

Once again, thanks to everyone who's helped shape it. Your inputs are invaluable to us ❤️

Planning wise we are working with a two-step plan for phasing the search abstractions into the Umbraco core:

1. Release them as an add-on for V16 and V17.
2. Include them in core from V18.

To maximize their potential value in V16 + V17, some minor changes will be made to the Umbraco core. For example, the collection views (formerly known as list views) could conditionally be powered by these search abstractions, provided we include proper hooks in the Umbraco core.