Happy New Year everyone! A light hearted look at enterprise search evolution to start the New Year. This is an extended summary, the full article will be published in “Search Insights” in March 2020 including the link to digital transformation.

The Cambrian explosion occurred more than 500 million years ago, a rapid burst that diversified life. There is significant evidence that environmental changes caused this, leading to a time of great body plan innovation, such as development of hard exoskeletons. Ecological change responding to these environmental factors.

Using this as a metaphor, environmental changes over the past few years in Enterprise Search, such as technological advances (compute power and machine learning), exponentially increasing information creation & capture and the OpenSource movement have arguably led to an explosion in ‘search species’.

Metadata Only Catalogues

Original document indexes of metadata provided early search interfaces in the enterprise. They were generally high precision, low recall, as only a small amount of information was searchable. The exponential increase in digital information creation particularly in the late 1990’s meant it was no longer feasible for centralized departments to manage all of an enterprise’s information. This environmental change saw the deployment of Electronic Document Management Systems (EDMS) and associated full body text search engines. Out-competed, the metadata catalogue declined in its popularity, some forms became extinct. There are still niche areas where this type of search remains useful, such as physical asset management.

General Purpose Corporate Google

The corporate Intranet was among the first drivers for a Corporate Google-like search. Over time this often merged with the search of EDMS systems, moving towards the text and image based ‘enterprise search’ concept we know today. This was of the form of a search box and ‘ten blue links’, mirroring the success seen in the Internet.

Despite often poor user satisfaction, this ‘search species’ flourishes today and remains the mainstay approach for any search deployment in the enterprise. Search ranking (not user interface) being the dominant criteria by which staff judge its success.

Social Search

Enterprise ‘white pages’ of people, expertise, Discussion Forums were among the central planks of Knowledge Management (KM) strategies in the late 1990’s and early 2000’s. There appears to be a move towards more cloud-based service adoption by enterprises, ‘information push feeds’ complementing traditional ‘information pull’. Some research indicates people are spending as much time (if not more) searching within social networks on the Internet than they are using Google Internet search, where deep context significantly aids ‘interestingness’ of results.

Answer Machine

This ‘search species’ marks an important evolutionary branch from the classic enterprise search deployments. The focus is answering questions not finding documents. This requires the use of Natural Language Processing (NLP) and Machine Learning (ML) to convert unstructured text into structured data and information. Concepts and entities dominate, rather than ‘the document’.

Whilst these answer machines can be voice activated (like Siri/Alexa), they can also be text based to suit the environment. On Smartphones or mobile devices, answer machines are more significant due to the real estate afforded to the user, where scrolling long lists of search results or viewing complex visualizations is problematic.

Search Task Applications

For many high value functions in enterprises, there is a need to mine information for insights and have new information needs stimulated by applications. These often involve rich domain dashboard-like visualizations emphasizing the meaningful (rather than returning a simple factual answer).

Rather than search being a ‘passive’ facility – meeting an existing need the user already has, search task applications are more intrusive; they are precognitive creating new information needs. These applications act as ‘assistants’, heavily curating what the user sees, notifying us, offering data driven informed opinions for important business activities based on past heuristics and information.

Why wouldn’t any professional want opinions from a machine that has read every document in the company?

The new terminology of ‘cognitive search’ and ‘insight engines’ from IT market analysts may have been attempts to define this new ‘search species’ to differentiate it from its ancestors. The mistake some may have made is to think of these ‘search species’ as replacing its antecedents. As the picture shows, my suggestion is that they co-exist in different niches.

Summary

A rich and varied ecosystem exists for enterprise search. If enterprise search is the body, a significant amount of ‘body plan’ innovation has occurred in response to changing environments. Measuring the success of enterprise search is perhaps evolving past simple user satisfaction metrics of search results lists. These remain important, but enterprise search capability is so much more.

The search network can be found Here. You can find the 2019 report on Search Insights here