Enterprise Search Engines in Transition – Processing Unstructured Data Better
Making knowledge accessible to all employees - made possible by new enterprise search engines in companies
This article explains why there is an extreme amount of movement in the market for corporate search engines right now.
Few technologies have developed as quickly in recent years as Natural Language Processing (NLP). Large transformation models make it possible. This gives companies completely new ways to make unstructured knowledge much more accessible to employees and to process it.
Through COVID-19, 75% of companies have introduced digital collaboration tools, according to a Bitkom study. As a result, not only has the number of data silos in companies increased, but also the amount of documented knowledge – but mostly in unstructured forms such as text, image, audio or video formats. With the increasing share of home offices, the possibility of simply asking colleagues for information has been permanently eliminated. Surprisingly often, knowledge is not only stored in the heads of employees, but already digitized in various software solutions. And not just since Corona. Intranets, archives, e-mail boxes, SharePoints or network drives have for years been an excellent data grave in which it is difficult to find information again. As a consequence, employees search extremely hard or simply recreate documents, etc. This is not only annoying for many employees. This not only annoys many employees, but also results in high costs for companies. The concept of an enterprise search engine is therefore not new to many software providers. It is about providing the right information via different data sources in the company.
Index
New technology – new possibilities
What has changed, however, is the technology that can be used to process unstructured file formats. Since transformer models were presented at the Neural Information Processing Systems conference in 2017, Natural Language Processing (NLP) has opened up new possibilities for processing unstructured information. In enterprise search, the user experience familiar from internet search engines can now be approximated in an enterprise context. What Internet search engines used to compensate with millions of data points in places where semantic understanding was not yet sufficient, is now becoming accessible and affordable for enterprises.
The use cases of NLP are diverse. With NLP, a computer not only recognizes language, but can capture and interpret sentences. Thus, not only question-answer models are possible, but also, for example, summaries or extremely good translations can be generated. This is especially beneficial for text-heavy data in companies and opens up various use cases in different areas.
In this article, I would like to take a closer look at the use cases in the area of enterprise search.
Changes in information retrieval
Many companies have tried different solutions over the years to store information in a retrievable way. No matter how well the employees are trained, after a few months some of them establish their own filing systems, which makes it very difficult for new employees in particular to find information again. In addition, demographic change means that experienced employees are leaving companies, which means that companies have to focus even more on digitizing knowledge and finding it again.
For this reason, some companies are already using internal search engines – sometimes limited to specific departments, sometimes as an enterprise search. What many established providers have in common is that they have been around since the early 2000s and thus helped shape the first wave of digitization in German companies.
Older solutions usually have difficulties with scaling and with the quality of search results. This is due on the one hand to the performance of computers and classical search algorithms, and on the other hand to the way software development has changed over the years. Where once a search engine could normalize words to the maximum (example of normalization: had – have) and words were not interpreted correctly in context, today neural networks are used to classify not only words but also whole sentences and paragraphs correctly.
Where previously a lot of filters were needed to filter information accordingly, new types of search engines do this with the help of semantic understanding. This gives the user exactly the same user experience that he is familiar with from Internet search engines. On the one hand, this makes everyday work easier in general, and on the other hand, it compensates for the loss of information exchange about projects at the coffee bar, especially in times of home offices.
Enterprise search is therefore not only about reducing search time (the costs of which are relatively easy to calculate), but also about accelerating projects and service through the correct provision of information or avoiding data waste through versioning.
The potential of start-ups
As Natural Language Processing grows, new use cases and opportunities arise, which in turn flushes various new vendors into the market. The unbeatable advantage of the young companies:
They build software around a technology and do not have to integrate the new technology into existing software. This gives them the great advantage of being able to exploit the full potential of new technologies.
Natural Language Processing is not the only type of AI being used by startups. Social media is leading the way: More and more content is being prepared visually or provided via audio files. This trend is also noticeable in companies. Be it flyers, webinars, corporate podcasts or recorded video conferences. Multimedia content is also becoming the norm in companies. Computer vision or machine learning models, for example, make these solutions searchable and much easier to find for employees. What only Google & Co could offer on the Internet a few years ago, because they could compensate for the weaknesses of the technology with millions of data points, young start-ups are now bringing to many companies as standard software.
This has the potential to revolutionize an entire industry. It remains unclear whether one large provider will ultimately prevail or whether there will be several players that will determine the market behind. At the very least, there are currently several start-ups with investor money that are counting on scaling quickly.
Added value through enterprise search
After first taking a closer look at the technical aspects of differences, we now turn to the concrete added values for companies.
A comprehensive search not only reduces search time, although this would be the easiest metric to measure. Comprehensive search enables employees to make better decisions based on better information. Knowledge that is quickly and easily accessible also means that decisions can be made more quickly. After all, if knowledge is not accessible via a central location, some employees usually only look at 1-2 places where the right information might be located, but not at all the relevant filing locations (see Figure 1).
Another problem that many employees face: Re-elaborating knowledge and creating documents: If employees were able to find the right document/information quickly and efficiently, they could quickly get back to tasks that add value. It also reduces data waste and the number of versions of different documents.
In addition, the daily work routine becomes easier for employees, since they have to constantly send e-mails or make phone calls to find information, but – especially in times of home offices – are given the opportunity to find knowledge on their own.
The use cases for enterprise search engines are diverse – they can help employees in the call center or in technical service, they help in the office or in the executive suite. However, which use case offers the greatest added value for the company in question must be looked at on a case-by-case basis.
Comments are closed.