In one sentence: our position search was not up to scratch! Users would often share feedback about how search could be improved. One major issue was keyword search - you’d search for a keyword and the results didn’t have anything to do with the search term.

We have upgraded our search infrastructure to AWS OpenSearch, driven by the need for a more effective and efficient search solution. Our previous system was struggling with slow performance and delivering irrelevant results, especially for keyword searches. AWS OpenSearch, built on the Elasticsearch open-source project, offers an advanced search and analytics engine that promises to address these challenges with real-time search capabilities, scalability, and enhanced relevance.

Tool evaluation and testing We started by evaluating multiple search tools through a series of tests. These tests included performance benchmarks, relevance assessments, and scalability simulations to ensure we selected the optimal tool. Each solution was rigorously tested to understand its strengths and limitations.

Why we chose AWS OpenSearch

AWS OpenSearch provides a significant boost in both search performance and relevance. It enhances our ability to deliver accurate and contextually relevant results, addressing the shortcomings of our previous search system. Its rich feature set means we could implement advanced search functionalities such as faceted search and custom scoring. This customisation helped tailor the search experience to our needs and provided deeper insights into search behaviours.

Integration with Data OS Once AWS OpenSearch was selected, we proceeded to integrate it with our data operating system (Data OS). This phase involved migrating and indexing our existing data into OpenSearch. We created mappings and indices to accurately represent our data schema, ensuring that the search engine was set up to handle our specific data requirements.

Platform integration Following the data integration, we worked on integrating OpenSearch with our application platform. This integration required updating our application's search functionality to interface with OpenSearch using its client libraries and APIs. We focused on ensuring seamless communication between the application and the search engine.

Closed Beta testing Currently, our implementation is in the closed beta stage. We have rolled out the new search functionality to a select group of users to gather feedback and test the system in a real-world environment. This closed beta allows us to refine and optimise the search experience based on user input before a broader release.

Monitoring and optimisation: During the closed beta, we are closely monitoring the performance of our OpenSearch cluster. We are using monitoring tools to track key metrics and gather insights to make ongoing improvements. Feedback from beta users will be used to fine-tune configurations and enhance the overall search experience.

Conclusion The transition to AWS OpenSearch represents a significant upgrade to our search capabilities, addressing previous limitations and providing a more efficient, relevant search experience. By thoroughly evaluating search tools, integrating with our data and platform, and currently testing in a closed beta, we are working to ensure that OpenSearch meets our needs and delivers an exceptional user experience. We are excited about the improvements and look forward to a successful full deployment.

Implementing OpenSearch marks the end of our ‘Position Discovery’ project. As a team, we’re really proud of what we’ve built (see more here and here!) to make discovering positions easier, faster and more enjoyable. We’ll now being moving onto a new project ‘Application Management’. We are building a kanban style board which enables users to keep track of all of their applications in one place. Keep an eye out for the MVP, launching soon!