Agencies have accumulated mass quantities of structured and unstructured data and content over many years. At each interaction with your agency, citizens create data and that data needs to be managed. The more data produced and captured, the more paper and ‘digital paper’ work needs to be completed, which means more people hours go into keeping up with this overwhelming volume than to reviewing and improving how we keep up with these demands.
Knowledge Mining is the process of discovering actionable information from large sets of unstructured data, like text or images. It uses Artificial Intelligence to detect hidden patterns and information
Risks
A Cognitive Search service is hosted on Azure and is typically accessed by client applications over public network connections. While that pattern is predominant, it's not the only traffic pattern that you need to care about. Understanding all points of entry as well as outbound traffic is necessary background for securing your development and production environments.
Cognitive Search has three basic network traffic patterns:
- Inbound requests made by a client to the search service (the predominant pattern)
- Outbound requests issued by the search service to other services on Azure and elsewhere
- Internal service-to-service requests over the secure Microsoft backbone network
Find out more about how to secure the data here: Security overview - Azure Cognitive Search | Microsoft Docs
Rationale
Knowledge mining is an emerging discipline in artificial intelligence (AI) that uses a combination of intelligent services to quickly learn from vast amounts of information. It allows organizations to deeply understand and easily explore information, uncover hidden insights, and find relationships and patterns at scale.
- Unlock valuable information lying latent in all your data in order to automate actions and make better decisions.
- Saves money by reducing time employees are searching for business-critical information, automating redundant tasks
- Find answers quickly across a heterogenous set of information spanning many content types
- Identify patterns, relationships, and insights in that information
As-Is
Knowledge mining refers to an emerging category of AI designed to simplify the process of accessing the latent insights contained within structured and unstructured data. Knowledge mining defines the process of using an AI pipeline to discover hidden patterns and actionable information from sets of structured and unstructured data in a scalable way. Knowledge mining includes a complex logical understanding and can connect information streams to form real world business insights.
Azure Cognitive Search is a managed cloud search service and solution that gives developers APIs and tools for adding a rich search experience over private, heterogeneous content in web, mobile, and enterprise applications. It offers capabilities such as scoring, faceting, suggestions, synonyms, and geo-search to provide a rich user experience. Azure Cognitive Search is also the only cloud search service with built-in knowledge mining capabilities. Azure Cognitive Search acts as the orchestrator for your knowledge mining enrichment pipeline by following the steps to ingest, enrich, and explore and analyze.
Key scenarios for knowledge mining include:
- Digital content management: Help customers consume content more quickly by providing them relevant search results in your content catalog.
- Customer support and feedback analysis: Quickly find the right answer in documents and discover trends of what customers are asking for to improve customer experiences.
- Data extraction and process management: Accelerate processing documents by extracting key information and populating it in other business documentation.
- Technical content review and research: Quickly review documents and extract key information to make informed decisions.
- Auditing and compliance management: Quickly identify key areas and flag important ideas or information in documents.
Stakeholders
IT, LOB, Data Researchers
Business Process Model
Please review Knowledge Mining Solution Accelerator! This accelerator provides developers with all of the resources they need to quickly build an initial Knowledge Mining prototype with Azure Cognitive Search. Use this accelerator to jump-start your development efforts with your own data or as a learning tool to better understand how you can use Cognitive Search to meet the unique needs of your business.
In this repository, we've provided you with all of the artifacts you need to quickly create a Cognitive Search Solution including: templates for deploying the appropriate Azure resources, assets for creating your first search index, templates for using custom skills, a basic web app, and PowerBI reports to monitor search solution performance. We've infused best practices throughout the documentation to help guide you. With Cognitive Search, you can easily index both digital data (such as documents and text files) and analog data (such as images and scanned documents).
Prerequisites
In order to successfully complete your solution, you'll need to gain access and provision the following resources:
- Azure subscription - Create one for free
- Visual Studio 2019 or later - Community edition or higher
- Postman for making API calls
- Documents uploaded to any data source supported by Azure Search Indexers. For a list of these, see Indexers in Azure Cognitive Search. This solution accelerator uses Azure Blob Storage as a container for source data files. You can find sample documents in the sample_documents/ folder
Find out more here: Knowledge Mining Solution Accelerator - Code Samples | Microsoft Docs
Lessons Learned
Find answers to commonly asked questions about concepts, code, and scenarios related to Azure Cognitive Search. FAQ - Azure Cognitive Search | Microsoft Docs