Project Overview

Roadside incident reporting is the process of recording information of a reported incident and creating a summary report that accurately represents the situation for effective resolution and case review.
For this project a prototype was developed for CUBIC Transportation Systems in which Natural Language Processing (NLP) and Automated Speech Recognition (ASR) were implemented using Machine Learning as a Service (MLaaS) provided by Microsoft Azure’s Cognitive cloud services for the automation of incoming caller transcription. Alongside the use of mapping APIs provided by HERE Technologies, the operator is provided with automatic geographical suggestions to assist in pinpointing the incidents’ location. The aim of which is to later integrate the demonstrated technologies in the future for semi-autonomous incident resolution, with the hope of improving response times, resolution efficiency, and ultimately allow for a shift in the operator’s job role from ‘Responder’ to ‘Overseer’. The developed artifact is designed around a client-server architecture using C# .NET Core for the web API, with a front-end client application developed within the JavaScript React framework.

 

Project Goals

1. Develop a web-based prototype application for the purpose of streamlining and/or automating aspects of the roadside incident report handling process for operational agents.
2. Take live audio input and transcribe it to highlight key information with a summary.
3. Use mapping tools to allow live location information delivery related to incoming calls.

 

Application Modules

Speech Processing & Text Analysis

The speech module was created to process audio input for automatic transcription. Using Microsoft Azure Cognitive Speech services an audio stream is sent for processing in the cloud and continuously returns a transcription that is then used with Azures Text Analytic services to recognize, categorize, and extract keywords for the creation of a summary report.

 

Mapping & Location Technologies

The mapping module was created to allow for the visualization of relevant location information such as congestion, roadworks, known incidents, etc, and is done using the HERE Maps API for JavaScript. This module grants operators the ability to manipulate the map including manually searching for locations using a suggestive search, viewing of additional location information through marker interaction, the showing/hiding of markers, and resetting the session to clear the map and all previous search queries.

 

Final Prototype

The final prototype combines the two individual modules mentioned previously to allow for autonomous map updates during live caller transcription and report generation, adding a marker for the greatest match and providing suggested relevant locations for the operator to filter at their discretion.  In turn allowing for the operator to better focus on incident management instead of information processing.

 

Below is a screenshot of the final prototype and the demonstration video.

 

Testing

In order to assert the applications proper functionality both black and white box testing were undertaken at multiple stages of development. On top of this, performance testing was undertaken to compare the effectiveness of the base text analytic services offered by Azure and the addition of a keyword phrase list used to potentially increase recognition of technical terms.

The figures below show  that whilst the inclusion of a phrase list can increase recognition of fixed technical terms it should be replaced with the training of a custom speech model for the best performance due to increased flexibility in recognition of proximally similar key phrases, as well as the benefit of tailoring for regional dialects and accents.

 

Issues & Challenges

The following video highlights the issues faced and the key decisions made during the project.

 

Future Possibilities

The following video highlights the future possibilities for the project given further time and access to existing systems.