Project Overview
Roadside incident reporting is the process of recording information of a reported incident and creating a summary report that accurately represents the situation for effective resolution and case review.
For this project a prototype was developed for CUBIC Transportation Systems in which Natural Language Processing (NLP) and Automated Speech Recognition (ASR) were implemented using Machine Learning as a Service (MLaaS) provided by Microsoft Azure’s Cognitive cloud services for the automation of incoming caller transcription. Alongside the use of mapping APIs provided by HERE Technologies, the operator is provided with automatic geographical suggestions to assist in pinpointing the incidents’ location. The aim of which is to later integrate the demonstrated technologies in the future for semi-autonomous incident resolution, with the hope of improving response times, resolution efficiency, and ultimately allow for a shift in the operator’s job role from ‘Responder’ to ‘Overseer’. The developed artifact is designed around a client-server architecture using C# .NET Core for the web API, with a front-end client application developed within the JavaScript React framework.
Project Goals
- Develop a web-based prototype application for the purpose of streamlining and/or automating aspects of the roadside incident report handling process for operational agents.
- Take live audio input and transcribe it to highlight key information with a summary.
- Use mapping tools to allow live location information delivery related to incoming calls.
Application Modules
Speech Processing & Text Analysis
The speech module was created to process audio input for automatic transcription. Using Microsoft Azure Cognitive Speech services an audio stream is sent for processing in the cloud and continuously returns a transcription that is then used with Azures Text Analytic services to recognize, categorize, and extract keywords for the creation of a summary report.
Mapping & Location Technologies
The mapping module was created to allow for the visualization of relevant location information such as congestion, roadworks, known incidents, etc, and is done using the HERE Maps API for JavaScript. This module grants operators the ability to manipulate the map including manually searching for locations using a suggestive search, viewing of additional location information through marker interaction, the showing/hiding of markers, and resetting the session to clear the map and all previous search queries.
Final Prototype
The final prototype combines the two individual modules mentioned previously to allow for autonomous map updates during live caller transcription and report generation, adding a marker for the greatest match and providing suggested relevant locations for the operator to filter at their discretion. In turn allowing for the operator to better focus on incident management instead of information processing.
Below is a screenshot of the final prototype and the demonstration video.
Testing
In order to assert the applications proper functionality both black and white box testing were undertaken at multiple stages of development. On top of this, performance testing was undertaken to compare the effectiveness of the base text analytic services offered by Azure and the addition of a keyword phrase list used to potentially increase recognition of technical terms.
The figures below show that whilst the inclusion of a phrase list can increase recognition of fixed technical terms it should be replaced with the training of a custom speech model for the best performance due to increased flexibility in recognition of proximally similar key phrases, as well as the benefit of tailoring for regional dialects and accents.
Issues & Challenges
The following video highlights the issues faced and the key decisions made during the project.
Future Possibilities
The following video highlights the future possibilities for the project given further time and access to existing systems.

ABOUT ME
Course
BSc(Hons) Computer Science
Biography
I am a final year student at Teesside University aiming to graduate in the summer of 2021 with a predicted first grade degree in BSc (Hons) Computer Science. I undertook a 16 month placement at the Wilton Business Center in Redcar with Sabisu/Aspentech in which I was seconded to SABIC, one of the worlds largest petrochemical manufacturers, during which I co-managed a queue of support tickets and project work ranging from critical system updates to the creation of new reporting tools. In my free time I enjoy thinking about game development and playing games both social and solitary.
Skill Proficiency
Software:
Visual Studio, Net Beans, Azure Cloud Services, Git, Visio, Android Studio, Microsoft Office
Languages:
Java, JavaScript (Base, Knockout.js, React), Python, HTML, CSS, C#, .NET, SQL, Visual Basic
Other:
Driving License, Completed Fire Team Project with Cleveland Fire Brigade
Employment, Work Experience & Volunteering
Relevant Work Experience
ASSOCIATE CONSULTANT ENGINEER • SABISU/ASPENTECH (STUDENT PLACEMENT) • 2019 (JUN) – 2020 (SEP)
Seconded to SABIC (a large multinational chemical company) I co-managed a support ticket queue, worked as a full stack developer using MySQL, C#, VB, .NET, MVC, Knockout.js, JavaScript, HTML and CSS, patched existing systems, constructed new systems, and consulted for customers and staff. With the reforms I made to the HR system being crucial to its deployment to the rest of the company worldwide
University Volunteering
PASS Leader in Java at Teesside University during 2nd year, with an open mind in further pursuing teaching or tutoring.
Vice-chair of Finance and Co-Founder of the Teesside University Magic the Gathering society.
Actively interacted with students and staff, providing troubleshooting and feedback in key areas, as part of my responsibilities as a 2nd year Computer Science course representative.
Other
Volunteered for Scope charity in Redcar (2015)
Interests
Board games, PC games, card games (Magic the Gathering), table top role playing games (Dungeons & Dragons), table top war games (Warhammer 40k), fiction (Sci-fi, Fantasy), cycling, game development, mental health & self improvement.