Software Example Patent for Semiautomated Relay Method and Apparatus

Invented by Robert M. Engelke, Kevin R. Colwell, Christopher Engelke, Ultratec Inc

The market for semiautomated relay method and apparatus is experiencing significant growth due to the increasing demand for efficient and reliable relay systems in various industries. Semiautomated relay systems are designed to automate the process of relaying information or signals between different components or devices, thereby reducing human intervention and improving overall efficiency. One of the key drivers of this market is the rising need for advanced relay systems in the power industry. With the increasing demand for electricity and the integration of renewable energy sources into the grid, there is a growing requirement for reliable and efficient relay systems to ensure the smooth transmission and distribution of power. Semiautomated relay systems offer advanced features such as fault detection, isolation, and restoration, which are crucial for maintaining the stability and reliability of the power grid. The industrial automation sector is also contributing to the growth of the semiautomated relay market. As industries strive to improve productivity and reduce operational costs, the adoption of advanced automation technologies becomes imperative. Semiautomated relay systems play a crucial role in industrial automation by enabling the seamless communication and coordination between different machines and processes. This not only enhances productivity but also ensures the safety of workers by minimizing the risk of accidents. Furthermore, the increasing complexity of modern vehicles is driving the demand for semiautomated relay systems in the automotive industry. With the integration of advanced driver-assistance systems (ADAS) and autonomous driving technologies, vehicles require sophisticated relay systems to facilitate the communication between various sensors, actuators, and control units. Semiautomated relay systems enable the efficient transmission of signals and data, ensuring the smooth operation of these advanced features. The healthcare sector is another significant contributor to the market for semiautomated relay method and apparatus. With the increasing adoption of digital health technologies and the need for seamless communication between medical devices and systems, semiautomated relay systems are becoming essential. These systems enable the secure and efficient transmission of patient data, ensuring accurate diagnosis and timely treatment. In terms of regional growth, North America and Europe are expected to dominate the semiautomated relay market due to the presence of established industries and advanced infrastructure. However, emerging economies in Asia Pacific, such as China and India, are witnessing rapid industrialization and urbanization, which is driving the demand for semiautomated relay systems. Key players in the market are continuously investing in research and development to introduce innovative products and gain a competitive edge. They are also focusing on strategic partnerships and collaborations to expand their market presence and cater to the evolving needs of customers. In conclusion, the market for semiautomated relay method and apparatus is witnessing significant growth due to the increasing demand for efficient and reliable relay systems across various industries. The power industry, industrial automation, automotive sector, and healthcare are the key drivers of this market. With advancements in technology and increasing investments, the semiautomated relay market is expected to continue its growth trajectory in the coming years.

The Ultratec Inc invention works as follows

The method of trancribing communications consists of the following steps: obtaining a number of hypothesis transcriptions from a voice signal generated using a speech recognition device; determining consistent word that is included in the first and second hypothesis transcriptions. In response to the determination of consistent words, the consistent word are provided to a device to be presented to an assisted user.

Background for Semiautomated Relay Method and Apparatus

The present invention is a relay system for providing captioning in voice for users with hearing impairments. More specifically, it is a relay that uses automatic voice-to text captioning software for voice transcription.

Many people suffer from some form of hearing loss. In the United States, for example, 3 out of 1000 people are functionally blind and approximately 17 percent (36 millions) of American adults have some degree of hearing impairment, which usually gets worse with age. Many people with hearing impairments have learned to adapt to the way their loss affects their ability to communicate. Many deaf individuals have developed ways to compensate for their hearing loss. They communicate via sign language, or read lips while someone speaks.

When it comes to remote communication using a phone, unfortunately, a hearing impaired individual (e.g. an assisted user) cannot use sight to compensate their hearing loss, as traditional telephones don’t allow an AU to view the person at the other end (e.g. no lip-reading or sign viewing). Some people with partial hearing loss can compensate by turning up the volume of their phone. Others with a more severe loss of hearing cannot be compensated by conventional telephones and telephone communication can be a poor choice.

In order to provide communication services to AUs, a new industry has developed where voice communications are transcribed and displayed in text on a screen. The AU can then read the text during a session. In some cases, the AU device broadcasts the voice of the person linked to it at the same time as the displayed text. This allows an AU with some hearing ability to understand most communication phrases.

U.S. Pat. No. Patent No. 6,603,835 (hereinafter “the ‘835″ patent) entitled?System For Text-Assisted Telephony?” The patent teaches AUs how to use several types of relay systems in order to provide text captioning. A single-line system is one type of captioning service. This is where a relay connects the AU’s device to the telephone being used by the other person. Unless otherwise indicated, the person who is communicating with the AU in the following paragraphs will be referred as a hearing-user (HU), even though it may be that the AU is communicating with another AU. In single-line systems, a line connects an HU to the relay, and a line (e.g. the single line), the relay to the AU. The voice from the HU device is sent to a relay assistant (CA), who transcribes it to text. This text is then transmitted to the AU display device. In some cases, the HU’s speech is also carried through the relay and broadcasted to the AU.

The other captioning system type described in the patent ‘835 is a 2-line system. In a 2-line system, a HU telephone is directly connected to the AU device via a 1st line. This allows voice communication between the AU & HU. The AU’s device can be linked to a relay via a captioning button to provide the HU voice on a second-line. A relay CA again listens to the HU’s voice message, and then transcribes it into text. This is sent back to the AU on the second line for the AU to view. The two-line system has the advantage of allowing the AU to add captioning during a call. It is crucial because many AUs may only be partially impaired, and only need captioning when it’s absolutely necessary. It is important to have the option of not having captioning in situations where an AU can be used like a regular telephone, and non-AUs who do not require captioning (e.g. a spouse living next door to an AU with good hearing ability) may use the AU.

With any relay service, accuracy, speed, and the cost of providing the service are the main factors that determine the value. Text should accurately reflect spoken messages from HUs, so that AUs reading the text can understand the meaning of the message. The use of incorrect words can lead to inaccurate messages, and confusion when an AU is reading transcribed texts.

In terms of speed, it is ideal that text be presented to an AU at the same time as the voice message corresponding with the text. This way an AU can see the text in conjunction with a message while the message is being heard. Text that follows a voice by several seconds may cause confusion. The captioned text is displayed by most systems relatively quickly, e.g. Most of the time, captioned text is presented within 1-3 seconds following the broadcasting of the voice message. Sometimes, a captioner can be behind, causing a longer delay (e.g. 10-15 seconds).

With regard to cost, current systems require that each communication session be handled by a CA who is highly-trained and unique. CAs must be able speak clearly, and type accurately and quickly. CA jobs also have high levels of pressure and turnover is high compared to other jobs. This increases the cost associated with running a relay.

The use of voice to text transcription software by relay CAs has improved captioning speed and reduced costs. Early relay systems required CAs type the entire text that was presented by an AU device. Typists with high levels of skill were needed to present text as fast as possible following the broadcast of a voice message. Normal conversations are conducted at a speed of between 110 and150 words per minute. During a normal conversation between an AU, and an HU only half of the words spoken must be transcribed. (The AU usually communicates with the HU for half a session.) Due to various inefficiencies, a CA must be able type 100 words or more per minute to keep up. Most professional typists can type between 50 and 80 words per minute, so they are able to keep up for a while with a typical conversation. Professional typists can be expensive. Even professional typists can fall behind in real-time transcription at times, even though they are able to keep pace with the conversation for most of the time.

In relay systems using voice-to text transcription software that is trained to the voice of a CA, the CA listens and then revoices a HU?s voice message into a computer running this software. As the software is trained to the CA?s voice, it transcribes a re-voiced text message faster and with fewer errors than a typer can. Revoicing techniques are much easier to learn and faster than typing at high speeds. This reduces the training costs as well as general costs for CAs. Revoicing is faster in many cases than typing, so voice-to text transcription can be accelerated by using revoicing.

At the very least, some previous systems contemplated furthering reducing costs associated relay services by replacing CAs with computers that run voice-to text software to convert HU voice message to text. This solution has been plagued by several issues in the past, which led to no one ever implementing an effective system. The majority of voice messages (e.g. an HU voice message) sent over the most common telephone lines are not compatible with direct voice-to text transcription software. Automated transcription software has been designed to work with voice signals that have a wider spectrum of frequencies. This is different from the typical range used for phone communication. Voice signals are typically delivered between 300 Hz and 3000 Hz. Automated transcription software is not able to work with voice signals that are delivered over telephone lines. This leads to a large number of errors. Noise on the telephone line, which is common, can also affect accuracy.

Second, to ensure accuracy in automated transcription programs, they must be trained on the voice of the speaker. There is no way that a relay can have trained the software on the HU’s voice before a new HU contacts an AU device. The software will not be able to accurately produce text when the HU speaks.

Third, many automated transcription packages use context to create text from a message. Software can use the context of each word to determine which word was spoken. In order to use the words surrounding a first-word to identify that word, it is necessary to obtain the words surrounding the first word. Many automated transcription systems delay presenting transcribed text to allow context to be used before presenting. The delay in text delivery caused by systems that wait to present text until after subsequent words have been transcribed to correct prior words is not compatible with relay systems’ need for real-time or near real-time text delivery.

It has been acknowledged that a semi-automated hybrid system is possible. When acceptable accuracy can easily be achieved with automated transcription software the system will automatically use this software to convert HU voice to text. If accuracy is not acceptable, a CA can be patched in to transcribe the voice to text. It is thought that the number CAs needed at large relay facilities can be significantly reduced (e.g. 30% or more) when software is able to perform a significant portion of transcription into text. The automated transcription software is improving over time. In some cases, it may also train itself to the voice of an HU and to the nuances of voice messages transmitted over the phone line. The training can be done in parallel to a CA (e.g. typing, revoicing etc.). Transcribing speech-to-text, and once accuracy reaches a threshold acceptable level, the system can automatically delink from a CA and use text generated by software to drive an AU display device.

It is now known that there are two processors in a relay that can perform automated voice recognition. This means that they could handle the automated voice-recognition part of a triage procedure involving an CA. In most cases, either a relay or AU device processor can perform the automated transcription part of a hybrid procedure. In some cases, an AU device may perform automated transcription while a relay assistant generates CA generated texts. The relay and AU device then work together to assess when the CA is removed from a call and the automated text replaces the CA generated words.

In other cases, where the communication device of a HU is a computer with a processor that can convert voice messages into text, the device of a CA may generate text simultaneously and the device of the HU and the relay could cooperate to produce text and determine the time when the CA’s call should be terminated.

The CA generated text can be used, regardless of the device performing the automated captioning to determine the accuracy of the automated text and when the CA is removed from the call. The CA generated text can also be used, no matter which device is doing automated text captioning. It will help to increase accuracy and speed up the process until the CA is removed from the call.

It has been acknowledged that there are instances when a hearing-impaired person listens to a HU without a device from an AU providing simultaneous text, when the AU becomes confused and wants transcriptions of recent voice messages by the HU. If an AU is using an AU device to make a call that is not captioned, and it has trouble understanding the voice message, the AU may initiate a captioning service in order to get text for the subsequent voice messages. While text is provided to the AU for the subsequent messages, he or she still does not understand the original voice message. In another case, where CA-generated text lags behind a current HU voice message by a significant amount, an AU can request that captioning catch up with the current message.

In order to provide captioning in such cases, at least in some embodiments, an AU’s devices stores the voice messages of an HU and, upon receiving a captioning request or a catch-up request, the recorded messages are either used to automatically generate text, or have a CA create text corresponding to recorded voice message.

When automated software is tuned to the voice of a HU, it may store a voice model that can later be used to tune automated software in order to transcribe that HU?s voice. It may also store a profile that allows the HU to be distinguished from other HUs. The profile can then be used by the HU to identify it when calling an AU device. The voice model is used to tune software to produce highly accurate text corresponding with the HU voice messages.

Click here to view the patent on Google Patents.