multi modal

The term multimodal is used in the application scenarios of the telecommunications services and in the user interfaces. In the case of telecommunications services, this refers to the media-technical mix of communication channels; in the case of user interfaces, it refers to the various operating options such as voice input or gesture recognition.

As far as telecommunication services are concerned, the example of a voice mail shows that the message is spoken as speech to a voice mailbox. The voice mailbox stores the message, which is then converted to text using speech-to-text and forwarded to the recipient as an e-mail and/or short message service( SMS). In this example, the message is first transmitted over a voice channel, then cached in a voice memory, and then forwarded over a message channel as an e-mail and over a data channel as a text message.

Another example shows that the conversion of one information channel into another brings a high benefit to the user. For example, a request via a cell phone for a theater program can be answered by playing a video sequence and followed up with the possibility of ordering tickets directly.

Such a mix of communication channels can be created customer-specific and improves the use of communication services. The customer can decide which mix best supports his way of working. An essential aspect of the use of multimodal applications is speech recognition, which provides for the conversion of speech information to another message channel.

The W3C consortium has taken up the issue of multimodal applications and, with the description language "Extensible Multimodal AnnotationMarkup Language", provides the basis for linking a wide variety of content multimodally.