Handwriting recognition
Encyclopedia
Handwriting recognition is the ability of a computer to receive and interpret intelligible handwritten input from sources such as paper
Paper
Paper is a thin material mainly used for writing upon, printing upon, drawing or for packaging. It is produced by pressing together moist fibers, typically cellulose pulp derived from wood, rags or grasses, and drying them into flexible sheets....

 documents, photograph
Photograph
A photograph is an image created by light falling on a light-sensitive surface, usually photographic film or an electronic imager such as a CCD or a CMOS chip. Most photographs are created using a camera, which uses a lens to focus the scene's visible wavelengths of light into a reproduction of...

s, touch-screens and other devices. The image of the written text may be sensed "off line" from a piece of paper by optical scanning (optical character recognition
Optical character recognition
Optical character recognition, usually abbreviated to OCR, is the mechanical or electronic translation of scanned images of handwritten, typewritten or printed text into machine-encoded text. It is widely used to convert books and documents into electronic files, to computerize a record-keeping...

) or intelligent word recognition
Intelligent word recognition
Intelligent Word Recognition, or IWR, is the recognition of unconstrained handwritten words. IWR recognizes entire handwritten words or phrases instead of character-by-character, like its predecessor, Optical Character Recognition...

. Alternatively, the movements of the pen tip may be sensed "on line", for example by a pen-based computer screen surface.

Handwriting recognition principally entails optical character recognition
Optical character recognition
Optical character recognition, usually abbreviated to OCR, is the mechanical or electronic translation of scanned images of handwritten, typewritten or printed text into machine-encoded text. It is widely used to convert books and documents into electronic files, to computerize a record-keeping...

. However, a complete handwriting recognition system also handles formatting, performs correct segmentation
Segment (handwriting)
A segment of handwriting is a piece of the pen-tip trajectory between two defined segmentation points. If the occurrence of a minimum in the absolute velocity is used as a heuristic for segmentation, the pen-tip trajectory can be subdivided into segments corresponding to ballistic strokes.In...

 into characters and finds the most plausible words.

Off-line recognition

Off-line handwriting recognition involves the automatic conversion of text in an image into letter codes which are usable within computer and text-processing applications. The data obtained by this form is regarded as a static representation of handwriting. Off-line handwriting recognition is comparatively difficult, as different people have different handwriting styles. And, as of today, OCR engines are primarily focused on machine printed text and ICR for hand "printed" (written in capital letters) text. There is no OCR/ICR engine that supports handwriting recognition as of today.

Problem domain reduction techniques

Narrowing the problem domain often helps increase the accuracy of handwriting recognition systems. A form field for a ZIP code for example, would contain only the characters 0-9. This fact would reduce the number of possible identifications.

Primary techniques:
  • Specifying specific character ranges
  • Utilization of specialized forms

Character extraction

Off-line character recognition often involves scanning a form or document written sometime in the past. This means the individual characters contained in the scanned image will need to be extracted. Tools exist that are capable of performing this step however, several common imperfections in this step. The most common being characters that are connected together are returned as a single sub-image containing both characters. This causes a major problem in the recognition stage. Yet many algorithms are available that reduce the risk of connected characters.

Character recognition

After the extraction of individual characters occurs a recognition engine is used to identify the corresponding computer character. Several different recognition techniques are currently available.

Neural networks

Neural network recognizers learn from an initial image training set. The trained network then makes the character identifications. Each neural network uniquely learns the properties that differentiate training images. It then looks for similar properties in the target image to be identified. Neural networks are quick to set up; however, they can be inaccurate if they learn properties that are not important in the target data.

Feature extraction

Feature extraction works in a similar fashion to neural network recognizers however, programmers must manually determine the properties they feel are important.

Some example properties might be:
  • Aspect Ratio
  • Percent of pixels above horizontal half point
  • Percent of pixels to right of vertical half point
  • Number of strokes
  • Average distance from image center
  • Is reflected y axis
  • Is reflected x axis


This approach gives the recognizer more control over the properties used in identification. Yet any system using this approach requires substantially more development time than a neural network because the properties are not learned automatically.

On-line recognition

On-line handwriting recognition involves the automatic conversion of text as it is written on a special digitizer or PDA
Personal digital assistant
A personal digital assistant , also known as a palmtop computer, or personal data assistant, is a mobile device that functions as a personal information manager. Current PDAs often have the ability to connect to the Internet...

, where a sensor picks up the pen-tip movements as well as pen-up/pen-down switching. That kind of data is known as digital ink and can be regarded as a dynamic representation of handwriting. The obtained signal is converted into letter codes which are usable within computer and text-processing applications.

The elements of an on-line handwriting recognition interface typically include:
  • a pen or stylus for the user to write with.
  • a touch sensitive surface, which may be integrated with, or adjacent to, an output display.
  • a software application which interprets the movements of the stylus across the writing surface, translating the resulting strokes into digital text.

Hardware

Commercial products incorporating handwriting recognition as a replacement for keyboard input were introduced in the early 1980s. Examples include handwriting terminals such as the Pencept Penpad
and the Inforite point-of-sale terminal.
With the advent of the large consumer market for personal computers, several commercial products were introduced to replace the keyboard and mouse on a personal computer with a single pointing/handwriting system, such as those from PenCept, CIC and others.
The first commercially available tablet-type portable computer was the GRiDPad from GRiD Systems, released in September 1989. Its operating system was based on MS-DOS
MS-DOS
MS-DOS is an operating system for x86-based personal computers. It was the most commonly used member of the DOS family of operating systems, and was the main operating system for IBM PC compatible personal computers during the 1980s to the mid 1990s, until it was gradually superseded by operating...

.

In the early 1990s, hardware makers including NCR
NCR Corporation
NCR Corporation is an American technology company specializing in kiosk products for the retail, financial, travel, healthcare, food service, entertainment, gaming and public sector industries. Its main products are self-service kiosks, point-of-sale terminals, automated teller machines, check...

, IBM
IBM
International Business Machines Corporation or IBM is an American multinational technology and consulting corporation headquartered in Armonk, New York, United States. IBM manufactures and sells computer hardware and software, and it offers infrastructure, hosting and consulting services in areas...

 and EO
EO Personal Communicator
The EO was an early commercial tablet computer created by GO/Eo and released in April 1993. Eo was the hardware spin-out of GO. Officially named the AT&T EO Personal Communicator, it was similar to a large personal digital assistant with wireless communications, and competed against the Apple...

 released tablet computer
Tablet computer
A tablet computer, or simply tablet, is a complete mobile computer, larger than a mobile phone or personal digital assistant, integrated into a flat touch screen and primarily operated by touching the screen...

s running the PenPoint
PenPoint OS
The PenPoint OS was a product of GO Corporation and was one of the earliest operating systems written specifically for graphical tablets and personal digital assistants...

 operating system developed by GO Corp.
GO Corp.
GO Corporation was founded in 1987 to create portable computers, an operating system, and software with a pen-based user interface. It was famous not only for its pioneering work in Pen-based computing but as well as being one of the most well-funded start-up companies of its time.Though the...

. PenPoint used handwriting recognition and gestures throughout and provided the facilities to third-party software. IBM's tablet computer was the first to use the ThinkPad
ThinkPad
ThinkPad is line of laptop computers originally sold by IBM but now produced by Lenovo. They are known for their boxy black design, which was modeled after a traditional Japanese lunchbox...

 name and used IBM's handwriting recognition. This recognition system was later ported to Microsoft Windows for Pen Computing
Windows for Pen Computing
Windows for Pen Computing was a software suite for Windows 3.1x, that Microsoft designed to incorporate pen computing capabilities into the Windows operating environment...

, and IBM's Pen for OS/2
OS/2
OS/2 is a computer operating system, initially created by Microsoft and IBM, then later developed by IBM exclusively. The name stands for "Operating System/2," because it was introduced as part of the same generation change release as IBM's "Personal System/2 " line of second-generation personal...

. None of these were commercially successful.

Advancements in electronics allowed the computing power necessary for handwriting recognition to fit into a smaller form factor than tablet computers, and handwriting recognition is often used as an input method for hand-held PDA
Personal digital assistant
A personal digital assistant , also known as a palmtop computer, or personal data assistant, is a mobile device that functions as a personal information manager. Current PDAs often have the ability to connect to the Internet...

s. The first PDA to provide written input was the Apple Newton
Apple Newton
The MessagePad was the first series of personal digital assistant devices developed by Apple for the Newton platform in 1993. Some electronic engineering and the manufacture of Apple's MessagePad devices was done in Japan by the Sharp Corporation...

, which exposed the public to the advantage of a streamlined user interface. However, the device was not a commercial success, owing to the unreliability of the software, which tried to learn a user's writing patterns. By the time of the release of the Newton OS
Newton OS
Newton OS was the operating system for the Apple Newton PDAs produced by Apple from 1993-1997. Newton OS was written entirely in C++ and trimmed to be low power consuming and use the available memory efficiently...

 2.0, wherein the handwriting recognition was greatly improved, including unique features still not found in current recognition systems such as modeless error correction, the largely negative first impression had been made. After discontinuation of Apple Newton
Apple Newton
The MessagePad was the first series of personal digital assistant devices developed by Apple for the Newton platform in 1993. Some electronic engineering and the manufacture of Apple's MessagePad devices was done in Japan by the Sharp Corporation...

, the feature has been ported to Mac OS X 10.2 or later in form of Inkwell (Macintosh)
Inkwell (Macintosh)
Inkwell, or simply Ink, is the name of the handwriting recognition technology developed by Apple Inc. and built into the Mac OS X operating system. Introduced in an update to Mac OS X v10.2 "Jaguar", Inkwell can translate English, French, and German writing...

.

Palm
Palm, Inc.
Palm, Inc., was a smartphone manufacturer headquartered in Sunnyvale, California, that was responsible for products such as the Pre and Pixi as well as the Treo and Centro smartphones. Previous product lines include the PalmPilot, Palm III, Palm V, Palm VII, Zire and Tungsten. While their older...

 later launched a successful series of PDA
Personal digital assistant
A personal digital assistant , also known as a palmtop computer, or personal data assistant, is a mobile device that functions as a personal information manager. Current PDAs often have the ability to connect to the Internet...

s based on the Graffiti
Graffiti (Palm OS)
Graffiti is an essentially single-stroke shorthand handwriting recognition system used in PDAs based on the Palm OS. Graffiti was originally written by Palm, Inc...

 recognition system. Graffiti improved usability by defining a set of "unistrokes", or one-stroke forms, for each character. This narrowed the possibility for erroneous input, although memorization of the stroke patterns did increase the learning curve for the user. The Graffiti handwriting recognition was found to infringe on a patent held by Xerox, and Palm replaced Graffiti with a licensed version of the CIC handwriting recognition which, while also supporting unistroke forms, pre-dated the Xerox patent. The court finding of infringement was reversed on appeal, and then reversed again on a later appeal. The parties involved subsequently negotiated a settlement concerning this and other patents Graffiti (Palm OS)
Graffiti (Palm OS)
Graffiti is an essentially single-stroke shorthand handwriting recognition system used in PDAs based on the Palm OS. Graffiti was originally written by Palm, Inc...

.

A Tablet PC
Tablet computer
A tablet computer, or simply tablet, is a complete mobile computer, larger than a mobile phone or personal digital assistant, integrated into a flat touch screen and primarily operated by touching the screen...

 is a special notebook computer that is outfitted with a digitizer tablet
Graphics tablet
A graphics tablet is a computer input device that enables a user to hand-draw images and graphics, similar to the way a person draws images with a pencil and paper. These tablets may also be used to capture data or handwritten signatures...

 and a stylus, and allows a user to handwrite text on the unit's screen. The operating system recognizes the handwriting and converts it into typewritten text. Windows Vista
Windows Vista
Windows Vista is an operating system released in several variations developed by Microsoft for use on personal computers, including home and business desktops, laptops, tablet PCs, and media center PCs...

 and Windows 7 include personalization features that learn a user's writing patterns and/or vocabulary for English, Japanese, Chinese Traditional, Chinese Simplified and Korean. The features include a "personalization wizard" that prompts for samples of a user's handwriting and uses them to retrain the system for higher accuracy recognition. This system is distinct from the less advanced handwriting recognition system employed in its Windows Mobile
Windows Mobile
Windows Mobile is a mobile operating system developed by Microsoft that was used in smartphones and Pocket PCs, but by 2011 was rarely supplied on new phones. The last version is "Windows Mobile 6.5.5"; it is superseded by Windows Phone, which does not run Windows Mobile software.Windows Mobile is...

 OS for PDAs.

In recent years, several attempts were made to produce ink pens that include digital elements, such that a person could write on paper
Paper
Paper is a thin material mainly used for writing upon, printing upon, drawing or for packaging. It is produced by pressing together moist fibers, typically cellulose pulp derived from wood, rags or grasses, and drying them into flexible sheets....

, and have the resulting text stored digitally. The best known of these use technology developed by Anoto
Anoto
Anoto Group AB is a Swedish technology company which has invented, developed and licensed a product line of digital pens and a product suite to manage information capturing utilizing these . The Anoto Group has around 110 employees...

, which has had some success in the education market. The general success of these products is yet to be determined.

Although handwriting recognition is an input form that the public has become accustomed to, it has not achieved widespread use in either desktop computers or laptops. It is still generally accepted that keyboard input is both faster and more reliable. , many PDAs offer handwriting input, sometimes even accepting natural cursive handwriting, but accuracy is still a problem, and some people still find even a simple on-screen keyboard
Virtual keyboard
A virtual keyboard is a software component that allows a user to enter characters. A virtual keyboard can usually be operated with multiple input devices, which may include a touchscreen, an actual keyboard and a computer mouse.- Types :...

 more efficient.

Software

Initial software modules could understand print handwriting where the characters were separated. Commercial examples came from companies such as Communications Intelligence Corporation and IBM. In the early 90s, two companies, ParaGraph International, and Lexicus came up with systems that could understand cursive handwriting recognition. ParaGraph was based in Russia and founded by computer scientist Stepan Pachikov while Lexicus was founded by Ronjon Nag and Chris Kortge who were students at Stanford University. The ParaGraph CalliGrapher system was deployed in the Apple Newton systems, and Lexicus Longhand system was made available commercially for the PenPoint and Windows operating system. Lexicus was acquired by Motorola in 1993 and went on to develop Chinese handwriting recognition and predictive text
Predictive text
Predictive text is an input technology used where one key or button represents many letters, such as on mobile phones and in accessibility technologies. Each key press results in a prediction rather than repeatedly sequencing through the same group of "letters" it represents, in the same,...

 systems for Motorola. ParaGraph was acquired in 1997 by SGI and its handwriting recognition team formed a P&I division, later acquired from SGI by Vadem. Microsoft has acquired CalliGrapher handwriting recognition and other digital ink technologies developed by P&I from Vadem in 1999.

A modern handwriting recognition system can be seen in Microsoft
Microsoft
Microsoft Corporation is an American public multinational corporation headquartered in Redmond, Washington, USA that develops, manufactures, licenses, and supports a wide range of products and services predominantly related to computing through its various product divisions...

's operating system running on Tablet PC
Tablet computer
A tablet computer, or simply tablet, is a complete mobile computer, larger than a mobile phone or personal digital assistant, integrated into a flat touch screen and primarily operated by touching the screen...

s (notably Windows XP Tablet PC Edition, Windows Vista
Windows Vista
Windows Vista is an operating system released in several variations developed by Microsoft for use on personal computers, including home and business desktops, laptops, tablet PCs, and media center PCs...

, and Windows 7). It is based on a Time delay neural network
Time delay neural network
Time delay neural network is an alternative neural network architecture whose primary purpose is to work on continuous data. The advantage of this architecture is to adapt the network online and hence helpful in many real time applications, like time series prediction, online spell check,...

 (TDNN) classifier, nicknamed "Inferno", built at Microsoft. Later on a version of CalliGrapher, a successor of the handwriting recognition software used on Newton OS
Newton OS
Newton OS was the operating system for the Apple Newton PDAs produced by Apple from 1993-1997. Newton OS was written entirely in C++ and trimmed to be low power consuming and use the available memory efficiently...

 2.0, was integrated as a secondary recognizer with the TDNN. The new generation of CalliGrapher software is currently shipped for Windows Mobile by PhatWare Corp, which licensed ParaGraph's technologies from Vadem in 2001.

The "third generation" riteScript handwriting recognition technology, built by EverNote Corporation (the successor of Pen&Internet division of Parascript) in 2000-2008, is included in the ritePen and EverNote
EverNote
Evernote is a suite of software and services designed for notetaking and archiving. A "note" can be a piece of formatted text, a full webpage or webpage excerpt, a photograph, a voice memo, or a handwritten "ink" note. Notes can also have file attachments...

 software. ritePen also includes fusion technology, which allows combining riteScript with the embedded handwriting recognition in Windows Vista and Windows 7 to improve recognition accuracy of each handwriting recognition engine. riteScript technology is available on Windows and Linux.

CellWriter is an Open Source handwriting recognition program written for Linux
Linux
Linux is a Unix-like computer operating system assembled under the model of free and open source software development and distribution. The defining component of any Linux system is the Linux kernel, an operating system kernel first released October 5, 1991 by Linus Torvalds...

. It was developed primarily by Michael Levin, working under the University of Minnesota
University of Minnesota
The University of Minnesota, Twin Cities is a public research university located in Minneapolis and St. Paul, Minnesota, United States. It is the oldest and largest part of the University of Minnesota system and has the fourth-largest main campus student body in the United States, with 52,557...

's Undergraduate Research Opportunity Program. CellWriter is designed to be writer-dependent, so input training is required prior to use. It includes an English word-recognition engine to aid recognition accuracy, an on-screen keyboard and the ability to fine tune or correct mistakes through a context menu. Unlike most recognition engines, the source code and algorithms are available and free to download.

MyScript Builder is a handwriting recognition SDK from VisionObjects http://www.visionobjects.com/ that is popular among companies developing software for digital pens based on Anoto technology.

Research

Handwriting Recognition has an active community of academics studying it. The biggest conferences for handwriting recognition are the International Conference on Frontiers in Handwriting Recognition (ICFHR), held in even-numbered years, and the International Conference on Document Analysis and Recognition (ICDAR), held in odd-numbered years. Both of these conferences are scrutinized by the IEEE. Active areas of research include:
  • Online Recognition
  • Offline Recognition
  • Signature Verification
  • Postal-Address Interpretation
  • Bank-Check Processing
  • Writer Recognition


A survey of research on handwriting recognition is by R Plamondon and S. N. Srihari
Sargur Srihari
S. N. Srihari is an American Computer scientist and educator who has made contributions to the field of pattern recognition. The principal impact of his work has been in handwritten address reading systems and in computer forensics...

.

Brief historical notes

  • 1915: U.S. Patent on handwriting recognition user interface with a stylus
  • 1957: Stylator tablet: Tom Dimond demonstrates electronic tablet with pen for computer input and handwriting recognition
  • 1961: RAND Tablet invented: better known than earlier Stylator system
  • 1962: Computer recognition of connected/script handwriting
  • 1969: GRAIL system: handwriting recognition with electronic ink display, gesture commands
  • 1973: Applicon CAD/CAM computer system using the Ledeen recognizer for handwriting recognition
  • 1980s: Retail handwriting-recognition systems: Pencept and CIC both offer PC computers for the consumer market using a tablet and handwriting recognition instead of a keyboard and mouse. Cadre System markets Inforite point-of-sale terminal using handwriting recognition and a small electronic tablet and pen.
  • 1989: Portable handwriting recognition computer: GRiDPad from GRiD Systems.
  • 1997: First handwritten address interpretation system(HWAI) deployed by United States Postal Service
  • 2007: First automatic writer recognition system: CEDAR-FOX
    CEDAR-FOX
    - Introduction :This is a software system for forensic comparison of handwriting. It was developed at CEDAR, the Center of Excellence for Document Analysis and Recognition at the University at Buffalo...

    .

More extensive information on the history of handwriting recognition technology can be found in the article on Pen computing
Pen computing
Pen computing refers to a computer user-interface using a pen and tablet, rather than devices such as a keyboard, joysticks or a mouse....

.

See also

  • Handwriting movement analysis
    Handwriting movement analysis
    Handwriting movement analysis is the study and analysis of the movements involved in handwriting and drawing. It forms an important part of graphonomics, which became established after the "International Workshop on Handwriting Movement Analysis" in 1982 in Nijmegen, The Netherlands. It would...

  • Neocognitron
    Neocognitron
    The neocognitron is a hierarchical multilayered neural network proposed by Professor Kunihiko Fukushima. It has been used for handwritten character recognition and other pattern recognition tasks....

  • Optical character recognition
    Optical character recognition
    Optical character recognition, usually abbreviated to OCR, is the mechanical or electronic translation of scanned images of handwritten, typewritten or printed text into machine-encoded text. It is widely used to convert books and documents into electronic files, to computerize a record-keeping...

  • Pen computing
    Pen computing
    Pen computing refers to a computer user-interface using a pen and tablet, rather than devices such as a keyboard, joysticks or a mouse....

  • Tablet PC
    Tablet computer
    A tablet computer, or simply tablet, is a complete mobile computer, larger than a mobile phone or personal digital assistant, integrated into a flat touch screen and primarily operated by touching the screen...

  • Sketch recognition
    Sketch recognition
    Sketch recognition is the automated recognition of hand-drawn diagrams by a computer. Research in sketch recognition lies at the crossroads of Artificial Intelligence and Human Computer Interaction...


External links

The source of this article is wikipedia, the free encyclopedia.  The text of this article is licensed under the GFDL.
 
x
OK