Text

MyScript Cloud can recognize handwritten text and turn it into digital text.

The diagram below shows the information processed by the recognition engine to convert handwriting to text.

Dear John Segmentation Characterrecognition Languagemodeling

The recognizer will process the handwriting intelligently. It refers to lexicons, and uses a language model to understand how parts of a language come together and interact. It will not just guess one character at a time, but a whole sentence, so you can allow your application to accept complex handwriting.

Some functions are deprecated in 3.0 for HTTP/REST requests. To have your system fully updated, please change your end point prefix and replace {"/api/myscript/v2.0/hwr/\*"} with {"/api/v3.0/recognition/rest/text/\*"}.

Making a Text recognition request

The request is made to the recognition server via POST, using the specific text recognition URL. The input is digital ink (you need to have in place your own system for capturing this ink and generating its XY coordinates). You pass the XY coordinates in the POST request.

You also need to give MyScript basic information about how your digital ink is structured and if there are any specific needs for the request. For example, do you want to see more detail in the result, or do you want to use a particular set of characters?

If you make a request without specifying any of the non-mandatory parameters, the recognizer will use a broad set of recognition resources to process it. The writing language must always be specified or it will return an error.

See the Reference guide for information on the mandatory and optional parameters needed for Text recognition.

See the example of a Text recognition request and response.

Here are the parameters needed for Text recognition:

Structure Parameter Possible values Description Mandatory? Example
applicationKey   [your application key] This key is provided on registration and must be included in all HTTP/REST requests and in the first WebSocket request. YES applicationKey= your_key
instanceId   [Instance ID get by a first request] This is an instance identification if there was a previous text recognition request. NO instanceId= your_instance_ID
textInput textParameter See the content of the textParameter structure below. These are the parameters to apply to the text recognition. YES textInput= {"textParameter": {...},"inputUnits": [...]}
textInput inputUnits See the content of the inputUnits structure below. This describes an inputUnit to send to the text recognition. YES textInput= {"textParameter": {...},"inputUnits": [...]}
inputUnits textInputType CHAR, WORD, SINGLE_LINE_TEXT, MULTI_LINE_TEXT This describes the type of input, if it is a single character, a single line of text or many lines of text. NO textInput= {"textParameter": {...},"inputUnits": {"textInputType": "CHAR","components": [...]}}
inputUnits components stroke, string, char Under components, specify a type. Either stroke followed by an XY array of ink points, string followed by a string of characters or char followed by a single character. These types allow you to add predefined digital text to the result. That input will pass through the recognizer and be integrated into the final result. YES "inputUnits": {"components": [{"type": "stroke", "x": [], "y": []}, {"type": "string", "string": "Today's date is "}, {"type": "char", "boundingBox": {"y": 50, "x": 50, "height": 100, "width": 100}, "character":"A"}]}or if there are several characters: "inputUnits": {"components": [{"type": "char", "boundingBox": {"y": 50, "x": 50, "height": 100, "width": 100}, "character":"A"}, {"type": "char", "boundingBox": {"y": 50, "x": 454, "height": 100, "width": 100}, "character":"B"}]}
textParameter language See the GET request to use to obtain the list This is the language in which the text is written. YES "textParameter": {"language": "en_US"}
textParameter textInputMode ISOLATED, CURSIVE, VERTICAL*, SUPERIMPOSED** This describes the type of writing - isolated (for example in boxed fields) and cursive (any other type of natural writing). See the code samples. *The recognition of characters handwritten vertically is only possible in Japanese. **Only possible in Latin and Cyrillic languages. NO "textParameter": {"textInputMode": "CURSIVE"}
textParameter contentTypes See the GET request to use to obtain the list The content type describes the expected linguistic type of input. The default is text which means that the recognizer will use a broad lexicon of general text. You can specify others for different types of fields such as surnames, city names, numbers, etc. You can specify more than one. NO "textParameter": {"contentTypes": ["text"]}
textParameter subset Knowledges See the GET request to use to obtain the list Subset knowledge is a way of telling the recognizer that you only want it to enable only certain characters. For example, for a phone number field, you may want only digits to be recognized. They are language-specific but generally offer digits, punctuation, alphabetical and alphanumerical characters and upper/lower case letters. NO "textParameter": {"subsetKnowledges": ["digits"]}
textParameter userResources See the list in: MyScript Cloud service dashboard > My Account tab > Custom Resources panel A user resource tells the recognizer that you want to use the given resources for recognition of the current request. NO "textParameter": {"userResources": ["user_resource_name"]}
textParameter userLkWords A list of string words A userLkWords list tells the recognizer that you want to use only the given words for recognition of the current request. NO "textParameter": {"userLkWords": ["any word"]}
textParameter resultDetail TEXT, WORD, CHARACTER This describes the level of result detail you want. If you ask for TEXT level, you only get the final result for the whole ink sample. Results can also be broken down into WORD or CHARACTER level, if you want to offer alternative recognition candidates. NO "textParameter": {"resultDetail": "TEXT"}
textParameter textProperties See the content of the textProperties structure below. This describes properties to send to the text recognition. NO "textParameter": {"textProperties": {...}}
textProperties textCandidate ListSize A number between 1 and 20 (1=default). Sets the size of the candidate lists given at text level in the recognition result. For this to be valid, ask for the corresponding level of ResultDetail. NO "textProperties": {"textCandidate ListSize": 4}
textProperties wordCandidate ListSize A number between 1 and 20 (1=default). Sets the size of the candidate lists given at word level in the recognition result. For this to be valid, ask for the corresponding level of ResultDetail. NO "textProperties": {"wordCandidate ListSize": 3}
textProperties character CandidateList Size A number between 1 and 20 (1=default). Sets the size of the candidate lists given at character level in the recognition result. For this to be valid, ask for the corresponding level of ResultDetail. NO "textProperties": {"character CandidateListSize": 3}
textProperties wordPrediction ListSize A number between 1 and 20 (1=default). Sets the size of the predicted words lists given at word level in the recognition result. NO "textProperties": {"wordPrediction ListSize": 3}
textProperties wordCompletion ListSize A number between 1 and 20 (1=default). Sets the size of the completed words lists given at word level in the recognition result. NO "textProperties": {"wordCompletion ListSize": 3}

A previous and deprecated version of this table is available here.

GET requests

The list of languages, subsets and content types can be fetched with a GET request. The subsets and content types that you can use are specific to each language. They respectively restrict the characters that are used by the recognizer or choose a specific lexicon, such as a list of nationalities.

Even if you are using the WebSocket protocol, lists of resources, languages, etc. can only be obtained via the HTTP/REST protocol.

To fetch the list of languages currently available:

https://cloud.myscript.com/api/v3.0/recognition/rest/text/languages.json?applicationKey=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX

To fetch the content types for a specific language:

https://cloud.myscript.com/api/v3.0/recognition/rest/text/language/[language_country_code]/contenttypes.json?applicationKey=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX

To fetch the character subsets for a specific language:

https://cloud.myscript.com/api/v3.0/recognition/rest/text/language/[language_country_code]/subsetknowledges.json?applicationKey=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX

Shape

MyScript Shape can recognize a fixed set of hand drawn shapes and lines.

Shape recognizer(neural network) Shape resources

A shape is related to one or more primitives, which is the basic geometrical unit of a shape (a line segment, an arc, etc.). Each type of shape recognized can be described by one or more of these basic primitives. For example, a rectangle is made up of 4 line segments, a circle is a type of arc, etc.

The recognizer will send you a result describing the type of shape, the primitives that constitute it and information on the shape’s orientation, size etc., so that it can be reconstituted geometrically. The resulting shape will be “beautified” to give a visually attractive geometrical shape as close as possible to the original. For example, roughly parallel lines will be made perfectly parallel, shapes that are almost touching will be connected and so on.

Some functions are deprecated in 3.0. To have your system fully updated, please change your end point prefix and replace {"/api/myscript/v2.0/shape/\*"} with {"/api/v3.0/recognition/rest/shape/\*"}

How does Shape recognition work?

The Shape recognizer uses resources describing a set of recognizable shapes, and components defining each type of shape. Then, it sends back components in a beautified form (if needed, see doBeautification), describing the shape’s type, which curves and line segments it uses, as well as their size and position. That way, you can reconstitute the shape as a vector object.

Making a Shape recognition request

In the POST request, you need to send only the strokes (XY coordinates of the digital ink). No need to give additional information to the Shape recognizer, as it assumes that all input is some kind of geometrical shape or line. The result will return a shape type and various elements that allow the shape to be reconstituted (constituting lines, size, orientation, etc.). Note that the form recognition is done iteratively.

See the Reference guide for information on the mandatory and optional parameters needed for Shape recognition.

See the example of a Shape recognition request and response.

Here are the parameters needed for Shape recognition:

Structure Parameter Possible values Description Mandatory? Example
applicationKey   [your application key] This key is provided on registration and must be included in all HTTP/REST requests and in the first WebSocket request. YES applicationKey=your_key
instanceId   [Instance ID get by a first request] This is an instance identification if there was a previous shape recognition request. NO instanceId=your_instance_ID
shapeInput components stroke Under components, specify a type, i.e. for Shape: stroke followed by an XY array of ink points. YES shapeInput={"components" : [{"type" : "stroke", "x" : [...], "y" : [...] , }
shapeInput userResources See the list in: MyScript Cloud service dashboard > My Account tab > Custom Resources panel A user resource tells the recognizer that you want to use the given resources for recognition of the current request. NO shapeInput={"userResources" : ["user_resource_name"]}

A previous and deprecated version of this table is available here.

Math

MyScript Math recognizes all kinds of mathematical expressions. It works as a neural network, using resources to identify various subsets of elements that make up typical math expressions. Eventually, it sends a LaTeX or MathML result.

The scratch-out gesture is supported, meaning that the end user can erase a character, a shape, a symbol, etc. by scratching it out. To see a working example of MyScript Cloud, visit our WebDemo site and try out our handwriting recognition in a variety of situations.
-b±√b²-4ac 2a ________ Math recognizer(neural network) Math resources(symbols, rules, etc.)

How does Math recognition work?

The Math recognizer uses resources to know how a math expression can be broken into various parts, laid out as fractions, using square roots, brackets, etc. and the various symbols and characters that can be found in each of those parts.

A list of limitations for Math recognition can be found in the appendix.

Some functions are deprecated in 3.0 for HTTP/REST requests. To have your system fully updated, please change your end point prefix and replace {"/api/myscript/v2.0/equation/\*"} with {"/api/v3.0/recognition/rest/math/\*"}.

Making a Math recognition request

A Math recognition request is straightforward, as you only need to send the strokes (XY coordinates of digital ink) and specify whether you want the result in LaTeX or MathML format (or both), or in a symbol tree ("resultTypes" :["LATEX","MATHML", "SYMBOLTREE"]).

See the Reference guide for information on the mandatory and optional parameters needed for Math recognition.

See the example of a Math recognition request and response.

Here are the parameters needed for Math recognition:

Structure Parameter Possible values Description Mandatory? Example
applicationKey   [your application key] This key is provided on registration and must be included in all HTTP/REST requests and in the first WebSocket request. YES applicationKey=your_key
instanceId   [Instance ID get by a first request] This is an instance identification if there was a previous math recognition request. NO instanceId=your_instance_ID
instanceId resultTypes LATEX MATHML SYMBOLTREE Describes the format(s) required for the recognition result. LATEX is default. NO mathInput={"resultTypes":["LATEX","MATHML","SYMBOLTREE"], "components":[{"type":"stroke", "x":[...], "y":[...]}]}
mathInput components stroke Under components, specify a type, i.e. for Math: stroke followed by an XY array of ink points. YES mathInput={"components":[{"type":"stroke", "x":[...], "y":[...]}]}
mathInput userResources See the list in: MyScript Cloud service dashboard > My Account tab > Custom Resources panel A user resource tells the recognizer that you want to use the given resources for recognition of the current request. NO mathInput={"userResources" : ["user_resource_name"]}

A previous and deprecated version of this table is available here.

Music

MyScript Music recognition engine analyzes the spatial relationship between the various parts of the music document and the recognizer sends you a result in MusicXML and ScoreTREE strings.

Music recognizer(neural network) Music resources

How does Music recognition work?

MyScript Music requires two resource files:

  • Music Alphabet Knowledge
    This is a special resource used for recognition of music elements, providing “music alphabet knowledge” to the recognizer so it is able to recognize the various elements that could possibly make up a music score.

  • Music Grammar Knowledge
    The way the score elements can be recognized together is specified by this grammar resource that is attached to the recognizer.

Resources are to be attached to the recognizer, in order to tell it how to perform the segmentation of all the input ink parts.

While digital ink is captured in real time, it is simultaneously recognized in the background. The Music recognizer will first analyze the spatial relationship between parts of the score. It will then use a symbol classifier that calculates the probabilities for all the elements in the suggested segmentation. Finally it will output a result.

Some functions are deprecated in 3.0. To have your system fully updated, please change your end point prefix and replace {"/api/myscript/v2.0/music/\*"} with {"/api/v3.0/recognition/rest/music/\*}.

See the Reference guide for information on the mandatory and optional parameters needed for Music recognition.

See the example of a Music recognition request and response.

Here are the parameters needed for Music recognition:

Structure Parameter Possible values Description Mandatory? Example
applicationKey   [your application key] This key is provided on registration and must be included in all HTTP/REST requests and in the first WebSocket request. YES applicationKey=your_key
instanceId   [Instance ID get by a first request] This is an instance identification if there was a previous music recognition request. NO instanceId=your_instance_ID
musicInput resultTypes MUSICXML SCORETREE Describes the format(s) required for the recognition result. NO musicInput={"resultTypes":["MUSICXML","SCORETREE"], "components":[{"type":"stroke", "x":[...], "y":[...]}]}
musicInput components stroke Under components, specify a type, i.e. for Music: stroke followed by an XY array of ink points. YES musicInput={"components":[{"type":"stroke", "x":[...], "y":[...]}]}
musicInput divisions   A division refers to a type of ornamentation or variation in which each note of a melodic line is “divided” into several shorter, faster-moving notes. YES musicInput={"divisions": 480}
musicInput staff   A staff is a set of five horizontal lines and four spaces that each represent a different musical pitch. YES musicInput={'staff': {'count': staff.count,'top': staff.top,'gap': staff.gap}}
musicInput userResources See the list in: MyScript Cloud service dashboard > My Account tab > Custom Resources panel A user resource tells the recognizer that you want to use the given resources for recognition of the current request. YES musicInput={"userResources" : ["user_resource_name"]}

A previous and deprecated version of this table is available here.

Recognition result

The result is returned in the requested format, ready for rendering, for example:

{
"instanceId":"38dada20-0500-44b9-9c6a-d53a7b3f8c00",
		"result":{
			"results":[
			{"type":"MUSICXML", "value": "..."},
			{"type":"SCORETREE","score":{...}
			]
		}
}

Analyzer

MyScript Analyzer enables the recognition of mixed content by identifying the different elements composing a document. For now, only a mix of Text and Shape can be recognized, excluding music and mathematical expressions.

The Analyzer recognizer sends a result describing:

  • the text candidates (see Text recognition) with related information about positioning, orientation, justification, bounding box and font size.
  • the type of shapes (see Shape recognition), “beautified” to give a visually attractive geometrical shape as close as possible to the original.
  • the structure of the document (tables, groups, underlines) with the links between detected objects.
Some functions are deprecated in 3.0. To have your system fully updated, please change your end point prefix and replace {"/api/myscript/v2.0/analyzer/\*"} with {"/api/v3.0/recognition/rest/analyzer/\*"}.

How does Analyzer recognition work?

Besides its own engine, the Analyzer recognizer uses a Text recognizer and a Shape recognizer.

The Text recognizer uses resources describing a set of recognizable text lines, and the components that determine each element. The Analyzer recognizer sends back components describing the textlines, providing their size and layout. That way, you can reconstitute textline as a font objects.

The Shape recognizer uses resources describing a set of recognizable shapes, and the components that determine each type of shape. The Analyzer recognizer sends back components in a beautified form, describing the shape’s type and which curves and line segments it uses, as well as their size and position. That way, you can reconstitute the shape as a vector object.

If required, the Analyzer recognizer can use resources that describe a set of recognizable links between objects, and the components that determine each type (groups, tables). The Analyzer recognizer sends back components describing the link types. It provides their description and references, so you can reconstitute the document layout.

Making an Analyzer recognition request

In the POST request, you need to send the strokes (XY coordinates of digital ink) some parameters (language, result type expected, etc.) to give information to the Analyzer recognizer. It assumes that all input is some kind of geometrical shape line or text line. The result will return a textline type, shape type, table type and group type with elements that allow the object to be reconstituted (constituting lines, size, orientation, etc.).

See the Reference guide for information on the mandatory and optional parameters needed for Text and Shape recognition.

See the example of an Analyzer recognition request and response.

Here are the parameters needed for Analyzer recognition:

Structure Parameter Possible values Description Mandatory? Example
applicationKey   [your application key] This key is provided on registration and must be included in all HTTP/REST requests and in the first WebSocket request. YES applicationKey=your_key
instanceId   [Instance ID get by a first request] This is an instance identification if there was a previous analyzer recognition request. NO instanceId=your_instance_ID
analyzerInput components stroke Under components, specify a type, i.e. for Shape: stroke followed by an XY array of ink points. YES "analyzerInput" : {"components" : [{"type" : "stroke", "x" : [...], "y" : [...] , }
analyzerInput parameter parameters Under parameter, specify recognition parameters. For Text, this is textParameter followed by language and textProperties. YES "analyzerInput" : {"components" : [...] , "parameter":{"textParameter":{"resultDetail":"TEXT", "textProperties":{}, "language":"en_US"}, "coordinateResolution":0}}
analyzerInput userResources See the list in: MyScript Cloud service dashboard > My Account tab > Custom Resources panel A user resource tells the recognizer that you want to use the given resources for recognition of the current request. NO analyzerInput={"userResources" : ["user_resource_name"]}

A previous and deprecated version of this table is available here.

The following tree shows the Analyzer result structure: