Description

Returns elements of a string as "tokens".

Tokenizing a string is the process of breaking apart a string into components that are meaningful to you. For example, if whole words are the components we are concerned with in a sentence, then we would expect white-space characters to separate, or delimit, the words of the sentence. Thus, in this scenario, the sentence "Here is some text" would be tokenized into "Here", "is", "some", and "text". To tokenize a string, call strTokArray with the string to be tokenized.

The default delimiters are the white-space characters (i.e. space, tab, and new line characters), but any delimiters may be used.

To match the full delimiter string (which will almost never be done with the default delimiters), specify the -matchAll flag.

This command is similar to the strTok command, except that strTok requires multiple calls to tokenize the string, returning one token of the string with each call. Use strTok if you need to change your delimiter in the middle of tokenizing.

Functional area

String

Command syntax

Syntax

strTokArray "string" [-delims string] [-matchAll]

Arguments

Name	Type	Required	Comments
`strToTokenize`	string	yes	The string to be tokenized.

Flags

Name	Flag arguments	Argument type	Exclusive to	Comments
`delims`	1	string	—	Overrides the default delimiters with user provided delimiters. The default delimiters are space, tab, and new line (any white space).
`matchAll`	0	—	—	Specifies that all the delimiter characters must be matched. The default is to match any of the delimiter characters.

Return value

string array

Returns all of the "tokens" found within "string".

Examples

// Demonstrate string tokenization.
string $str = "This is some text";
string $filepath = "C:/Program Files/ShogunPost#.#/SampleFiles/Day1/Test.hdf";
string $tokens[];
 
// Tokenize the first string.
$tokens = `strTokArray $str`;
print $tokens;
 
// Now tokenize the file path. Tokenize it by
// folders, so use / as a delimiter
$tokens = `strTokArray $filepath -delims "/"`;
print $tokens;

Shogun 1.10 Documentation

strTokArray