Description

Returns an element of a string as a "token".

Tokenizing a string is the process of breaking apart a string into components that are meaningful to you. For example, if whole words are the components we are concerned with in a sentence, then we would expect white-space characters to separate, or delimit, the words of the sentence. Thus, in this scenario, the sentence "Here is some text" would be tokenized into "Here", "is", "some", and "text".

To tokenize a string, first call strTok with the strToTokenize argument, which "seeds" the tokenizer for subsequent calls to strTok. It also returns the first token, if one is found.

Then call strTok repeatedly to get the elements of the string, until an empty string is returned.

The default delimiters are the white-space characters (i.e. space, tab, and new line characters), but any delimiters may be used.

To match the full delimiter string (which will almost never be done with the default delimiters), specify the -matchAll flag.

This command is similar to the strTokArray command, except that strTokArray tokenizes the string in one call, returning an array of all the elements. Use strTokArray if you won't change your delimiter in the middle of tokenizing.

Functional area

String

Command syntax

Syntax

strTok "string"[-delims string] [-matchAll]

Arguments

Name	Type	Required	Comments
`strToTokenize`	string	no	The string to be tokenized. Only needs to be provided on the first call to strTok. Subsequent calls operate on this string, until this argument is provided again.

Flags

Name	Flag arguments	Argument type	Exclusive to	Comments
`delims`	1	string	—	Overrides the default delimiters with user provided delimiters. The default delimiters are space, tab, and new line (any white space).
`matchAll`	0	—	—	Specifies that all the delimiter characters must be matched. The default is to match any of the delimiter characters.

Return value

string

Returns the "token" found with "string", or an empty string if the end of the string is reached.

Examples

// Demonstrate string tokenization.
string $str = "This is some text";
string $filepath = "c:/Program Files/ShogunPost#.#/SampleFiles/Day1/Test.hdf";
string $token;
 
// Tokenize the first string. Start off by "seeding" the strTok command with str.
// It will return the first token if one was found.
$token = `strTok $str`;
 
// Keep tokenizing until we have no more tokens
while( `strLength $token` != 0 )
{ 
     // Print out our token 
     print $token; 
     // Get the next token. Don't pass in $str, as we only need to seed it
     // in the beginning 
     $token = `strTok`;
}
// Now tokenize the file path. Tokenize it by folders, so use / as a delimiter
$token = `strTok $filepath -delims "/"`;
while( `strLength $token` != 0 )
{ 
     // Print out our token 
     print $token; 
     // Get the next token. don't pass in $filepath, as we only need to seed
     // it in the beginning 
     $token = `strTok -delims "/"`;
}

Shogun 1.11 Documentation

strTok