Function tokenize

tokenize(string, wordDiff?): string[]
Tokenizes a string into an array of tokens.
Parameters
- string: string
  The string to tokenize.
- wordDiff: boolean = false
  If true, performs word-based tokenization. Default is false.
Returns string[]
An array of tokens.
Example: Usage
```
import { tokenize } from "@std/internal/diff-str";
import { assertEquals } from "@std/assert";

assertEquals(tokenize("Hello\nWorld"), ["Hello\n", "World"]);
```
- Defined in internal/diff_str.ts:57