Definitions
from Wiktionary, Creative Commons Attribution/Share-Alike License.
- noun computing A system that parses an input stream into tokens.
Etymologies
Sorry, no etymologies found.
Support
Help support Wordnik (and make this page ad-free) by adopting the word tokenizer.
Examples
-
The word_tokenize function uses NLTK’s currently recommended word tokenizer, TreebankWordTokenizer, and should be fed one sentence at a time.
-
Reason column shows that the first request was from the original html lookahead tokenizer while the subsequent successful downloads were triggered by the restarted html lookahead tokenizer:
Site Home EricLaw [MSFT] 2011
-
It tries to be as compliant as possible but uses some (helpful) parts of the CSS 2.1 tokenizer.
Softpedia - Windows - All Softpedia Linux 2010
-
· Or just use the included message, string, dataIO, hashtable, dequeue, string-tokenizer, reference-count, regular expression and pattern matching parser, and object-pool code by themselves, as handy cross-platform utility classes.
-
A tokenizer for Hailo which splits on whitespace, mostly.
search.cpan.org 2010
-
The cssutils tokenizer is a customized implementation of CSS3 Module: Syntax (W3C Working Draft 13 August 2003) which itself is based on the CSS 2.1 tokenizer.
Softpedia - Windows - All Softpedia Linux 2010
-
The usage file parser and the input line tokenizer are built on the excellent
Planet Debian Matt Brubeck 2010
-
SQLite standard tokenizer xml xmlreader xmlwriter zlib
-
Jonathan Howard was probably operating under too much stress when he created Like, Python using Python's tokenizer to add keywords to Python's lexical understanding.
fullasagoog.com full roast blend DigitalMediaMinute 2010
-
Implement a tokenizer (Difficulty: medium/high, priority: low)
Comments
Log in or sign up to get involved in the conversation. It's quick and easy.