Tokenizer doesn't handle 'S very well when there are spaces in front

[dominickng]

These are sourced from tokenising Clueweb 09 and attempting to apply FACC annotations. Each one has the pattern <space> <apostrophe> <S>.

: 'Stille Nacht' -> : 'S tille Nacht
track 'Straight to Hell -> track 'S traight to Hell
, 'SB' for a Snowboardtour, 'SK' -> , 'S B ' for a Snowboardtour , 'S K '

[jnothman]

Yes, this is among issues that should be ported from redmine

On 30 April 2014 15:37, Dominick Ng [email protected] wrote:

These are sourced from tokenising Clueweb 09 and attempting to apply
FACC annotations. Each one has the pattern .

: 'Stille Nacht' -> : 'S tille Nacht
track 'Straight to Hell -> track 'S traight to Hell
, 'SB' for a Snowboardtour, 'SK' -> , 'S B ' for a Snowboardtour , 'S K '

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/8
.

[timdawborn]

Related to, but not identical to #23.