Added_token (u.95651758f926ec81d627efae6ee3c604.saga.1.0.0~alpha1.doc.saga.tokenizers.Saga_tokenizers.Added

Sourcetype t

Added token with customization, matching HF Added_token.

val create : 
  ?content:string ->
  ?single_word:bool ->
  ?lstrip:bool ->
  ?rstrip:bool ->
  ?normalized:bool ->
  ?special:bool ->
  unit ->
  t

Create added token with options and defaults.

Sourceval content : t -> string

Sourceval lstrip : t -> bool

Sourceval normalized : t -> bool

Sourceval rstrip : t -> bool

Sourceval single_word : t -> bool