Module Torch.Text_helperSource

Sourcetype t
Sourceval create : filename:Base.string -> t

create ~filename reads text file filename and stores its content in t. The tensors returned in this module use compact labels: e.g. if filename use n different characters, the values used in the tensors will range from 0 to n-1.

Sourceval iter : ?device:Device.t -> t -> f:(Base.int -> xs:Tensor.t -> ys:Tensor.t -> Base.unit) -> seq_len:Base.int -> batch_size:Base.int -> Base.unit

iter t ~f ~seq_len ~batch_size iterates f over the whole dataset. f is given two different tensors xs and ys which shapes are both seq_len; batch_size; labels. ys is shifted by one compared to xs. The dataset is shuffled on each call to iter.

Sourceval char : t -> label:Base.int -> Base.char

char t ~label returns the character from the original file that has been mapped to label.

Sourceval total_length : t -> Base.int
Sourceval labels : t -> Base.int

labels t returns the number of different labels, i.e. the number of distinct chars in the original file.