Skip to main content

sentence_to_pairs

word2vec::model

Function sentence_to_pairs 

Source 
pub fn sentence_to_pairs(
    tokens: &[usize],
    window_size: usize,
    rng: &mut SmallRng,
) -> Vec<(usize, Vec<usize>)>
Expand description

Generate training examples from a tokenised sentence.

Returns (center_idx, context_indices) pairs using a dynamic window (window size drawn uniformly from [1, window_size]).