t-SNE is described in (Van der Maaten & Hinton 2008), while the Barnes-Hut t-SNE implementation is described in (Van der Maaten 2014). To cite the Rtsne package specifically, use (Krijthe 2015).

van der Maaten L, Hinton G (2008). “Visualizing High-Dimensional Data Using t-SNE.” Journal of Machine Learning Research, 9, 2579-2605.

van der Maaten L (2014). “Accelerating t-SNE using Tree-Based Algorithms.” Journal of Machine Learning Research, 15, 3221-3245.

Krijthe JH (2015). Rtsne: T-Distributed Stochastic Neighbor Embedding using Barnes-Hut Implementation. R package version 0.16, https://github.com/jkrijthe/Rtsne.

Corresponding BibTeX entries:

  @Article{,
    title = {Visualizing High-Dimensional Data Using t-SNE},
    volume = {9},
    pages = {2579-2605},
    year = {2008},
    author = {L.J.P. {van der Maaten} and G.E. Hinton},
    journal = {Journal of Machine Learning Research},
  }

  @Article{,
    title = {Accelerating t-SNE using Tree-Based Algorithms},
    volume = {15},
    pages = {3221-3245},
    year = {2014},
    author = {L.J.P. {van der Maaten}},
    journal = {Journal of Machine Learning Research},
  }

  @Manual{,
    title = {{Rtsne}: T-Distributed Stochastic Neighbor Embedding using
      Barnes-Hut Implementation},
    author = {Jesse H. Krijthe},
    year = {2015},
    note = {R package version 0.16},
    url = {https://github.com/jkrijthe/Rtsne},
  }