To use RTextDoc with LaTeX, install MiKTeX or TeX Live (for Windows). For Linux or Mac, follow the corresponding instructions. RTextDoc autodetects LaTeX installation, so it is wise to install LaTeX first. If you installed first RTextDoc and then LaTeX, use the RTextDoc menu [Edit]⇒[Option]⇒[LaTeX] to check the LaTeX installation (click “Autodetect” button).
Note that RTextDoc also supports LaTeX without native LaTeX installed, but such usage is limited. Please read the section LaTeX to HTML.