-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathlook-under-hood.qmd
817 lines (584 loc) · 34.6 KB
/
look-under-hood.qmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
---
title: Looking under the hood
engine: markdown
code-overflow: wrap
---
So far, we haven't talked much about what happens behind the scenes when you run `quarto render`.
In this chapter, you'll learn more about what Quarto does under the hood, so you can troubleshoot problems when they occur.
## What does Quarto do?
At a high level, rendering a Quarto document involves two steps:
1. **Code execution via the computational engine (`knitr`, `jupyter`, or `julia`):** The computational engine executes all the code cells in the source document and replaces the cells with static Markdown content containing the results.
The result is a Markdown file (`.md`).
2. **Format conversion via Pandoc:** The Markdown file generated in the previous step is passed to Pandoc which converts from Markdown to the required output format.
To help you understand this process, you'll follow a series of documents through these steps.
You'll start with a document that targets the HTML format and has only R code chunks, then see how the process alters for a document with Python code cells and a document that targets PDF.
## An HTML document with R code cells
The document in @lst-render-html-r-raw, `html-r.qmd`, is a Quarto document that targets the HTML format and contains two R code cells.
::: {#lst-render-html-r-raw}
````{.markdown}
{{< include _examples/look-under-hood/html-r.qmd >}}
````
`html-r.qmd` is a Quarto document with R code cells and placeholder text, targeting HTML output.
:::
The process this file undergoes when you call `quarto render` is illustrated in @fig-render-html-r.
You'll dive into the details of each step below.
{#fig-render-html-r fig-alt="Diagram showing a `.qmd` file entering a box labelled Quarto, it flows through `knitr` to a `.html.md` file, then through Pandoc, exiting the box as a `.html` file along with a folder with the name `_files.`."}
### Before the computational engine
Before anything else, Quarto parses your file to extract the document header and code cells.
Quarto determines the set of formats that need to be rendered and select the appropriate computational engine.
::: callout-tip
## Include shortcodes are also resolved
Include shortcodes are also resolved at this stage because they may contribute code cells to the document.
:::
It's at this stage you'll get errors about malformed YAML in your document header--- these start with `ERROR: YAMLError:`.
You can learn more about common YAML issues in @sec-yaml.
Each input file is only ever run through one computational engine: `knitr`, `jupyter`, or `julia`.
[^look-under-hood-1] With no additional information from the metadata (e.g. none of `engine`, `jupyter`, nor `knitr`), Quarto will choose the engine based on the file extension and the language of executable code cells in the document.
[^look-under-hood-1]: We use lower-case engine names, e.g. `knitr` vs. Knitr, because a computational engine is more than the tool that does the execution.
For example, although the Jupyter project [NBClient](https://github.com/jupyter/nbclient) does the execution when the engine is `jupyter`, the `jupyter` engine also encompasses translating the `.qmd` to an `.ipynb` and back again after execution---tasks that are handled by Quarto itself.
For `.qmd` documents, Quarto will use:
- `knitr`, if there are **any** `{r}` code cells.
- `jupyter`, if there are any executable code cells other than `{r}`.
The Jupyter kernel will be chosen based on the first executable code cell.
- The `markdown` engine, equivalent to using no computational engine, if there are no executable cells.
For `html-r.qmd`, HTML is the only format specified in the header, and assuming no `--to` argument was specified in the `quarto render` call, Quarto will know to target it.
Since, `html-r.qmd` contains executable R code cells (and doesn't specify engine otherwise) Quarto selects the `knitr` engine.
### Computational engine
The selected computational engine is informed of the target format.
It runs the executable code cells and any inline code expressions, replacing them with appropriate Markdown, abiding by the execution options specified in the metadata and code cells.
The result is a Markdown file.
You can examine this Markdown file by adding `keep-md` to your document metadata:
``` yaml
---
title: My document
keep-md: true
---
```
@lst-render-html-r-md shows the resulting Markdown file for `html-r.qmd`.
Notice that the document header and Markdown content have passed through unchanged.
However, there are now no `{r}` code cells.
These are replaced by Markdown, albeit with some special classes attached.
::: {#lst-render-html-r-md}
```` markdown
{{< include _examples/look-under-hood/html-r.html.md >}}
````
`html-r.html.md` the intermediate Markdown created after passing `html-r.qmd` through the `knitr` engine.
:::
For example, the executable code cell that does some addition in @lst-code-cell-raw is replaced in @lst-code-cell-md by a fenced div with the class `.cell` that contains:
1. A non-executable code cell `{.r}` with class `.cell-code` and the attribute `code-line-numbers="true"` that contains the code --- this is the "echo".
2. A fenced div with classes `.cell-output` and `.cell-output-stdout` that contains a code cell with the result.
::::: {#lst-code-cell .column-body-outset layout-ncol="2"}
::: {#lst-code-cell-raw}
```` markdown
```{r}
#| code-line-numbers: true
1 + 1
```
````
The original code cell.
:::
::: {#lst-code-cell-md}
```` markdown
::: {.cell}
```{.r .cell-code code-line-numbers="true"}
1 + 1
```
::: {.cell-output .cell-output-stdout}
```
[1] 2
```
:::
:::
````
The code cell after being processed by the `knitr` engine.
:::
A code cell in `html-r.qmd` before and after being processed by the `knitr` computational engine.
:::::
You can see a similar transformation has occurred on the code cell creating a plot.
However, now there is no "echo", because the code cell included the option `echo: false`, and the output is an image stored in `html-r_files/figure-html/`.
The name of the intermediate file, `html-r.html.md`, includes the format, `.html` because the engine is run separately for each format.
The Markdown output and supporting files are format specific.
You'll see an example of this in @sec-pdf-r.
If you receive an error from the computational engine, you won't get the get the intermediate Markdown file.
Common errors that arise during execution by the computational engine are discussed in @sec-computational-errors.
### Pandoc
The Markdown file is passed along to Pandoc[^look-under-hood-2] for conversion into the desired output format.
Pandoc's role in rendering provides many of the features for [authoring](#sec-authoring) Quarto documents.
For example, the syntax you learned for headings, quotations, lists, tables, links and footnotes is the syntax defined by Pandoc (see [Pandoc's Markdown](https://pandoc.org/MANUAL.html#pandocs-markdown) on the Pandoc documentation site).
Other more complicated features, like the default handling of citations with `citeproc`, also come directly from Pandoc (see [Citations](https://pandoc.org/MANUAL.html#citations) on the Pandoc documentation site).
[^look-under-hood-2]: Quarto includes a particular version of Pandoc.
You don't need to install or manage it separately, and it won't interfere with any other version of Pandoc you have installed.
If you ever need to run Quarto's Pandoc, you can with `quarto pandoc`.
However, Quarto adds many of its own features beyond those available in Pandoc.
These are added via an established mechanism for customizing Pandoc output: filters.
When Pandoc reads an input file, it translates it into an abstract representation known as an abstract syntax tree.
Filters are functions that operate on this abstract syntax tree, transforming it before it is written out to the desired format.
Cross-references, shortcodes[^look-under-hood-3], and code annotation are examples of features that Quarto adds via filters.
[^look-under-hood-3]: Most shortcodes are implemented as filters, but there are two exceptions `include` and `embed`.
For documents targeting the HTML format the output file returned to you is the output returned from Pandoc.
An excerpt from `html-r.html`, the HTML file returned from `quarto render html-r.qmd` is shown in @lst-render-html-r-html.
There are more than 100 lines of HTML prior to this excerpt, but we've excluded that code here to focus on the part of the document that reflects our content.
::: {#lst-render-html-r-html}
``` html
<section id="some-computation" class="level2">
<h2 class="anchored" data-anchor-id="some-computation">Some computation</h2>
<div class="cell">
<div class="sourceCode cell-code" id="cb1"><pre class="sourceCode numberSource r number-lines code-with-copy"><code class="sourceCode r"><span id="cb1-1"><a href="#cb1-1"></a><span class="dv">1</span> <span class="sc">+</span> <span class="dv">1</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
<div class="cell-output cell-output-stdout">
<pre><code>[1] 2</code></pre>
</div>
</div>
<div class="cell">
<div class="cell-output-display">
<div>
<figure class="figure">
<p><img src="html-r_files/figure-html/unnamed-chunk-2-1.png" class="img-fluid figure-img" width="672"></p>
</figure>
</div>
</div>
</div>
</section>
```
An excerpt from `html-r.html`---the final output file from rendering `html-r.qmd`
:::
The Markdown is now converted to HTML.
For example, the Markdown section heading `## A computation` is represented by a `<h2>` heading:
``` html
<h2 class="anchored" data-anchor-id="some-computation">Some computation</h2>
```
The fenced divs have been literally converted to `<div>` elements, and code blocks to `<pre>` elements.
This HTML file requires some additional resources for a browser to correctly render it, but Quarto provides them all in `html-r_files/`.
When you open `html-r.qmd` in a browser you'll see the final rendered document (@fig-html-r-html).
{#fig-html-r-html fig-alt="Screenshot of a webpage. Page starts with the heading 'My document', shows the output of two R code cells, and ends with a paragraph of text."}
::: callout-tip
## Templates contribute document boilerplate
Converting the Markdown to HTML commands isn't enough to have a complete HTML document.
The converted content is pasted into a template that adds the boilerplate required to form a complete document.
Quarto has default templates it uses unless otherwise specified.
You'll learn more about templates and customizing them in @sec-templates.
:::
### Monitoring the process on the command line
You can also see this process in the messages Quarto prints as it runs.
These messages are generated when you run `quarto render` or `quarto preview`, but you may have missed them because they scroll quickly past or are hidden in a pane in your IDE.
In @fig-r-output, you can see the messages Quarto prints in the Terminal during a successful render of `html-r.qmd`.
Knitr generates the first chunk of output as it processes the document and runs the code in the three (unnamed) chunks.
The remaining output gives some information on the settings being passed along to pandoc.
:::: {#fig-r-output}
::: {style="text-align: left"}
``` {.default filename="Terminal"}
$ quarto render html-r.qmd #<1>
processing file: html-r.qmd #<2>
1/5 #<2>
2/5 [unnamed-chunk-1] #<2>
3/5 #<2>
4/5 [unnamed-chunk-2] #<2>
5/5 #<2>
output file: html-r.knit.md #<2>
pandoc #<3>
to: html #<3>
output-file: html-r.html #<3>
standalone: true #<3>
section-divs: true #<3>
html-math-method: mathjax #<3>
wrap: none #<3>
default-image-extension: png #<3>
#<3>
metadata #<3>
document-css: false #<3>
link-citations: true #<3>
date-format: long #<3>
lang: en #<3>
title: My document #<3>
Output created: html-r.html #<4>
```
1. Quarto starts working, you may hang here for awhile as Quarto parses what it needs to plan its work.
2. Knitr executes the code cells
3. The information passed along to Pandoc
4. A final message from Quarto indicating success
:::
The output printed by `quarto render` when run on `html-r.qmd`.
::::
## An HTML document with Python code cells
@lst-render-html-python-raw shows `html-python.qmd`, a file that is very similar to `html-r.qmd` except instead of R code cells, it has Python code cells.
::: {#lst-render-html-python-raw}
```` {.markdown}
{{< include _examples/look-under-hood/html-python.qmd >}}
````
`html-python.qmd` a Quarto document with Python code cells and placeholder text, targeting HTML output.
:::
In @fig-render-html-python we show the process of rendering `html-python.qmd`, greying those parts that are the same as the process for `html-r.qmd`.
{#fig-render-html-python fig-alt="Diagram showing a `.qmd` file entering a box labelled Quarto, it flows through `jupyter` to a `.html.md` file, then through Pandoc, exiting the box as a `.html` file along with a folder with the name `_files.`. Everything is greyed out except the `jupyter` box."}
### Before computational engine
Like `html-r.qmd`, the only format specified in the header of `html-python.qmd` is HTML.
Unlike `html-r.qmd`, `html-python.qmd` contains only Python code cells, so Quarto will select the `jupyter` engine.
Since the first code cell is a Python code cell, the `python` kernel will be selected.
### Computational engine
Now rather than `knitr`, the `jupyter` engine is responsible for executing the code cells and replacing them with Markdown.
The intermediate file `html-python.md.html` is shown in @lst-render-html-python-md.
Just like when `knitr` processed `html-r.qmd`, the `jupyter` engine has replaced the executable code cells (`{python}`) with non-executable ones (`{.python}`), and placed the code and output in fenced divs with appropriate classes.
::: {#lst-render-html-python-md}
```` {.markdown}
{{< include _examples/look-under-hood/html-python.html.md >}}
````
`html-python.html.md`, the intermediate Markdown created after passing `html-python.qmd` through the `jupyter` engine.
:::
Functionally, the different engines should produce very similar output given the same code, but they do have their own idiosyncrasies.
For instance, you can see the `jupyter` engine has added unique code cell identifiers and an `execution_count` attribute to the code cells.
### Pandoc
Nothing changes in the Pandoc step.
The intermediate Markdown is passed to Pandoc with the exact same settings as in the `html-r.qmd` example.
### Monitoring the process on the command line
You can also see the similarities in the process through messages Quarto prints on the command line (@fig-render-html-python-output).
The only difference is where the computational engine runs.
:::: {#fig-render-html-python-output}
::: {style="text-align: left"}
``` {.default filename="Terminal"}
$ quarto render html-python.qmd #<1>
Starting python3 kernel...Done #<2>
#<2>
Executing 'html-python.quarto_ipynb' #<2>
Cell 1/3: ''...Done #<2>
Cell 2/3: ''...Done #<2>
Cell 3/3: ''...Done #<2>
pandoc #<3>
to: html #<3>
output-file: html-python.html #<3>
standalone: true #<3>
section-divs: true #<3>
html-math-method: mathjax #<3>
wrap: none #<3>
default-image-extension: png #<3>
toc: true #<3>
#<3>
metadata #<3>
document-css: false #<3>
link-citations: true #<3>
date-format: long #<3>
lang: en #<3>
title: My document #<3>
Output created: html-python.html #<4>
```
1. Quarto starts working
2. Jupyter executes the code cells
3. The information passed along to Pandoc
4. A final message from Quarto indicating success
:::
The output printed by `quarto render` when run on `html-python.qmd`.
::::
## A PDF document with R code cells {#sec-pdf-r}
Now consider the document `pdf-r.qmd` shown in @lst-render-pdf-r-raw.
This document is identical to `html-r.qmd` except instead of targeting the HTML format, it targets PDF.
::: {#lst-render-pdf-r-raw}
```` {.markdown}
{{< include _examples/look-under-hood/pdf-r.qmd >}}
````
`pdf-r.qmd` a Quarto document with R code cells that targets the PDF format.
:::
@fig-render-pdf-r shows the process of rendering `pdf-r.qmd`, again greying those parts that are the same as the process for `html-r.qmd`.
{#fig-render-pdf-r fig-alt="Diagram showing a `.qmd` file entering a box labelled Quarto, it flows through `knitr` to a `.pdf.md` file, through Pandoc to a `.tex` file, then through TeX and exiting the box as a `.pdf` file. Everything is greyed out except the `.pdf.md`, `.tex` and `.pdf` files, and the TeX box."}
### Before the computational engine
Quarto identifies the target format as PDF, and selects the `knitr` engine.
### Computational engine
There isn't any difference in how the computational engine, `knitr`, is run compared to when the target format was HTML, but the engine is aware the target format is now PDF.
This means the the engine can tailor output to the target format.
In @lst-render-pdf-r-md you can see the intermediate Markdown file that is produced.
::: {#lst-render-pdf-r-md}
```` {.markdown}
{{< include _examples/look-under-hood/pdf-r.pdf.md >}}
````
`pdf-r.pdf.md` the intermediate Markdown file after the `knitr` engine has exectuted the code in `pdf-r.qmd`
:::
It is mostly identical to that produced when the target format was HTML, but there are some differences.
In particular, compare the Markdown used to include the plot (@fig-plot-md).
::::: {#fig-plot-md}
::: {#fig-plot-html}
``` markdown
{width=672}
```
Targeting HTML
:::
::: {#fig-plot-pdf}
``` markdown
{fig-pos='H'}
```
Targeting PDF
:::
Markdown produced by `knitr` engine for when targeting (a) HTML, and (b) PDF.
:::::
When targeting PDF there is a LaTeX specific attribute added, `fig-pos='H'`, and the file included is a PDF, compared to a PNG.
### Pandoc
When the target format was HTML, Pandoc handled the conversion, and Quarto returned the HTML document directly from Pandoc.
When the target format is PDF, Pandoc only converts to LaTeX, and Quarto handles the remaining conversion from LaTeX to PDF.
This means Pandoc is called on `pdf-r.pdf.md` in the same way as on `pdf-r.html.md`, but with the format set to `latex`.
You can examine the `.tex` file Pandoc returns by setting `keep-tex` to `true`:
``` yaml
---
title: My document
format: pdf
keep-tex: true
---
```
An excerpt of the resulting `.tex` file is shown in @lst-render-pdf-r-tex.
Above this excerpt is almost 200 lines of other TeX commands setting up the document.
::: {#lst-render-pdf-r-tex}
``` tex
\subsection{Some computation}\label{some-computation}
\begin{Shaded}
\begin{Highlighting}[numbers=left,,]
\DecValTok{1} \SpecialCharTok{+} \DecValTok{1}
\end{Highlighting}
\end{Shaded}
\begin{verbatim}
[1] 2
\end{verbatim}
\includegraphics{pdf-r_files/figure-pdf/unnamed-chunk-2-1.pdf}
```
An **excerpt** from `pdf-r.tex`, the TeX file created after passing `pdf-r.pdf.md` through Pandoc.
:::
Now the Markdown syntax has been converted to TeX syntax: section headings are created with the TeX command `\subsection`, code cells use `Shaded` and `Highlighting` environments, and output uses appropriate commands like the `verbatim` environment for code output, and the `\includegraphics` command for the image file.
### After Pandoc
There is an additional step now required---Quarto runs the `.tex` document through LaTeX to produce the PDF.
Quarto will look for, and use an installation of TeX.
If TeX is not installed, you can install it with:
``` {.bash filename="Terminal"}
quarto install tinytex
```
Quarto will prefer the version installed this way over other installations.
Quarto will use the detected installation and process the `.tex` file as many times as needed to resolve all cross references and citations.
@fig-render-process-pdf shows part of the PDF output that results for `pdf-r.qmd`.
[{#fig-render-process-pdf fig-alt="A screenshot of a PDF document with the title 'My document'. Below the title is a table of contents followed by a section called `A computation`."}](_examples/look-under-hood/render-process.pdf)
## A multi-format document
You've seen what happens when a single format is declared.
But, what if more than one is declared?
For example, you might have specified both PDF and HTML in your document header:\
``` yaml
---
format:
pdf: default
html: default
---
```
@fig-render-multiformat illustrates what happens.
Before the computational engine, Quarto recognizes that more than one format is a target.
Each format is then generated in series and passes through the render process independently: the computational engine is run for each format; an intermediate Markdown file is generated for each format; Pandoc is run for each format; and any post-Pandoc steps are completed for each format.
{#fig-render-multiformat fig-alt="Diagram of a quarto file entering Quarto and splitting into two processes: render to HTML; and render to PDF."}
## Other variations
### Other types of input file
You've seen the render process specifically for `.qmd` inputs, but Quarto accepts other types of input files.
For `.Rmd`, `.ipynb`, and `.md`, the primary difference is that the engine is selected based on the file type: `jupyter` for `.ipynb` , but see @nte-ipynb; `knitr` for `.Rmd` ; and `markdown` for`.md`.
For script files (`.R`,`.py`,`.jl` etc.) the file is pre-processed into a `.qmd` file before continuing the render process as described above.
::: {#nte-ipynb .callout-note}
## Python notebooks aren't normally executed
Since `.ipynb` files store their output alongside the source, the code cells are not executed by default when you run `quarto render`.
To execute the cells, pass the `--execute` flag to `quarto render`:
``` {.bash filename="Terminal"}
quarto render document.ipynb --execute
```
:::
### Other target formats
Most formats are like HTML, they are simply passed along to Pandoc with the Markdown file, Pandoc handles the conversion to the target format, and Quarto does no further processing.
One exception is Typst.
Typst is like PDF: Pandoc handles the conversion into the Typst format `.typ`, and Quarto handles the final conversion to PDF.
In contrast to `format: pdf` where Quarto looks for an external TeX installation, for `format: typst` Quarto runs the Typst version provided internally with Quarto.
This is equivalent to:
``` {.bash filename="Terminal"}
quarto typst compile document.typ
```
### Preview vs. render
When you are working with a Quarto document, you'll often be running `quarto preview` as opposed to `quarto render`.
Think about `quarto preview` as a two-step process: first, run `quarto render`; then, serve the document.
For HTML documents this means starting up a webserver, for other formats, opening some other form of viewer.
### Rendering projects
Think about rendering a project, as rendering a folder of documents, where each one goes through the process above.
Quarto often needs to do some processing before, or after (or both) rendering each document, depending on the type of project.
For example, for websites Quarto needs to run through all documents to build the list of navigation items, before rendering individual pages.
Some projects may also involve `freeze`.
When freezing a document it is the intermediate Markdown file that gets put in the freezer, effectively removing the computational engine step when the freezer is activated.
## Troubleshooting
In this section, you'll learn some tips for troubleshooting problems when they do occur.
### Run `quarto check`
As you've seen, Quarto potentially accesses and runs many other tools.
The `quarto check` command is a good way to check the versions of these tools and the exact ones Quarto is finding.
Run `quarto check` on the command line in your project after activating the relevant environment (if any):
``` {.bash filename="Terminal"}
quarto check
```
@fig-quarto-check shows example output from `quarto check` with the following highlighted:
- **Quarto version**: Check your Quarto version against the latest release, particularly when documented features don't seem to work for you.
Before reporting a bug, you should confirm that your problem exists on the latest release, and it is good practice to also confirm that it exists on the latest pre-release version.
- **Internal dependencies**: Tools listed here are bundled with Quarto, and you cannot link Quarto to newer versions of those tools yourself.
However, knowing these versions can be useful for troubleshooting. For instance, you might be trying to use a feature of Typst that is newer than the version bundled with Quarto.
- **Quarto installation**: The version here will match the one reported on the first line, but adds the path to Quarto.
If you manage multiple installations of Quarto, you should check that you are getting the one you expect here.
- **Tools installed by Quarto**: These tools aren't included with Quarto but they can be installed with `quarto install`.
TinyTex is recommended for producing PDFs via `format: pdf`; and Chromium is required for producing versions of Mermaid and Grapviz diagrams suitable for use in PDF or DOCX.
You can also check these tools by running `quarto tools` which will inform you if they are out of date.
You can update them with `quarto update`.
- **LaTeX**: Information about the TeX distribution Quarto will use.
If you are having trouble with `format: pdf` output, and this isn't TinyTex, you should install TinyTex with `quarto install tinytex`.
- **Python**: The Python 3 interpreter discovered by Quarto—check it is the Python you expect.
This is most often controlled by a virtual environment, but can also be set with the environment variable `QUARTO_PYTHON`.
If you are using a language other than Python with the `jupyter` engine, confirm that the kernel you need is listed here.
- **R**: The R interpreter discovered by Quarto—check it is the R you expect.
It can be set with the environment variable `QUARTO_R`.
:::: {#fig-quarto-check}
::: {style="text-align: left"}
``` default
Quarto 1.5.56 #<1>
[✓] Checking versions of quarto binary dependencies...
Pandoc version 3.2.0: OK #<2>
Dart Sass version 1.70.0: OK #<2>
Deno version 1.41.0: OK #<2>
Typst version 0.11.0: OK #<2>
[✓] Checking versions of quarto dependencies......OK
[✓] Checking Quarto installation......OK
Version: 1.5.56 #<3>
Path: /Applications/quarto/bin #<3>
[✓] Checking tools....................OK
TinyTeX: v2024.03.13 #<4>
Chromium: (not installed) #<4>
[✓] Checking LaTeX....................OK #<5>
Using: TinyTex #<5>
Path: /xxx/Library/TinyTeX/bin/universal-darwin #<5>
Version: 2024 #<5>
[✓] Checking basic markdown render....OK
[✓] Checking Python 3 installation....OK #<6>
Version: 3.12.2 #<6>
Path: /xxx/.pyenv/versions/3.12.2/bin/python3 #<6>
Jupyter: 5.7.2 #<6>
Kernels: julia-1.10, python3 #<6>
[✓] Checking Jupyter engine render....OK
[✓] Checking R installation...........OK #<7>
Version: 4.3.3 #<7>
Path: /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources #<7>
LibPaths: #<7>
- /xxx/Library/R/arm64/4.3/library #<7>
- /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library #<7>
knitr: 1.44 #<7>
rmarkdown: 2.25 #<7>
[✓] Checking Knitr engine render......OK
```
1. Quarto version
2. Internal dependencies
3. Quarto installation
4. Tools installed by Quarto
5. LaTeX
6. Python
7. R
:::
An example of the result of running `quarto check`
::::
### Errors from the computational engine {#sec-computational-errors}
If your Quarto document fails to make it through the computational engine, the culprit is likely one of the following three issues:
- The wrong engine is being used.
- The environment Quarto sees isn't the one you expect.
- Your code has errors in it.
#### Wrong Engine
The heuristics Quarto uses when selecting an engine can sometimes be wrong.
If you see some indication Knitr is running when you expected Jupyter, override the automatic choice by explicitly setting the engine with the `engine` key:
``` yaml
---
engine: jupyter
---
```
If you want the `julia` engine, Quarto will never automatically select it; you must opt-in via the `engine` key.
::: callout-tip
## Avoiding execution
For the purpose of choosing an engine, a code cell with the option `eval: false` is still considered an executable code cell, even though the code inside will not be executed.
This means you might see an engine starting even if you've set `eval: false` at the document level.
To avoid even starting an engine, and executing any code, you can explicitly use the `markdown` engine:
``` yaml
---
engine: markdown
---
```
:::
#### Wrong Environment
The usual symptom of Quarto not using the right environment is the failure to load packages you are sure you installed.
Troubleshooting your environment is language-specific, and you should refer back to the relevant sections of @sec-r, @sec-python, and @sec-julia.
#### Code Errors
When the code in your executable code cells generates an error, the computational engine will halt and Quarto will exit.
@fig-errors shows an example of this for the three engines--—in all cases the code cell producing the error requests the variable `x` which hasn't been defined.
The output differs by engine, but you'll generally see the engine start progressing through the cells (e.g. `Cell 1/1` in @fig-errors-jupyter) then an error report native to the language of your cells (e.g. `NameError: name 'x' is not defined` from Python in @fig-errors-jupyter).
::: {#fig-errors}
::: {#fig-errors-jupyter}
``` default
Executing 'python-error.quarto_ipynb'
Cell 1/1: ''...ERROR:
An error occurred while executing the following cell:
------------------
x
------------------
-------------------------------------------------------
NameError Traceback (most recent call last)
Cell In[2], line 1
----> 1 x
NameError: name 'x' is not defined
```
A computational error when using the `jupyter` engine
:::
::: {#fig-errors-knitr}
``` default
processing file: r-error.qmd
1/2
2/2 [unnamed-chunk-1]
Quitting from lines 7-8 [unnamed-chunk-1] (r-error.qmd)
Error:
! object 'x' not found
Execution halted
```
A computational error when using the `knitr` engine
:::
:::{#fig-errors-julia}
``` default
Running [1/1] at line 7: x
ERROR: Julia server returned error after receiving "run" command:
Failed to run notebook: /Users/charlottewickham/Documents/posit/weekly-log/2024-09-30/julia-error.qmd
ERROR: EvaluationError: Encountered 1 error during evaluation
Error 1 of 1
@ /Users/charlottewickham/Documents/posit/weekly-log/2024-09-30/julia-error.qmd:7
UndefVarError: `x` not defined
```
A computational error when using the `julia` engine
:::
Example output from `quarto render` when there is an execution error.
:::
::: callout-tip
## Keeping an error in your output
If you want to show an error in your document, set the option `error: true` in the corresponding code cell.
This will prevent the error from halting execution, and include the error message in your rendered document.
:::
You are now in the realm of debugging your R, Python, or Julia code.
The first way to make your life easier is to label your code cells:
````{.markdown}
```{r}
#| label: get-x
x
```
````
When the engines process your code cells, they will include the label, helping you to identify which code cell caused the error quickly.
If the error isn't obvious, run your code from top to bottom interactively in a clean session of your interpreter.
::: {.callout-tip}
## Run all
In RStudio, use the command **Restart R Session and Run All Chunks**.
In the Quarto extension for VS Code and Positron, use the command **Quarto: Run All Cells**.
:::
### Things that aren't errors
Some problems don't rise to the level of errors.
You might get output from `quarto render` but it might not look right.
In this case, a general troubleshooting process might look like:
* Verify you're working with the current release of Quarto (e.g. with `quarto check`).
* Find an example of what you are trying to do in the documentation and verify it works. If it does work, gradually alter it to match your example, until it no longer works---you'll either discover what you are doing wrong, or you'll produce a minimal reproducible example.
* Ask for help using your reproducible example.
### Asking for help
If you can't figure out the problem yourself, your next step is to ask someone.
A good question includes:
* A description of your problem, including what you expected to happen and what actually happened.
* A minimal reproducible example (see @sec-repro).
* The output from `quarto check`.
Some good venues to ask questions are the [discussion board on the `quarto-cli` repository](https://github.com/quarto-dev/quarto-cli/discussions) and [Posit community](https://forum.posit.co).
<!--
## Misc
What about the `./quarto` folder?
Why is it automatically added to `.gitignore`? -->
## Wrapping Up