`file-embed`: Additions to `extra-source-files` don't trigger rebuild of module #6689

andreasabel · 2025-02-21T09:24:23Z

Report against Stack-3.3.1 (latest).
I noticed that Stack does not seem to honor the extra-source-files when deciding whether to rebuild a project, in contrast to Cabal which does so.

This is relevant when using file-embed to embed files into the executable. Changes to the embedded files should trigger a recompilation to keep the executable up-to-date.

E.g. I want that these files are always embedded in their latest version:

extra-source-files:
    data/README.md
    data/sub/Foo.md

I added the second one in a second step. I see the following behavioral difference between Stack and Cabal:

$ stack run                                                                              ✔ │ ghc-9.4.8 hs │ 10:06:41 
File: README.md
Content: "This is a test of the `file-embed` package,\napplied to the `data-dir` of a Haskell project.\n"

$ cabal run                                                                              ✔ │ ghc-9.4.8 hs │ 10:14:10 
File: README.md
Content: "This is a test of the `file-embed` package,\napplied to the `data-dir` of a Haskell project.\n"
File: sub/Foo.md
Content: "This is a file in a subdirectory.\nIt contains 2 lines.\n"

WANT: Changes to the amount and content of the extra-source-files should trigger a rebuild.

I attach a reproducer to play with.

embed-datadir-0.1.tar.gz

The text was updated successfully, but these errors were encountered:

andreasabel · 2025-02-21T13:13:56Z

This issue might be releated:

qAddDependentFile doesn't cause re-compilation without extra-source-files #4360

Via Template Haskell we embed the files in `src/data` as `ByteString`s into the Agda binary. They will be dumped into `$AGDA_DIR/share/$VERSION` when Agda starts and this directory does not exist, or via `agda --setup`. The $Agda_datadir becomes obsolete. Agda needs to be recompiled now when a datafile is added or changed. Under Cabal, this is taken care of by their listing in `extra-source-files`, Stack however needs some explicit nudge to recompile (see commercialhaskell/stack#6689).

mpilgrem · 2025-02-22T19:26:46Z

Possibly relevant history:

The Cabal user guide currently documents that both extra-files (added cabal-verison: 3.14) and extra-source-files are to add files to an sdist archive, but the latter are also tracked by cabal build (first documented with Cabal 3.10, introduced in Cabal 3.4:

Rebuild on changes in all extra-source-files haskell/cabal#6889

).

mpilgrem · 2025-02-22T20:56:56Z

On Windows 11, I can't reproduce stack build (or stack run) not respecting changes to extra-source-files, per se:

❯ stack run
testESF-0.1.0.0: unregistering (local file changes: README.md)
testESF> build (lib + exe) with ghc-9.8.4

Also, I can't reproduce§ with package file-embed and $(embedDir "data") with lts-23.9 (GHC 9.8.4, template-haskell-2.21.0.0, file-embed-0.0.16.0) or lts-21.25 (GHC 9.4.8, template-haskell-2.19.0.0, file-embed-0.0.15.0). In each case, stack build recognises a change to README.md and embeds the changed file into the built executable.

§ EDIT: OK, I now see there is a problem when a new file is added to extra-source-files and Main.hs has already been compiled. I see that the Haddock documentation for embedFileIfExists has:

Warning: When a build is compiled with the file missing, a recompile when the file exists might not trigger an embed of the file. You might try to fix this by doing a clean build.

I am wondering if that is a more general problem with file-embed.

EDIT2: If I understand correctly, file-embed-0.0.16.0 has:

pairToExp :: FilePath -> (FilePath, B.ByteString) -> Q Exp
pairToExp _root (path, bs) = do
#if MIN_VERSION_template_haskell(2,7,0)
    qAddDependentFile $ _root ++ '/' : path -- <<< This tells GHC to track changes to file contents
#endif
    exp' <- bsToExp bs
    return $! TupE
#if MIN_VERSION_template_haskell(2,16,0)
      $ map Just
#endif
      [LitE $ StringL path, exp']

So, if a file is added subsequently: [1] GHC does not know to recompile Main.hs; and [2] Stack has no reason to recompile Main.hs either (as it has not changed).

andreasabel · 2025-02-23T08:44:10Z

@mpilgrem Thanks for the analysis!

I see that you flagged this as resolution: upstream issue, does this mean that you think that the problem is solely in file-embed?
Then one would wonder why Cabal can do this correctly, but Stack not.

It seems that changes to extra-source-files are more aggressively triggering rebuilds in Cabal compared to Stack.
This was maybe annoying in the time when extra-doc-files (and now extra-files) did not exist so folks needed to abuse extra-source-files for the purpose of just adding files to the sdist tarball.

Just from a theoretical perspective, file-embed by itself has no means to get the case right when files are added to a directory embedded by embedDir. It can, when executed, mark all the files in the directory as involved in the build (qAddDependentFile), so that changes to these files will trigger a rebuild subsequently. But as it cannot know the future, it cannot prepare for files that are added to the directory. So in this case, the trigger for a rebuild has to come externally, namely from the build orchestrator (Stack or Cabal) that sees a new entry in the file list denoted by extra-source-files (so, after expansion of wildcards) and will force a total rebuild.

I see that you found the respective PR for cabal:

Rebuild on changes in all extra-source-files haskell/cabal#6889

So you are likely aware of all of this already; I just wanted to write it down for the sake of clarity.

mpilgrem · 2025-02-23T10:42:22Z

@andreasabel, I think Cabal (the tool) (version 3.14.1.0) has identical behaviour to Stack's, given the following experiment (no package.yaml file):

❯ stack exec --no-ghc-package-path -- cabal --version # Record Cabal version ...
cabal-install version 3.14.1.0
compiled using version 3.14.1.0 of the Cabal library

❯ stack exec --no-ghc-package-path -- ghc --version # Record GHC version in environment ...
The Glorious Glasgow Haskell Compilation System, version 9.8.4

❯ stack exec --no-ghc-package-path -- cabal run # First build of embed-datadir-0.1
Resolving dependencies...
Build profile: -w ghc-9.8.4 -O1
In order, the following will be built (use -v for more details):
 - embed-datadir-0.1 (exe:embed-datadir) (first run)
Configuring executable 'embed-datadir' for embed-datadir-0.1...
Preprocessing executable 'embed-datadir' for embed-datadir-0.1...
Building executable 'embed-datadir' for embed-datadir-0.1...
[1 of 2] Compiling Main             ( Main.hs, dist-newstyle\build\x86_64-windows\ghc-9.8.4\embed-datadir-0.1\x\embed-datadir\build\embed-datadir\embed-datadir-tmp\Main.o )
[2 of 2] Compiling Paths_embed_datadir ( dist-newstyle\build\x86_64-windows\ghc-9.8.4\embed-datadir-0.1\x\embed-datadir\build\embed-datadir\autogen\Paths_embed_datadir.hs, dist-newstyle\build\x86_64-windows\ghc-9.8.4\embed-datadir-0.1\x\embed-datadir\build\embed-datadir\embed-datadir-tmp\Paths_embed_datadir.o )
[3 of 3] Linking dist-newstyle\\build\\x86_64-windows\\ghc-9.8.4\\embed-datadir-0.1\\x\\embed-datadir\\build\\embed-datadir\\embed-datadir.exe
File: README.md
Content: "This is a test of the `file-embed` package,\napplied to the `data-dir` of a Haskell project.\n"
File: sub\Foo.md
Content: "This is a file in a subdirectory.\nIt contains 2 lines.\n"

❯ # Add new Foo2.md to data and *.cabal file

❯ stack exec --no-ghc-package-path -- cabal run # Rebuild, but no recompilation of Main ...
Resolving dependencies...
Build profile: -w ghc-9.8.4 -O1
In order, the following will be built (use -v for more details):
 - embed-datadir-0.1 (exe:embed-datadir) (configuration changed)
Configuring executable 'embed-datadir' for embed-datadir-0.1...
Preprocessing executable 'embed-datadir' for embed-datadir-0.1...
Building executable 'embed-datadir' for embed-datadir-0.1...
File: README.md
Content: "This is a test of the `file-embed` package,\napplied to the `data-dir` of a Haskell project.\n"
File: sub\Foo.md
Content: "This is a file in a subdirectory.\nIt contains 2 lines.\n"

❯ stack exec --no-ghc-package-path -- cabal clean # Start afresh ...

❯ stack exec --no-ghc-package-path -- cabal run # Now new Foo2.md is recognised ...
Resolving dependencies...
Build profile: -w ghc-9.8.4 -O1
In order, the following will be built (use -v for more details):
 - embed-datadir-0.1 (exe:embed-datadir) (first run)
Configuring executable 'embed-datadir' for embed-datadir-0.1...
Preprocessing executable 'embed-datadir' for embed-datadir-0.1...
Building executable 'embed-datadir' for embed-datadir-0.1...
[1 of 2] Compiling Main             ( Main.hs, dist-newstyle\build\x86_64-windows\ghc-9.8.4\embed-datadir-0.1\x\embed-datadir\build\embed-datadir\embed-datadir-tmp\Main.o )
[2 of 2] Compiling Paths_embed_datadir ( dist-newstyle\build\x86_64-windows\ghc-9.8.4\embed-datadir-0.1\x\embed-datadir\build\embed-datadir\autogen\Paths_embed_datadir.hs, dist-newstyle\build\x86_64-windows\ghc-9.8.4\embed-datadir-0.1\x\embed-datadir\build\embed-datadir\embed-datadir-tmp\Paths_embed_datadir.o )
[3 of 3] Linking dist-newstyle\\build\\x86_64-windows\\ghc-9.8.4\\embed-datadir-0.1\\x\\embed-datadir\\build\\embed-datadir\\embed-datadir.exe
File: README.md
Content: "This is a test of the `file-embed` package,\napplied to the `data-dir` of a Haskell project.\n"
File: sub\Foo.md
Content: "This is a file in a subdirectory.\nIt contains 2 lines.\n"
File: sub\Foo2.md
Content: "Added\r\n"

That would be consistent with the documentation in the Cabal User Guide, which says only 'partial' rebuilds:

... Files listed here are tracked by cabal build; changes in these files cause (partial) rebuilds.

andreasabel · 2025-02-23T11:48:18Z

Thanks for clarifying this!

I think this is a bug in Cabal too, so I reported the issue:

Additions to extra-source-files do not trigger rebuild haskell/cabal#10801

andreasabel · 2025-02-23T11:55:06Z

That would be consistent with the documentation in the Cabal User Guide, which says only 'partial' rebuilds:

... Files listed here are tracked by cabal build; changes in these files cause (partial) rebuilds.

Ha, I added this piece of documentation myself. But I guess I did not mean anything specific by "partial". I think I wanted to relativize "rebuilds" a bit since I had (and have) no clear model of Cabal's rebuild logic.

https://github.com/haskell/cabal/pull/8765/files

mpilgrem · 2025-02-23T14:28:47Z

@andreasabel, you asked about my labelling this as an 'upstream issue'. It seems to me that both Stack and Cabal (the tool) are behaving as expected:

they are rebuilding when a new file is added to extra-source-files;
they are recompiling a built module when the past use of qAddDependentFile has told GHC that the module depends on a file and that file's contents have changed subsequently; and
they are not recompiling a built module when there is nothing to trigger a recompilation.

It also seems to me that what you are experiencing is a known limitation of file-embed, albeit one that is not documented as well in that package's Haddock documentation as it might be.

My immediate solution would be:

to improve Stack's online documentation to cover this use case (that is in hand); and
to raise a pull request to improve the Haddock documentation of the file-embed package.

andreasabel · 2025-02-23T16:17:56Z

Ah thanks. I think have misunderstood what triggering a rebuild can do.

So then using embedDir might not be a good idea. In my actual use case, I already departed from this, but I thought I am dealing with a limitation of Stack/Cabal.

andreasabel · 2025-02-23T16:18:58Z

Oops I have closed this by accident and reopening of an issue is not permitted here.
But I am ok either way.

mpilgrem · 2025-02-23T16:28:27Z

@andreasabel, the solution (of sorts) is to add ghc-options: -fforce-recomp to the package description of the package making use of file-embed's embedDir function.

andreasabel mentioned this issue Feb 21, 2025

Document that file paths need to be added to extra-source-files list to get recompilation detection in cabal projects snoyberg/file-embed#42

Open

mpilgrem added needs reproduction further investigation required component: build resolution: upstream issue and removed needs reproduction further investigation required labels Feb 22, 2025

mpilgrem changed the title ~~Changes to the extra-source-files do not trigger a rebuild~~ file-embed: Additions to extra-source-files don't trigger rebuild of module Feb 22, 2025

andreasabel mentioned this issue Feb 23, 2025

Additions to extra-source-files do not trigger rebuild haskell/cabal#10801

Closed

andreasabel closed this as completed Feb 23, 2025

mpilgrem added a commit that referenced this issue Feb 23, 2025

Re #6689 Document GHC's recompilation checker and Template Haskell

28ee6f3

mpilgrem reopened this Feb 23, 2025

mpilgrem mentioned this issue Feb 23, 2025

Extend Haddock documentation warning snoyberg/file-embed#52

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`file-embed`: Additions to `extra-source-files` don't trigger rebuild of module #6689

`file-embed`: Additions to `extra-source-files` don't trigger rebuild of module #6689

andreasabel commented Feb 21, 2025 •

edited by mpilgrem

Loading

andreasabel commented Feb 21, 2025

mpilgrem commented Feb 22, 2025 •

edited

Loading

mpilgrem commented Feb 22, 2025 •

edited

Loading

andreasabel commented Feb 23, 2025

mpilgrem commented Feb 23, 2025

andreasabel commented Feb 23, 2025

andreasabel commented Feb 23, 2025

mpilgrem commented Feb 23, 2025

andreasabel commented Feb 23, 2025

andreasabel commented Feb 23, 2025

mpilgrem commented Feb 23, 2025

file-embed: Additions to extra-source-files don't trigger rebuild of module #6689

file-embed: Additions to extra-source-files don't trigger rebuild of module #6689

Comments

andreasabel commented Feb 21, 2025 • edited by mpilgrem Loading

andreasabel commented Feb 21, 2025

mpilgrem commented Feb 22, 2025 • edited Loading

mpilgrem commented Feb 22, 2025 • edited Loading

andreasabel commented Feb 23, 2025

mpilgrem commented Feb 23, 2025

andreasabel commented Feb 23, 2025

andreasabel commented Feb 23, 2025

mpilgrem commented Feb 23, 2025

andreasabel commented Feb 23, 2025

andreasabel commented Feb 23, 2025

mpilgrem commented Feb 23, 2025

`file-embed`: Additions to `extra-source-files` don't trigger rebuild of module #6689

`file-embed`: Additions to `extra-source-files` don't trigger rebuild of module #6689

andreasabel commented Feb 21, 2025 •

edited by mpilgrem

Loading

mpilgrem commented Feb 22, 2025 •

edited

Loading

mpilgrem commented Feb 22, 2025 •

edited

Loading