[dvsim] Generate FUSESOC_IGNORE at top of scratch root
This is a bit of a work-around for a race condition that was causing
all sorts of confusing CI failures. This crops up the second time you
run a "primary config" (list of configurations), if you do that second
run with a high parallelism.
Suppose you've done a dvsim lint run already and you have two IP
blocks, A and B. At this point, you'll have generated .core files
somewhere inside A's build directory, protected by a FUSESOC_IGNORE
file.
Now you start a new run. Two fusesoc processes (for A and B,
respectively) start to race. The one for A cleans its build directory,
essentially running "rm -rf" on the generated tree. This works
top-down, deleting files and directories. Within a directory, it works
in the order that files come back from readdir. What happens if
FUSESOC_IGNORE appears in the list before the directories containing
the core files?
The "rm -rf" process deletes the FUSESOC_IGNORE file. Now suppose that
the kernel suspends that process and switches over to the fusesoc
process for block B. That process can now merrily read its way down
the build directory. There's no FUSESOC_IGNORE file, so it looks in
core A's generated core files and finds some. Oops.
Now the process for block B gets suspended and we're back to A, which
finishes up deleting stuff. When we get back to B again, it tries to
load that juicy core file that it found... which isn't there any
more. Total confusion!
Debugging this was made much worse by the fact that the time stamps
have all since been trashed by core A's fusesoc run, which has since
helpfully re-created all the files!
One "proper" fix would be to delete the old build directory more
carefully, ensuring that the generated core files get deleted before
FUSESOC_IGNORE. That seems rather difficult, though. Ideally, we'd
want to move that logic back to fusesoc, rather than calling "rm -rf"
ourselves.
Another possible fix would be to move this "touch" operation into the
various flow configurations that need it. I'm not so keen on that,
though, because now we duplicate the logic all over the place.
I'm hoping this commit will be reverted and replaced by a better
solution soon :-)
Signed-off-by: Rupert Swarbrick <rswarbrick@lowrisc.org>
diff --git a/util/dvsim/dvsim.py b/util/dvsim/dvsim.py
index 7f2fdea..64e6e3e 100755
--- a/util/dvsim/dvsim.py
+++ b/util/dvsim/dvsim.py
@@ -23,6 +23,7 @@
import datetime
import logging as log
import os
+from pathlib import Path
import shutil
import shlex
import subprocess
@@ -604,6 +605,11 @@
proj_root_src, proj_root = resolve_proj_root(args)
log.info("[proj_root]: %s", proj_root)
+ # Create an empty FUSESOC_IGNORE file in scratch_root. This ensures that
+ # any fusesoc invocation from a job won't search within scratch_root for
+ # core files.
+ (Path(args.scratch_root) / 'FUSESOC_IGNORE').touch()
+
args.cfg = os.path.abspath(args.cfg)
if args.remote:
cfg_path = args.cfg.replace(proj_root_src + "/", "")
