使用SCons的VariantDir和Repository来构建使用自定义Generator

Question

我有一个几乎可用的SConstruct文件。目前我没有使用任何SConscript文件，并且希望不需要在我的源代码库（git，而不是SCons）中使用它们。

简要概述-当更改某些参数然后返回到先前的参数时，相同的文件将被重新构建。

我运行scons -f Builder_repo/SConstruct 'NameOfTargetLibrary.b'来构建一个名为NameOfTargetLibrary.b的库，该库源文件为NameOfTargetLibrary.src。 <lib>.b应该放置在依赖于各种标志(Debug/Release, 32/64位，平台(从列表中))的位置上，如下所示:

topdir
|\-- Builder_repo (containing SConstruct, site_scons\...)
|\-- Lib1 (contains lib1.src, bunch of source files)
|\-- Lib2 (lib2.src, lib2 sources)
\--- BuiltLibs
     |\-- Windows
     |    |\-- Release_32
     |    |    |\-- lib1.b
     |    |    |\-- lib2.b
     |    |    \--- libn.b
     |    |\-- Debug_64
     |    |    |\-- lib1.b
     |    |    |\-- lib2.b
     |    |    \--- libn.b
     |    \--- (Debug_32, Release_64)
     \--- (etc, other targets, currently just one)

命令行类似于以下内容（为了易读性而拆分为多行，但在SCons/cmdLine中只有一行）：

"abspath to exe" "static path and args" --x64 -- 
    "abspath(lib1.src)" "abspath(BuiltLibs)"
    "abspath(BuiltLibs/Windows/Release_64)" 
    "flags for debug/release, target, bitness"

这个“工作中”的SConstruct使用了一个带有generate(env)的工具，大致如下：

• 构建目标目录（例如：BuiltLibs\Windows\Release_32），并将其存储在env中。
• 查找.src文件。
• 获取包含目录（使用os.path.dirname）。
• 添加到env.Repositories(dirname(lib.src))中。
• tgt = env.BLDR(<TARGETDIR>/lib.b, lib.src)
• env.Alias(lib.b, tgt)

然后Builder使用Emitter将任何取决于lib.src的<TARGETDIR>/libx.b文件添加到source列表中（从源文件中读取）。如果需要，这些文件也可以被添加为libx.b。

Generator解析输入目标和源列表以形成命令行，并返回它。根据当前的配置，目标和源都是相对路径，因此可能不需要Repository调用。

当我运行

scons -f Builder_repo\SConstruct 'lib2.b' DEBUG=0 BITNESS=32 TRGT=Windows

(由于发射器，lib2.src 依赖于 lib1.b)，正确的 lib1.b 和 lib2.b 已经构建并放置在 BuiltLibs\Windows\Release_32\lib{1,2}.b 中。

如果我重复执行命令，则不会构建任何内容，'lib2.b is up to date'。

然后，我尝试使用 scons -f <..> 'lib2.b' DEBUG=1 <other args same> 命令。两个库被构建并放置在 BuiltLibs\Windows\Debug_32\lib{1,2}.b 中，与预期相同。

当我再次尝试第一个命令（DEBUG=0）时，我希望不会构建任何东西（lib1.b，lib2.b 仍然是最新的 - 没有更改源文件，之前构建的文件仍然在 Release_32 中），但实际上它们被重新构建了。

我尝试通过在 for_signature 为 true 时返回一个简化的命令行来解决这个问题，以使得该值在这种情况下返回更像：

"abspath to exe" "static path and args" --
     "abspath(lib1.src)" "abspath(BuiltLibs)" "version string" 

“版本字符串”不受调试/发布、32/64位、平台标志影响（但会随着源代码的更改而变化）。这似乎没有什么区别。

我尝试使用env.VariantDir(<TARGETDIR>, '.', duplicate=0)的一些变化，然后tgt = env.BLDR(lib.b, Lib1/lib.src)，env.Alias(<TARGETDIR>/lib.b, tgt)或类似的方法，但我没有成功改进任何东西（有些配置总是重新构建，其他则使依赖项无法找到并导致SCons出错）。

我应该怎么做呢？

---

SConstruct：

import os

Decider('make')
Default(None)

# Add command line arguments with default values.
# These can be specified as, for example, LV_TARGET=cRIO
cmdVars = Variables(None, ARGUMENTS)
cmdVars.AddVariables(
    EnumVariable('LV_TARGET', 'Choose the target for LabVIEW packages', 
        'Windows', allowed_values=('Windows', 'cRIO')),
    BoolVariable('DEBUG', 'Set to 1 to build a debug-enabled package', 0),
    EnumVariable('BITNESS', 'Choose the bitness for LabVIEW packages', 
        '32', allowed_values=('32', '64')),
    EnumVariable('LV_VER', 'Choose the version of LabVIEW to use', 
        '2017', allowed_values=('2017',))
)

# Define a list of source extensions
src_exts = ['.vi', '.ctl', '.lvlib', '.vim', '.vit']

env = Environment(variables = cmdVars, ENV = os.environ, tools=['PPL'], PPLDIR='PPLs', SRC_EXTS=' '.join(src_exts))

---

init.py是PPL工具的初始化文件：

""" SCons.Tool.PPL
Tool-specific initialization for compilation of lvlibp files from lvlib files,
using the Wiresmith 'labview-cli.exe' and the LabVIEW code stored in the 
PPL_Builder GitHub repository.
This module should not usually be imported directly.
It can be imported using a line in SConstruct or SConscript like
env = Environment(tools=['PPL'])
"""

# A reference for this code can be found at
# https://github.com/SCons/scons/wiki/ToolsForFools
# which describes the creation of a Tool for JALv2 compilation.

import SCons.Builder
from SCons.Script import GetOption
import SCons.Node
import SCons.Util
import os.path
import textwrap
import re

import contextlib
import subprocess

# Add warning classes
class ToolPPLWarning(SCons.Warnings.Warning):
    pass

class LabVIEW_CLI_ExeNotFound(ToolPPLWarning):
    pass

SCons.Warnings.enableWarningClass(ToolPPLWarning)

__verbose = False

class LV_BuildPaths:
    """ A simple class to contain the build paths 
        and configuration flags for PPL compilation

        Contains the attributes:
        hwTarget{,Dir}, debug{Opt,Flag,String}, bitness{,Flag}, lv_ver,
        {ppl,storage,copy,topData,data}Dir
    """
    def __init__(self, env):
        # Set the target parameters
        self.hwTarget = env.get('LV_TARGET')
        copyDirExtra = ""
        if self.hwTarget == "cRIO":
            self.hwTargetDir = "cRIO-9045"
            copyDirExtra = os.path.join('home','lvuser','natinst','bin')
        else:
            self.hwTargetDir = self.hwTarget
        # Set the debug parameters
        self.debugOpt = env.get('DEBUG')
        self.debugFlag = int(self.debugOpt)
        self.debugString = "Debug" if self.debugOpt else "Release"
        # Set the bitness parameters
        self.bitness = env.get('BITNESS')
        self.bitnessFlag = ''
        if self.bitness == '64':
            self.bitnessFlag = '--x64'
        # Get the LabVIEW year
        self.lv_ver = env.get('LV_VER')

        # Get important build directory paths.
        # PPL files should be searched for in storageDir
        self.pplDir = os.path.normpath(env.get('PPLDIR', 'PPLs'))
        self.storageDir = os.path.join(self.pplDir, self.hwTargetDir, f'{self.debugString}_{self.bitness}', copyDirExtra)
        self.copyDir = os.path.join(self.pplDir, self.hwTargetDir, copyDirExtra)
        self.topDataDir = os.path.join(self.pplDir, 'Data')
        self.dataDir = os.path.join(self.copyDir, 'Data')

    def __str__(self):
        return (textwrap.dedent(f"""\
        The directories are as follows...
            PPL Dir:      {self.pplDir}
            Storage Dir:  {self.storageDir}
            Copy Dir:     {self.copyDir}""")
        ) 


def _print_info(message):
    """ Disable the output of messages if '--quiet', '-s' or '--silent'
        are passed to the command line """
    if not GetOption('silent'):
        print(message)

def _detectCLI(env):
    """ Search for the labview-cli.exe installed by Wiresmith's VIPackage """
    try:
        # If defined in the environment, use this
        _print_info(f"labview-cli defined in the environment at {env['LV_CLI']}")
        return env['LV_CLI']
    except KeyError:
        pass

    cli = env.WhereIs('labview-cli')
    if cli:
        _print_info(f"Found labview-cli at {cli}")
        return cli

    raise SCons.Errors.StopError(
        LabVIEW_CLI_ExeNotFound,
        "Could not detect the labview-cli executable")
    return None

@contextlib.contextmanager
def pushd(new_dir):
    previous_dir = os.getcwd()
    os.chdir(new_dir)
    yield
    os.chdir(previous_dir)

def _getHash(env, dir):
    if env['GIT_EXE']:
        with pushd(dir):
            #cmdLine = env['git_showref']
            cmdLine = env['git_describe']
            return subprocess.run(cmdLine, shell=True, capture_output=True, text=True).stdout
    return ''

def _detectGit(env):
    """ Search for a git executable. This is not required for usage """
    git = None
    try:
        # If defined in the environment, use this
        _print_info(f"git executable defined in the environment at {env['GIT_EXE']}")
        git = env['GIT_EXE']
    except KeyError:
        pass

    cli = env.WhereIs('git')
    if cli:
        _print_info(f"Found git at {cli}")
        git = cli

    if git:
        hash_len = 12
        env['GIT_EXE'] = f"'{git}'" # I edited this line compared to the version in the repository, but I don't think it's relevant.
        env['git_describe'] = f'"{git}" describe --dirty="*" --long --tags --always --abbrev={hash_len}'
        env['git_showref'] = f'"{git}" show-ref --hash={hash_len} --head head'
    return None

#
# Builder, Generator and Emitter
#
def _ppl_generator(source, target, env, for_signature):
    """ This function generates the command line to build the PPL.
        It should expect to receive a target as a relative path
        ['<SD>/A.lvlibp'], and source will be either None, or
        ['<src>/A.lvlib'].

        When for_signature == 0, the PPL will actually be built.
    """

    # Get these parameters properly
    run_vi = os.path.abspath(os.path.join('.','PPL_Builder','Call_Builder_Wiresmith.vi'))
    cliOpts = ''
    package_ver = "0.0.0.0#sconsTest"

    # These are extracted from the environment
    cli = env['LV_CLI']
    bp = env['LV_Dirs']
    ver = bp.lv_ver
    pplDir = f'{os.path.abspath(bp.pplDir)}'
    storageDir = f'{os.path.abspath(bp.storageDir)}'
    # Dependencies are parsed for the command line. They are already dependencies of the target.
    pplSrcs = source[1:]
    depsString = ""
    if pplSrcs:
        if __verbose:
            _print_info("Adding PPL dependencies: %s" % [ str(ppl) for ppl in pplSrcs ])
        depsString = " ".join([f'"{os.path.basename(ppl.get_string(for_signature))}"' for ppl in pplSrcs])

    cmdLine = f'"{cli}" --lv-ver {ver} {bp.bitnessFlag} {run_vi} {cliOpts} -- '
    lvlib_relpath = str(source[0])
    lvlib_abspath = os.path.abspath(lvlib_relpath)
    git_ver = _getHash(env, os.path.dirname(lvlib_abspath))
    print("git version is " + str(git_ver).strip())

    argsLine = f'"{lvlib_abspath}" "{pplDir}" "{storageDir}" {bp.debugFlag} {bp.hwTarget} "{package_ver}" {depsString}'

    if not for_signature:
        _print_info(f"Making {lvlib_abspath}")

    return cmdLine + argsLine
    #return cmdLine + argsLine

def _ppl_emitter(target, source, env):
    """ Appends any dependencies found in the .mk file to the list of sources.
        The target should be like [<SD>/A.lvlibp], 
        and the source should be like [<src>/A.lvlib]
    """
    if not source:
        return target, source
    exts_tuple = tuple(env['SRC_EXTS'].split(' '))
    src_files = _get_other_deps(source, exts_tuple)
    if __verbose:
        _print_info("Adding " + str(src_files) + " as dependencies")
    env.Depends(target, src_files)
    depsList, nodeDepsList = _get_ppl_deps(str(source[0]), env)
    if nodeDepsList:
        source += [os.path.normpath(os.path.join(env['LV_Dirs'].storageDir, str(pplNode))) for pplNode in nodeDepsList]
    return target, source

_ppl_builder = SCons.Builder.Builder(generator = _ppl_generator, emitter = _ppl_emitter)

def lvlibpCreator(env, target, source=None, *args, **kw):
    """ A pseudo-Builder for the labview-cli executable
        to build .lvlibp files from .lvlib sources, with
        accompanying dependency checks on appropriate source files

        Anticipate this being called via env.PPL('<SD>/A.lvlibp'),
        where target is a string giving a relative path, or
        env.PPL('<SD>/A.lvlibp', '<src>/A.lvlib')
    """
    bPaths = env['LV_Dirs']

    # Ensure that if source exists, it is a list
    if source and not SCons.Util.is_List(source):
        source = [source]

    if __verbose:
        _print_info(f"Target = {target}")
        if source:
            _print_info("Sources = %s" % [ str(s) for s in source])
    if __verbose:
        _print_info("args: %s" % [ str(s) for s in args ])
        _print_info("kw: %s" % str(kw.items()))

    tgt = _ppl_builder.__call__(env, target, source, **kw)
    return tgt

def _scanForLvlibs(env, topdir=None):
    # Maybe check this...
    if not topdir:
        topdir = '.'

    bPaths = env['LV_Dirs']
    lvlibList = []
    for root, dirs, files in os.walk(topdir):
        # if any of files ends with .lvlib, add to the list
        lvlibList += map(lambda selected: os.path.join(root, selected), filter(lambda x: x[-6:] == '.lvlib', files))
    for lib in lvlibList:
        # Set up the possibility of building the lvlib
        (srcDir, libnameWithExt) = os.path.split(lib)
        # Add the source repository
        if __verbose:
            _print_info("Adding repository at: " + srcDir)
        env.Repository(srcDir)
        # Add the build instruction
        lvlibpName = libnameWithExt + 'p'
        tgt = env.PPL(os.path.normpath(os.path.join(bPaths.storageDir, lvlibpName)),lib)
        if __verbose:
            _print_info(f"Adding alias from {libnameWithExt+'p'} to {str(tgt)}")
        env.Alias(lvlibpName, tgt)

def _get_ppl_deps(lvlib, env):
    lvlib_s = str(lvlib)
    lvlib_name = os.path.basename(lvlib_s)
    mkFile = lvlib_s.replace('.lvlib','.mk')
    if os.path.isfile(mkFile):
        # load dependencies from file
        depVarName = lvlib_name.replace(' ',r'\+').replace('.lvlib','_Deps')
        f = open(mkFile, "r")
        content = f.readlines() # Read all lines (not just first)
        depsList = []
        for line in content:
            matchedDeps = re.match(depVarName+r'[ ]?:=[ ]?(.*)$', line)
            if matchedDeps:
                listDeps = matchedDeps.group(1).replace(r'\ ','+').split(' ')
                depsList = ['"' + elem.replace('+', ' ') + '"' for elem in listDeps]
                nodeList = [ env.File(elem.replace('+', ' ')) for elem in listDeps]
                return (depsList, nodeList)
        raise RuntimeError("Found a .mk file ({mkFile}) but could not parse it to get dependencies.")
    #print(f"No .mk file for {lvlib_name}")
    return ('', None)

def _get_other_deps(source, exts):
    parent_dir = os.path.dirname(str(source[0]))
    if __verbose:
        _print_info(f"Searching {parent_dir} for source files...")
        _print_info(f"Acceptable extensions are {exts}")
    src_files = []
    for root, dirs, files in os.walk(parent_dir):
        src_files += [os.path.join(root, file) for file in files if file.endswith(exts)]
    return src_files

def generate(env):
    '''Add builders and construction variables to the Environment.'''
    env['LV_CLI'] = _detectCLI(env)
    env.AddMethod(lvlibpCreator, "PPL")
    _detectGit(env)

    bp = LV_BuildPaths(env)
    _print_info(bp)
    env['LV_Dirs'] = bp

    # Search for all lvlib files
    _scanForLvlibs(env)

def exists(env):
    return _detectCLI(env)

- chrisb2244

---

请发布相关工具的源代码。没有它，我们将会束手无策。 - bdbaddog

@bdbaddog 已添加所有源代码。有些地方可能有点啰嗦，某些部分在粘贴之前几乎肯定可以被削减，但显然，既然我正在寻求帮助，那么在你要求源代码后，我可能不应该决定什么是或不是相关的。 - chrisb2244

那里的逻辑太复杂了。为什么你需要在这个逻辑中使用git？如果你想要一个独立的git工具，就将其分离出来。同时运行scons --debug=explain并发布结果。还要运行--taskmastertrace=trace.log并将其发布到pastebin上。 - bdbaddog

常常不出所料，问题存在于椅子和键盘之间... Emitter 包含的依赖项包括一个由构建生成的文件。这会导致重新构建。从列表中删除该文件进行测试会产生预期结果（即在返回相同的命令行选项时不会重新构建）。也许使用默认值而不是 Decider('make') 可能会有所帮助。我会再次查看维基百科，因为我很确定我曾经读过一个讨论版本生成文件标记的示例。一旦我找到并实施了它，我会回复。 - chrisb2244

是的，我认为有几种版本文件的实现方式。AddPreAction是其中一种方法。 - bdbaddog

Answer 1

正如评论中简要描述的那样，重建的原因是使用了Decider('make')（即通过时间戳检查）并且有效地捕获了自动生成的文件的源文件。

当按照bdbaddog在问题评论中建议的运行scons --debug=explain时，这很容易看到。

虽然有点脆弱，但最简单的解决方案是修改emitter，保留以下内容（请参见--->标记）：

def _ppl_emitter(target, source, env):
    """ Appends any dependencies found in the .mk file to the list of sources.
        The target should be like [<SD>/A.lvlibp], 
        and the source should be like [<src>/A.lvlib]
    """
    if not source:
        return target, source
    exts_tuple = tuple(env['SRC_EXTS'].split(' '))
    src_files = _get_other_deps(source, exts_tuple)
--->filtered_files = list(filter(lambda x: "Get PPL Version.vi" not in x, src_files))
    if __verbose:
        _print_info("Adding " + str(filtered_files) + " as dependencies")
    env.Depends(target, filtered_files)
    depsList, nodeDepsList = _get_ppl_deps(str(source[0]), env)
    if nodeDepsList:
        source += [os.path.normpath(os.path.join(env['LV_Dirs'].storageDir, str(pplNode))) for pplNode in nodeDepsList]
    return target, source

通过删除此文件，目标不再对生成的文件具有显式依赖关系（这与Decider调用无关）。

此外，从SConstruct文件中删除Decider('make')行允许删除整个源代码库并重新下载而不触发重建。

顺便说一下，Git特定的代码也被移除并放置在Builder调用的代码内部 - 这样，仅在需要重建时才会调用它（而不是每次运行SCons时都调用），同时还能减少代码量。

- chrisb2244