diff options
author | Matt A. Tobin <email@mattatobin.com> | 2022-03-25 00:01:05 -0500 |
---|---|---|
committer | Matt A. Tobin <email@mattatobin.com> | 2022-03-25 00:01:05 -0500 |
commit | 46de659d63f8485dc01175ae0e36235bd2b11e58 (patch) | |
tree | cf71ea28cafdecb502918bb1685eb255d9b51d56 /build | |
parent | 8a34848ce0f6d4006947c51227b05bec6d12bfdb (diff) | |
download | aura-central-46de659d63f8485dc01175ae0e36235bd2b11e58.tar.gz |
Rebrand the Platform Codebase
Diffstat (limited to 'build')
-rw-r--r-- | build/docs/build-overview.rst | 117 | ||||
-rw-r--r-- | build/docs/build-targets.rst | 62 | ||||
-rw-r--r-- | build/docs/defining-binaries.rst | 351 | ||||
-rw-r--r-- | build/docs/environment-variables.rst | 31 | ||||
-rw-r--r-- | build/docs/files-metadata.rst | 178 | ||||
-rw-r--r-- | build/docs/glossary.rst | 48 | ||||
-rw-r--r-- | build/docs/index.rst | 41 | ||||
-rw-r--r-- | build/docs/jar-manifests.rst | 97 | ||||
-rw-r--r-- | build/docs/locales.rst | 21 | ||||
-rw-r--r-- | build/docs/mozbuild-files.rst | 176 | ||||
-rw-r--r-- | build/docs/mozbuild-symbols.rst | 7 | ||||
-rw-r--r-- | build/docs/mozbuild/dumbmake.rst | 38 | ||||
-rw-r--r-- | build/docs/mozbuild/index.rst | 41 | ||||
-rw-r--r-- | build/docs/mozconfigs.rst | 70 | ||||
-rw-r--r-- | build/docs/mozinfo.rst | 154 | ||||
-rw-r--r-- | build/docs/pgo.rst | 40 | ||||
-rw-r--r-- | build/docs/preprocessor.rst | 244 | ||||
-rw-r--r-- | build/docs/python.rst | 178 | ||||
-rw-r--r-- | build/docs/rust.rst | 79 | ||||
-rw-r--r-- | build/docs/slow.rst | 179 | ||||
-rw-r--r-- | build/docs/supported-configurations.rst | 30 | ||||
-rw-r--r-- | build/docs/test_manifests.rst | 207 | ||||
-rw-r--r-- | build/docs/toolchains.rst | 56 |
23 files changed, 0 insertions, 2445 deletions
diff --git a/build/docs/build-overview.rst b/build/docs/build-overview.rst deleted file mode 100644 index a81531bc9..000000000 --- a/build/docs/build-overview.rst +++ /dev/null @@ -1,117 +0,0 @@ -.. _build_overview: - -===================== -Build System Overview -===================== - -This document provides an overview on how the build system works. It is -targeted at people wanting to learn about internals of the build system. -It is not meant for persons who casually interact with the build system. -That being said, knowledge empowers, so consider reading on. - -The build system is composed of many different components working in -harmony to build the source tree. We begin with a graphic overview. - -.. graphviz:: - - digraph build_components { - rankdir="LR"; - "configure" -> "config.status" -> "build backend" -> "build output" - } - -Phase 1: Configuration -====================== - -Phase 1 centers around the ``configure`` script, which is a bash shell script. -The file is generated from a file called ``configure.in`` which is written in M4 -and processed using Autoconf 2.13 to create the final configure script. -You don't have to worry about how you obtain a ``configure`` file: the build -system does this for you. - -The primary job of ``configure`` is to determine characteristics of the system -and compiler, apply options passed into it, and validate everything looks OK to -build. The primary output of the ``configure`` script is an executable file -in the object directory called ``config.status``. ``configure`` also produces -some additional files (like ``autoconf.mk``). However, the most important file -in terms of architecture is ``config.status``. - -The existence of a ``config.status`` file may be familiar to those who have worked -with Autoconf before. However, Mozilla's ``config.status`` is different from almost -any other ``config.status`` you've ever seen: it's written in Python! Instead of -having our ``configure`` script produce a shell script, we have it generating -Python. - -Now is as good a time as any to mention that Python is prevalent in our build -system. If we need to write code for the build system, we do it in Python. -That's just how we roll. For more, see :ref:`python`. - -``config.status`` contains 2 parts: data structures representing the output of -``configure`` and a command-line interface for preparing/configuring/generating -an appropriate build backend. (A build backend is merely a tool used to build -the tree - like GNU Make or Tup). These data structures essentially describe -the current state of the system and what the existing build configuration looks -like. For example, it defines which compiler to use, how to invoke it, which -application features are enabled, etc. You are encouraged to open up -``config.status`` to have a look for yourself! - -Once we have emitted a ``config.status`` file, we pass into the realm of -phase 2. - -Phase 2: Build Backend Preparation and the Build Definition -=========================================================== - -Once ``configure`` has determined what the current build configuration is, -we need to apply this to the source tree so we can actually build. - -What essentially happens is the automatically-produced ``config.status`` Python -script is executed as soon as ``configure`` has generated it. ``config.status`` -is charged with the task of tell a tool how to build the tree. To do this, -``config.status`` must first scan the build system definition. - -The build system definition consists of various ``moz.build`` files in the tree. -There is roughly one ``moz.build`` file per directory or per set of related directories. -Each ``moz.build`` files defines how its part of the build config works. For -example it says *I want these C++ files compiled* or *look for additional -information in these directories.* config.status starts with the ``moz.build`` -file from the root directory and then descends into referenced ``moz.build`` -files by following ``DIRS`` variables or similar. - -As the ``moz.build`` files are read, data structures describing the overall -build system definition are emitted. These data structures are then fed into a -build backend, which then performs actions, such as writing out files to -be read by a build tool. e.g. a ``make`` backend will write a -``Makefile``. - -When ``config.status`` runs, you'll see the following output:: - - Reticulating splines... - Finished reading 1096 moz.build files into 1276 descriptors in 2.40s - Backend executed in 2.39s - 2188 total backend files. 0 created; 1 updated; 2187 unchanged - Total wall time: 5.03s; CPU time: 3.79s; Efficiency: 75% - -What this is saying is that a total of *1096* ``moz.build`` files were read. -Altogether, *1276* data structures describing the build configuration were -derived from them. It took *2.40s* wall time to just read these files and -produce the data structures. The *1276* data structures were fed into the -build backend which then determined it had to manage *2188* files derived -from those data structures. Most of them already existed and didn't need -changed. However, *1* was updated as a result of the new configuration. -The whole process took *5.03s*. Although, only *3.79s* was in -CPU time. That likely means we spent roughly *25%* of the time waiting on -I/O. - -For more on how ``moz.build`` files work, see :ref:`mozbuild-files`. - -Phase 3: Invokation of the Build Backend -======================================== - -When most people think of the build system, they think of phase 3. This is -where we take all the code in the tree and produce Firefox or whatever -application you are creating. Phase 3 effectively takes whatever was -generated by phase 2 and runs it. Since the dawn of Mozilla, this has been -make consuming Makefiles. However, with the transition to moz.build files, -you may soon see non-Make build backends, such as Tup or Visual Studio. - -When building the tree, most of the time is spent in phase 3. This is when -header files are installed, C++ files are compiled, files are preprocessed, etc. diff --git a/build/docs/build-targets.rst b/build/docs/build-targets.rst deleted file mode 100644 index dacd46c7f..000000000 --- a/build/docs/build-targets.rst +++ /dev/null @@ -1,62 +0,0 @@ -.. _build_targets: - -============= -Build Targets -============= - -When you build with ``mach build``, there are some special targets that can be -built. This page attempts to document them. - -Partial Tree Targets -==================== - -The targets in this section only build part of the tree. Please note that -partial tree builds can be unreliable. Use at your own risk. - -export - Build the *export* tier. The *export* tier builds everything that is - required for C/C++ compilation. It stages all header files, processes - IDLs, etc. - -compile - Build the *compile* tier. The *compile* tier compiles all C/C++ files. - -libs - Build the *libs* tier. The *libs* tier performs linking and performs - most build steps which aren't related to compilation. - -tools - Build the *tools* tier. The *tools* tier mostly deals with supplementary - tools and compiled tests. It will link tools against libXUL, including - compiled test binaries. - -binaries: - Recompiles and relinks C/C++ files. Only works after a complete normal - build, but allows for much faster rebuilds of C/C++ code. For performance - reasons, however, it skips nss, nspr, icu and ffi. This is targeted to - improve local developer workflow when touching C/C++ code. - -install-manifests - Process install manifests. Install manifests handle the installation of - files into the object directory. - - Unless ``NO_REMOVE=1`` is defined in the environment, files not accounted - in the install manifests will be deleted from the object directory. - -install-tests - Processes the tests install manifest. - -Common Actions -============== - -The targets in this section correspond to common build-related actions. Many -of the actions in this section are effectively frontends to shell scripts. -These actions will likely all be replaced by mach commands someday. - -buildsymbols - Create a symbols archive for the current build. - - This must be performed after a successful build. - -check - Run build system tests. diff --git a/build/docs/defining-binaries.rst b/build/docs/defining-binaries.rst deleted file mode 100644 index 97dadbbd5..000000000 --- a/build/docs/defining-binaries.rst +++ /dev/null @@ -1,351 +0,0 @@ -.. _defining_binaries: - -====================================== -Defining Binaries for the Build System -====================================== - -One part of what the build system does is compile C/C++ and link the resulting -objects to produce executables and/or libraries. This document describes the -basics of defining what is going to be built and how. All the following -describes constructs to use in moz.build files. - - -Source files -============ - -Source files to be used in a given directory are registered in the ``SOURCES`` -and ``UNIFIED_SOURCES`` variables. ``UNIFIED_SOURCES`` have a special behavior -in that they are aggregated by batches of 16, requiring, for example, that there -are no conflicting variables in those source files. - -``SOURCES`` and ``UNIFIED_SOURCES`` are lists which must be appended to, and -each append requires the given list to be alphanumerically ordered. - -.. code-block:: python - - UNIFIED_SOURCES += [ - 'FirstSource.cpp', - 'SecondSource.cpp', - 'ThirdSource.cpp', - ] - - SOURCES += [ - 'OtherSource.cpp', - ] - -``SOURCES`` and ``UNIFIED_SOURCES`` can contain a mix of different file types, -for C, C++, and Objective C. - - -Static Libraries -================ - -To build a static library, other than defining the source files (see above), one -just needs to define a library name with the ``Library`` template. - -.. code-block:: python - - Library('foo') - -The library file name will be ``libfoo.a`` on UNIX systems and ``foo.lib`` on -Windows. - -If the static library needs to aggregate other static libraries, a list of -``Library`` names can be added to the ``USE_LIBS`` variable. Like ``SOURCES``, it -requires the appended list to be alphanumerically ordered. - -.. code-block:: python - - USE_LIBS += ['bar', 'baz'] - -If there are multiple directories containing the same ``Library`` name, it is -possible to disambiguate by prefixing with the path to the wanted one (relative -or absolute): - -.. code-block:: python - - USE_LIBS += [ - '/path/from/topsrcdir/to/bar', - '../relative/baz', - ] - -Note that the leaf name in those paths is the ``Library`` name, not an actual -file name. - -Note that currently, the build system may not create an actual library for -static libraries. It is an implementation detail that shouldn't need to be -worried about. - -As a special rule, ``USE_LIBS`` is allowed to contain references to shared -libraries. In such cases, programs and shared libraries linking this static -library will inherit those shared library dependencies. - - -Intermediate (Static) Libraries -=============================== - -In many cases in the tree, static libraries are built with the only purpose -of being linked into another, bigger one (like libxul). Instead of adding all -required libraries to ``USE_LIBS`` for the bigger one, it is possible to tell -the build system that the library built in the current directory is meant to -be linked to that bigger library, with the ``FINAL_LIBRARY`` variable. - -.. code-block:: python - - FINAL_LIBRARY = 'xul' - -The ``FINAL_LIBRARY`` value must match a unique ``Library`` name somewhere -in the tree. - -As a special rule, those intermediate libraries don't need a ``Library`` name -for themselves. - - -Shared Libraries -================ - -Sometimes, we want shared libraries, a.k.a. dynamic libraries. Such libraries -are defined similarly to static libraries, using the ``SharedLibrary`` template -instead of ``Library``. - -.. code-block:: python - - SharedLibrary('foo') - -When this template is used, no static library is built. See further below to -build both types of libraries. - -With a ``SharedLibrary`` name of ``foo``, the library file name will be -``libfoo.dylib`` on OSX, ``libfoo.so`` on ELF systems (Linux, etc.), and -``foo.dll`` on Windows. On Windows, there is also an import library named -``foo.lib``, used on the linker command line. ``libfoo.dylib`` and -``libfoo.so`` are considered the import library name for, resp. OSX and ELF -systems. - -On OSX, one may want to create a special kind of dynamic library: frameworks. -This is done with the ``Framework`` template. - -.. code-block:: python - - Framework('foo') - -With a ``Framework`` name of ``foo``, the framework file name will be ``foo``. -This template however affects the behavior on all platforms, so it needs to -be set only on OSX. - - -Executables -=========== - -Executables, a.k.a. programs, are, in the simplest form, defined with the -``Program`` template. - -.. code-block:: python - - Program('foobar') - -On UNIX systems, the executable file name will be ``foobar``, while on Windows, -it will be ``foobar.exe``. - -Like static and shared libraries, the build system can be instructed to link -libraries to the executable with ``USE_LIBS``, listing various ``Library`` -names. - -In some cases, we want to create an executable per source file in the current -directory, in which case we can use the ``SimplePrograms`` template - -.. code-block:: python - - SimplePrograms([ - 'FirstProgram', - 'SecondProgram', - ]) - -Contrary to ``Program``, which requires corresponding ``SOURCES``, when using -``SimplePrograms``, the corresponding ``SOURCES`` are implied. If the -corresponding ``sources`` have an extension different from ``.cpp``, it is -possible to specify the proper extension: - -.. code-block:: python - - SimplePrograms([ - 'ThirdProgram', - 'FourthProgram', - ], ext='.c') - -Please note this construct was added for compatibility with what already lives -in the mozilla tree ; it is recommended not to add new simple programs with -sources with a different extension than ``.cpp``. - -Similar to ``SimplePrograms``, is the ``CppUnitTests`` template, which defines, -with the same rules, C++ unit tests programs. Like ``SimplePrograms``, it takes -an ``ext`` argument to specify the extension for the corresponding ``SOURCES``, -if it's different from ``.cpp``. - - -Linking with system libraries -============================= - -Programs and libraries usually need to link with system libraries, such as a -widget toolkit, etc. Those required dependencies can be given with the -``OS_LIBS`` variable. - -.. code-block:: python - - OS_LIBS += [ - 'foo', - 'bar', - ] - -This expands to ``foo.lib bar.lib`` when building with MSVC, and -``-lfoo -lbar`` otherwise. - -For convenience with ``pkg-config``, ``OS_LIBS`` can also take linker flags -such as ``-L/some/path`` and ``-llib``, such that it is possible to directly -assign ``LIBS`` variables from ``CONFIG``, such as: - -.. code-block:: python - - OS_LIBS += CONFIG['MOZ_PANGO_LIBS'] - -(assuming ``CONFIG['MOZ_PANGO_LIBS']`` is a list, not a string) - -Like ``USE_LIBS``, this variable applies to static and shared libraries, as -well as programs. - - -Libraries from third party build system -======================================= - -Some libraries in the tree are not built by the moz.build-governed build -system, and there is no ``Library`` corresponding to them. - -However, ``USE_LIBS`` allows to reference such libraries by giving a full -path (like when disambiguating identical ``Library`` names). The same naming -rules apply as other uses of ``USE_LIBS``, so only the library name without -prefix and suffix shall be given. - -.. code-block:: python - - USE_LIBS += [ - '/path/from/topsrcdir/to/third-party/bar', - '../relative/third-party/baz', - ] - -Note that ``/path/from/topsrcdir/to/third-party`` and -``../relative/third-party/baz`` must lead under a subconfigured directory (a -directory with an AC_OUTPUT_SUBDIRS in configure.in), or ``security/nss``. - - -Building both static and shared libraries -========================================= - -When both types of libraries are required, one needs to set both -``FORCE_SHARED_LIB`` and ``FORCE_STATIC_LIB`` boolean variables. - -.. code-block:: python - - FORCE_SHARED_LIB = True - FORCE_STATIC_LIB = True - -But because static libraries and Windows import libraries have the same file -names, either the static or the shared library name needs to be different -than the name given to the ``Library`` template. - -The ``STATIC_LIBRARY_NAME`` and ``SHARED_LIBRARY_NAME`` variables can be used -to change either the static or the shared library name. - -.. code-block:: python - - Library('foo') - STATIC_LIBRARY_NAME = 'foo_s' - -With the above, on Windows, ``foo_s.lib`` will be the static library, -``foo.dll`` the shared library, and ``foo.lib`` the import library. - -In some cases, for convenience, it is possible to set both -``STATIC_LIBRARY_NAME`` and ``SHARED_LIBRARY_NAME``. For example: - -.. code-block:: python - - Library('mylib') - STATIC_LIBRARY_NAME = 'mylib_s' - SHARED_LIBRARY_NAME = CONFIG['SHARED_NAME'] - -This allows to use ``mylib`` in the ``USE_LIBS`` of another library or -executable. - -When refering to a ``Library`` name building both types of libraries in -``USE_LIBS``, the shared library is chosen to be linked. But sometimes, -it is wanted to link the static version, in which case the ``Library`` name -needs to be prefixed with ``static:`` in ``USE_LIBS`` - -:: - - a/moz.build: - Library('mylib') - FORCE_SHARED_LIB = True - FORCE_STATIC_LIB = True - STATIC_LIBRARY_NAME = 'mylib_s' - b/moz.build: - Program('myprog') - USE_LIBS += [ - 'static:mylib', - ] - - -Miscellaneous -============= - -The ``SDK_LIBRARY`` boolean variable defines whether the library in the current -directory is going to be installed in the SDK. - -The ``SONAME`` variable declares a "shared object name" for the library. It -defaults to the ``Library`` name or the ``SHARED_LIBRARY_NAME`` if set. When -linking to a library with a ``SONAME``, the resulting library or program will -have a dependency on the library with the name corresponding to the ``SONAME`` -instead of the ``Library`` name. This only impacts ELF systems. - -:: - - a/moz.build: - Library('mylib') - b/moz.build: - Library('otherlib') - SONAME = 'foo' - c/moz.build: - Program('myprog') - USE_LIBS += [ - 'mylib', - 'otherlib', - ] - -On e.g. Linux, the above ``myprog`` will have DT_NEEDED markers for -``libmylib.so`` and ``libfoo.so`` instead of ``libmylib.so`` and -``libotherlib.so`` if there weren't a ``SONAME``. This means the runtime -requirement for ``myprog`` is ``libfoo.so`` instead of ``libotherlib.so``. - - -Gecko-related binaries -====================== - -Some programs or libraries are totally independent of Gecko, and can use the -above mentioned templates. Others are Gecko-related in some way, and may -need XPCOM linkage, mozglue. These things are tedious. A set of additional -templates exists to ease defining such programs and libraries. They are -essentially the same as the above mentioned templates, prefixed with "Gecko": - - - ``GeckoProgram`` - - ``GeckoSimplePrograms`` - - ``GeckoCppUnitTests`` - - ``GeckoSharedLibrary`` - - ``GeckoFramework`` - -There is also ``XPCOMBinaryComponent`` for XPCOM components, which is a -special kind of library. - -All the Gecko-prefixed templates take the same arguments as their -non-Gecko-prefixed counterparts, and can take a few more arguments -for non-standard cases. See the definition of ``GeckoBinary`` in -build/gecko_templates.mozbuild for more details, but most usecases -should not require these additional arguments. diff --git a/build/docs/environment-variables.rst b/build/docs/environment-variables.rst deleted file mode 100644 index c46339159..000000000 --- a/build/docs/environment-variables.rst +++ /dev/null @@ -1,31 +0,0 @@ -.. _environment_variables: - -================================================ -Environment Variables Impacting the Build System -================================================ - -Various environment variables have an impact on the behavior of the -build system. This document attempts to document them. - -AUTOCLOBBER - If defines, the build system will automatically clobber as needed. - The default behavior is to print a message and error out when a - clobber is needed. - - This variable is typically defined in a :ref:`mozconfig <mozconfig>` - file via ``mk_add_options``. - -REBUILD_CHECK - If defined, the build system will print information about why - certain files were rebuilt. - - This feature is disabled by default because it makes the build slower. - -MACH_NO_TERMINAL_FOOTER - If defined, the terminal footer displayed when building with mach in - a TTY is disabled. - -MACH_NO_WRITE_TIMES - If defined, mach commands will not prefix output lines with the - elapsed time since program start. This option is equivalent to - passing ``--log-no-times`` to mach. diff --git a/build/docs/files-metadata.rst b/build/docs/files-metadata.rst deleted file mode 100644 index 5af4d96fd..000000000 --- a/build/docs/files-metadata.rst +++ /dev/null @@ -1,178 +0,0 @@ -.. _mozbuild_files_metadata: - -============== -Files Metadata -============== - -:ref:`mozbuild-files` provide a mechanism for attaching metadata to -files. Essentially, you define some flags to set on a file or file -pattern. Later, some tool or process queries for metadata attached to a -file of interest and it does something intelligent with that data. - -Defining Metadata -================= - -Files metadata is defined by using the -:ref:`Files Sub-Context <mozbuild_subcontext_Files>` in ``moz.build`` -files. e.g.:: - - with Files('**/Makefile.in'): - BUG_COMPONENT = ('Core', 'Build Config') - -This working example says, *for all Makefile.in files in every directory -underneath this one - including this directory - set the Bugzilla -component to Core :: Build Config*. - -For more info, read the -:ref:`docs on Files <mozbuild_subcontext_Files>`. - -How Metadata is Read -==================== - -``Files`` metadata is extracted in :ref:`mozbuild_fs_reading_mode`. - -Reading starts by specifying a set of files whose metadata you are -interested in. For each file, the filesystem is walked to the root -of the source directory. Any ``moz.build`` encountered during this -walking are marked as relevant to the file. - -Let's say you have the following filesystem content:: - - /moz.build - /root_file - /dir1/moz.build - /dir1/foo - /dir1/subdir1/foo - /dir2/foo - -For ``/root_file``, the relevant ``moz.build`` files are just -``/moz.build``. - -For ``/dir1/foo`` and ``/dir1/subdir1/foo``, the relevant files are -``/moz.build`` and ``/dir1/moz.build``. - -For ``/dir2``, the relevant file is just ``/moz.build``. - -Once the list of relevant ``moz.build`` files is obtained, each -``moz.build`` file is evaluated. Root ``moz.build`` file first, -leaf-most files last. This follows the rules of -:ref:`mozbuild_fs_reading_mode`, with the set of evaluated ``moz.build`` -files being controlled by filesystem content, not ``DIRS`` variables. - -The file whose metadata is being resolved maps to a set of ``moz.build`` -files which in turn evaluates to a list of contexts. For file metadata, -we only care about one of these contexts: -:ref:`Files <mozbuild_subcontext_Files>`. - -We start with an empty ``Files`` instance to represent the file. As -we encounter a *files sub-context*, we see if it is appropriate to -this file. If it is, we apply its values. This process is repeated -until all *files sub-contexts* have been applied or skipped. The final -state of the ``Files`` instance is used to represent the metadata for -this particular file. - -It may help to visualize this. Say we have 2 ``moz.build`` files:: - - # /moz.build - with Files('*.cpp'): - BUG_COMPONENT = ('Core', 'XPCOM') - - with Files('**/*.js'): - BUG_COMPONENT = ('Firefox', 'General') - - # /foo/moz.build - with Files('*.js'): - BUG_COMPONENT = ('Another', 'Component') - -Querying for metadata for the file ``/foo/test.js`` will reveal 3 -relevant ``Files`` sub-contexts. They are evaluated as follows: - -1. ``/moz.build - Files('*.cpp')``. Does ``/*.cpp`` match - ``/foo/test.js``? **No**. Ignore this context. -2. ``/moz.build - Files('**/*.js')``. Does ``/**/*.js`` match - ``/foo/test.js``? **Yes**. Apply ``BUG_COMPONENT = ('Firefox', 'General')`` - to us. -3. ``/foo/moz.build - Files('*.js')``. Does ``/foo/*.js`` match - ``/foo/test.js``? **Yes**. Apply - ``BUG_COMPONENT = ('Another', 'Component')``. - -At the end of execution, we have -``BUG_COMPONENT = ('Another', 'Component')`` as the metadata for -``/foo/test.js``. - -One way to look at file metadata is as a stack of data structures. -Each ``Files`` sub-context relevant to a given file is applied on top -of the previous state, starting from an empty state. The final state -wins. - -.. _mozbuild_files_metadata_finalizing: - -Finalizing Values -================= - -The default behavior of ``Files`` sub-context evaluation is to apply new -values on top of old. In most circumstances, this results in desired -behavior. However, there are circumstances where this may not be -desired. There is thus a mechanism to *finalize* or *freeze* values. - -Finalizing values is useful for scenarios where you want to prevent -wildcard matches from overwriting previously-set values. This is useful -for one-off files. - -Let's take ``Makefile.in`` files as an example. The build system module -policy dictates that ``Makefile.in`` files are part of the ``Build -Config`` module and should be reviewed by peers of that module. However, -there exist ``Makefile.in`` files in many directories in the source -tree. Without finalization, a ``*`` or ``**`` wildcard matching rule -would match ``Makefile.in`` files and overwrite their metadata. - -Finalizing of values is performed by setting the ``FINAL`` variable -on ``Files`` sub-contexts. See the -:ref:`Files documentation <mozbuild_subcontext_Files>` for more. - -Here is an example with ``Makefile.in`` files, showing how it is -possible to finalize the ``BUG_COMPONENT`` value.:: - - # /moz.build - with Files('**/Makefile.in'): - BUG_COMPONENT = ('Core', 'Build Config') - FINAL = True - - # /foo/moz.build - with Files('**'): - BUG_COMPONENT = ('Another', 'Component') - -If we query for metadata of ``/foo/Makefile.in``, both ``Files`` -sub-contexts match the file pattern. However, since ``BUG_COMPONENT`` is -marked as finalized by ``/moz.build``, the assignment from -``/foo/moz.build`` is ignored. The final value for ``BUG_COMPONENT`` -is ``('Core', 'Build Config')``. - -Here is another example:: - - with Files('*.cpp'): - BUG_COMPONENT = ('One-Off', 'For C++') - FINAL = True - - with Files('**'): - BUG_COMPONENT = ('Regular', 'Component') - -For every files except ``foo.cpp``, the bug component will be resolved -as ``Regular :: Component``. However, ``foo.cpp`` has its value of -``One-Off :: For C++`` preserved because it is finalized. - -.. important:: - - ``FINAL`` only applied to variables defined in a context. - - If you want to mark one variable as finalized but want to leave - another mutable, you'll need to use 2 ``Files`` contexts. - -Guidelines for Defining Metadata -================================ - -In general, values defined towards the root of the source tree are -generic and become more specific towards the leaves. For example, -the ``BUG_COMPONENT`` for ``/browser`` might be ``Firefox :: General`` -whereas ``/browser/components/preferences`` would list -``Firefox :: Preferences``. diff --git a/build/docs/glossary.rst b/build/docs/glossary.rst deleted file mode 100644 index f846effe5..000000000 --- a/build/docs/glossary.rst +++ /dev/null @@ -1,48 +0,0 @@ -======== -Glossary -======== - -.. glossary:: - :sorted: - - object directory - A directory holding the output of the build system. The build - system attempts to isolate all file modifications to this - directory. By convention, object directories are commonly - directories under the source directory prefixed with **obj-**. - e.g. **obj-firefox**. - - mozconfig - A shell script used to configure the build system. - - configure - A generated shell script which detects the current system - environment, applies a requested set of build configuration - options, and writes out metadata to be consumed by the build - system. - - config.status - An executable file produced by **configure** that takes the - generated build config and writes out files used to build the - tree. Traditionally, config.status writes out a bunch of - Makefiles. - - install manifest - A file containing metadata describing file installation rules. - A large part of the build system consists of copying files - around to appropriate places. We write out special files - describing the set of required operations so we can process the - actions effeciently. These files are install manifests. - - clobber build - A build performed with an initially empty object directory. All - build actions must be performed. - - incremental build - A build performed with the result of a previous build in an - object directory. The build should not have to work as hard because - it will be able to reuse the work from previous builds. - - mozinfo - An API for accessing a common and limited subset of the build and - run-time configuration. See :ref:`mozinfo`. diff --git a/build/docs/index.rst b/build/docs/index.rst deleted file mode 100644 index fa1434b85..000000000 --- a/build/docs/index.rst +++ /dev/null @@ -1,41 +0,0 @@ -============ -Build System -============ - -Important Concepts -================== -.. toctree:: - :maxdepth: 1 - - glossary - build-overview - supported-configurations - Mozconfig Files <mozconfigs> - mozbuild-files - mozbuild-symbols - files-metadata - Profile Guided Optimization <pgo> - slow - environment-variables - build-targets - python - test_manifests - mozinfo - preprocessor - jar-manifests - defining-binaries - toolchains - locales - rust - -mozbuild -======== - -mozbuild is a Python package containing a lot of the code for the -Mozilla build system. - -.. toctree:: - :maxdepth: 1 - - mozbuild/index - mozbuild/dumbmake diff --git a/build/docs/jar-manifests.rst b/build/docs/jar-manifests.rst deleted file mode 100644 index 89776a101..000000000 --- a/build/docs/jar-manifests.rst +++ /dev/null @@ -1,97 +0,0 @@ -.. _jar_manifests: - -============= -JAR Manifests -============= - -JAR Manifests are plaintext files in the tree that are used to package chrome -files into the correct JARs, and create -`Chrome Registration <https://developer.mozilla.org/en-US/docs/Chrome_Registration>`_ -manifests. JAR Manifests are commonly named ``jar.mn``. They are -declared in ``moz.build`` files using the ``JAR_MANIFESTS`` variable. - -``jar.mn`` files are automatically processed by the build system when building a -source directory that contains one. The ``jar``.mn is run through the -:ref:`preprocessor` before being passed to the manifest processor. In order to -have ``@variables@`` expanded (such as ``@AB_CD@``) throughout the file, add -the line ``#filter substitution`` at the top of your ``jar.mn`` file. - -The format of a jar.mn is fairly simple; it consists of a heading specifying -which JAR file is being packaged, followed by indented lines listing files and -chrome registration instructions. - -To see a simple ``jar.mn`` file at work, see ``toolkit/profile/jar.mn``. A much -more complex ``jar.mn`` is at ``toolkit/locales/jar.mn``. - -Shipping Chrome Files -===================== - -To ship chrome files in a JAR, an indented line indicates a file to be packaged:: - - <jarfile>.jar: - path/in/jar/file_name.xul (source/tree/location/file_name.xul) - -The JAR location may be preceded with a base path between square brackets:: - [base/path] <jarfile>.jar: - path/in/jar/file_name.xul (source/tree/location/file_name.xul) - -In this case, the jar will be directly located under the given ``base/bath``, -while without a base path, it will be under a ``chrome`` directory. - -If the JAR manifest and packaged file live in the same directory, the path and -parenthesis can be omitted. In other words, the following two lines are -equivalent:: - - path/in/jar/same_place.xhtml (same_place.xhtml) - path/in/jar/same_place.xhtml - -The source tree location may also be an *absolute* path (taken from the -top of the source tree:: - - path/in/jar/file_name.xul (/path/in/sourcetree/file_name.xul) - -An asterisk marker (``*``) at the beginning of the line indicates that the -file should be processed by the :ref:`preprocessor` before being packaged:: - - * path/in/jar/preprocessed.xul (source/tree/location/file_name.xul) - -Preprocessed files always replace existing files, to ensure that changes in -``#expand`` or ``#include`` directives are picked up. - -There is a special source-directory format for localized files (note the -percent sign in the source file location): this format reads ``localized.dtd`` -from the ``en-US`` directory if building an English version, and reads the -file from the alternate localization source tree -``/l10n/<locale>/path/localized.dtd`` if building a localized version:: - - locale/path/localized.dtd (%localized/path/localized.dtd) - -The source tree location can also use wildcards, in which case the path in -jar is expected to be a base directory. Paths before the wildcard are not -made part of the destination path:: - - path/in/jar/ (source/tree/location/*.xul) - -The above will install all xul files under ``source/tree/location`` as -``path/in/jar/*.xul``. - -Register Chrome -=============== - -`Chrome Registration <https://developer.mozilla.org/en-US/docs/Chrome_Registration>`_ -instructions are marked with a percent sign (``%``) at the beginning of the -line, and must be part of the definition of a JAR file. Any additional percents -signs are replaced with an appropriate relative URL of the JAR file being -packaged:: - - % content global %path/in/jar/ - % overlay chrome://blah/content/blah.xul chrome://foo/content/overlay.xul - -There are two possible locations for a manifest file. If the chrome is being -built into a standalone application, the ``jar.mn`` processor creates a -``<jarfilename>.manifest`` next to the JAR file itself. This is the default -behavior. - -If the build specifies ``USE_EXTENSION_MANIFEST = 1``, the ``jar.mn`` processor -creates a single ``chrome.manifest`` file suitable for registering chrome as -an extension. diff --git a/build/docs/locales.rst b/build/docs/locales.rst deleted file mode 100644 index 86c740218..000000000 --- a/build/docs/locales.rst +++ /dev/null @@ -1,21 +0,0 @@ -.. _localization: - -=================== -Localization (l10n) -=================== - -Single-locale language repacks -============================== - -To save on build time, the build system and automation collaborate to allow -downloading a packaged en-US Firefox, performing some locale-specific -post-processing, and re-packaging a locale-specific Firefox. Such artifacts -are termed "single-locale language repacks". There is another concept of a -"multi-locale language build", which is more like a regular build and less -like a re-packaging post-processing step. - -There are scripts in-tree in mozharness to orchestrate these re-packaging -steps for `Desktop -<https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/scripts/desktop_l10n.py>`_ -but they rely heavily on buildbot information so they are almost impossible to -run locally. diff --git a/build/docs/mozbuild-files.rst b/build/docs/mozbuild-files.rst deleted file mode 100644 index 3550c51ef..000000000 --- a/build/docs/mozbuild-files.rst +++ /dev/null @@ -1,176 +0,0 @@ -.. _mozbuild-files: - -=============== -moz.build Files -=============== - -``moz.build`` files are the mechanism by which tree metadata (notably -the build configuration) is defined. - -Directories in the tree contain ``moz.build`` files which declare -functionality for their respective part of the tree. This includes -things such as the list of C++ files to compile, where to find tests, -etc. - -``moz.build`` files are actually Python scripts. However, their -execution is governed by special rules. This is explained below. - -moz.build Python Sandbox -======================== - -As mentioned above, ``moz.build`` files are Python scripts. However, -they are executed in a special Python *sandbox* that significantly -changes and limits the execution environment. The environment is so -different, it's doubtful most ``moz.build`` files would execute without -error if executed by a vanilla Python interpreter (e.g. ``python -moz.build``. - -The following properties make execution of ``moz.build`` files special: - -1. The execution environment exposes a limited subset of Python. -2. There is a special set of global symbols and an enforced naming - convention of symbols. -3. Some symbols are inherited from previously-executed ``moz.build`` - files. - -The limited subset of Python is actually an extremely limited subset. -Only a few symbols from ``__builtins__`` are exposed. These include -``True``, ``False``, and ``None``. Global functions like ``import``, -``print``, and ``open`` aren't available. Without these, ``moz.build`` -files can do very little. *This is by design*. - -The execution sandbox treats all ``UPPERCASE`` variables specially. Any -``UPPERCASE`` variable must be known to the sandbox before the script -executes. Any attempt to read or write to an unknown ``UPPERCASE`` -variable will result in an exception being raised. Furthermore, the -types of all ``UPPERCASE`` variables is strictly enforced. Attempts to -assign an incompatible type to an ``UPPERCASE`` variable will result in -an exception being raised. - -The strictness of behavior with ``UPPERCASE`` variables is a very -intentional design decision. By ensuring strict behavior, any operation -involving an ``UPPERCASE`` variable is guaranteed to have well-defined -side-effects. Previously, when the build configuration was defined in -``Makefiles``, assignments to variables that did nothing would go -unnoticed. ``moz.build`` files fix this problem by eliminating the -potential for false promises. - -After a ``moz.build`` file has completed execution, only the -``UPPERCASE`` variables are used to retrieve state. - -The set of variables and functions available to the Python sandbox is -defined by the :py:mod:`mozbuild.frontend.context` module. The -data structures in this module are consumed by the -:py:class:`mozbuild.frontend.reader.MozbuildSandbox` class to construct -the sandbox. There are tests to ensure that the set of symbols exposed -to an empty sandbox are all defined in the ``context`` module. -This module also contains documentation for each symbol, so nothing can -sneak into the sandbox without being explicitly defined and documented. - -Reading and Traversing moz.build Files -====================================== - -The process for reading ``moz.build`` files roughly consists of: - -1. Start at the root ``moz.build`` (``<topsrcdir>/moz.build``). -2. Evaluate the ``moz.build`` file in a new sandbox. -3. Emit the main *context* and any *sub-contexts* from the executed - sandbox. -4. Extract a set of ``moz.build`` files to execute next. -5. For each additional ``moz.build`` file, goto #2 and repeat until all - referenced files have executed. - -From the perspective of the consumer, the output of reading is a stream -of :py:class:`mozbuild.frontend.reader.context.Context` instances. Each -``Context`` defines a particular aspect of data. Consumers iterate over -these objects and do something with the data inside. Each object is -essentially a dictionary of all the ``UPPERCASE`` variables populated -during its execution. - -.. note:: - - Historically, there was only one ``context`` per ``moz.build`` file. - As the number of things tracked by ``moz.build`` files grew and more - and more complex processing was desired, it was necessary to split these - contexts into multiple logical parts. It is now common to emit - multiple contexts per ``moz.build`` file. - -Build System Reading Mode -------------------------- - -The traditional mode of evaluation of ``moz.build`` files is what's -called *build system traversal mode.* In this mode, the ``CONFIG`` -variable in each ``moz.build`` sandbox is populated from data coming -from ``config.status``, which is produced by ``configure``. - -During evaluation, ``moz.build`` files often make decisions conditional -on the state of the build configuration. e.g. *only compile foo.cpp if -feature X is enabled*. - -In this mode, traversal of ``moz.build`` files is governed by variables -like ``DIRS`` and ``TEST_DIRS``. For example, to execute a child -directory, ``foo``, you would add ``DIRS += ['foo']`` to a ``moz.build`` -file and ``foo/moz.build`` would be evaluated. - -.. _mozbuild_fs_reading_mode: - -Filesystem Reading Mode ------------------------ - -There is an alternative reading mode that doesn't involve the build -system and doesn't use ``DIRS`` variables to control traversal into -child directories. This mode is called *filesystem reading mode*. - -In this reading mode, the ``CONFIG`` variable is a dummy, mostly empty -object. Accessing all but a few special variables will return an empty -value. This means that nearly all ``if CONFIG['FOO']:`` branches will -not be taken. - -Instead of using content from within the evaluated ``moz.build`` -file to drive traversal into subsequent ``moz.build`` files, the set -of files to evaluate is controlled by the thing doing the reading. - -A single ``moz.build`` file is not guaranteed to be executable in -isolation. Instead, we must evaluate all *parent* ``moz.build`` files -first. For example, in order to evaluate ``/foo/moz.build``, one must -execute ``/moz.build`` and have its state influence the execution of -``/foo/moz.build``. - -Filesystem reading mode is utilized to power the -:ref:`mozbuild_files_metadata` feature. - -Technical Details ------------------ - -The code for reading ``moz.build`` files lives in -:py:mod:`mozbuild.frontend.reader`. The Python sandboxes evaluation results -(:py:class:`mozbuild.frontend.context.Context`) are passed into -:py:mod:`mozbuild.frontend.emitter`, which converts them to classes defined -in :py:mod:`mozbuild.frontend.data`. Each class in this module defines a -domain-specific component of tree metdata. e.g. there will be separate -classes that represent a JavaScript file vs a compiled C++ file or test -manifests. This means downstream consumers of this data can filter on class -types to only consume what they are interested in. - -There is no well-defined mapping between ``moz.build`` file instances -and the number of :py:mod:`mozbuild.frontend.data` classes derived from -each. Depending on the content of the ``moz.build`` file, there may be 1 -object derived or 100. - -The purpose of the ``emitter`` layer between low-level sandbox execution -and metadata representation is to facilitate a unified normalization and -verification step. There are multiple downstream consumers of the -``moz.build``-derived data and many will perform the same actions. This -logic can be complicated, so we have a component dedicated to it. - -:py:class:`mozbuild.frontend.reader.BuildReader`` and -:py:class:`mozbuild.frontend.reader.TreeMetadataEmitter`` have a -stream-based API courtesy of generators. When you hook them up properly, -the :py:mod:`mozbuild.frontend.data` classes are emitted before all -``moz.build`` files have been read. This means that downstream errors -are raised soon after sandbox execution. - -Lots of the code for evaluating Python sandboxes is applicable to -non-Mozilla systems. In theory, it could be extracted into a standalone -and generic package. However, until there is a need, there will -likely be some tightly coupled bits. diff --git a/build/docs/mozbuild-symbols.rst b/build/docs/mozbuild-symbols.rst deleted file mode 100644 index 4e9a8853a..000000000 --- a/build/docs/mozbuild-symbols.rst +++ /dev/null @@ -1,7 +0,0 @@ -.. _mozbuild_symbols: - -======================== -mozbuild Sandbox Symbols -======================== - -.. mozbuildsymbols:: mozbuild.frontend.context diff --git a/build/docs/mozbuild/dumbmake.rst b/build/docs/mozbuild/dumbmake.rst deleted file mode 100644 index 7840695ad..000000000 --- a/build/docs/mozbuild/dumbmake.rst +++ /dev/null @@ -1,38 +0,0 @@ -dumbmake -======== - -*dumbmake* is a simple dependency tracker for make. - -It turns lists of make targets into longer lists of make targets that -include dependencies. For example: - - netwerk, package - -might be turned into - - netwerk, netwerk/build, toolkit/library, package - -The dependency list is read from the plain text file -`topsrcdir/build/dumbmake-dependencies`. The format best described by -example: - - build_this - when_this_changes - -Interpret this to mean that `build_this` is a dependency of -`when_this_changes`. More formally, a line (CHILD) indented more than -the preceding line (PARENT) means that CHILD should trigger building -PARENT. That is, building CHILD will trigger building first CHILD and -then PARENT. - -This structure is recursive: - - build_this_when_either_change - build_this_only_when - this_changes - -This means that `build_this_when_either_change` is a dependency of -`build_this_only_when` and `this_changes`, and `build_this_only_when` -is a dependency of `this_changes`. Building `this_changes` will build -first `this_changes`, then `build_this_only_when`, and finally -`build_this_when_either_change`. diff --git a/build/docs/mozbuild/index.rst b/build/docs/mozbuild/index.rst deleted file mode 100644 index 86f38940b..000000000 --- a/build/docs/mozbuild/index.rst +++ /dev/null @@ -1,41 +0,0 @@ -======== -mozbuild -======== - -mozbuild is a Python package providing functionality used by Mozilla's -build system. - -Modules Overview -================ - -* mozbuild.backend -- Functionality for producing and interacting with build - backends. A build backend is an entity that consumes build system metadata - (from mozbuild.frontend) and does something useful with it (typically writing - out files that can be used by a build tool to build the tree). -* mozbuild.compilation -- Functionality related to compiling. This - includes managing compiler warnings. -* mozbuild.frontend -- Functionality for reading build frontend files - (what defines the build system) and converting them to data structures - which are fed into build backends to produce backend configurations. -* mozpack -- Functionality related to packaging builds. - -Overview -======== - -The build system consists of frontend files that define what to do. They -say things like "compile X" "copy Y." - -The mozbuild.frontend package contains code for reading these frontend -files and converting them to static data structures. The set of produced -static data structures for the tree constitute the current build -configuration. - -There exist entities called build backends. From a high level, build -backends consume the build configuration and do something with it. They -typically produce tool-specific files such as make files which can be used -to build the tree. - -Piecing it all together, we have frontend files that are parsed into data -structures. These data structures are fed into a build backend. The output -from build backends is used by builders to build the tree. - diff --git a/build/docs/mozconfigs.rst b/build/docs/mozconfigs.rst deleted file mode 100644 index 1befca9ec..000000000 --- a/build/docs/mozconfigs.rst +++ /dev/null @@ -1,70 +0,0 @@ -.. _mozconfig: - -=============== -mozconfig Files -=============== - -mozconfig files are used to configure how a build works. - -mozconfig files are actually shell scripts. They are executed in a -special context with specific variables and functions exposed to them. - -API -=== - -Functions ---------- - -The following special functions are available to a mozconfig script. - -ac_add_options -^^^^^^^^^^^^^^ - -This function is used to declare extra options/arguments to pass into -configure. - -e.g.:: - - ac_add_options --disable-tests - ac_add_options --enable-optimize - -mk_add_options -^^^^^^^^^^^^^^ - -This function is used to inject statements into client.mk for execution. -It is typically used to define variables, notably the object directory. - -e.g.:: - - mk_add_options AUTOCLOBBER=1 - -ac_add_options -^^^^^^^^^^^^^^ - -This is a variant of ac_add_options() which only adds configure options -for a specified application. This is only used when building multiple -applications through client.mk. This function is typically not needed. - -Special mk_add_options Variables --------------------------------- - -For historical reasons, the method for communicating certain -well-defined variables is via mk_add_options(). In this section, we -document what those special variables are. - -MOZ_OBJDIR -^^^^^^^^^^ - -This variable is used to define the :term:`object directory` for the current -build. - -Finding the active mozconfig -============================ - -Multiple mozconfig files can exist to provide different configuration -options for different tasks. The rules for finding the active mozconfig -are defined in the -:py:func:`mozbuild.mozconfig.MozconfigLoader.find_mozconfig` method: - -.. autoclass:: mozbuild.mozconfig.MozconfigLoader - :members: find_mozconfig diff --git a/build/docs/mozinfo.rst b/build/docs/mozinfo.rst deleted file mode 100644 index 745476bef..000000000 --- a/build/docs/mozinfo.rst +++ /dev/null @@ -1,154 +0,0 @@ -.. _mozinfo: - -======= -mozinfo -======= - -``mozinfo`` is a solution for representing a subset of build -configuration and run-time data. - -``mozinfo`` data is typically accessed through a ``mozinfo.json`` file -which is written to the :term:`object directory` during build -configuration. The code for writing this file lives in -:py:mod:`mozbuild.mozinfo`. - -``mozinfo.json`` is an object/dictionary of simple string values. - -The attributes in ``mozinfo.json`` are used for many purposes. One use -is to filter tests for applicability to the current build. For more on -this, see :ref:`test_manifests`. - -.. _mozinfo_attributes: - -mozinfo.json Attributes -================================= - -``mozinfo`` currently records the following attributes. - -appname - The application being built. - - Value comes from ``MOZ_APP_NAME`` from ``config.status``. - - Optional. - -asan - Whether address sanitization is enabled. - - Values are ``true`` and ``false``. - - Always defined. - -bin_suffix - The file suffix for binaries produced with this build. - - Values may be an empty string, as not all platforms have a binary - suffix. - - Always defined. - -bits - The number of bits in the CPU this build targets. - - Values are typically ``32`` or ``64``. - - Universal Mac builds do not have this key defined. - - Unkown processor architectures (see ``processor`` below) may not have - this key defined. - - Optional. - -buildapp - The path to the XUL application being built. - - For desktop Firefox, this is ``browser``. - -crashreporter - Whether the crash reporter is enabled for this build. - - Values are ``true`` and ``false``. - - Always defined. - -debug - Whether this is a debug build. - - Values are ``true`` and ``false``. - - Always defined. - -mozconfig - The path of the :ref:`mozconfig file <mozconfig>` used to produce this build. - - Optional. - -nightly_build - Whether this is a nightly build. - - Values are ``true`` and ``false``. - - Always defined. - -os - The operating system the build is produced for. Values for tier-1 - supported platforms are ``linux`` and ``win``. For other platforms, - the value is the lowercase version of the ``OS_TARGET`` variable - from ``config.status``. - - Always defined. - -processor - Information about the processor architecture this build targets. - - Values come from ``TARGET_CPU``, however some massaging may be - performed. - - If the build is a universal build on Mac (it targets both 32-bit and - 64-bit), the value is ``universal-x86-x86_64``. - - If the value starts with ``arm``, the value is ``arm``. - - If the value starts with a string of the form ``i[3-9]86]``, the - value is ``x86``. - - Always defined. - -release_or_beta - Whether this is a release or beta build. - - Values are ``true`` and ``false``. - - Always defined. - -tests_enabled - Whether tests are enabled for this build. - - Values are ``true`` and ``false``. - - Always defined. - -toolkit - The widget toolkit in case. The value comes from the - ``MOZ_WIDGET_TOOLKIT`` ``config.status`` variable. - - Always defined. - -topsrcdir - The path to the source directory the build came from. - - Always defined. - -wave - Whether Wave audio support is enabled. - - Values are ``true`` and ``false``. - - Always defined. - -webm - Whether WebM support is enabled. - - Values are ``true`` and ``false``. - - Always defined. diff --git a/build/docs/pgo.rst b/build/docs/pgo.rst deleted file mode 100644 index 51d03c625..000000000 --- a/build/docs/pgo.rst +++ /dev/null @@ -1,40 +0,0 @@ -.. _pgo: - -=========================== -Profile Guided Optimization -=========================== - -:abbr:`PGO (Profile Guided Optimization)` is the process of adding -probes to a compiled binary, running said binary, then using the -run-time information to *recompile* the binary to (hopefully) make it -faster. - -How PGO Builds Work -=================== - -The supported interface for invoking a PGO build is to evaluate the -*build* target of client.mk with *MOZ_PGO* defined. e.g.:: - - $ make -f client.mk MOZ_PGO=1 - -This is equivalent to:: - - $ make -f client.mk profiledbuild - -Which is roughly equivalent to: - -#. Perform a build with *MOZ_PROFILE_GENERATE=1* and *MOZ_PGO_INSTRUMENTED=1* -#. Package with *MOZ_PGO_INSTRUMENTED=1* -#. Performing a run of the instrumented binaries -#. $ make maybe_clobber_profiledbuild -#. Perform a build with *MOZ_PROFILE_USE=1* - -Differences between toolchains -============================== - -There are some implementation differences depending on the compiler -toolchain being used. - -The *maybe_clobber_profiledbuild* step gets its name because of a -difference. On Windows, this step merely moves some *.pgc* files around. -Using GCC or Clang, it is equivalent to a *make clean*. diff --git a/build/docs/preprocessor.rst b/build/docs/preprocessor.rst deleted file mode 100644 index ab2d8ecf7..000000000 --- a/build/docs/preprocessor.rst +++ /dev/null @@ -1,244 +0,0 @@ -.. _preprocessor: - -================= -Text Preprocessor -================= - -The build system contains a text preprocessor similar to the C preprocessor, -meant for processing files which have no built-in preprocessor such as XUL -and JavaScript documents. It is implemented at ``python/mozbuild/mozbuild/preprocessor.py`` and -is typically invoked via :ref:`jar_manifests`. - -While used to preprocess CSS files, the directives are changed to begin with -``%`` instead of ``#`` to avoid conflict of the id selectors. - -Directives -========== - -Variable Definition -------------------- - -define -^^^^^^ - -:: - - #define variable - #define variable value - -Defines a preprocessor variable. - -Note that, unlike the C preprocessor, instances of this variable later in the -source are not automatically replaced (see #filter). If value is not supplied, -it defaults to ``1``. - -Note that whitespace is significant, so ``"#define foo one"`` and -``"#define foo one "`` is different (in the second case, ``foo`` is defined to -be a four-character string). - -undef -^^^^^ - -:: - - #undef variable - -Undefines a preprocessor variable. - -Conditionals ------------- - -if -^^ - -:: - - #if variable - #if !variable - #if variable==string - #if variable!=string - -Disables output if the conditional is false. This can be nested to arbitrary -depths. Note that in the equality checks, the variable must come first, and -the comparison operator must not be surrounded by any whitespace. - -else -^^^^ - -:: - - #else - -Reverses the state of the previous conditional block; for example, if the -last ``#if`` was true (output was enabled), an ``#else`` makes it off -(output gets disabled). - -.. warning:: An ``#else`` is relative to the last conditional block only, - unlike the C preprocessor. - - It does not matter whether any blocks before it were true. This behavior - changed on trunk (Gecko 1.9) on 2006-12-07; see Bug 277122 for details. - -:: - - #if 1 - always included - #elif 1 - never included - #else - always included - #endif - -endif -^^^^^ - -:: - - #endif - -Ends the conditional block. - -ifdef / ifndef -^^^^^^^^^^^^^^ - -:: - - #ifdef variable - #ifndef variable - -An ``#if`` conditional that is true only if the preprocessor variable -variable is defined (in the case of ``ifdef``) or not defined (``ifndef``). - -elif / elifdef / elifndef -^^^^^^^^^^^^^^^^^^^^^^^^^ - -:: - - #elif variable - #elif !variable - #elif variable == string - #elif variable != string - #elifdef variable - #elifndef variable - -A shorthand to mean an ``#else`` combined with the relevant conditional. -The following two blocks are equivalent:: - - #ifdef foo - block 1 - #elifdef bar - block 2 - #endif - -:: - - #ifdef foo - block 1 - #else - #ifdef bar - block 2 - #endif - #endif - -.. warning:: An ``#elif``, ``#elifdef``, or ``#elifndef`` is relative to - the last conditional block only (as well as the condition it implies), - unlike the C preprocessor. It does not matter whether any blocks before - it were true. This behavior changed on trunk (Gecko 1.9) on 2006-12-07. - See Bug 277122 for details. - -File Inclusion --------------- - -include -^^^^^^^ - -:: - - #include filename - -The file specified by filename is processed as if the contents was placed -at this position. This also means that preprocessor conditionals can even -be started in one file and ended in another (but is highly discouraged). -There is no limit on depth of inclusion, or repeated inclusion of the same -file, or self inclusion; thus, care should be taken to avoid infinite loops. - -includesubst -^^^^^^^^^^^^ - -:: - - #includesubst @variable@filename - -Same as a ``#include`` except that all instances of variable in the included -file is also expanded as in ``#filter`` substitution - -expand -^^^^^^ - -:: - - #expand string - -All variables wrapped in ``__`` are replaced with their value, for this line -only. If the variable is not defined, it expands to an empty string. For -example, if ``foo`` has the value ``bar``, and ``baz`` is not defined, then:: - - #expand This <__foo__> <__baz__> gets expanded - -Is expanded to:: - - This <bar> <> gets expanded - -filter / unfilter -^^^^^^^^^^^^^^^^^ - -:: - - #filter filter1 filter2 ... filterN - #unfilter filter1 filter2 ... filterN - -``#filter`` turns on the given filter. - -Filters are run in alphabetical order on a per-line basis. - -``#unfilter`` turns off the given filter. Available filters are: - -emptyLines - strips blank lines from the output -slashslash - strips everything from the first two consecutive slash (``/``) - characters until the end of the line -spaces - collapses consecutive sequences of spaces into a single space, - and strips leading and trailing spaces -substitution - all variables wrapped in @ are replaced with their value. If the - variable is not defined, it is a fatal error. Similar to ``#expand`` - and ``#filter`` -attemptSubstitution - all variables wrapped in ``@`` are replaced with their value, or an - empty string if the variable is not defined. Similar to ``#expand``. - -literal -^^^^^^^ - -:: - - #literal string - -Output the string (i.e. the rest of the line) literally, with no other fixups. -This is useful to output lines starting with ``#``, or to temporarily -disable filters. - -Other ------ - -#error -^^^^^^ - -:: - - #error string - -Cause a fatal error at this point, with the error message being the -given string. diff --git a/build/docs/python.rst b/build/docs/python.rst deleted file mode 100644 index 098541706..000000000 --- a/build/docs/python.rst +++ /dev/null @@ -1,178 +0,0 @@ -.. _python: - -=========================== -Python and the Build System -=========================== - -The Python programming language is used significantly in the build -system. If we need to write code for the build system or for a tool -related to the build system, Python is typically the first choice. - -Python Requirements -=================== - -The tree requires Python 2.7.3 or greater but not Python 3 to build. -All Python packages not in the Python distribution are included in the -source tree. So all you should need is a vanilla Python install and you -should be good to go. - -Only CPython (the Python distribution available from www.python.org) is -supported. - -We require Python 2.7.3 (and not say 2.7.2) to build because Python -2.7.3 contains numerous bug fixes, especially around the area of Unicode -handling. These bug fixes are extremely annoying and have to be worked -around. The build maintainers were tired of doing this, so the minimum -version requirement was upped (bug 870420). - -We intend to eventually support Python 3. This will come by way of dual -2.7/3.x compatibility because a single flag day conversion to 3.x will -be too cumbersome given the amount of Python that would need converted. -We will not know which 3.x minor release we are targeting until this -effort is underway. This is tracked in bug 636155. - -Compiled Python Packages -======================== - -There are some features of the build that rely on compiled Python packages -(packages containing C source). These features are currently all -optional because not every system contains the Python development -headers required to build these extensions. - -We recommend you have the Python development headers installed (``mach -bootstrap`` should do this for you) so you can take advantage of these -features. - -Issues with OS X System Python -============================== - -The Python that ships with OS X has historically been littered with -subtle bugs and suboptimalities. Furthermore, OS X up through 10.8 don't -ship with Python 2.7.3 (10.8 ships with 2.7.2). - -OS X 10.8 and below users will be required to install a new Python -distribution. This may not be necessary for OS X 10.9+. However, we -still recommend installing a separate Python because of the history with -OS X's system Python issues. - -We recommend installing Python through Homebrew or MacPorts. If you run -``mach bootstrap``, this should be done for you. - -Virtualenvs -=========== - -The build system relies heavily on -`virtualenvs <http://www.virtualenv.org/en/latest/>`_. Virtualenvs are -standalone and isolated Python environments. The problem a virtualenv -solves is that of dependencies across multiple Python components. If two -components on a system relied on different versions of a package, there -could be a conflict. Instead of managing multiple versions of a package -simultaneously, Python and virtualenvs take the route that it is easier -to just keep them separate so there is no potential for conflicts. - -Very early in the build process, a virtualenv is created inside the -:term:`object directory`. The virtualenv is configured such that it can -find all the Python packages in the source tree. The code for this lives -in :py:mod:`mozbuild.virtualenv`. - -Deficiencies ------------- - -There are numerous deficiencies with the way virtualenvs are handled in -the build system. - -* mach reinvents the virtualenv. - - There is code in ``build/mach_bootstrap.py`` that configures ``sys.path`` - much the same way the virtualenv does. There are various bugs tracking - this. However, no clear solution has yet been devised. It's not a huge - problem and thus not a huge priority. - -* They aren't preserved across copies and packaging. - - If you attempt to copy an entire tree from one machine to another or - from one directory to another, chances are the virtualenv will fall - apart. It would be nice if we could preserve it somehow. Instead of - actually solving portable virtualenvs, all we really need to solve is - encapsulating the logic for populating the virtualenv along with all - dependent files in the appropriate place. - -* .pyc files written to source directory. - - We rely heavily on ``.pth`` files in our virtualenv. A ``.pth`` file - is a special file that contains a list of paths. Python will take the - set of listed paths encountered in ``.pth`` files and add them to - ``sys.path``. - - When Python compiles a ``.py`` file to bytecode, it writes out a - ``.pyc`` file so it doesn't have to perform this compilation again. - It puts these ``.pyc`` files alongside the ``.pyc`` file. Python - provides very little control for determing where these ``.pyc`` files - go, even in Python 3 (which offers customer importers). - - With ``.pth`` files pointing back to directories in the source tree - and not the object directory, ``.pyc`` files are created in the source - tree. This is bad because when Python imports a module, it first looks - for a ``.pyc`` file before the ``.py`` file. If there is a ``.pyc`` - file but no ``.py`` file, it will happily import the module. This - wreaks havoc during file moves, refactoring, etc. - - There are various proposals for fixing this. See bug 795995. - -Installing Python Manually -========================== - -We highly recommend you use your system's package manager or a -well-supported 3rd party package manager to install Python for you. If -these are not available to you, we recommend the following tools for -installing Python: - -* `buildout.python <https://github.com/collective/buildout.python>`_ -* `pyenv <https://github.com/yyuu/pyenv>`_ -* An official installer from http://www.python.org. - -If all else fails, consider compiling Python from source manually. But this -should be viewed as the least desirable option. - -Common Issues with Python -========================= - -Upgrading your Python distribution breaks the virtualenv --------------------------------------------------------- - -If you upgrade the Python distribution (e.g. install Python 2.7.5 -from 2.7.3, chances are parts of the virtualenv will break. -This commonly manifests as a cryptic ``Cannot import XXX`` exception. -More often than not, the module being imported contains binary/compiled -components. - -If you upgrade or reinstall your Python distribution, we recommend -clobbering your build. - -Packages installed at the system level conflict with build system's -------------------------------------------------------------------- - -It is common for people to install Python packages using ``sudo`` (e.g. -``sudo pip install psutil``) or with the system's package manager -(e.g. ``apt-get install python-mysql``. - -A problem with this is that packages installed at the system level may -conflict with the package provided by the source tree. As of bug 907902 -and changeset f18eae7c3b27 (September 16, 2013), this should no longer -be an issue since the virtualenv created as part of the build doesn't -add the system's ``site-packages`` directory to ``sys.path``. However, -poorly installed packages may still find a way to creep into the mix and -interfere with our virtualenv. - -As a general principle, we recommend against using your system's package -manager or using ``sudo`` to install Python packages. Instead, create -virtualenvs and isolated Python environments for all of your Python -projects. - -Python on $PATH is not appropriate ----------------------------------- - -Tools like ``mach`` will look for Python by performing ``/usr/bin/env -python`` or equivalent. Please be sure the appropriate Python 2.7.3+ -path is on $PATH. On OS X, this likely means you'll need to modify your -shell's init script to put something ahead of ``/usr/bin``. diff --git a/build/docs/rust.rst b/build/docs/rust.rst deleted file mode 100644 index 970cf3268..000000000 --- a/build/docs/rust.rst +++ /dev/null @@ -1,79 +0,0 @@ -.. _rust: - -============================== -Including Rust Code in Firefox -============================== - -The build system has support for building and linking Rust crates. -Rust code is built using ``cargo`` in the typical way, so it is -straightforward to take an existing Rust crate and integrate it -into Firefox. - -.. important:: - - Rust code is not currently enabled by default in Firefox builds. - This should change soon (`bug 1283898 <https://bugzilla.mozilla.org/show_bug.cgi?id=1283898>`_), - but the option to build without Rust code will likely last a little longer - (`bug 1284816 <https://bugzilla.mozilla.org/show_bug.cgi?id=1284816>`_), - so Rust code cannot currently be used for required components. - - -Linking Rust Crates into libxul -=============================== - -Rust crates that you want to link into libxul should be listed in the -``dependencies`` section of `toolkit/library/rust/shared/Cargo.toml <https://dxr.mozilla.org/mozilla-central/source/toolkit/library/rust/shared/Cargo.toml>`_. -You'll also need to add an ``extern crate`` reference to `toolkit/library/rust/shared/lib.rs <https://dxr.mozilla.org/mozilla-central/source/toolkit/library/rust/shared/lib.rs>`_. -This ensures that the Rust code will be linked properly into libxul as well -as the copy of libxul used for gtests. - -Linking Rust Crates into something else -======================================= - -There currently is not any Rust code being linked into binaries other than -libxul. If you would like to do so, you'll need to create a directory with -a ``Cargo.toml`` file for your crate, and a ``moz.build`` file that contains: - -.. code-block:: python - - RustLibrary('crate_name') - -Where *crate_name* matches the name from the ``[package]`` section of your -``Cargo.toml``. You can refer to `the moz.build file <https://dxr.mozilla.org/mozilla-central/rev/3f4c3a3cabaf94958834d3a8935adfb4a887942d/toolkit/library/rust/moz.build#7>`_ and `the Cargo.toml file <https://dxr.mozilla.org/mozilla-central/rev/3f4c3a3cabaf94958834d3a8935adfb4a887942d/toolkit/library/rust/Cargo.toml>`_ that are used for libxul. - -You can then add ``USE_LIBS += ['crate_name']`` to the ``moz.build`` file -that defines the binary as you would with any other library in the tree. - -.. important:: - - You cannot link a Rust crate into an intermediate library that will wind - up being linked into libxul. The build system enforces that only a single - ``RustLibrary`` may be linked into a binary. If you need to do this, you - will have to add a ``RustLibrary`` to link to any standalone binaries that - link the intermediate library, and also add the Rust crate to the libxul - dependencies as in `linking Rust Crates into libxul`_. - -Where Should I put my Crate? -============================ - -If your crate's canonical home is mozilla-central, you can put it next to the -other code in the module it belongs to. - -If your crate is mirrored into mozilla-central from another repository, and -will not be actively developed in mozilla-central, you can simply list it -as a ``crates.io``-style dependency with a version number, and let it be -vendored into the ``third_party/rust`` directory. - -If your crate is mirrored into mozilla-central from another repository, but -will be actively developed in both locations, you should send mail to the -dev-builds mailing list to start a discussion on how to meet your needs. - - -Crate dependencies -================== - -All dependencies for in-tree Rust crates are vendored into the -``third_party/rust`` directory. Currently if you add a dependency on a new -crate you must run ``mach vendor rust`` to vendor the dependencies into -that directory. In the future we hope to make it so that you only need to -vendor the dependencies in order to build your changes in a CI push. diff --git a/build/docs/slow.rst b/build/docs/slow.rst deleted file mode 100644 index 546fe26ce..000000000 --- a/build/docs/slow.rst +++ /dev/null @@ -1,179 +0,0 @@ -.. _slow: - -============================ -Why the Build System is Slow -============================ - -A common complaint about the build system is that it's slow. There are -many reasons contributing to its slowness. We will attempt to document -them here. - -First, it is important to distinguish between a :term:`clobber build` -and an :term:`incremental build`. The reasons for why each are slow can -be different. - -The build does a lot of work -============================ - -It may not be obvious, but the main reason the build system is slow is -because it does a lot of work! The source tree consists of a few -thousand C++ files. On a modern machine, we spend over 120 minutes of CPU -core time compiling files! So, if you are looking for the root cause of -slow clobber builds, look at the sheer volume of C++ files in the tree. - -You don't have enough CPU cores and MHz -======================================= - -The build should be CPU bound. If the build system maintainers are -optimizing the build system perfectly, every CPU core in your machine -should be 100% saturated during a build. While this isn't currently the -case (keep reading below), generally speaking, the more CPU cores you -have in your machine and the more total MHz in your machine, the better. - -**We highly recommend building with no fewer than 4 physical CPU -cores.** Please note the *physical* in this sentence. Hyperthreaded -cores (an Intel Core i7 will report 8 CPU cores but only 4 are physical -for example) only yield at most a 1.25x speedup per core. - -We also recommend using the most modern CPU model possible. Haswell -chips deliver much more performance per CPU cycle than say Sandy Bridge -CPUs. - -This cause impacts both clobber and incremental builds. - -You are building with a slow I/O layer -====================================== - -The build system can be I/O bound if your I/O layer is slow. Linking -libxul on some platforms and build architectures can perform gigabytes -of I/O. - -To minimize the impact of slow I/O on build performance, **we highly -recommend building with an SSD.** Power users with enough memory may opt -to build from a RAM disk. Mechanical disks should be avoided if at all -possible. - -Some may dispute the importance of an SSD on build times. It is true -that the beneficial impact of an SSD can be mitigated if your system has -lots of memory and the build files stay in the page cache. However, -operating system memory management is complicated. You don't really have -control over what or when something is evicted from the page cache. -Therefore, unless your machine is a dedicated build machine or you have -more memory than is needed by everything running on your machine, -chances are you'll run into page cache eviction and you I/O layer will -impact build performance. That being said, an SSD certainly doesn't -hurt build times. And, anyone who has used a machine with an SSD will -tell you how great of an investment it is for performance all around the -operating system. On top of that, some automated tests are I/O bound -(like those touching SQLite databases), so an SSD will make tests -faster. - -This cause impacts both clobber and incremental builds. - -You don't have enough memory -============================ - -The build system allocates a lot of memory, especially when building -many things in parallel. If you don't have enough free system memory, -the build will cause swap activity, slowing down your system and the -build. Even if you never get to the point of swapping, the build system -performs a lot of I/O and having all accessed files in memory and the -page cache can significantly reduce the influence of the I/O layer on -the build system. - -**We recommend building with no less than 8 GB of system memory.** As -always, the more memory you have, the better. For a bare bones machine -doing nothing more than building the source tree, anything more than 16 -GB is likely entering the point of diminishing returns. - -This cause impacts both clobber and incremental builds. - -You are building on Windows -=========================== - -New processes on Windows are about a magnitude slower to spawn than on -UNIX-y systems such as Linux. This is because Windows has optimized new -threads while the \*NIX platforms typically optimize new processes. -Anyway, the build system spawns thousands of new processes during a -build. Parts of the build that rely on rapid spawning of new processes -are slow on Windows as a result. This is most pronounced when running -*configure*. The configure file is a giant shell script and shell -scripts rely heavily on new processes. This is why configure on Windows -can run over a minute slower on Windows. - -Another reason Windows builds are slower is because Windows lacks proper -symlink support. On systems that support symlinks, we can generate a -file into a staging area then symlink it into the final directory very -quickly. On Windows, we have to perform a full file copy. This incurs -much more I/O. And if done poorly, can muck with file modification -times, messing up build dependencies. As of the summer of 2013, the -impact of symlinks is being mitigated through the use -of an :term:`install manifest`. - -These issues impact both clobber and incremental builds. - -Recursive make traversal is slow -================================ - -The build system has traditionally been built by employing recursive -make. Recursive make involves make iterating through directories / make -files sequentially and executing each in turn. This is inefficient for -directories containing few targets/tasks because make could be *starved* -for work when processing these directories. Any time make is starved, -the build isn't using all available CPU cycles and the build is slower -as a result. - -Work has started in bug 907365 to fix this issue by changing the way -make traverses all the make files. - -The impact of slow recursive make traversal is mostly felt on -incremental builds. Traditionally, most of the wall time during a -no-op build is spent in make traversal. - -make is inefficient -=================== - -Compared to modern build backends like Tup or Ninja, make is slow and -inefficient. We can only make make so fast. At some point, we'll hit a -performance plateau and will need to use a different tool to make builds -faster. - -Please note that clobber and incremental builds are different. A clobber -build with make will likely be as fast as a clobber build with e.g. Tup. -However, Tup should vastly outperform make when it comes to incremental -builds. Therefore, this issue is mostly seen when performing incremental -builds. - -C++ header dependency hell -========================== - -Modifying a *.h* file can have significant impact on the build system. -If you modify a *.h* that is used by 1000 C++ files, all of those 1000 -C++ files will be recompiled. - -Our code base has traditionally been sloppy managing the impact of -changed headers on build performance. Bug 785103 tracks improving the -situation. - -This issue mostly impacts the times of an :term:`incremental build`. - -A search/indexing service on your machine is running -==================================================== - -Many operating systems have a background service that automatically -indexes filesystem content to make searching faster. On Windows, you -have the Windows Search Service. On OS X, you have Finder. - -These background services sometimes take a keen interest in the files -being produced as part of the build. Since the build system produces -hundreds of megabytes or even a few gigabytes of file data, you can -imagine how much work this is to index! If this work is being performed -while the build is running, your build will be slower. - -OS X's Finder is notorious for indexing when the build is running. And, -it has a tendency to suck up a whole CPU core. This can make builds -several minutes slower. If you build with ``mach`` and have the optional -``psutil`` package built (it requires Python development headers - see -:ref:`python` for more) and Finder is running during a build, mach will -print a warning at the end of the build, complete with instructions on -how to fix it. diff --git a/build/docs/supported-configurations.rst b/build/docs/supported-configurations.rst deleted file mode 100644 index 7645b2e66..000000000 --- a/build/docs/supported-configurations.rst +++ /dev/null @@ -1,30 +0,0 @@ -.. _build_supported_configurations: - -======================== -Supported Configurations -======================== - -This page attempts to document supported build configurations. - -Windows -======= - -We support building on Windows 7 and newer operating systems using -Visual Studio 2015. - -The following are not fully supported by Mozilla (but may work): - -* Building without the latest *MozillaBuild* Windows development - environment -* Building with Mingw or any other non-Visual Studio toolchain. - -Linux -===== - -Linux 2.6 and later kernels are supported. - -Most distributions are supported as long as the proper package -dependencies are in place. Running ``mach bootstrap`` should install -packages for popular Linux distributions. ``configure`` will typically -detect missing dependencies and inform you how to disable features to -work around unsatisfied dependencies. diff --git a/build/docs/test_manifests.rst b/build/docs/test_manifests.rst deleted file mode 100644 index 8c85579be..000000000 --- a/build/docs/test_manifests.rst +++ /dev/null @@ -1,207 +0,0 @@ -.. _test_manifests: - -============== -Test Manifests -============== - -Many test suites have their test metadata defined in files called -**test manifests**. - -Test manifests are divided into two flavors: :ref:`manifestparser_manifests` -and :ref:`reftest_manifests`. - -Naming Convention -================= - -The build system does not enforce file naming for test manifest files. -However, the following convention is used. - -mochitest.ini - For the *plain* flavor of mochitests. - -chrome.ini - For the *chrome* flavor of mochitests. - -browser.ini - For the *browser chrome* flavor of mochitests. - -a11y.ini - For the *a11y* flavor of mochitests. - -xpcshell.ini - For *xpcshell* tests. - -.. _manifestparser_manifests: - -ManifestParser Manifests -========================== - -ManifestParser manifests are essentially ini files that conform to a basic -set of assumptions. - -The `reference documentation <http://mozbase.readthedocs.org/en/latest/manifestparser.html>`_ -for manifestparser manifests describes the basic format of test manifests. - -In summary, manifests are ini files with section names describing test files:: - - [test_foo.js] - [test_bar.js] - -Keys under sections can hold metadata about each test:: - - [test_foo.js] - skip-if = os == "win" - [test_foo.js] - skip-if = os == "linux" && debug - [test_baz.js] - fail-if = os == "mac" || os == "android" - -There is a special **DEFAULT** section whose keys/metadata apply to all -sections/tests:: - - [DEFAULT] - property = value - - [test_foo.js] - -In the above example, **test_foo.js** inherits the metadata **property = value** -from the **DEFAULT** section. - -Recognized Metadata -------------------- - -Test manifests can define some common keys/metadata to influence behavior. -Those keys are as follows: - -head - List of files that will be executed before the test file. (Used in - xpcshell tests.) - -tail - List of files that will be executed after the test file. (Used in - xpcshell tests.) - -support-files - List of additional files required to run tests. This is typically - defined in the **DEFAULT** section. - - Unlike other file lists, *support-files* supports a globbing mechanism - to facilitate pulling in many files with minimal typing. This globbing - mechanism is activated if an entry in this value contains a ``*`` - character. A single ``*`` will wildcard match all files in a directory. - A double ``**`` will descend into child directories. For example, - ``data/*`` will match ``data/foo`` but not ``data/subdir/bar`` where - ``data/**`` will match ``data/foo`` and ``data/subdir/bar``. - - Support files starting with ``/`` are placed in a root directory, rather - than a location determined by the manifest location. For mochitests, - this allows for the placement of files at the server root. The source - file is selected from the base name (e.g., ``foo`` for ``/path/foo``). - Files starting with ``/`` cannot be selected using globbing. - - Some support files are used by tests across multiple directories. In - this case, a test depending on a support file from another directory - must note that dependency with the path to the required support file - in its own **support-files** entry. These use a syntax where paths - starting with ``!/`` will indicate the beginning of the path to a - shared support file starting from the root of the srcdir. For example, - if a manifest at ``dom/base/test/mochitest.ini`` has a support file, - ``dom/base/test/server-script.sjs``, and a mochitest in - ``dom/workers/test`` depends on that support file, the test manifest - at ``dom/workers/test/mochitest.ini`` must include - ``!/dom/base/test/server-script.sjs`` in its **support-files** entry. - -generated-files - List of files that are generated as part of the build and don't exist in - the source tree. - - The build system assumes that each manifest file, test file, and file - listed in **head**, **tail**, and **support-files** is static and - provided by the source tree (and not automatically generated as part - of the build). This variable tells the build system not to make this - assumption. - - This variable will likely go away sometime once all generated files are - accounted for in the build config. - - If a generated file is not listed in this key, a clobber build will - likely fail. - -dupe-manifest - Record that this manifest duplicates another manifest. - - The common scenario is two manifest files will include a shared - manifest file via the ``[include:file]`` special section. The build - system enforces that each test file is only provided by a single - manifest. Having this key present bypasses that check. - - The value of this key is ignored. - - -skip-if - Skip this test if the specified condition is true. - See :ref:`manifest_filter_language`. - -fail-if - Expect test failure if the specified condition is true. - See :ref:`manifest_filter_language`. - -run-sequentially - If present, the test should not be run in parallel with other tests. - - Some test harnesses support parallel test execution on separate processes - and/or threads (behavior varies by test harness). If this key is present, - the test harness should not attempt to run this test in parallel with any - other test. - - By convention, the value of this key is a string describing why the test - can't be run in parallel. - -.. _manifest_filter_language: - -Manifest Filter Language ------------------------- - -Some manifest keys accept a special filter syntax as their values. These -values are essentially boolean expressions that are evaluated at test -execution time. - -The expressions can reference a well-defined set of variables, such as -``os`` and ``debug``. These variables are populated from the -``mozinfo.json`` file. For the full list of available variables, see -the :ref:`mozinfo documentation <mozinfo_attributes>`. - -See -`the source <https://hg.mozilla.org/mozilla-central/file/default/testing/mozbase/manifestparser/manifestparser/manifestparser.py>`_ for the full documentation of the -expression syntax until it is documented here. - -.. todo:: - - Document manifest filter language. - -.. _manifest_file_installation: - -File Installation ------------------ - -Files referenced by manifests are automatically installed into the object -directory into paths defined in -:py:func:`mozbuild.frontend.emitter.TreeMetadataEmitter._process_test_manifest`. - -Relative paths resolving to parent directory (e.g. -``support-files = ../foo.txt`` have special behavior. - -For ``support-files``, the file will be installed to the default destination -for that manifest. Only the file's base name is used to construct the final -path: directories are irrelevant. Files starting with ``/`` are an exception, -these are installed relative to the root of the destination; the base name is -instead used to select the file.. - -For all other entry types, the file installation is skipped. - -.. _reftest_manifests: - -Reftest Manifests -================= - -See `MDN <https://developer.mozilla.org/en-US/docs/Creating_reftest-based_unit_tests>`_. diff --git a/build/docs/toolchains.rst b/build/docs/toolchains.rst deleted file mode 100644 index 6ad460a1e..000000000 --- a/build/docs/toolchains.rst +++ /dev/null @@ -1,56 +0,0 @@ -.. _build_toolchains: - -=========================== -Creating Toolchain Archives -=========================== - -There are various scripts in the repository for producing archives -of the build tools (e.g. compilers and linkers) required to build. - -Clang -===== - -See the ``build/build-clang`` directory. Read ``build/build-clang/README`` -for more. - -Windows -======= - -The ``build/windows_toolchain.py`` script is used to build and manage -Windows toolchain archives containing Visual Studio executables, SDKs, -etc. - -The way Firefox build automation works is an archive containing the -toolchain is produced and uploaded to an internal Mozilla server. The -build automation will download, verify, and extract this archive before -building. The archive is self-contained so machines don't need to install -Visual Studio, SDKs, or various other dependencies. Unfortunately, -Microsoft's terms don't allow Mozilla to distribute this archive -publicly. However, the same tool can be used to create your own copy. - -Configuring Your System ------------------------ - -It is **highly** recommended to perform this process on a fresh installation -of Windows 7 or 10 (such as in a VM). Installing all updates through -Windows Update is not only acceptable - it is encouraged. Although it -shouldn't matter. - -Next, install Visual Studio 2015 Community. The download link can be -found at https://www.visualstudio.com/en-us/products/visual-studio-community-vs.aspx. -Be sure to follow these install instructions: - -1. Choose a ``Custom`` installation and click ``Next`` -2. Select ``Programming Languages`` -> ``Visual C++`` (make sure all sub items are - selected) -3. Under ``Windows and Web Development`` uncheck everything except - ``Universal Windows App Development Tools`` and the items under it - (should be ``Tools (1.3.1)...`` and the ``Windows 10 SDK``). - -Once Visual Studio 2015 Community has been installed, from a checkout -of mozilla-central, run something like the following to produce a ZIP -archive:: - - $ ./mach python build/windows_toolchain.py create-zip vs2015u3 - -The produced archive will be the argument to ``create-zip`` + ``.zip``. |