early-access version 2853

This commit is contained in:
pineappleEA
2022-07-23 03:01:36 +02:00
parent 1f2b5081b5
commit 1f111bb69c
8955 changed files with 418777 additions and 999 deletions

View File

@@ -0,0 +1,159 @@
# Binary Caching v1.1 (Jul 14, 2020)
**Note: this is the feature as it was initially specified and does not necessarily reflect the current behavior.**
**Up-to-date documentation is available at [Binarycaching](../users/binarycaching.md).**
## Motivation
The primary motivation of binary caching is to accelerate two broad scenarios in an easily accessible way
- Continuous Integration
- Developer Environment Changes (first-time or branch change)
We generally believe both of these scenarios are addressed with the same feature set, however when differences arise they will be discussed in the individual scenarios.
It should also be explicitly noted that this specification does not intend to propose a "Microsoft Sanctioned Public Binaries Service" such as nuget.org we only intend to enable users to leverage services they already have access to, such as GitHub, local file shares, Azure Artifacts, etc.
## Key User Stories
### CI -> CI
In this story, a CI build using either persistent or non-persistent machines wants to potentially reuse binaries built in a previous run of the pipeline. This is partially covered by the Cache tasks in GitHub Actions or Azure DevOps Pipelines, however the Cache task is all-or-nothing: a single package change will prevent restoration and require rebuilding the entire graph which is unacceptable in many scenarios (such as if actively developing one of the packages).
### CI -> Developer
In this story, the developer wants to reuse binaries built during a CI run. Given appropriate CI coverage, most developers will always have any needed dependencies pre-built by the CI system.
Notably, this scenario indicates a need for Read/Write access granularity on the remote storage solution. Developers should not need write access to the output from the CI system for security reasons.
### Single Developer (same machine reuse)
With the introduction of manifest files, each project will have separate instances of Vcpkg. The performance costs of rebuilding binaries across each cloned project can be debilitating for those working in micro-repos or open source; for the monolithic enterprise developer it is simply frustrating.
User-wide binary caching alleviates the pain of this scenario by ensuring the same binaries arent built multiple times (as long as the projects truly overlap with respect to versions/packages/etc).
### Developer <-> Developer (multi-machine / team scenario)
In a small team scenario, it's reasonable that multiple developer machines can trust each other enough to share binaries. This also applies to developers that have multiple machines and wish to share binaries between them (given a similar enough environment).
## Solution Aspects
### Tracking Compilers
In order to provide reliable binary caching, vcpkg must determine if the produced binaries are appropriate for the current context. Currently, we consider many factors, including:
- All files in the port directory
- The toolchain file contents
- The triplet contents
- All dependency binaries
- The version of the CMake tool used to build
and a few others.
However, we notably do not currently track the compiler used. This is critical for all cross-machine scenarios, as the environment is likely to change incompatibly from machine to machine. We propose hashing the compiler that will used by CMake. This can be accomplished either by reimplementing the logic of CMake or running some partial project and extracting the results. For performance reasons, we will prefer first using heuristics to approximate the CMake logic with accompanying documentation for users that fall outside those bounds.
Another aspect of the environment we don't currently track is the CRT version on Linux systems. Currently, we believe this will not cause as many problems in most practices (thus not suitable for an MVP), since the compiler will (generally) link against the system CRT and should sufficiently reflect any differences. This can also be easily worked around by the user with documentation the toolchain file can simply have a comment such as "# this uses muslc", which will cause it to hash differently.
### Better control over source modifications
Currently, vcpkg caches sources inside `buildtrees/$PORT/src/`. The built-in helpers, such as `vcpkg_extract_archive_ex()` assume that if the appropriately named source folder exists, it is true, accurate, and without modification.
However, the basic workflow for working on ports (specifically, developing patches) breaks this assumption by directly editing whatever extracted source directory the tool is currently using until a successful build is achieved. The user then usually builds a patch file from their changes, then checks it in to the port directory (adding the changes to one of the tracked locations above) and everything is restored to normal.
However, this causes serious issues with the current tracking system, because modifications to this cached source are not detected and tracked into the binary package.
Our proposed solution is to force source re-extraction each time during builds that have uploading to any protocol enabled. Uploading/downloading can then be disabled on the command line via the --editable switch to reuse extracted sources and enable the current workflow.
### Protocols
To service different scenarios and user requirements, we need to support multiple backends. Currently, our CI system uses our only implemented backend: file-based archives.
#### Backend #1: File-Based Archives
This backend simply stores .zip files in a hierarchy similar to git objects: `$VCPKG_ROOT/archives/$XX/$YYYY.zip` with `$XX` being the first two characters of the computed package hash, and `$YYYY` being the full expanded hash. It also supports storing failure logs as `$VCPKG_ROOT/archives/fail/$XX/$YYYY.zip`, however we consider this an internal feature that is not relevant to the key User Stories.
Our CI system uses this backend by symlinking this directory to an Azure Files share, enabling built binaries and failure logs to be shared by all machines in the pool. Credentials are handled at the time of mounting the Azure Files share, so this does not require interactive authentication.
This protocol is ideal due to simplicity for same-machine reuse and simple serverless scenarios such as using networked SMB folders across multiple machines for very small teams. However, it has three significant limitations in the current incarnation:
- It uses the hardcoded directory `$VCPKG_ROOT/archives` (redirectable using symlinks, but unwieldy)
- It cannot use multiple directories
- There is no ability to treat directories as "read-only"/immutable
These second two points are required to implement the very useful concept of "fallback" folders (see https://github.com/NuGet/Home/wiki/%5BSpec%5D-Fallback-package-folders for NuGets spec on this topic).
#### Backend #2: NuGet (Azure DevOps Artifacts, GitHub Packages, etc)
This backend packages binaries into a "raw" NuGet package (not suitable for direct import by MSBuild projects) and uploads them to supported NuGet servers such as Azure DevOps Artifacts and GitHub Packages. We believe this will best satisfy the CI scenarios both CI -> CI as well as CI -> Developer by relying on powerful, centralized, managed hosting.
There is a difference in this case between the developer and CI scenarios. The developer generally wants to configure their remotes for the project and then be able to run vcpkg commands as normal, with packages automatically being downloaded and uploaded to optimize the experience. This is similar to File-Based Archives.
While a CI system could use the same workflow as a developer, there are a few key differences. First, a CI system must use a stored secret for authentication, because it cannot interactively authenticate. Second, to enable more complex interactions with systems such as package signing and task-based restores, we must also support a 4-step workflow:
1. Vcpkg computes hashes of any potentially required packages and writes them to a file
2. An unspecified service/task/etc can parse this file and download any appropriate packages
3. vcpkg is then invoked a second time, with any downloaded packages. This consumes the packages, performs any installations and builds, and potentially produces new packages to an output folder.
4. Finally, another unspecified service/task/etc can take these output packages, sign them, and upload them.
This flow enables arbitrarily complex, user-defined authentication and signing schemes, such as the tasks provided by GitHub Actions and Azure DevOps Pipelines or manual signing as documented in the NuGet documentation: https://docs.microsoft.com/en-us/nuget/create-packages/sign-a-package.
#### Configuration
Currently, our file-based backend is enabled by passing the undocumented `--binarycaching` flag to any Vcpkg command or setting the undocumented environment variable `VCPKG_FEATURE_FLAGS` to `binarycaching`. We will replace this feature flag with an on-by-default user-wide behavior, plus command line and environment-based configurability.
The on-by-default configuration will specify the file-based archive protocol on either `%LOCALAPPDATA%/vcpkg/archives` (Windows) or `$XDG_CACHE_HOME/vcpkg/archives` (Unix). If `XDG_CACHE_HOME` is not defined on Unix, we will fall back to `$HOME/.cache/vcpkg/archives` based on the [XDG Base Directory Specification][1]. This can be redirected with a symlink, or completely overridden with the command line or environment. In the future we can also consider having a user-wide configuration file, however we do not believe this is important for any of our key scenarios.
On the command line, a backend can be specified via `--binarysource=<config>`. Multiple backends can be specified by passing the option multiple times and the order of evaluation is determined by the order on the command line. Writes will be performed on all upload backends, but only for packages that were built as part of this build (the tool will not repackage/reupload binaries downloaded from other sources).
The environment variable `VCPKG_BINARY_SOURCES` can be set to a semicolon-delimited list of `<config>`. Empty `<config>` strings are valid and ignored, to support appending like `set VCPKG_BINARY_SOURCES=%VCPKG_BINARY_SOURCES%;foo` or `export VCPKG_BINARY_SOURCES="$VCPKG_BINARY_SOURCES;foo"`
`<config>` can be any of:
- `clear` - ignore all lower priority sources (lowest priority is default, then env, then command line)
- `default[,<readwrite>]` - Reintroduce the default ~/.vcpkg/packages (as read-only or with uploading)
- `files,<path>[,<readwrite>]` - Add a file-based archive at `<path>`
- `nuget,<url>[,<readwrite>]` - Add a nuget-based source at `<url>`. This url has a similar semantic as `nuget.exe restore -source <url>` for reads and `nuget.exe push -source <url>` for writes; notably it can also be a local path.
- `nugetconfig,<path>[,<readwrite>]` - Add a nuget-based source using the NuGet.config file at `<path>`. This enables users to fully control NuGet's execution in combination with the documented NuGet environment variables. This has similar semantics to `nuget.exe push -ConfigFile <path>` and `nuget.exe restore -ConfigFile <path>`.
- `interactive` - Enables interactive mode (such as manual credential entry) for all other configured backends.
`<readwrite>` can be any of `read`, `write`, or `readwrite` to control whether packages will be consumed or published.
Backtick (`) can be used as an escape character within config strings, with double backtick (``) inserting a single backtick. All paths must be absolute.
For all backends, noninteractive operation will be the default and the vcpkg tool will take a `--interactive` parameter to enable prompting for user credentials (if needed by the backend).
To enable the 4-step flow, `vcpkg install` will take a command `--write-nuget-packages-config=<path>` which can be used in combination with `--dry-run`. This path can be relative and will resolve with respect to the current working directory.
[1]: https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html
#### Example 4-step flow
```
PS> vcpkg install --dry-run pkg1 pkg2 pkg3 --write-nuget-packages-config=packages.config
```
An unspecified process, such as `nuget.exe restore packages.config -packagedirectory $packages` or the [ADO task][2], restores the packages to `$packages`.
```
PS> vcpkg install pkg1 pkg2 pkg3 --binarysource=clear --binarysource=nuget,$outpkgs,upload --binarysource=nuget,$packages
```
Another unspecified process such as `nuget.exe sign $outpkgs/*.nupkg` and `nuget.exe push $outpkgs/*.nupkg` or the ADO task uploads the packages for use in future CI runs.
[2]: https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/package/nuget?view=azure-devops

View File

@@ -0,0 +1,174 @@
# Binary Export (Apr 28, 2017)
**Note: this is the feature as it was initially specified and does not necessarily reflect the current behavior.**
## 1. Motivation
### A. Build once and share
Customers want to be able to build their set of required libraries once, and then distribute the resulting binaries to all members of the "group". This has been brought up in
- Enterprise environments, in which there are dedicated teams to acquire libraries and then share them with other teams to consume them
- Academic environments, in which the professor/teacher wants to build the required libraries and then provide them to all the students
- CI Systems, in which developers want to quickly distribute their exact set of dependencies to a cloud-based farm of build machines
Building once and sharing ensures that everyone gets exactly the same binaries, isolates the building effort to a small number of people and minimizes friction to obtain them. Therefore, there is value in enabling users to easily export ready-to-share binaries from `vcpkg`.
### B. Very large libraries
Libraries like [Qt][] can take a very long time to build (5+ hours). Therefore, having the ability to build them and then distribute the binaries can save a lot of time.
### C. Flexibility and uses without `vcpkg`
`vcpkg` currently handles cases where you have a `vcpkg` enlistment on your machine and use it for acquiring libraries and integrating into Visual Studio, CMake etc. However, users need the ability to build the libraries and then use them outside of and independently of `vcpkg`. For example:
- Use `vcpkg` for the build, then host the binaries in a website (similarly to nuget)
- Use `vcpkg` for the build, then put the binaries in an installer and distribute the installer
Consuming the libraries outside of `vcpkg` forfeits the ability to install new libraries or update existing ones, but this can be:
- not a concern, like in a short term project or assignment
- explicitly desired, like in the development of a game where libraries and their versions are sealed for a particular release, never to be modified
### D. Easy consumption in Visual Studio for NuGet users
Customers have requested C++ NuGet packages to integrate into their project. This has come from:
- Customers than have used NuGet (e.g. in C#) and find it very convenient
- Customers who are working on a C# project that has a few dependencies on C++ and just want those dependencies to be satisfied in the most automatic way possible
Providing a way to create NuGet packages provides great value to those customers. In an enterprise environment which focuses on C#, the dedicated acquisition team can create the NuGet packages with `vcpkg` and provide them to the other developers. For the "end-developer", this makes the consumption of C++ libraries the same as C# ones.
[Qt]: https://www.qt.io/
## 2. Other design concerns
- The `vcpkg` root may have a variety of packages built and many of them might be unrelated to the current task. Providing an easy way to export a subset of them will enhance user experience.
- Since binary compatibility is not guaranteed, it is not safe to individually export packages. Therefore, when exporting a particular package, all of the dependencies that it was built against must also be present in the export format (e.g. zip file). When a `vcpkg export` command succeeds, there is a guarantee that all required headers/binaries are available in the target bundle.
## 3. Proposed solution
This document proposes the `vcpkg export` command to pack the desired binaries in a convenient format. It is not the goal of this document to discuss binary distribution for C++ in a similar way that NuGet does for C#. It proposes exporting "library sets" instead of individual libraries as a solution to the C++ binary incompatibility problem.
From a user experience perspective, the user expresses interest in exporting a particular library (e.g. `vcpkg export cpprestsdk`). `vcpkg export` should then make sure that the output contains `cpprestsdk` along with all dependencies it was actually built against.
## 4. Proposed User experience
### i. User knows what libraries he needs and wants to export them to an archive format (zip)
Developer Bob needs gtest and cpprestsdk and has been manually building them and their dependencies, then using the binaries in his project via applocal deployment. Bob has been experimenting with `vcpkg` and wants to use `vcpkg` for the building part only.
Bob tries to export the libraries:
```no-highlight
> vcpkg export gtest cpprestsdk --zip
The following packages are already built and will be exported:
* boost:x86-windows
* bzip2:x86-windows
cpprestsdk:x86-windows
* openssl:x86-windows
* websocketpp:x86-windows
* zlib:x86-windows
The following packages need to be built:
gtest:x86-windows
Additional packages (*) need to be exported to complete this operation.
There are packages that have not been built.
To build them, run:
vcpkg install gtest:x86-windows
```
Bob proceeds to install the missing libraries:
```no-highlight
> vcpkg install gtest:x86-windows
// -- omitted build information -- //
Package gtest:x86-windows is installed.
```
Bob then returns to export the libraries:
```no-highlight
> vcpkg export gtest cpprestsdk --zip
The following packages are already built and will be exported:
* boost:x86-windows
* bzip2:x86-windows
cpprestsdk:x86-windows
gtest:x86-windows
* openssl:x86-windows
* websocketpp:x86-windows
* zlib:x86-windows
Additional packages (*) need to be exported to complete this operation.
Exporting package zlib:x86-windows...
Exporting package zlib:x86-windows... done
Exporting package openssl:x86-windows...
Exporting package openssl:x86-windows... done
Exporting package bzip2:x86-windows...
Exporting package bzip2:x86-windows... done
Exporting package boost:x86-windows...
Exporting package boost:x86-windows... done
Exporting package websocketpp:x86-windows...
Exporting package websocketpp:x86-windows... done
Exporting package cpprestsdk:x86-windows...
Exporting package cpprestsdk:x86-windows... done
Exporting package gtest:x86-windows...
Exporting package gtest:x86-windows... done
Creating zip archive...
Creating zip archive... done
zip archive exported at: C:/vcpkg/vcpkg-export-20170428-155351.zip
```
Bob takes the zip file and extracts the contents next to his other dependencies. Bob can now proceed with building his own project as before.
### ii. User has a vcpkg root that works and wants to share it
Developer Alice has been using `vcpkg` and has a Visual Studio project that consumes libraries from it (via `vcpkg integrate`). The project is built for both 32-bit and 64-bit architectures. Alice wants to quickly share the dependencies with Bob so he can test the project.
```no-highlight
> vcpkg export gtest zlib gtest:x64-windows zlib:x64-windows --nuget
The following packages are already built and will be exported:
gtest:x86-windows
gtest:x64-windows
zlib:x86-windows
zlib:x64-windows
Exporting package zlib:x86-windows...
Exporting package zlib:x86-windows... done
Exporting package zlib:x64-windows...
Exporting package zlib:x64-windows... done
Exporting package gtest:x86-windows...
Exporting package gtest:x86-windows... done
Exporting package gtest:x64-windows...
Exporting package gtest:x64-windows... done
Creating nuget package...
Creating nuget package... done
Nuget package exported at: C:/vcpkg/scripts/buildsystems/tmp/vcpkg-export-20170428-164312.nupkg
```
Alice gives to Bob: a) The link to her project and b) The NuGet package "vcpkg-export-20170428-164312.nupkg". Bob clones the project and then installs the NuGet package. Bob is now ready to build Alice's project.
### iii. User has a vcpkg root that works and wants to share it #2
Developer Alice has been using `vcpkg` and has a CMake project that consumes libraries from it (via CMake toolchain file). Alice wants to quickly share the dependencies with Bob so he can test the project.
```no-highlight
> vcpkg export cpprestsdk zlib --zip
The following packages are already built and will be exported:
* boost:x86-windows
* bzip2:x86-windows
cpprestsdk:x86-windows
* openssl:x86-windows
* websocketpp:x86-windows
zlib:x86-windows
Additional packages (*) need to be exported to complete this operation.
Exporting package zlib:x86-windows...
Exporting package zlib:x86-windows... done
Exporting package openssl:x86-windows...
Exporting package openssl:x86-windows... done
Exporting package bzip2:x86-windows...
Exporting package bzip2:x86-windows... done
Exporting package boost:x86-windows...
Exporting package boost:x86-windows... done
Exporting package websocketpp:x86-windows...
Exporting package websocketpp:x86-windows... done
Exporting package cpprestsdk:x86-windows...
Exporting package cpprestsdk:x86-windows... done
Creating zip archive...
Creating zip archive... done
zip archive exported at: C:/vcpkg/vcpkg-export-20170428-155351.zip
```
Alice gives to Bob: a) The links to her project and b) The zip file "vcpkg-export-20170428-155351.zip". Bob clones the project, extracts the zip file and uses the provided (in the zip) CMake toolchain file to make the dependencies available to CMake. Bob is now ready to build Alice's project.
## 5. Technical model
- Each exported library, must be accompanied with all of its dependencies, even if they are not explicitly specified in the `vcpkg export` command.
- When exporting a library, a dependency graph will be built, similarly to install, to figure out which packages need to be exported.
- It is allowed to have packages from different triplets, so users can include 32/64-bit and dynamic/static binaries in the same export.
- The exported archives also include the files needed to integrate with MSBuild and/or CMake.

View File

@@ -0,0 +1,291 @@
# Proposal: Features / Feature packages (Feb 23 2017)
**Note: this is the feature as it was initially specified and does not necessarily reflect the current behavior.**
**Up-to-date documentation is available at [Selecting Library Features](../users/selecting-library-features.md).**
## 1. Motivation
### A. OpenCV + CUDA
[OpenCV][] is a computer vision library that can optionally be built with CUDA support to massively accelerate certain tasks when using computers with NVidia GPUs. For users without NVidia GPUs, building with CUDA support provides no benefit. [CUDA][] is provided only via a 1.3 GB installer (at the time of this authoring), which requires administrator access to install and modifies the global system state.
Therefore, there is significant value in enabling users to choose whether they find CUDA support valuable for their particular scenario.
### B. OpenCV + OpenCV\_contrib
The community around [OpenCV][] has built up a library of extensions called [OpenCV_contrib][]. However, these extensions are a source-level patch onto the main OpenCV codebase and therefore must be applied _during_ the core OpenCV build. Further confounding the problem, it is the author's understanding that these community extensions have only been developed with [CUDA][] enabled and cannot be built without that dependency.
Therefore, if CUDA is disabled, OpenCV\_contrib must also be disabled. Likewise, when a user requests OpenCV\_contrib, CUDA must be enabled. It would be convenient, but not a requirement, to enable CUDA without enabling the community extensions.
Finally, these extensions add additional exports and headers which could be depended upon by other libraries. For maintainers, there must be a way to specify this requirement such that `vcpkg install mylib-depends-ocv-contrib` will verify/build/rebuild OpenCV with the community extensions enabled.
### C. C++ REST SDK + SignalR
The [C++ REST SDK][cpprestsdk] is a networking library that provides (among other features) HTTP and Websockets clients. To implement the HTTP client functionality on Windows Desktop, only the core Win32 platform APIs are needed (`zlib` is optional).
However, the websockets client is based on [Websockets++][], which adds mandatory dependencies on `boost`, `openssl`, and `zlib`. Many users of the C++ REST SDK do not use the websockets component, so to minimize their overall dependency footprint it can be disabled at build time. Ideally, these kinds of options would be easily accessible to users in Vcpkg who are concerned about the final size or licensing of their deployment.
[SignalR-Client-Cpp][SignalR] depends on the websockets functionality provided by the C++ REST SDK. Therefore, the maintainers of the `signalrclient` port would ideally like to express this dependency such that `cpprestsdk` will be automatically correctly built for their needs. Note that `signalrclient` does not _inherently_ care about `boost`, `websocketspp` or `openssl` -- it depends only on the public websocket client APIs provided by `cpprestsdk`. It would be much more maintainable to declare dependencies based on the public APIs rather than the dependencies themselves.
[OpenCV]: http://opencv.org/
[CUDA]: http://www.nvidia.com/object/cuda_home_new.html
[OpenCV_contrib]: https://github.com/opencv/opencv_contrib
[cpprestsdk]: https://github.com/Microsoft/cpprestsdk
[Websockets++]: https://www.zaphoyd.com/websocketpp/
[SignalR]: https://github.com/aspnet/SignalR-Client-Cpp
## 2. Other design concerns
- General-purpose Open Source projects must be able to easily and succinctly describe their build dependencies inside Vcpkg. This should be no more verbose than a single `vcpkg install` line and, when that command succeeds, there is a strong expectation that all required functionality/headers/imports are available.
- The internal state of the Vcpkg enlistment must be either extremely transparent OR managed by version control (git). This enables larger projects to efficiently transfer the entire state of their customized Vcpkg system between machines (and onto build servers) by having the destination clone and then run a single `vcpkg install` line for the subset of dependencies required. The results of this operation should be as repeatable as reasonably achievable given the current limits of the underlying toolchain.
## 3. Proposed solution
A key summary of the above motivations is that they are all scenarios surrounding APIs that are not independently buildable from each other. We have an existing solution for APIs that are independently buildable: separate packages. Therefore, we seek to extend the user-facing notion of "packages" to include capabilities and contracts that cannot be made into independent builds.
This document proposes "features" (also called feature packages). These features are intended to model semi-independently toggleable API sets/contracts such that they can be sanely depended upon by other packages. It is not a goal to model exclusive alternatives (such as implementation choices that are not directly user-observable) through this mechanism.
- Individual libraries within `boost` may be reasonably represented as features.
- Whether a graphics library is built on DirectX xor OpenGL (where one but not both must be chosen) is not representable as a feature.
From a user experience perspective (i.e. from `vcpkg install`) feature packages act as much as possible like completely independent packages. However, internally, any change to a package's features will result in a rebuild of the associated "parent" package. This will invoke a package rebuild experience similar to upgrading.
When using `vcpkg install <package>`, some features will be enabled by default. These default features can be avoided by referring to the packages as `<package>[core]` and features can be added by supplying them on the same installation line.
### A. Proposed User experience
#### i. User with no preference about options
Install of a library with default features:
```no-highlight
> vcpkg install cpprestsdk
// -- omitted build information -- //
Package cpprestsdk[core]:x86-windows is installed.
Package cpprestsdk[compression]:x86-windows is installed.
Package cpprestsdk[ws-client]:x86-windows is installed.
```
Removal of that library:
```no-highlight
> vcpkg remove cpprestsdk
The following packages will be removed:
cpprestsdk:x86-windows
Removing package cpprestsdk:x86-windows...
Removing package cpprestsdk:x86-windows... done
Purging package cpprestsdk:x86-windows...
Cleaned up D:\src\vcpkg\packages\cpprestsdk_x64-windows
Purging package cpprestsdk:x86-windows... done
```
Installation of a library with optional features:
```no-highlight
> vcpkg install opencv
// -- omitted build information -- //
Package opencv[core]:x86-windows is installed.
```
#### ii. User desires CUDA support for OpenCV directly, and is unfamiliar with feature packages
Developer Bob knows he wants OpenCV, so he guesses what the package is called
```no-highlight
> vcpkg install opencv
// -- omitted build information -- //
Package opencv[core]:x86-windows is installed.
```
Bob attempts to build his application against OpenCV (assuming CUDA), which fails at runtime or compile time indicating that OpenCV wasn't built with CUDA.
Bob comes back to vcpkg, not knowing about the "feature packages" feature. The primary inquiry tools for Vcpkg are `search` and `list`, so he runs `vcpkg search`:
```no-highlight
> vcpkg search opencv
opencv 3.2.0 computer vision library
opencv[cuda] support for NVidia CUDA
opencv[contrib] community supported extensions for OpenCV
If your library is not listed, please open an issue at:
https://github.com/Microsoft/vcpkg/issues
```
He isn't immediately sure what the lack of a version number means, but anything in `vcpkg search` can be applied to `vcpkg install`, so he runs:
```no-highlight
> vcpkg install opencv[cuda]
The following packages will be rebuilt:
opencv:x86-windows
To rebuild with this feature, use:
vcpkg remove opencv:x86-windows
vcpkg install opencv[core,cuda]:x86-windows
```
Bob follows the instructions...
```no-highlight
> vcpkg remove opencv:x86-windows
// -- omitted results as above -- //
> vcpkg install opencv[core,cuda]:x86-windows
// -- omitted build information -- //
Package opencv[core]:x86-windows is installed.
Package opencv[cuda]:x86-windows is installed.
```
and he can now use OpenCV's CUDA support in his application.
#### iii. User is familiar with feature packages, and wants to opt-out of a feature
Developer Alice has used `cpprestsdk`, built it from source, and she knows about the option to disable websockets. She uses `search` to find the complete list of features:
```
> vcpkg search cpprestsdk
cpprestsdk 2.9.0-2 C++11 JSON, REST, and OAuth library The C++ RES...
cpprestsdk[compression] Gzip compression support in the HTTP client.
cpprestsdk[ws-client] Websocket client support based on websocketspp.
If your library is not listed, please open an issue at:
https://github.com/Microsoft/vcpkg/issues
```
She decided she only wants `cpprestsdk[compression]`, so she installs only that feature:
```no-highlight
> vcpkg install cpprestsdk[compression]
// -- omitted build information -- //
Package cpprestsdk[core]:x86-windows is installed.
Package cpprestsdk[compression]:x86-windows is installed.
```
She receives a quick recursive build that only depends on `zlib`.
She's now interested in some additional libraries built on top of cpprestsdk: `azure-storage-cpp` and `signalrclient`.
```no-highlight
> vcpkg install azure-storage-cpp
// -- omitted build information -- //
Package azure-storage-cpp[core]:x86-windows is installed.
> vcpkg install signalrclient
Package signalrclient:x86-windows depends on cpprestsdk[ws-client]:x86-windows.
The following packages will be rebuilt:
* azure-storage-cpp:x86-windows
* cpprestsdk:x86-windows
To rebuild the current package graph with this feature, use:
vcpkg remove cpprestsdk:x86-windows azure-storage-cpp:x86-windows
vcpkg install cpprestsdk[core,compression,ws-client]:x86-windows
vcpkg install azure-storage-cpp[core]:x86-windows
vcpkg install signalrclient[core]:x86-windows
```
She follows the above script and can use both `azure-storage-cpp` and `signalrclient` in her code.
Some time has passed, she decided not to use `signalrclient`, and she's interested in shipping her application. She wants to minimize her final install size, so she'd like to remove all unneeded packages like `boost` and `openssl`.
```no-highlight
> vcpkg remove boost openssl
The following packages and features will be removed:
* signalrclient[core]:x86-windows
* cpprestsdk[ws-client]:x86-windows
boost[core]:x86-windows
openssl[core]:x86-windows
The following packages will be rebuilt:
* azure-storage-cpp:x86-windows
* cpprestsdk:x86-windows
Removing features requires rebuilding packages.
To rebuild the current package graph without these features, use:
vcpkg remove cpprestsdk:x86-windows azure-storage-cpp:x86-windows signalrclient:x86-windows openssl:x86-windows boost:x86-windows
vcpkg install cpprestsdk[core,compression]:x86-windows
vcpkg install azure-storage-cpp[core]:x86-windows
```
In the end, her final `vcpkg list` outputs:
```no-highlight
> vcpkg list
zlib[core]:x86-windows 1.2.11 A compression library
azure-storage-cpp[core]:x86-windows 2.6.0 Microsoft Azure Storage Client SDK for ...
cpprestsdk[core]:x86-windows 2.9.0-2 C++11 JSON, REST, and OAuth library
cpprestsdk[compression]:x86-windows Gzip compression support in the HTTP client.
```
### B. Technical model
- Each package can have any number "features".
- Features follow the same naming conventions as packages, but when referenced are always "namespaced" by the parent package.
- `cpprestsdk[ws-client]` is a completely orthogonal feature from `poco[ws-client]`.
- Features are valid dependencies.
- `signalrclient` depends on `cpprestsdk[ws-client]`
- Features can have dependencies (including other features).
- `cpprestsdk[ws-client]` depends on `boost`, `openssl`, and `websocketspp`
- `opencv[cuda]` depends on `cuda`
- `opencv[contrib]` depends on `opencv[cuda]`
- `boost[python]` depends on `libpython`
- Every package has an implicit feature called `core`, which covers the core library with a minimum set of features. All features implicitly depend on the `core` feature of their parent package
- `azure-storage-cpp` depends on `cpprestsdk[core]`
- `cpprestsdk[ws-client]` implicitly depends on `cpprestsdk[core]`
- Each package declares a list of default features that are enabled when the package is referred to by its raw name, and `core` is always a default feature.
- `cpprestsdk` declares `ws-client` and `compression` to be default features. Any unqualified reference `cpprestsdk` implicitly means `cpprestsdk[core]` _and_ `cpprestsdk[ws-client]` _and_ `cpprestsdk[compression]`.
- `opencv` does not declare `cuda` nor `contrib` to be default features.
As a conclusion of the above, it is expected that all packages will be buildable with all features disabled (just the `core` feature) and with all features enabled.
### C. Proposed Control File Syntax
#### OpenCV and CUDA
To add the feature CUDA to OpenCV, we will adopt the following syntax in the CONTROL file:
```no-highlight
# opencv/CONTROL
Source: opencv
Version: 3.2.0-1
Build-Depends: zlib, libpng, libjpeg-turbo, tiff
Description: computer vision library
Default-Features:
Feature: cuda
Build-Depends: cuda
Description: parallel computing platform
Feature: contrib
Build-Depends: opencv[cuda]
Description: library of OpenCV Extensions
```
#### Signalrclient
```no-highlight
# signalrclient/CONTROL
Source: signalrclient
Version: 1.0.0-beta1
Build-Depends: cpprestsdk[ws-client]
Description: C++ client for SignalR.
```
```no-highlight
# cpprestsdk/CONTROL
Source: cpprestsdk
Version: 2.9.0-2
Build-Depends:
Description: C++11 JSON, REST, and OAuth library ...
Default-Features: compression, ws-client
Feature: compression
Build-Depends: zlib (windows)
Description: Gzip compression support in the HTTP client.
Feature: ws-client
Build-Depends: boost (windows), openssl (windows), websocketpp (windows)
Description: Websocket client support based on websocketspp
```
### D. Additional Control File Technical Details
- If any feature paragraphs exist, the field `Default-Features` must be present.
## 4. Related Work
### Cargo's Features (from Rust): <http://doc.crates.io/manifest.html#the-features-section>
The proposed feature packages are exceedingly similar to Cargo's Features, with the following changes:
- We avoid any collision problems because features are always namespaced by the owning package
- We do not have a concept of "feature groups", instead we allow dependencies from one feature to another within the same package (Note: This may be how "feature groups" are implemented internally to Cargo -- it was not clear from the documentation).
- Because of the nature of C and C++, it is extremely commonplace that large software packages can have features disabled to remove their dependencies upon other libraries. Changing this configuration requires a rebuild of the package and potentially rippling ABI changes to any downstream dependencies. Therefore, we expect significantly more use of this feature to manage optional API contracts instead of the intended use in Cargo (curation).
- We do not intend feature packages to be used to express the curation relationship, beyond the notion of a "default" set within a package.
### Gentoo's USE flags: <https://wiki.gentoo.org/wiki/Handbook:X86/Working/USE>
Gentoo's USE flags can be shortly summarized as a global set of keywords that is used to make cross-cutting changes to the entire package graph's build configuration. This system standardizes many common settings such that they can be simultaneously toggled for the entire graph.
The most common example of this would be using KDE vs Gnome. A user who knows that, given the choice, they would prefer the KDE/Qt interface can manage the massive space of package configuration efficiently without learning the particular term that each package has decided to call "build using Qt instead of GTK".
USE flags can be customized hierarchically when needed, including at the per-package level. They can be depended upon by other packages, both positively and negatively. USE flags themselves can be used in any boolean expression to determine the complete set of package dependencies, including removing dependencies when flags are enabled.
Problems with USE flags:
- They require coordination from package maintainers to achieve the goal of "portable" flags. This increases the burden of adding a package -- to author a good package, I need to be aware of every uncommon USE flag and evaluate how those could map onto my local configuration space.
- Based on research online, it seems extremely common that users need to tweak flags at a per-package level. This calls into question how valuable the cross-cutting power above is.
- The vast majority of common USE flags are essentially a list of all the common packages and focus on giving the user a view of dependencies (which a package manager is designed to abstract when possible) instead of APIs (which is what users code against).
- Dependency analysis with USE flags becomes a SAT problem with an enormous state space -- P*F bits -- which compounds with any versioning relations. This may work acceptably in practice via heuristics, but it implies that a) there is a looming performance wall which could suddenly create a poor user experience and b) the heuristics may incorrectly model the user's needs, causing a disconnect in desire vs practice, which again leads to a poor user experience.

View File

@@ -0,0 +1,302 @@
# Manifests -- `vcpkg.json`
**Note: this is the feature as it was initially specified and does not necessarily reflect the current behavior.**
**Up-to-date documentation is available at [Manifests](../users/manifests.md).**
For many other language package managers, there exists a way of writing one's dependencies in a declarative
manifest format; we want something similar for vcpkg. What follows is the specification of that feature;
this should mean that vcpkg becomes far more user and enterprise-friendly, and is additionally an important
first step for versioning and package federation. Our primary concern, beyond implementability, is ease-of-use;
it is important that using this feature is all of:
* Easy for existing users
* Easy for new users to set up
* Easy to extend later for new features like versioning and federation
* _Declarative_, not _Imperative_.
## Reasoning
### Why JSON?
We choose JSON for five main reasons:
* Everybody knows JSON, and if one doesn't, it's really easy to learn
* Every tool supports JSON in the standard library, or in a commonly used support library
* This means writing tooling should be trivial in any language one is comfortable with
* Most configuration formats don't have a COBOL implementation 😉
* Specified in an international standard
* There is _one_ right way to parse JSON
* There are no ambiguities of what the parse tree _should_ be
* Simple and secure
* Unlike YAML, for example, there's no weird ACE issues
* Easy to write a parser -- important since we can't depend on external libraries
* Schemas are almost a necessity
Some have suggested allowing comments or commas in our parser; we chose to use JSON proper
rather than JSON5 or JSON with comments because JSON is the everywhere-supported international
standard. That is not necessarily true of JSON with comments. Additionally, if one needs
to write a comment, they can do so via `"$reason"` or `"$comment"` fields.
## Specification
A manifest file shall have the name `vcpkg.json`, and shall be in the root directory of a package.
It also replaces CONTROL files, though existing CONTROL files will still be
supported; there will be no difference between ports and packages, except
that packages do not need to supply portfile.cmake (eventually we would like
to remove the requirement of portfile.cmake for ports that already use
CMake).
The specification uses definitions from the [Definitions](#definitions) section in order
to specify the shape of a value. Note that any object may contain any directives, written as
a field key that starts with a `$`; these directives shall be ignored by `vcpkg`. Common
directives may include `"$schema"`, `"$comment"`, `"$reason"`.
A manifest must be a top-level object, and must have at least:
* `"name"`: a `<package-name>`
* One (and only one) of the following version fields:
* `"version-string"`: A `string`. Has no semantic meaning.
Equivalent to `CONTROL`'s `Version:` field.
* Other version fields will be defined by the Versions RFC
The simplest vcpkg.json looks like this:
```json
{
"name": "mypackage",
"version-string": "0.1.0-dev"
}
```
Additionally, it may contain the following properties:
* `"port-version"`: A non-negative integer. If this field doesn't exist, it's assumed to be `0`.
* Note that this is a change from existing CONTROL files, where versions were a part of the version string
* `"maintainers"`: An array of `string`s which contain the authors of a package
* `"maintainers": [ "Nicole Mazzuca <nicole@example.com>", "שלום עליכם <shalom@example.com>" ]`
* `"description"`: A string or array of strings containing the description of a package
* `"description": "mypackage is a package of mine"`
* `"homepage"`: A url which points to the homepage of a package
* `"homepage": "https://github.com/strega-nil/mypackage"`
* `"documentation"`: A url which points to the documentation of a package
* `"documentation": "https://readthedocs.io/strega-nil/mypackage"`
* `"license"`: A `<license-string>`
* `"license": "MIT"`
* `"dependencies"`: An array of `<dependency>`s
* `"dev-dependencies"`: An array of `<dependency>`s which are required only for developers (testing and the like)
* `"features"`: An array of `<feature>`s that the package supports
* `"default-features"`: An array of `<identifier>`s that correspond to features, which will be used by default.
* `"supports"`: A `<platform-expression>`
* `"supports": "windows & !arm"`
Any properties which are not listed, and which do not start with a `$`,
will be warned against and are reserved for future use.
The following is an example of an existing port CONTROL file rewritten as a vcpkg.json file:
```
Source: pango
Version: 1.40.11-6
Homepage: https://ftp.gnome.org/pub/GNOME/sources/pango/
Description: Text and font handling library.
Build-Depends: glib, gettext, cairo, fontconfig, freetype, harfbuzz[glib] (!(windows&static)&!osx)
```
```json
{
"name": "pango",
"version-string": "1.40.11",
"port-version": 6,
"homepage": "https://ftp.gnome.org/pub/GNOME/sources/pango/",
"description": "Text and font handling library.",
"dependencies": [
"glib",
"gettext",
"cairo",
"fontconfig",
"freetype",
{
"name": "harfbuzz",
"features": [ "glib" ],
"platform": "!(windows & static) & !osx"
}
]
}
```
## Behavior of the Tool
There will be two "modes" for vcpkg from this point forward: "classic", and "manifest".
The former will act exactly like the existing vcpkg workflow, so as to avoid breaking
anyone. The latter will be the mode only when the user either:
* Passes `--manifest-root=<directory>` (initially, `x-manifest-root`)
* Runs `vcpkg` in a directory that contains a file named `vcpkg.json`, or in a
child directory of a directory containing `vcpkg.json`.
* For this, initially vcpkg will warn that the behavior will change in the
future, and simply run in classic mode, unless the feature flag `manifests` is
passed via:
* The environment variable `VCPKG_FEATURE_FLAGS`
* The option `--feature-flags`
* (e.g., `--feature-flags=binarycaching,manifests`)
* If someone wants to use classic mode and silence the warning, they can add the
`-manifests` feature flag to disable the mode.
When in "manifest" mode, the `installed` directory will be changed to
`<manifest-root>/vcpkg_installed` (name up for bikeshedding).
The following commands will change behavior:
* `vcpkg install` without any port arguments will install the dependencies listed in
the manifest file, and will remove any dependencies
which are no longer in the dependency tree implied by the manifest file.
* `vcpkg install` with port arguments will give an error.
The following commands will not work in manifest mode, at least initially:
* `vcpkg x-set-installed`: `vcpkg install` serves the same function
* `vcpkg remove`
* `vcpkg export`
We may add these features back for manifest mode once we understand how best to
implement them.
### Behavior of the Toolchain
Mostly, the toolchain file stays the same; however, we shall add
two public options:
```cmake
VCPKG_MANIFEST_MODE:BOOL=<we found a manifest>
VCPKG_MANIFEST_INSTALL:BOOL=ON
```
The first option either explicitly turns on, or off, manifest mode;
otherwise, we default to looking for a manifest file in the directory
tree upwards from the source directory.
The `VCPKG_MANIFEST_INSTALL` option tells the toolchain whether to
install the packages or not -- if you wish to install the manifest
dependencies manually, you can set this to off, and we also turn it
off for packages installed by vcpkg.
Additionally, if `-manifests` is set in the feature flags environment
variable, we turn off manifest mode in the toolchain, and we act like
the classic toolchain.
### Example - CMake Integration
An example of using the new vcpkg manifests feature for a new
project follows:
The filesystem structure should look something like:
```
example/
src/
main.cxx
CMakeLists.txt
vcpkg.json
```
Then, `main.cxx` might look like:
```cpp
#include <fmt/format.h>
int main() {
fmt::print("Hello, {}!", "world");
}
```
Therefore, in `vcpkg.json`, we'll need to depend on `fmt`:
```json
{
"name": "example",
"version-string": "0.0.1",
"dependencies": [
"fmt"
]
}
```
Then, let's write our `CMakeLists.txt`:
```cmake
cmake_minimum_required(VERSION 3.14)
project(example CXX)
add_executable(example src/main.cxx)
find_package(fmt REQUIRED)
target_link_libraries(example
PRIVATE
fmt::fmt)
```
And finally, to configure and build:
```sh
$ cd example
$ cmake -B build -S . -DCMAKE_TOOLCHAIN_FILE=$VCPKG_ROOT/scripts/buildsystems/vcpkg.cmake
... configuring and installing...
$ cmake --build build
```
and we're done! `fmt` will get installed into
`example/build/vcpkg_installed`, and we can run our executable with:
```sh
$ build/example
Hello, world!
```
## Definitions
* `<identifier>`: A `string` which:
* Is entirely ASCII
* Contains only lowercase alphabetic characters, digits, and hyphen-minus
* Does not have multiple consecutive hyphens
* Does not begin nor end with a hyphen
* Is not a Windows filesystem reserved name
* Is not a vcpkg reserved name: "default" or "core".
* In other words, it must follow the regex `[a-z0-9]+(-[a-z0-9]+)*`, and must not be any of:
* `{ prn, aux, nul, con, lpt[1-9], com[1-9], core, default }`
* `<package-name>`: A `string` consisting of a non-zero number of `<identifier>`s, separated by `.`.
* `a.b.c` is valid
* `a` is valid
* `a/b` is not valid
* `Boost.Beast` is not valid, but `boost.beast` is
* `<dependency>`: Either a `<package-name>`, or an object:
* A dependency always contains the following:
* `"name"`: A `<package-name>`
* Optionally, `"features"`: an array of `<identifier>`s corresponding to features in the package.
* Optionally, `"default-features"`: a `boolean`. If this is false, then don't use the default features of the package; equivalent to core in existing CONTROL files. If this is true, do the default thing of including the default features.
* Optionally, `"platform"`: a `<platform-expression>`
* `<dependency.port>`: No extra fields are required.
* `<license-string>`: An SPDX license expression at version 3.9.
* `<platform-expression>`: A specification of a set of platforms; used in platform-specific dependencies and supports fields. A string that is parsed as follows:
* `<platform-expression>`:
* `<platform-expression.not>`
* `<platform-expression.and>`
* `<platform-expression.or>`
* `<platform-expression.simple>`:
* `( <platform-expression> )`
* `<platform-expression.identifier>`
* `<platform-expression.identifier>`:
* regex: `/^[a-z0-9]+$/`
* `<platform-expression.not>`:
* `<platform-expression.simple>`
* `! <platform-expression.simple>`
* `<platform-expression.and>`
* `<platform-expression.not>`
* `<platform-expression.and> & <platform-expression.not>`
* `<platform-expression.or>`
* `<platform-expression.not>`
* `<platform-expression.or> | <platform-expression.not>`
* `<feature>`: An object containing the following:
* `"name"`: An `<identifier>`, the name of the feature
* `"description"`: A `string` or array of `string`s, the description of the feature
* Optionally, `"dependencies"`: An array of `<dependency>`s, the dependencies used by this feature

View File

@@ -0,0 +1,183 @@
# Ports Overlay (Jun 19, 2019)
**Note: this is the feature as it was initially specified and does not necessarily reflect the current behavior.**
## 1. Motivation
### A. Allow users to override ports with alternate versions
It's a common scenario for `vcpkg` users to keep specific versions of libraries to use in their own projects. The current recommendation for users is to fork `vcpkg`'s repository and create tags for commits containing the specific versions of the ports they want to use.
This proposal adds an alternative to solve this problem. By allowing `vcpkg` users to specify additional locations in their file system containing ports for:
* older or newer versions of libraries,
* modified libraries, or
* libraries not available in `vcpkg`.
These locations will be searched when resolving port names during package installation, and override ports in `<vcpkg-root>/ports`.
### B. Allow users to keep unmodified upstream ports
Users will be able to keep unmodified versions of the ports shipped with `vcpkg` and update them via `vcpkg update` and `vcpkg upgrade` without having to solve merge conflicts.
## 2. Other design concerns
* Allow a set of `vcpkg` commands to optionally accept additional paths to be used when searching for ports.
* Additional paths must take precedence when resolving names of ports to install.
* Allow users to specify multiple additional paths.
* Provide a simple disambiguation mechanism to resolve ambiguous port names.
* After resolving a port name, the installation process has to work the same as for ports shipped by `vcpkg`.
* This **DOES NOT ENABLE MULTIPLE VERSIONS** of a same library to be **INSTALLED SIDE-BY-SIDE**.
## 3. Proposed solution
This document proposes allowing additional locations to search for ports during package installation that will override and complement the set of ports provided by `vcpkg` (ports under the `<vcpkg_root>/ports` directory).`
A new option `--overlay-ports` will be added to the `vcpkg install`, `vcpkg update`, `vcpkg upgrade`, `vcpkg export`, and `vcpkg depend-info` commands to specify additional paths containing ports.
From a user experience perspective, a user expresses interest in adding additional lookup locations by passing the `--overlay-ports` option followed by a path to:
* an individual port (directory containing a `CONTROL` file),
* `vcpkg install sqlite3 --overlay-ports="C:\custom-ports\sqlite3"`
* a directory containing ports,
* `vcpkg install sqlite3 --overlay-ports=\\share\org\custom-ports`
* a file listing paths to the former two.
> NOTE: Reading paths from a text file is not available in the current implementation, some revisions to this part of the specification are being made and will be implemented in a future date.
* `vcpkg install sqlite3 --overlay-ports=..\port-repos.txt`
_port-repos.txt_
```
.\experimental-ports\sqlite3
C:\custom-ports
\\share\team\custom-ports
\\share\org\custom-ports
```
*Relative paths inside this file are resolved relatively to the file's location. In this case a `experimental-ports` directory should exist at the same level as the `port-repos.txt` file.*
_NOTE: It is not the goal of this document to discuss library versioning or project dependency management solutions, which require the ability to install multiple versions of a same library side-by-side._
### Multiple additional paths
Users can provide multiple additional paths by repeating the `--overlay-ports` option.
```
vcpkg install sqlite3
--overlay-ports="..\experimental-ports\sqlite3"
--overlay-ports="C:\custom-ports"
--overlay-ports="\\share\team\custom-ports
```
### Overlaying ports
Port name resolution follows the order in which additional paths are specified, with the first match being selected for installation, and falling back to `<vcpkg-root>/ports` if the port is not found in any of the additional paths.
No effort is made to compare version numbers inside the `CONTROL` files, or to determine which port contains newer or older files.
### Examples
Given the following directory structure:
```
team-ports/
|-- sqlite3/
|---- CONTROL
|-- rapidjson/
|---- CONTROL
|-- curl/
|---- CONTROL
my-ports/
|-- sqlite3/
|---- CONTROL
|-- rapidjson/
|---- CONTROL
vcpkg
|-- ports/
|---- <upstream ports>
|-- vcpkg.exe
|-- preferred-ports.txt
```
* #### Example #1:
Running:
```
vcpkg/vcpkg.exe install sqlite3 --overlay-ports=my-ports --overlay-ports=team-ports
```
Results in `my-ports/sqlite3` getting installed as that location appears first in the command line arguments.
* #### Example #2:
A specific version of a port can be given priority by adding its path first in the list of arguments:
```
vcpkg/vcpkg.exe install sqlite3 rapidjson curl
--overlay-ports=my-ports/rapidjson
--overlay-ports=vcpkg/ports/curl
--overlay-ports=team-ports
```
Installs:
* `sqlite3` from `team-ports/sqlite3`
* `rapidjson` from `my-ports/rapidjson`
* `curl` from `vcpkg/ports/curl`
* #### Example #3:
> NOTE: Reading paths from a text file is not available in the current implementation, some revisions to this part of the specification are being made and will be implemented in a future date.
Given the content of `preferred-ports.txt` as:
```
./ports/curl
/my-ports/rapidjson
/team-ports
```
A location can be appended or prepended to those included in `preferred-ports.txt` via the command line, like this:
```
vcpkg/vcpkg.exe install sqlite3 curl --overlay-ports=my-ports --overlay-ports=vcpkg/preferred-ports.txt
```
Which results in `my-ports/sqlite3` and `vcpkg/ports/curl` getting installed.
## 4. Proposed User experience
### i. User wants to preserve an older version of a port
Developer Alice and her team use `vcpkg` to acquire **OpenCV** and some other packages. She has even contributed many patches to add features to the **OpenCV 3** port in `vcpkg`. But, one day, she notices that a PR to update **OpenCV** to the next major version has been merged.
Alice wants to update some packages available in `vcpkg`. Unfortunately, updating her project to use the latest **OpenCV** is not immediately possible.
Alice creates a private GitHub repository and checks in the set of ports that she wants to preserve. Then provides her teammates with the link to clone her private ports repository.
```
mkdir vcpkg-custom-ports
cd vcpkg-custom-ports
git init
cp -r %VCPKG_ROOT%/ports/opencv .
git add .
git commit -m "[opencv] Add OpenCV 3 port"
git remote add origin https://github.com/<Alice's GitHub username>/vcpkg-custom-ports.git
git push -u origin master
```
Now her team is able to use:
```
git clone https://github.com/<Alice's GitHub username>/vcpkg-custom-ports.git
vcpkg update --overlay-ports=./vcpkg-custom-ports
vcpkg upgrade --no-dry-run --overlay-ports=./vcpkg-custom-ports
```
to upgrade their packages and preserve the old version of **OpenCV** they require.

160
externals/vcpkg/docs/specifications/prefab.md vendored Executable file
View File

@@ -0,0 +1,160 @@
# Vcpkg: export Android prefab Archives (AAR files)
**Note: this is the feature as it was initially specified and does not necessarily reflect the current behavior.**
Vcpkg can export android archives ([AAR files](https://developer.android.com/studio/projects/android-library)). Once an archive is created, it can imported in Android Studio as a native dependent. The archive is automatically consumed using [android studio's prefab tool](https://github.com/google/prefab).
For more information on Prefab, refer to:
* The [official prefab documentation](https://google.github.io/prefab).
* a blog post from Android developers blog: [Native Dependencies in Android Studio 4.0](https://android-developers.googleblog.com/2020/02/native-dependencies-in-android-studio-40.html)
_Note for Android Studio users: prefab packages are supported on Android Studio 4+_
## Requirements
1. `ndk <required>`
Set environment variable `ANDROID_NDK_HOME` to your android ndk installation. For example:
````
export ANDROID_NDK_HOME=/home/your-account/Android/Sdk/ndk-bundle
````
2. `7zip <required on windows>` or `zip <required on linux>`
3. `maven <optional>`
4. Android triplets
See [android.md](../users/android.md) for instructions on how to install the triplets.
*Please note that in order to use "prefab" (see below), the four architectures are required. If any is missing the export will fail*
## Example exporting [jsoncpp]
First "vcpkg install" the 4 android architectures (it is mandatory to export all 4 of them)
````
./vcpkg install jsoncpp:arm-android jsoncpp:arm64-android jsoncpp:x64-android jsoncpp:x86-android
````
Then, export the prefab:
Note:
* The `--prefab-maven` flag is optional. Call it if you maven is installed.
* The `--prefab-debug` flag will output instructions on how to use the prefab archive via gradle.
```
./vcpkg export --triplet x64-android jsoncpp --prefab --prefab-maven --prefab-debug
```
You will see an output like this:
```
The following packages are already built and will be exported:
jsoncpp:arm64-android
Exporting package jsoncpp...
[DEBUG] Found 4 triplets
arm64-android
x64-android
x86-android
arm-android
...
... Lots of output...
...
[INFO] Scanning for projects...
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-clean-plugin/2.5/maven-clean-plugin-2.5.pom
...
... Lots of output...
...
[INFO] BUILD SUCCESS
[INFO] Total time: 2.207 s
[INFO] Finished at: 2020-05-10T14:42:28+02:00
...
... Lots of output...
...
[DEBUG] Configuration properties in Android Studio
In app/build.gradle
com.vcpkg.ndk.support:jsoncpp:1.9.2
And cmake flags
externalNativeBuild {
cmake {
arguments '-DANDROID_STL=c++_shared'
cppFlags "-std=c++17"
}
}
In gradle.properties
android.enablePrefab=true
android.enableParallelJsonGen=false
android.prefabVersion=${prefab.version}
Successfully exported jsoncpp. Checkout .../vcpkg/prefab
```
#### The output directory after export
````
prefab
└── jsoncpp/
├── aar/
│   ├── AndroidManifest.xml
│   ├── META-INF/
│   │   └── LICENSE
│   └── prefab/
│   ├── modules/
│   │   └── jsoncpp/
│   │   ├── libs/
│   │   │   ├── android.arm64-v8a/
│   │   │   │   ├── abi.json
│   │   │   │   ├── include/
│   │   │   │   │   └── json/
│   │   │   │   │   ├── json.h
│   │   │   │   │   └── ....
│   │   │   │   └── libjsoncpp.so
│   │   │   ├── android.armeabi-v7a/
│   │   │   │   ├── abi.json
│   │   │   │   ├── include/
│   │   │   │   │   └── json/
│   │   │   │   │   ├── json.h
│   │   │   │   │   └── ....
│   │   │   │   └── libjsoncpp.so
│   │   │   ├── android.x86/
│   │   │   │   ├── abi.json
│   │   │   │   ├── include/
│   │   │   │   │   └── json/
│   │   │   │   │   ├── json.h
│   │   │   │   │   └── ....
│   │   │   │   └── libjsoncpp.so
│   │   │   └── android.x86_64/
│   │   │   ├── abi.json
│   │   │   ├── include/
│   │   │   │   └── json/
│   │   │   │   │   ├── json.h
│   │   │   │   │   └── ....
│   │   │   └── libjsoncpp.so
│   │   └── module.json
│   └── prefab.json
├── jsoncpp-1.9.2.aar
└── pom.xml
````
## Example consuming [jsoncpp] via vcpkg and prefab
See the example repo here:
https://github.com/atkawa7/prefab-vpkg-integration-sample

View File

@@ -0,0 +1,561 @@
# Registries: Take 2 (including Git Registries)
**Note: this is the feature as it was initially specified and does not necessarily reflect the current behavior.**
**Up-to-date documentation is available at [Registries](../users/registries.md).**
Originally, the design of registries was decided upon and written up in the [Registries RFC](registries.md).
However, as we've gotten further into the design process of git registries and versioning,
and discussed the interaction of versioning with registries,
it's become clear that the existing design was lacking.
We need to have an on-disk port database that is not tied to the ports tree.
This RFC is a new design for registries, that includes this registry database.
It also includes the design for git registries,
which are likely to be the predominant form of registries in the wild.
They are also what we will start to treat the default registry as,
to allow for updating ports without updating the vcpkg executable
(likely necessary for binary releases).
## Design Considerations
After internal discussions of the relationship between versioning and registries,
it was clear that the existing design of registries does not play well with versioning.
It was also clear that it was necessary to have metadata about ports in a separate place from the ports tree;
in fact, after discussion, it was clear that the ports tree should be considered an implementation detail;
a backing store for build process information (e.g., `portfile.cmake` and the patches) and the manifest.
From this, it's clear that vcpkg needs to add a new set of metadata.
The versioning implementation has decided on `port_versions`, and thus that's what this RFC uses.
Since we're replacing the existing ports directory with a new method of describing ports,
this means that the ports directory is no longer anything but a data store.
This also means that the existing rules around locations of ports is no longer required;
however, it will still keep getting followed for the main repository,
and it's recommended that other registries follow the same pattern to make contributing easier.
## What does the registry database look like?
We don't wish to have the same problem as we do right now,
where there are nearly 1500 entries in a single directory.
We solve this by placing each database entry into `port_versions/<first character of port name>-/<port name>.json`.
For example, the database entry for 7zip is in `port_versions/7-/7zip.json`.
Each of these database entries contains all of the versions of the port throughout history,
along with versioning and feature metadata, so that we do not have to check out old manifests or CONTROL files
to get at that information.
Each database entry file must be a top-level array of port version objects, which contain the following entries:
* A version field: `"version-string"`, `"version"`, etc. Same as in the manifest.
* Optionally, `"port-version"`: Same as in the manifest.
And also contain a description of where to find the build files for this port; the possibilities include:
* `"git-tree"`: The [git object ID] of a tree object; this is only allowed for git registries.
Note that this ID must be an ID from the repository where the registry is located.
* `"path"`: A path describing where to find the build files.
The first entry in this path should be `$`, which means "this path starts at the root of the registry".
No other kinds of paths are allowed.
* For example: `$/foo/bar` gives you `foo/bar` underneath the folder containing the `port_versions` directory.
* `/foo/bar` and `foo/bar` are both disallowed.
Using a `"git-tree"` as a backend in a non-git registry, and using a `"path"` in a git registry,
is not permitted. Future extensions may include things like remote archives or git repositories,
or may allow `"path"` in git registries.
Note that a registry entry should _always_ be additive;
deleting existing entries is unsupported and may result in bad behavior.
The only modification to existing entries that is allowable is moving the backing store
for the build files, assuming that the new build files are equivalent to the old build files.
(For example, a filesystem registry might have a new way of laying out where ports are).
Additionally, we'd like a new way of describing the set of ports that make up a "baseline".
This is currently done with the reference of the vcpkg git repository -
each reference has a set of versions that are tested against each other,
and this is a major feature of vcpkg.
We wish to have the same feature in the new versioning world,
and so we'll have a set of baseline versions in the registry database.
Baselines act differently between git registries or the builtin registry,
and in filesystem registries.
In git registries and the builtin registry,
since there's a history that one can access,
a baseline is the `"default"` entry in the baseline at the reference specified.
In filesystem registries, since there is no accessible history,
the baseline identifiers are mapped directly to entries in the baseline file,
without translation; by default, the `"default"` entry is used.
These baselines are placed in `port_versions/baseline.json`.
This is an object mapping baseline names to baseline objects,
where baseline objects map port names to version objects.
A version object contains `"baseline"`, which is un-schemed version,
and optionally `"port-version"`.
[git object ID]: https://git-scm.com/book/en/v2/Git-Internals-Git-Objects
### Example of a baseline file
The following is a reasonable baseline.json for a filesystem registry that only has two ports:
```json
{
"default": {
"abseil": { "baseline": "2020-03-03" },
"zlib": { "baseline": "1.2.11", "port-version": 9 }
},
"old": {
"abseil": { "baseline": "2019-02-11" },
"zlib": { "baseline": "1.2.11", "port-version": 3 }
},
"really-old": {
"zlib": { "baseline": "1.2.9" }
}
}
```
### Example of a registry database entry file
Note: This file assumes that the versions RFC has been implemented,
and thus that minimum versions are required;
the syntax may change in the time between now and finishing the implementation.
This example is of `ogre`, since this port has both features and dependencies;
remember that this file would be `port_versions/o-/ogre.json`.
```json
[
{
"version-string": "1.12.7",
"git-tree": "466e96fd2e17dd2453aa31dc0bc61bdcf53e7f61",
},
{
"version-string": "1.12.1",
"port-version": 1,
"git-tree": "0de81b4f7e0ec24966e929c2ea64e16c15e71d5e",
},
...
]
```
#### Filesystem Registry Databases
Filesystem registries are the simplest possible registry;
they have a `port_versions` directory at the top-level, which contains the registry database.
It's expected that the filesystem registry would have a filesystem backing store:
something like the existing `ports` directory, except with separate versions.
There won't be a specific way to lay the ports tree out as mandated by the tool,
as we are treating the ports tree as an implementation detail of the registry;
it's simply a way to get the files for a port.
As an example, let's assume that the registry is laid out something like this:
```
<registry>/
port_versions/
baseline.json
a-/
abseil.json
asmjit.json
o-/
ogre.json
ports/
a-/
abseil/
2020-03-03_7/
vcpkg.json
portfile.cmake
...
2020-03-03_8/
vcpkg.json
portfile.cmake
...
...
asmjit/
2020-05-08/
CONTROL
portfile.cmake
...
2020-07-22/
vcpkg.json
portfile.cmake
...
o-/
ogre/
1.12.7/
...
1.12.1/
...
...
...
```
Then, let's look at updating `asmjit` to latest.
The current manifest file, in `asmjit/2020-07-22/vcpkg.json` looks like:
```json
{
"name": "asmjit",
"version-string": "2020-07-22",
"description": "Complete x86/x64 JIT and Remote Assembler for C++",
"homepage": "https://github.com/asmjit/asmjit",
"supports": "!arm"
}
```
while the current `port_versions/a-/asmjit.json` looks like:
```json
[
{
"version-string": "2020-07-22",
"path": "$/ports/a-/asmjit/2020-07-22"
},
{
"version-string": "2020-05-08",
"path": "$/ports/a-/asmjit/2020-05-08"
}
]
```
with `port_versions/baseline.json` looking like:
```json
{
"default": {
...,
"asmjit": { "baseline": "2020-07-22" },
...
}
}
```
and we'd like to update to `2020-10-08`.
We should first copy the existing implementation to a new folder:
```sh
$ cp -r ports/a-/asmjit/2020-07-22 ports/a-/asmjit/2020-10-08
```
then, we'll make the edits required to `ports/a-/asmjit/2020-10-08` to update to latest.
We should then update `port_versions/a-/asmjit.json`:
```json
[
{
"version-string": "2020-10-08",
"path": "$/ports/a-/asmjit/2020-10-08"
},
{
"version-string": "2020-07-22",
"path": "$/ports/a-/asmjit/2020-07-22"
},
{
"version-string": "2020-05-08",
"path": "$/ports/a-/asmjit/2020-05-08"
}
]
```
and update `port_versions/baseline.json`:
```json
{
"default": {
...,
"asmjit": { "baseline": "2020-10-08" },
...
}
}
```
and we're done 😊.
#### Git Registry Databases
Git registries are not quite as simple as filesystem registries,
but they're still pretty simple, and are likely to be the most common:
the default registry is a git registry, for example.
There is not a specific way the tool requires one to lay out the backing store,
as long as it's possible to get an object hash that corresponds to a checked-in git tree
of the build information.
This allows, for example, the current vcpkg default registry way of laying out ports,
where the latest version of a port `<P>` is at `ports/<P>`,
and it also allows for any number of other designs.
One interesting design, for example,
is having an `old-ports` branch which is updated whenever someone want to backfill versions;
then, one could push the old version to the `old-ports` branch,
and then update the HEAD branch with the git tree of the old version in `port_versions/p-/<P>`.
As above, we want to update `asmjit` to latest; let's assume we're working in the default vcpkg registry
(the <https://github.com/microsoft/vcpkg> repository):
The current manifest file for `asmjit` looks like:
```json
{
"name": "asmjit",
"version-string": "2020-07-22",
"description": "Complete x86/x64 JIT and Remote Assembler for C++",
"homepage": "https://github.com/asmjit/asmjit",
"supports": "!arm"
}
```
and the current `port_versions/a-/asmjit.json` looks like:
```json
[
{
"version-string": "2020-07-22",
"git-tree": "fa0c36ba15b48959ab5a2df3463299e1d2473b6f"
}
]
```
Now, let's update it to the latest version:
```json
{
"name": "asmjit",
"version-string": "2020-10-08",
"description": "Complete x86/x64 JIT and Remote Assembler for C++",
"homepage": "https://github.com/asmjit/asmjit",
"supports": "!arm"
}
```
and make the proper edits to the portfile.cmake. Then, let's commit the changes:
```cmd
> git add ./ports/asmjit
> git commit -m "[asmjit] update asmjit to 2020-10-08"
```
In `git-tree` mode, one needs to commit the new version of the port to get the git tree hash;
we use `git rev-parse` to do so:
```cmd
> git rev-parse HEAD:ports/asmjit
2bb51d8ec8b43bb9b21032185ca8123da10ecc6c
```
and then modify `port_versions/a-/asmjit.json` as follows:
```json
[
{
"version-string": "2020-10-08",
"git-tree": "2bb51d8ec8b43bb9b21032185ca8123da10ecc6c"
},
{
"version-string": "2020-07-22",
"git-tree": "fa0c36ba15b48959ab5a2df3463299e1d2473b6f"
}
]
```
Then we can commit and push this new database with:
```sh
$ git add port_versions
$ git commit --amend --no-edit
$ git push
```
## Consuming Registries
The `vcpkg-configuration.json` file from the [first registries RFC](registries.md)
is still the same, except that the registries have a slightly different layout.
A `<configuration>` is still an object with the following fields:
* Optionally, `"default-registry"`: A `<registry-implementation>` or `null`
* Optionally, `"registries"`: An array of `<registry>`s
Additionally, `<registry>` is still the same;
a `<registry-implementation>` object, plus the following properties:
* Optionally, `"baseline"`: A named baseline. Defaults to `"default"`.
* Optionally, `"packages"`: An array of `<package-name>`s
however, `<registry-implementation>`s are now slightly different:
* `<registry-implementation.builtin>`:
* `"kind"`: The string `"builtin"`
* `<registry-implementation.filesystem>`:
* `"kind"`: The string `"filesystem"`
* `"path"`: A path
* `<registry-implementation.git>`:
* `"kind"`: The string `"git"`
* `"repository"`: A URI
The `"packages"` field of distinct registries must be disjoint,
and each `<registry>` must have at the `"packages"` property,
since otherwise there's no point.
As an example, a package which uses a different default registry, and a different registry for boost,
might look like the following:
```json
{
"default-registry": {
"kind": "filesystem",
"path": "vcpkg-ports"
},
"registries": [
{
"kind": "builtin",
"packages": [ "cppitertools" ]
}
]
}
```
This will install `fmt` from `<directory-of-vcpkg-configuration.json>/vcpkg-ports`,
and `cppitertools` and the `boost` ports from the registry that ships with vcpkg.
Notably, this does not replace behavior up the tree -- only the `vcpkg-configuration.json`s
for the current invocation do anything.
### Filesystem Registries
A filesystem registry takes on the form:
* `"kind"`: The string `"filesystem"`
* `"path"`: The path to the filesystem registry's root, i.e. the directory containing the `port_versions` directory.
```json
{
"kind": "filesystem",
"path": "vcpkg-registry"
}
```
Unlike git registries, where there's quite a bit of interesting stuff going on,
there isn't much stuff to do with filesystem registries.
We simply use the registry database at `<registry root>/port_versions` to get information about ports.
### Git Registries
A git registry takes on the form:
* `"kind"`: The string `"git"`
* `"repository"`: The URL at which the git repository lives. May be any kind of URL that git understands
```json
{
"kind": "git",
"repository": "https://github.com/microsoft/vcpkg"
}
```
Whenever the first vcpkg command is run with a git registry,
vcpkg notes down the exact commit that HEAD points to at the time of the run in the `vcpkg-lock.json` file.
This will be used as the commit which vcpkg takes the `"default"` baseline from,
and vcpkg will only update that commit when `vcpkg update` is run.
Since the `"versions"` field is strictly additive, we don't consider older refs than `HEAD`.
We update the repository on some reasonable clip.
Likely, whenever a command is run that will change the set of installed ports.
#### `vcpkg-lock.json`
This file will contain metadata that we need to save across runs,
to allow us to keep a "state-of-the-world" that doesn't change unless one explicitly asks for it to change.
This means that, even across different machines, the same registries will be used.
We will also be able to write down version resolution in this file as soon as that feature is added.
It is recommended that one adds this `vcpkg-lock.json` to one's version control.
This file is machine generated, and it is not specified how it's laid out;
however, for purposes of this RFC, we will define how it relates to git registries.
In `vcpkg-lock.json`, in the top level object,
there will be a `"registries"` property that is an object.
This object will contain a `"git"` field, which is an array of git-registry objects,
that contain:
* `"repository"`: The `"repository"` field from the git registry object
* `"baseline"`: The name of the baseline that we've used
* `"baseline-ref"`: The ref which we've gotten the specific baseline from.
For example, a `vcpkg-lock.json` might look like:
```json
{
"registries": {
"git": [
{
"repository": "https://github.com/microsoft/vcpkg",
"baseline": "default",
"baseline-ref": "6185aa76504a5025f36754324abf307cc776f3da"
}
]
}
}
```
#### `vcpkg update`
You'll notice that once the repository is added the first time,
there is only one way to update the repository to the tag at a later date - deleting the lock file.
We additionally want to add support for the user updating the registry by themselves -
they will be able to do this via the `vcpkg update` command.
The `vcpkg update` command will, for each git registry,
update the registry and repoint the `"commit"` field in `vcpkg-lock.json` to the latest `HEAD`.
There is no way to update only one git registry to a later date, since versions are strictly additive.
## Git Registries: Implementation on Disk
There are two implementations on disk to consider here: the implementation of the registry database,
and once we have the database entries for the ports, accessing the port data from the git tree object.
Both of these implementations are placed in the vcpkg cache home (shared by binary caching archives).
On unix, this is located at `$XDG_CACHE_HOME/vcpkg` if the environment variable exists,
otherwise `$HOME/.cache/vcpkg`; on Windows, it's located at `%LOCALAPPDATA%\vcpkg`.
In this document, we use the variable `$CACHE_ROOT` to refer to this folder.
We will add a new folder, `$CACHE_ROOT/registries`, which will contain all the data we need.
First, we'll discuss the registry database.
### Registry Database
At `$CACHE_ROOT/registries/git`,
we'll create a new git repository root which contains all information from all git registries,
since the hashes should be unique, and this allows for deduplication
across repositories which have the same commits (e.g., for mirrors).
In order to get the data from git registries, we simply `fetch` the URL of the git registry.
In order to grab a specific database entry from a git registry, `git show` is used to grab the
file from the right commit: `git show <commit id> -- port_versions/<first character>-/<portname>.json`.
One unfortunate thing about having one directory being used for all vcpkg instances on a machine is
that it's possible to have an issue with concurrency - for example, after `fetch`ing the latest HEAD
of `https://github.com/microsoft/vcpkg`, another vcpkg process might fetch the latest HEAD of
`https://github.com/meow/vcpkg` before the first vcpkg process has the chance to `git rev-parse FETCH_HEAD`.
Since the first vcpkg process will run `git rev-parse` after the second fetch is done,
instead of getting the `HEAD` of `microsoft/vcpkg`, they instead get the `HEAD` of `meow/vcpkg`.
We will solve this by having a mutex file in `$CACHE_ROOT/registries/git`
that vcpkg locks before any fetches (and unlocks after `rev-parse`ing).
### Accessing Port Data from `git-tree`s
Once we've done version resolution and everything with the database,
we then need to access the port data from the git history.
We will add a new folder, `$CACHE_ROOT/registries/git-trees`, into which we'll check out the port data.
In this `git-trees` directory, we will have all of the trees we check out, at their hashes.
For example, the asmjit port data from above will be located at
`git-trees/2bb51d8ec8b43bb9b21032185ca8123da10ecc6c`.
We will add a mutex file in this `git-trees` directory as well which is taken whenever
we are checking out a new git tree.
We wish to allow multiple vcpkg instances to read port data at a time,
and thus we do the check outs semi-atomically - if `git-trees/<hash>` exists,
then the `<hash>` must be completely checked out.
vcpkg does this by first checking out to a temporary directory,
and then renaming to the actual hash.
## Future Extensions
The way forward for this is to allow the `"builtin"` registry to be a git registry,
in order to support packaging and shipping vcpkg as a binary.
This is currently our plan, although it definitely is still a ways out.
Git registries _are_ an important step on that road,
but are also a good way to support both enterprise,
and experimentation by our users.
They allow us a lot more flexibility than we've had in the past.

View File

@@ -0,0 +1,289 @@
# Package Federation: Custom Registries
**Note: this is the feature as it was initially specified and does not necessarily reflect the current behavior.**
**Up-to-date documentation is available at [Registries](../users/registries.md).**
As it is now, vcpkg has over 1400 ports in the default registry (the `/ports` directory).
For the majority of users, this repository of packages is enough. However, many enterprises
need to more closely control their dependencies for one reason or another, and this document
lays out a method which we will build into vcpkg for exactly that reason.
## Background
A registry is simply a set of packages. In fact, there is already a registry in vcpkg: the default one.
Package federation, implemented via custom registries, allows one to add new packages,
edit existing packages, and have as much or as little control as one likes over the dependencies that one uses.
It gives the control over dependencies that an enterprise requires.
### How Does the Current Default Registry Work?
Of course, the existing vcpkg tool does have packages in the official,
default registry. The way we describe these packages is in the ports tree
at the base of the vcpkg install directory, there is a directory named ports,
which contains on the order of 1300 directories, one for each package. Then,
in each package directory, there are at least two files: a CONTROL or
vcpkg.json file, which contains the name, version, description, and features
of the package; and a portfile.cmake file which contains the information on
how to download and build the package. There may be other files in this
registry, like patches or usage instructions, but only those two files are
needed.
### Existing vcpkg Registry-like Features
There are some existing features in vcpkg that act somewhat like a custom
registry. The most obvious feature that we have is overlay ports this
feature allows you to specify any number of directories as "overlays", which
either contain a package definition directly, or which contain some number of
package directories; these overlays will be used instead of the ports tree
for packages that exist in both places, and are specified exclusively on the
command line. Additionally, unfortunately, if one installs a package from
overlay ports that does not exist in the ports tree, one must pass these
overlays to every vcpkg installation command.
There is also the less obvious "feature" which works by virtue of the ports
tree being user-editable: one can always edit the ports tree on their own
machine, and can even fork vcpkg and publish their own ports tree.
Unfortunately, this then means that any updates to the source tree require
merges, as opposed to being able to fast-forward to the newest sources.
### Why Registries?
There are many reasons to want custom registries; however, the most important reasons are:
* Legal requirements a company like Microsoft or Google
needs the ability to strictly control the code that goes into their products,
making certain that they are following the licenses strictly.
* There have been examples in the past where a library which is licensed under certain terms contains code
which is not legally allowed to be licensed under those terms (see [this example][legal-example],
where a person tried to merge Microsoft-owned, Apache-licensed code into the GPL-licensed libstdc++).
* Technical requirements a company may wish to run their own tests on the packages they ship,
such as [fuzzing].
* Other requirements an organization may wish to strictly control its dependencies for a myriad of other reasons.
* Newer versions vcpkg may not necessarily always be up to date for all libraries in our registry,
and an organization may require a newer version than we ship;
they can very easily update this package and have the version that they want.
* Port modifications vcpkg has somewhat strict policies on port modifications,
and an organization may wish to make different modifications than we do.
It may allow that organization to make certain that the package works on triplets
that our team does not test as extensively.
* Testing just like port modifications, if a team wants to do specific testing on triplets they care about,
they can do so via their custom registry.
Then, there is the question of why vcpkg needs a new solution for custom registries,
beyond the existing overlay ports feature. There are two big reasons
the first is to allow a project to define the registries that they use for their dependencies,
and the second is the clear advantage in the user experience of the vcpkg tool.
If a project requires specific packages to come from specific registries,
they can do so without worrying that a user accidentally misses the overlay ports part of a command.
Additionally, beyond a feature which makes overlay ports easier to use,
custom registries allow for more complex and useful infrastructure around registries.
In the initial custom registry implementation, we will allow overlay ports style paths,
as well as git repositories, which means that people can run and use custom registries
without writing their own infrastructure around getting people that registry.
It is the intention of vcpkg to be the most user-friendly package manager for C++,
and this allows us to fulfill on that intention even further.
As opposed to having to write `--overlay-ports=path/to/overlay` for every command one runs,
or adding an environment variable `VCPKG_OVERLAY_PORTS`,
one can simply write vcpkg install and the registries will be taken care of for you.
As opposed to having to use git submodules, or custom registry code for every project,
one can write and run the infrastructure in one place,
and every project that uses that registry requires only a few lines of JSON.
[legal-example]: https://gcc.gnu.org/legacy-ml/libstdc++/2019-09/msg00054.html
[fuzzing]: https://en.wikipedia.org/wiki/Fuzzing
## Specification
We will be adding a new file that vcpkg understands - `vcpkg-configuration.json`.
The way that vcpkg will find this file is different depending on what mode vcpkg is in:
in classic mode, vcpkg finds this file alongside the vcpkg binary, in the root directory.
In manifest mode, vcpkg finds this file alongside the manifest. For the initial implementation,
this is all vcpkg will look for; however, in the future, vcpkg will walk the tree and include
configuration all along the way: this allows for overriding defaults.
The specific algorithm for applying this is not yet defined, since currently only one
`vcpkg-configuration.json` is allowed.
The only thing allowed in a `vcpkg-configuration.json` is a `<configuration>` object.
A `<configuration>` is an object:
* Optionally, `"default-registry"`: A `<registry-implementation>` or `null`
* Optionally, `"registries"`: An array of `<registry>`s
Since this is the first RFC that adds anything to this field,
as of now the only properties that can live in that object will be
these.
A `<registry-implementation>` is an object matching one of the following:
* `<registry-implementation.builtin>`:
* `"kind"`: The string `"builtin"`
* `<registry-implementation.directory>`:
* `"kind"`: The string `"directory"`
* `"path"`: A path
* `<registry-implementation.git>`:
* `"kind"`: The string `"git"`
* `"repository"`: A URI
* Optionally, `"path"`: An absolute path into the git repository
* Optionally, `"ref"`: A git reference
A `<registry>` is a `<registry-implementation>` object, plus the following properties:
* Optionally, `"scopes"`: An array of `<package-name>`s
* Optionally, `"packages"`: An array of `<package-name>`s
The `"packages"` and `"scopes"` fields of distinct registries must be disjoint,
and each `<registry>` must have at least one of the `"scopes"` and `"packages"` property,
since otherwise there's no point.
As an example, a package which uses a different default registry, and a different registry for boost,
might look like the following:
```json
{
"default-registry": {
"kind": "directory",
"path": "vcpkg-ports"
},
"registries": [
{
"kind": "git",
"repository": "https://github.com/boostorg/vcpkg-ports",
"ref": "v1.73.0",
"scopes": [ "boost" ]
},
{
"kind": "builtin",
"packages": [ "cppitertools" ]
}
]
}
```
This will install `fmt` from `<directory-of-vcpkg.json>/vcpkg-ports`,
`cppitertools` from the registry that ships with vcpkg,
and any `boost` dependencies from `https://github.com/boostorg/vcpkg-ports`.
Notably, this does not replace behavior up the tree -- only the `vcpkg-configuration.json`s
for the current invocation do anything.
### Behavior
When a vcpkg command requires the installation of dependencies,
it will generate the initial list of dependencies from the package,
and then run the following algorithm on each dependency:
1. Figure out which registry the package should come from by doing the following:
1. If there is a registry in the registry set which contains the dependency name in the `"packages"` array,
then use that registry.
2. For every scope, in order from most specific to least,
if there is a registry in the registry set which contains that scope in the `"scopes"` array,
then use that registry.
(For example, for `"cat.meow.cute"`, check first for `"cat.meow.cute"`, then `"cat.meow"`, then `"cat"`).
3. If the default registry is not `null`, use that registry.
4. Else, error.
2. Then, add that package's dependencies to the list of packages to find, and repeat for the next dependency.
vcpkg will also rerun this algorithm whenever an install is run with different configuration.
### How Registries are Laid Out
There are three kinds of registries, but they only differ in how the registry gets onto one's filesystem.
Once the registry is there, package lookup runs the same, with each kind having it's own way of defining its
own root.
In order to find a port `meow` in a registry with root `R`, vcpkg first sees if `R/meow` exists;
if it does, then the port root is `R/meow`. Otherwise, see if `R/m-` exists; if it does,
then the port root is `R/m-/meow`. (note: this algorithm may be extended further in the future).
For example, given the following port root:
```
R/
abseil/...
b-/
boost/...
boost-build/...
banana/...
banana/...
```
The port root for `abseil` is `R/abseil`; the port root for `boost` is `R/b-/boost`;
the port root for `banana` is `R/banana` (although this duplication is not recommended).
The reason we are making this change to allow more levels in the ports tree is that ~1300
ports are hard to look through in a tree view, and this allows us to see only the ports we're
interested in. Additionally, no port name may end in a `-`, so this means that these port subdirectories
will never intersect with actual ports. Additionally, since we use only ASCII for port names,
we don't have to worry about graphemes vs. code units vs. code points -- in ASCII, they are equivalent.
Let's now look at how different registry kinds work:
#### `<registry.builtin>`
For a `<registry.builtin>`, there is no configuration required.
The registry root is simply `<vcpkg-root>/ports`.
#### `<registry.directory>`
For a `<registry.directory>`, it is again fairly simple.
Given `$path` the value of the `"path"` property, the registry root is either:
* If `$path` is absolute, then the registry root is `$path`.
* If `$path` is drive-relative (only important on Windows), the registry root is
`(drive of vcpkg.json)/$path`
* If `$path` is relative, the registry root is `(directory of vcpkg.json)/$path`
Note that the path to vcpkg.json is _not_ canonicalized; it is used exactly as it is seen by vcpkg.
#### `<registry.git>`
This registry is the most complex. We would like to cache existing registries,
but we don't want to ignore new updates to the registry.
It is the opinion of the author that we want to find more updates than not,
so we will update the registry whenever the `vcpkg.json` or `vcpkg-configuration.json`
is modified. We will do so by keeping a sha512 of the `vcpkg.json` and `vcpkg-configuration.json`
inside the `vcpkg-installed` directory.
We will download the specific ref of the repository to a central location (and update as needed),
and the root will be either: `<path to repository>`, if the `"path"` property is not defined,
or else `<path to repository>/<path property>` if it is defined.
The `"path"` property must be absolute, without a drive, and will be treated as relative to
the path to the repository. For example:
```json
{
"kind": "git",
"repository": "https://github.com/microsoft/vcpkg",
"path": "/ports"
}
```
is the correct way to refer to the registry built in to vcpkg, at the latest version.
The following are all incorrect:
```json
{
"$reason": "path can't be drive-absolute",
"kind": "git",
"repository": "https://github.com/microsoft/vcpkg",
"path": "F:/ports"
}
```
```json
{
"$reason": "path can't be relative",
"kind": "git",
"repository": "https://github.com/microsoft/vcpkg",
"path": "ports"
}
```
```json
{
"$reason": "path _really_ can't be relative like that",
"kind": "git",
"repository": "https://github.com/microsoft/vcpkg",
"path": "../../meow/ports"
}
```

View File

@@ -0,0 +1,66 @@
# Scripts Tree Extraction
**Note: this is the feature as it was initially specified and does not necessarily reflect the current behavior.**
## Background
We extracted vcpkg-tool as part of a future wherein Registries are the primary mechanism for interacting with the ports tree, which would allow the vcpkg tool and associated artifacts to be deployed and figure the rest out on their own. Unfortunately, we have concurrently edited things in the so called "scripts" tree which lives in support of ports but really probably belongs in the vcpkg-tool repo.
Moreover, as part of stabilizing registries, the interface exposed by the scripts tree becomes contractual rather than something we can change in concert with ports, since we can no longer see the universe of ports to validate that changes are correct.
To that end we are auditing the contents of the scripts tree to make sure it is a solid foundation for future work.
The work list is contained in [Issue #16188].
[Issue #16188]: https://github.com/microsoft/vcpkg/issues/16188
## Audit Points
The following are assertions we want to be able to make about the contents of the scripts tree. Note that this does *not* refer to `vcpkg.cmake` since that needs to work with older versions of cmake.
These are design ideals that we may break in some limited cases where that makes sense.
- We always use `cmake_parse_arguments` rather than function parameters, or referring to `${ARG<N>}`.
- Exception: there are exclusively positional parameters. This should be _very rare_.
- In this case, positional parameters should be put in the function declaration
(rather than using `${ARG<N>}`), and should be named according to local rules
(i.e. `snake_case`).
- Exception: positional parameters that are optional should be given a name via
`set(argument_name "${ARG<N>}") after checking `${ARGC}`.
- Note: in cases where there are positional parameters along with non-positional parameters, positional parameters should be referred to by `arg_UNPARSED_ARGUMENTS`.
- All `cmake_parse_arguments` use `PARSE_ARGV` for resistance to embedded semicolons.
- All `foreach` loops use `IN LISTS` for resistance to embedded semicolons.
- The variable `${ARGV}` is unreferenced.
- We use functions, not macros or top level code.
- Scripts in the scripts tree should not be expected to need changes as part of normal operation. (For example, `vcpkg_acquire_msys` has hard coded specific packages and versions thereof used which we believe is unacceptable)
- All non-splat variable expansions are in quotes "".
- There are no "pointer" parameters (where a user passes a variable name rather than the contents) except for out parameters.
- Undefined names are not referenced.
- Out parameters only set `PARENT_SCOPE`.
- `CACHE` variables are not used.
- `include()`s are removed and fixes to `port.cmake` et al. are made as necessary to avoid this.
- `foreach(RANGE)`'s arguments _must always be_ natural numbers, and `<start>` _must always be_ less than or equal to `<stop>`.
- This should be checked.
### Naming Variables
- `cmake_parse_arguments`: set prefix to `"arg"`
- local variables are named `snake_case`
- Internal global variable names are named `Z_VCPKG_`.
- External experimental global variable names are named `X_VCPKG_`.
- Internal functions are named `z_vcpkg_*`
- Functions which are internal to a single function (i.e., helper functions)
are named `[z_]<func>_<name>`, where `<func>` is the name of the function they are
a helper to, and `<name>` is what the helper function does.
- `z_` should be added to the front if `<func>` doesn't have a `z_`,
but don't name a helper function `z_z_foo_bar`.
- Public global variables are named `VCPKG_`.
## Prognosis
Not everything should remain in the scripts tree. As part of this audit, each helper will be dealt with in one of several ways:
- Stay in scripts tree
- Deleted outright
- Moved to a tool port
- Deprecated

View File

@@ -0,0 +1,357 @@
# Versioning Specification
**Note: this is the feature as it was initially specified and does not necessarily reflect the current behavior.**
**Up-to-date documentation is available at [Versioning](../users/versioning.md).**
## Glossary
Some of the terms used in this document have similar meaning when discussed by the community, and because of that, they can cause confusion and ambiguity. To solve this issue, we will assign specific meaning to these terms and try to keep a consistent usage through the document.
**Library**: A piece of software (source code, binary files, documentation, license, etc.) that is intended to be reused by other software.
**Package**: A package can contain a library, collection of libraries, build scripts, software tools, or other components necessary for their use. The goal of vcpkg is to facilitate the installation of these packages in the user's environment.
**Port**: A vcpkg specific term, a port contains:
* Metadata about a package: package version, supported features, dependencies, etc.
* Instructions to acquire, build if necessary, and install the package.
## 1 Enabling package versioning
On launch, the versioning feature will be disabled by default. Users can enable this feature by setting the `versions` feature flag.
Example:
```
vcpkg --feature-flags=versions install
```
### 1.1 Proposed experience
This feature requires the use of manifests to declare project dependencies. To allow versioning, the following features are added to manifests:
* Ability to declare a package's versioning scheme.
* Ability to declare version constraints on dependencies.
* Ability for a top-level manifest to override all other version constraints.
* Ability to declare a baseline for all versions.
Example: A manifest (`vcpkg.json`) using versioning features.
```json
{
"name": "versions-test",
"version": "1.0.0",
"dependencies": ["fmt", {"name": "zlib", "version>=": "1.2.11"}],
"$x-default-baseline": "9fd3bd594f41afb8747e20f6ac9619f26f333cbe"
}
```
The example above shows some new manifest properties:
* `"version"`: Declares a version using a dot-separated versioning scheme (`1.0.0`).
* `"version>="`: Declares a minimum version constraint on package `zlib`.
* `"$x-default-baseline"`: Declares a baseline version for all packages.
All these new features are described in more detail in this document.
## 2 Specifying package versions
Through the years, C++ software authors have adopted multiple versioning schemes and practices that sometimes conflict between each other. On vcpkg, the most recurrent versioning schemes found are:
* Semantic versions
* Dates
* Repository commits
* Arbitrary strings
For vcpkg to achieve wide adoption and compatibility with existing projects, it is important that we respect the versioning schemes used by each of the packages contained in our ports catalog.
### 2.1 Port versions
Package versioning information is divided in two parts: a version string and a port version.
Port versions are a concept exclusive to vcpkg, they do not form part of a packages upstream. But allow for versioning of the vcpkg ports themselves.
Packages can also include the port version as part of a version constraint by using the “port-version” property on their dependencies.
#### `port-version`
An integer value that increases each time a vcpkg-specific change is made to the port.
The rules for port versions are:
* Start at 0 for the original version of the port,
* increase by 1 each time a vcpkg-specific change is made to the port that does not increase the version of the package,
* and reset to 0 each time the version of the package is updated.
Defaults to 0 if omitted.
### 2.2 Package versions
Versions are an important part of a packages upstream metadata. Ports in vcpkg should attempt to follow the versioning conventions used by the packages authors. For that reason, when declaring a packages version the appropriate scheme should be used.
Each versioning scheme defines their own rules on what is a valid version string and more importantly the rules for how to sort versions using the same scheme.
The versioning schemes understood by vcpkg are:
Manifest property | Versioning scheme
------------------|------------------------------------
`version` | For dot-separated numeric versions
`version-semver` | For SemVer compliant versions
`version-date` | For dates in the format YYYY-MM-DD
`version-string` | For arbitrary strings
A manifest must contain only one version declaration.
#### `version`
Accepts version strings that follow a relaxed, dot-separated-, semver-like scheme.
The version is logically composed of dot-separated (`.`) numeric sections. Each section must contain an integer positive number with no leading zeroes.
The regex pattern for this versioning scheme is: `(0|[1-9]\d*)(\.(0|[1-9]\d*))*`
_Sorting behavior_: When comparing two versions, each section is compared from left to right by their numeric value, until the first difference is found. A version with the smallest set of sections takes precedence over another with a larger set of sections, given that all their preceding sections compare equally.
Example:
`0` < `0.1` < `0.1.0` < `1` < `1.0.0` < `1.0.1` < `1.1`< `2.0.0`
#### `version-semver`
Accepts version strings that follow semantic versioning conventions as described in the [semantic versioning specification](https://semver.org/#semantic-versioning-specification-semver).
_Sorting behavior_: Strings are sorted following the rules described in the semantic versioning specification.
Example:
`1.0.0-1` < `1.0.0-alpha` < `1.0.0-beta` < `1.0.0` < `1.0.1` < `1.1.0`
#### `version-date`
Accepts version strings that can be parsed to a date following the ISO-8601 format `YYYY-MM-DD`. Disambiguation identifiers are allowed in the form of dot-separated-, positive-, integer-numbers with no leading zeroes.
The regex pattern for this versioning scheme is: `\d{4}-\d{2}-\d{2}(\.(0|[1-9]\d*))*`.
_Sorting behavior_: Strings are sorted first by their date part, then by numeric comparison of their disambiguation identifiers. Disambiguation identifiers follow the rules of the relaxed (version) scheme.
Examples:
`2020-01-01` < `2020-01-01.1` < `2020-02-01.1.2` < `2020-02-01.1.3` < `2020-02-01`
#### `version-string`
For packages using version strings that do not fit any of the other schemes, it accepts most arbitrary strings, but some special characters like `#` are disallowed.
_Sorting behavior_: No sorting is attempted on the version string itself. However, if the strings match exactly, the port versions can be compared and sorted.
Examples:
`apple` <> `orange` <> `orange.2` <> `orange2`
`watermelon` (`port-version`: 0) < `watermelon` (`port-version`: 1)
##### Example: Manifests using different versioning schemes
```json
{
"name": "openssl",
"version": "1.1.1",
"port-version": 0
}
```
```json
{
"name": "bzip2",
"version-semver": "1.0.8",
}
```
```json
{
"name": "abseil",
"version-date": "2020-03-03",
"port-version": 8
}
```
```json
{
"name": "d3dx12",
"version-string": "may2020",
"port-version": 0
}
```
## 3 Specifying dependency versions
### 3.1 On manifest files
Manifest files help users specify complex versioned dependency graphs in a declarative manner. In this document we define a top-level manifest as the manifest file written by a user to declare their projects dependencies. This is opposed to a ports manifest file, which is used by ports to declare the dependencies of the package it contains.
There are three mechanisms you can use in your manifest files to control which versions of your packages are installed: **version constraints, registry baselines and overrides**.
#### Version constraints
Specifying a version constraint is the most direct way to control which version of a package is installed, in vcpkg you can declare minimum version constraints using the syntax `"version>=": "1.0.0"`.
#### Registry baseline
Baselines are used to set lower boundaries on package versions. A baseline effectively adds a minimum version constraint on all the packages declared in it.
But what is a baseline?
In the main registry, the baseline is a file located in `${VCPKG_ROOT}/versions/baseline.json`. This file contains a version declaration for each package in vcpkg. The format of this file is the following:
```json
{
"default": [
{
...
"fmt": { "version-semver": "7.1.2", "port-version": 0},
...
}
]
}
```
The baseline file is tracked under source control. For any given revision of the registry, the versions declared in the baseline file must match the current versions of the ports in the registry at that revision.
Old revisions of vcpkg that do not contain a baseline file can still work with versioning. As a fallback, if no baseline is available at a given revision, vcpkg will use its local baseline file. If a local baseline file does not exist, the local version of the port will be used as the baseline version.
Baselines define a minimum version constraint an all packages contained in it.
For example, if the baseline contains the entry:
```
“fmt”: { “version-semver”: “7.1.2”, “port-version”: 0 }
```
A minimum version constraint will be added to `fmt` so that vcpkg wont install a version lower than `7.1.2` with port version `0`.
#### Overrides
Declaring an override forces vcpkg to ignore all other constraints, both top-level and transitive constraints, and use the version specified in the override. This is useful for pinning exact versions and for resolving version conflicts.
## 4 Version constraints
### 4.1 Declaring a baseline
For the initial implementation, the method to declare a baseline is to set the `“$x-default-baseline”` property.
The use of `“$x-default-baseline”` is temporary and will very likely change in the future, as we work on implementing custom registries.
#### `$x-default-baseline`
Accepts a Git commit ID. Vcpkg will try to find a baseline file in the given commit ID and use that to set the baseline versions (lower bound versions) of all declared dependencies.
When resolving version constraints for a package, vcpkg will look for a baseline version:
* First by looking at the baseline file in the given commit ID.
* If the given commit ID does not contain a baseline file, vcpkg will fallback to use the local baseline file instead.
* If theres no local baseline file, vcpkg will use the version currently available in the ports directory.
_NOTE: If a baseline file is found, but it does not contain an entry for the package, the vcpkg invocation will fail._
Example:
```json
{
"name": "project",
"version": "1.0.0",
"dependencies": ["zlib", "fmt"],
"$x-default-baseline":"9fd3bd594f41afb8747e20f6ac9619f26f333cbe"
}
```
Baselines can be used without any other version constraints to obtain behavior close to using “classic” mode.
### 4.2 Declaring minimum version constraints
A minimum version requirement puts a lower boundary on the versions that can be used to satisfy a dependency. This means that any version that is newer than the requirement is valid (including major version changes).
Vcpkg will use the oldest identified version that can satisfy all the version requirements in a build graph. Using a minimum version approach has the following advantages:
* Is predictable and easy to understand.
* User controls when upgrades happen, as in, no upgrades are performed automatically when a new version is released.
* Avoids using a SAT solver.
Minimum version requirements are expressed by using the `"version>="` property in the dependencies list.
Example:
```json
{
"name": "project",
"version-semver": "1.0.0",
"dependencies": [
{ "name": "zlib", "version>=": "1.2" },
{ "name": "rapidjson", "version>=": "2020-02-01" }
]
}
```
### 4.3 Declaring port version constraints
To be consistent with the minimum version approach, vcpkg uses the lowest available port version that matches the package version. There are many scenarios where a higher port version is desirable, e.g.: support for new platforms, fixing installation issues, among others.
As part of the dependency object a port version can be specified. An error will be emitted if a non-existent port-version for the given package version is requested.
Example:
```json
{
"name": "project",
"version-semver": "1.0.0",
"dependencies": [
{ "name": "zlib", "version>=": "1.2" },
{ "name": "rapidjson", "version=": "2020-02-01", "port-version": 2 }
]
}
```
### 4.4 Declaring overrides
Overrides are declared as an array of package version declarations.
For an override to take effect, the overridden package must form part of the dependency graph. That means that a dependency must be declared either by the top-level manifest or be part of a transitive dependency.
Example:
```json
{
"name": "project",
"version": "1.0.0",
"dependencies": ["cpprestsdk"],
"overrides": [{"name":"zlib", "version-semver":"1.2.10"}],
"$x-default-baseline":"9fd3bd594f41afb8747e20f6ac9619f26f333cbe"
}
```
In the previous example, `zlib` is not a direct dependency of the project but it is a dependency for `cpprestsdk`, so the override takes effect forcing `zlib` to version `1.2.10`.
## 5 Design considerations
### 5.1 Constraint resolution
Given a manifest with a set of versioned dependencies, vcpkg will attempt to calculate a package installation plan that satisfies all the constraints. Constraints can be declared in the top-level manifest but can also be added transitively by indirect dependencies.
Vcpkg roughly follows the steps below to compute an installation plan, the installation plan will either contain a valid set of package versions, or a list of version conflicts.
* Add all top-level constraints to the plan.
* Recursively add transitive constraints to the plan.
* Each time a constraint is added for a package, also add its baseline version as a minimum constraint.
* Each time a constraint is added:
* If an override exists for the package, select the version in the override.
* Otherwise:
* If there is no previous version selected.
* Select the minimal version that satisfies the constraint.
* If there is a previous version selected:
* If the versioning scheme of the new constraint does not match that of the previously selected version:
* Add a version conflict.
* If the constraints version is not comparable to the previously selected version. For example, comparing “version-string: apple” to “version-string: orange”:
* Add a version conflict.
* If the constraints version is higher than the previously selected version:
* Select the highest version.
* Otherwise, keep the previous selection.
* Review the plan:
* If there are no conflicts, install the selected packages.
* Otherwise, report the conflicts to the user.
### 5.2 Acquiring port versions
Although the concept of package versions has always been present in vcpkg, the concept of version constraints has been not.
With the introduction of versioning constraints, it is now possible that a package depends on a port version that does not match the one available locally. This raises a problem as vcpkg needs to know how to acquire the port files for the requested version.
To solve this problem, a new set of metadata needs to be introduced. This specification proposes a that a new "versions" folder is added as part of a registry. In the main vcpkg registry, this means a new root level versions directory.
The versions directory, from here on referred as the versions database, will contain JSON files for each one of the ports available in the registry. Each file will list all the versions available for a package and contain a Git tree-ish object that vcpkg can check out to obtain that versions portfiles.
As part of the versioning implementation, a generator for these database files will be implemented. The generator will extract from our repositorys Git history, all the versions of each port that had been available at any moment in time and compile them into these database files.
Example: generated `zlib.json`
```json
{
"versions": [
{
"git-tree": "2dfc991c739ab9f2605c2ad91a58a7982eb15687",
"version-string": "1.2.11",
"port-version": 9
},
{ “$truncated for brevity” },
{
"git-tree": "a516e5ee220c8250f21821077d0e3dd517f02631",
"version-string": "1.2.10",
"port-version": 0
},
{
"git-tree": "3309ec82cd96d752ff890c441cb20ef49b52bf94",
"version-string": "1.2.8",
"port-version": 0
}
]
}
```
For each port, its corresponding versions file should be located in `versions/{first letter of port name}-/{port name}.json`. For example, zlibs version file will be located in `versions/z-/zlib.json`.
Aside from port version files, the current baseline file is located in `versions/baseline.json`.