Saturday 15 March 2025
As technology continues to advance at a rapid pace, the importance of software integrity and reliability cannot be overstated. With the rise of open-source software and the increasing reliance on digital systems, ensuring that our code is trustworthy has become a top priority. A recent study sheds light on one crucial aspect of this issue: reproducibility.
Reproducibility refers to the ability to rebuild software from its source code and obtain identical results. This may seem like a straightforward concept, but it’s surprisingly complex in practice. Many factors can influence the outcome of a build process, making it challenging to achieve consistency across different environments and systems.
Researchers have been working to address this issue by developing functional package managers (FPMs) like Nix. FPMs allow for the creation of reproducible builds by encapsulating dependencies and ensuring that they are consistently resolved. However, until now, there has been limited understanding of how well these systems perform in real-world scenarios.
A team of researchers set out to investigate the reproducibility of Nix packages over a six-year period. They rebuilt 709,816 packages from 17 different revisions of the nixpkgs software repository, sampling every 4.1 months between 2017 and 2023. This massive undertaking allowed them to identify trends and patterns in reproducibility.
The results were striking. The study found that bitwise reproducibility – the ability to rebuild software and obtain identical binary code – has increased significantly over time. In 2017, only 69% of packages were bitwise reproducible, but by 2023, this number had risen to 91%. Mere rebuildability, which includes both reproducible and non-reproducible builds, remained consistently high, at around 99.8%.
The researchers also discovered that certain ecosystems within nixpkgs exhibited different reproducibility rates. For example, packages from the Python ecosystem showed a significant increase in bitwise reproducibility over time, while those from the Haskell ecosystem remained relatively stable.
To better understand why some packages were not reproducible, the team developed heuristics to identify common causes of unreproducibility. They found that 15% of non-reproducible packages contained embedded build dates, which can affect the outcome of a build process. Other factors, such as environment variables and uname outputs, also played a role.
The study’s findings have important implications for software development and maintenance.
Cite this article: “Reproducibility of Nix Packages Over Six Years: Trends and Patterns”, The Science Archive, 2025.
Software Integrity, Reproducibility, Nix Packages, Functional Package Managers, Fpms, Bitwise Reproducibility, Rebuildability, Nixpkgs Software Repository, Python Ecosystem, Haskell Ecosystem
