7dfceceea6 build: Remove #undef hack for ASM in the precomputation programs (Tim Ruffing)
bb36fe9be0 ci: Test `make precomp` (Tim Ruffing)
d94a37a20c build: Remove CC_FOR_BUILD stuff (Tim Ruffing)
ad63bb4c29 build: Prebuild and distribute ecmult_gen table (Tim Ruffing)
ac49361ed0 prealloc: Get rid of manual memory management for prealloc contexts (Tim Ruffing)
6573c08f65 ecmult_gen: Tidy precomputed file and save space (Tim Ruffing)
5eba83f17c ecmult_gen: Precompute tables for all values of ECMULT_GEN_PREC_BITS (Tim Ruffing)
fdb33dd122 refactor: Make PREC_BITS a parameter of ecmult_gen_build_prec_table (Tim Ruffing)
a4875e30a6 refactor: Move default callbacks to util.h (Tim Ruffing)
4c94c55bce doc: Remove obsolete hint for valgrind stack size (Tim Ruffing)
5106226991 exhaustive_tests: Fix with ecmult_gen table with custom generator (Tim Ruffing)
e1a76530db refactor: Make generator a parameter of ecmult_gen_create_prec_table (Tim Ruffing)
9ad09f6911 refactor: Rename program that generates static ecmult_gen table (Tim Ruffing)
8ae18f1ab3 refactor: Rename file that contains static ecmult_gen table (Tim Ruffing)
00d2fa116e ecmult_gen: Make code consistent with comment (Tim Ruffing)
3b0c2185ea ecmult_gen: Simplify ecmult_gen context after making table static (Tim Ruffing)
e43ba02cfc refactor: Decouple table generation and ecmult_gen context (Tim Ruffing)
22dc2c0a0d ecmult_gen: Move table creation to new file and force static prec (Tim Ruffing)
Pull request description:
This resolves#893, resolves#692 (and also resolvesbitcoin/bitcoin#22854).
- [x] Extract table generation to separate function in separate file (to be used by generation script and exhaustive tests)
- [x] Tidy up
- [x] Remove code that deals with non-static tables
- [x] Make functions that need ecmult_gen not depend on signing context
- [x] Rename stuff to make it fit the new structure and consistent with how we hande verification tables (#956)
- [x] Fix exhaustive tests
- [x] Make table generation function take generator as input
- [x] Overwrite the static tables with a table with custom generator in exhaustive tests
- [x] Overhaul script that generates table files
- [x] Make table generation function take PREC_BITS as input (I have some code already, just not yet in this branch)
- [x] Change generation script to generate three tables (for all three values of ECMULT_GEN_PREC_BITS)
- [x] Ship pre-built tables
- [x] Add pregenerated table file to repo
- [x] Remove generation of table file from build process (like in #956)
- [x] Remove left-over stuff (e.g., detecting a compiler running on the build machine) from build system
- [x] Final cleanups (copyright headers, commit, messages, etc.)
- [ ] (separate PR:) Make sure link-time optimization remove corresponding static tables (and code) when no signing/verifcation function is called
- [ ] (separate PR:) Compile precomputation as a separate object file and link it (https://github.com/bitcoin-core/secp256k1/pull/988#issuecomment-977813538)
- [ ] (separate PR:) Document the backwards-compatible API changes made in this PR and in #956.
- [ ] Maybe deprecate the static context
ACKs for top commit:
sipa:
ACK 7dfceceea6
robot-dreams:
ACK 7dfceceea6 (based on range-diff between 56284c7d44c0ed46e636588bfbf6c403b7dfa6c1 and 7dfceceea6)
Tree-SHA512: 6efb3f36f05efe3b79bbd877881fe1409f71fd6488d24c811b2e77d9f053bed78670dd1dcbb42ad780458a51c4ffa36de9cd6567271b22041dc7a122ceb677c5
- Improve Makefile.am for both prebuilt tables files
- On the way, tidy EXTRA_DIST: Move the header files to noinst_HEADERS,
where they conceptually belong, and add missing SECURITY.md to EXTRA_DIST
c0cd7de6d4 build: add -no-undefined to libtool LDFLAGS (fanquake)
fe32a79d35 build: pass win32-dll to LT_INIT (fanquake)
Pull request description:
This takes care of two of the outstanding issues in #923. One being initializing libtool with `win32-dll` and the other being the addition of `-no-undefined` to the libtool LDFLAGS. See each commit for more details.
Builders cross-compiling for Windows (including Core) will no-longer see:
```bash
libtool: warning: undefined symbols not allowed in x86_64-w64-mingw32 shared libraries; building static only
```
I'm planning on making some related changes downstream.
ACKs for top commit:
sipa:
utACK c0cd7de6d4. We indeed have done the work to propertly mark exported symbols, and AFAIK have no imported symbols apart from standard library ones.
real-or-random:
ACK c0cd7de6d4
hebasto:
ACK c0cd7de6d4
Tree-SHA512: 6756bc88ac439a27117a1341d82a801cef70354a9e7a563592ab3ac7298fbefdaa0a2c410ea3fba8953d53f254c449dc491069f30468db12791027a65dd02f80
This is only needed for the very oldest of non-POSIX-compatible shells.
Note that this code will also only be executed on macOS, where it'd be
very unlikely to run into such a shell.
Followup to #1019.
See:
https://github.com/koalaman/shellcheck/wiki/SC2006
Co-authored-by: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>
214042a170 build: don't append valgrind CPPFLAGS if not installed (fanquake)
Pull request description:
Valgrinds CPPFLAGS, i.e `-I/usr/local/opt/valgrind/include`, are currently added to CPPFLAGS, regardless of whether valgrind is installed. This changes configure so that they are only added if valgrind is available. i.e the output of `brew list --versions valgrind` is non-null.
ACKs for top commit:
real-or-random:
ACK 214042a170
hebasto:
ACK 214042a170, tested on macOS Big Sur 11.6.1 (20G224, Intel).
Tree-SHA512: 5101636a0a12f1941b01967ca8eab7aa20f44db0d1ef4571a5ad6026bb89494b983465d34d93c8b17a260b695116792991da53d135bc19a3c9e974f5266a90af
This header contains a static array that replaces the ecmult_context pre_g and pre_g_128 tables.
The gen_ecmult_static_pre_g program generates this header file.
Fixes one of the items in #923, namely the warnings of the form
'_putenv' redeclared without dllimport attribute:
previous dllimport ignored [-Wattributes]
This also cleans up the way we add CFLAGS, in particular flags enabling
warnings. Now we perform some more fine-grained checking for flag
support, which is not strictly necessary but the changes also help to
document autoconf.ac.
Bitcoin Core's `configure` script uses `AC_CHECK_PROG` to find brew in the `PATH` [1]. If found, this macro will set `BREW=brew`. When building with dependencies however the `BREW` variable is set to `no` on macOS via `depends/<host_prefix>/share/config.site` [2] and this overrides `AC_CHECK_PROG` results [3]. Ideally, secp256k1's `configure` script should follow the same logic but this is not what happens because secp256k1's `configure` uses `AC_PATH_PROG` instead which respects preset variable values (in this case for variable `BREW`) only if they are a valid path (i.e., they match `[\\/*] | ?:[\\/]*` [4]), and `no` is not a path.
This commit changes `AC_PATH_PROG` to `AC_CHECK_PROG` to be consistent with Core's `AC_CHECK_PROG`. Both of these macros are supposed to find executables in the `PATH` but the difference is that former is supposed to return the full path whereas the latter is supposed to find only the program. As a result, the latter will accept even non-paths `no` as an override. Not knowing the full path is not an issue for the `configure` script because it will only execute `BREW` immediately afterwards, which works fine without the full path. (In particular, `PATH` cannot have changed in between [5].)
[1] https://github.com/bitcoin/bitcoin/blob/master/configure.ac#L684
[2] https://github.com/bitcoin/bitcoin/blob/master/depends/config.site.in#L73-L76
[3] 6d38e9fa2b/lib/autoconf/programs.m4 (L47)
[4] 6d38e9fa2b/lib/autoconf/programs.m4 (L127)
[5] [3ab1178](3ab1178d54)
This compiler flag is available for clang but not gcc.
Test plan:
```
autogen.sh
./configure
make check
CC=clang ./configure
make check
```
If a variable is used uninitialized, the warning should look something
like:
```
CC src/tests-tests.o
src/tests.c:4336:15: warning: variable 'recid' may be uninitialized when used here [-Wconditional-uninitialized]
CHECK(recid >= 0 && recid < 4);
^~~~~
./src/util.h:54:18: note: expanded from macro 'CHECK'
if (EXPECT(!(cond), 0)) { \
^~~~
./src/util.h:41:39: note: expanded from macro 'EXPECT'
^
src/tests.c:4327:14: note: initialize the variable 'recid' to silence this warning
int recid;
^
= 0
1 warning generated.
```
This commits simply uses CC as CC_FOR_BUILD and the same for
corresponding flags if we're not cross-compiling. This has a number of
benefits in this common case:
- It avoids strange cases where very old compilers are used (#768).
- Flags are consistently set for CC and CC_FOR_BUILD.
- ./configure is faster.
- You get compiler x consistently if you set CC=x; we got this wrong
in CI in the past.
./configure warns if a _FOR_BUILD variable is set but ignored because
we're not cross-compiling.
The change exposed that //-style comments are used in gen_context.c,
which is also fixed by this commit.
This commit also reorganizes code in configure.ac to have a cleaner
separation of sections.
Valgrind is typically installed using brew on macOS. This commit
makes ./configure detect this case set the appropriate include
directory (in the same way as we already do for openssl and gmp).
412bf874d0 configure: Allow specifying --with[out]-valgrind explicitly (Luke Dashjr)
Pull request description:
ACKs for top commit:
sipa:
ACK 412bf874d0. Tested by running configure on a system with and without valgrind, and with no argument, with `--with-valgrind`, and with `--without-valgrind`.
real-or-random:
ACK 412bf874d0
jonasnick:
ACK 412bf874d0
Tree-SHA512: 92417609751e5af813faff1661055cd37f3d00dbcf109a8f14f8ba59d9f3d620c9c6b67d2b1629b6ab75e2afcd47d2b3898a0427931567fb505bc92fa5ee3532
0dccf98a21 Use preprocessor macros instead of autoconf to detect endianness (Tim Ruffing)
Pull request description:
This does not fix any particular issue but it's preferable to not
rely on autoconf. This avoids endianness mess for users on BE hosts
if they use their build without autoconf.
The macros are carefully written to err on the side of the caution,
e.g., we #error if the user manually configures a different endianness
than what we detect.
Supersedes #770 .
ACKs for top commit:
sipa:
ACK 0dccf98a21
gmaxwell:
ACK 0dccf98a21
Tree-SHA512: 6779458de5cb6eaef2ac37f9d4b8fa6c9b299f58f6e5b72f2b0d7e36c12ea06074e483acfb85085a147e0f4b51cd67d897f61a67250ec1cea284a0f7680eb2e8
This does not fix any particular issue but it's preferable to not
rely on autoconf. This avoids endianness mess for users on BE hosts
if they use their build without autoconf.
The macros are carefully written to err on the side of the caution,
e.g., we #error if the user manually configures a different endianness
than what we detect.
Instead of supporting configuration of the field and scalar size independently,
both are now controlled by the availability of a 64x64->128 bit multiplication
(currently only through __int128). This is autodetected from the C code through
__SIZEOF_INT128__, but can be overridden using configure's
--with-test-override-wide-multiply, or by defining
USE_FORCE_WIDEMUL_{INT64,INT128} manually.
ca739cba23 Compile with optimization flag -O2 by default instead of -O3 (Jonas Nick)
83fb1bcef4 Remove -O2 from default CFLAGS because this would override the -O3 flag (see AC_PROG_CC in the Autoconf manual) (Jonas Nick)
ecba8138ec Append instead of Prepend user-CFLAGS to default CFLAGS allowing the user to override default variables (Jonas Nick)
613c34cd86 Remove test in configure.ac because it doesn't have an effect (Jonas Nick)
Pull request description:
Right now, it's not easy to reduce the optimization level with `CFLAGS` because `configure` overwrites any optimization flag with `-O3`. The [automake documentation](https://www.gnu.org/software/automake/manual/html_node/Flag-Variables-Ordering.html) states that:
> The reason ‘$(CPPFLAGS)’ appears after ‘$(AM_CPPFLAGS)’ or ‘$(mumble_CPPFLAGS)’ in the compile command is that users should always have the last say.
and also that it's incorrect to redefine CFLAGS in the first place
> You should never redefine a user variable such as CPPFLAGS in Makefile.am. [...] You should not add options to these user variables within configure either, for the same reason
With this PR `CFLAGS` is still redefined, but user-provided flags appear after the default `CFLAGS` which means that they override the default flags (at least in clang and gcc). Otherwise, the default configuration is not changed. This also means that if CFLAGS are defined by the user, then -g is not added (which does not seem to make much sense). In order to keep the `-O3` despite the reordering we need to explicitly tell autoconf to not append `-O2` by setting the default to `-g` with `: ${CFLAGS="-g"}` as per [the manual](https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.69/autoconf.html#C-Compiler) (EDIT: link fix).
ACKs for top commit:
real-or-random:
ACK ca739cba23
theuni:
ACK ca739cba23.
elichai:
ACK ca739cba23
Tree-SHA512: be92589faa461d245203385d44b489c7d6917b0c68472b8d7576806c0250cf5ff61d5c99ce04eebb8ff5279b9987185d4e5d2da979683fb1c489fdf3e5b59630
Valgrind does bit-level tracking of the "uninitialized" status of memory,
property tracks memory which is tainted by any uninitialized memory, and
warns if any branch or array access depends on an uninitialized bit.
That is exactly the verification we need on secret data to test for
constant-time behaviour. All we need to do is tell valgrind our
secret key is actually uninitialized memory.
This adds a valgrind_ctime_test which is compiled if valgrind is installed:
Run it with libtool --mode=execute:
$ libtool --mode=execute valgrind ./valgrind_ctime_test
ECDSA signing has a retry loop for the exceptionally unlikely case
that S==0. S is not a secret at this point and this case is so
rare that it will never be observed but branching on it will trip
up tools analysing if the code is constant time with respect to
secrets.
Derandomized ECDSA can also loop on k being zero or overflowing,
and while k is a secret these cases are too rare (1:2^255) to
ever observe and are also of no concern.
This adds a function for marking memory as no-longer-secret and
sets it up for use with the valgrind memcheck constant-time
test.
Jonas Nick
Pieter Wuille
Tim Ruffing
fanquake
Dimitris Apostolou
Russell O'Connor
UdjinM6
PiRK
Elichai Turkel
Luke Dashjr
Gregory Maxwell