nim-libp2p-experimental/azure-pipelines.yml
Giovanni Petrantoni b99d2039a8
Gossip one one (#240)
* allow multiple codecs per protocol (without breaking things)

* add 1.1 protocol to gossip

* explicit peering part 1

* explicit peering part 2

* explicit peering part 3

* PeerInfo and ControlPrune protocols

* fix encodePrune

* validated always, even explicit peers

* prune by score (score is stub still)

* add a way to pass parameters to gossip

* standard setup fixes

* take into account explicit direct peers in publish

* add floodPublish logic

* small fixes, publish still half broken

* make sure to waitsub in sparse test

* use var semantics to optimize table access

* wip... lvalues don't work properly sadly...

* big publish refactor, replenish and balance

* fix internal tests

* use g.peers for fanout (todo: don't include flood peers)

* exclude non gossip from fanout

* internal test fixes

* fix flood tests

* fix test's trypublish

* test interop fixes

* make sure to not remove peers from gossip table

* restore old replenishFanout

* cleanups

* restore utility module import

* restore trace vs debug in gossip

* improve fanout replenish behavior further

* triage publish nil peers (issue is on master too but just hidden behind a if/in)

* getGossipPeers fixes

* remove topics from pubsubpeer (was unused)

* simplify rebalanceMesh (following spec) and make it finally reach D_high

* better diagnostics

* merge new pubsubpeer, copy 1.1 to new module

* fix up merge

* conditional enable gossip11 module

* add back topics in peers, re-enable flood publish

* add more heartbeat locking to prevent races

* actually lock the heartbeat

* minor fixes

* with sugar

* merge 1.0

* remove assertion in publish

* fix multistream 1.1 multi proto

* Fix merge oops

* wip

* fix gossip 11 upstream

* gossipsub11 -> gossipsub

* support interop testing

* tests fixing

* fix directchat build

* control prune updates (pb)

* wip parameters

* gossip internal tests fixes

* parameters wip

* finishup with params

* cleanups/wip

* small sugar

* grafted and pruned procs

* wip updateScores

* wip

* fix logging issue

* pubsubpeer, chronicles explicit override

* fix internal gossip tests

* wip

* tables troubleshooting

* score wip

* score wip

* fixes

* fix test utils generateNodes

* don't delete while iterating in score update

* fix grafted defect

* add a handleConnect in subscribeTopic

* pruning improvements

* wip

* score fixes

* post merge - builds gossip tests

* further merge fixes

* rebalance improvements and opportunistic grafting

* fix test for now

* restore explicit peering

* implement peer exchange graft message

* add an hard cap to PX

* backoff time management

* IWANT cap/budget

* Adaptive gossip dissemination

* outbound mesh quota, internal tests fixing

* oversub prune score based, finish outbound quota

* finishup with score and ihave budget

* use go daemon 0.3.0

* import fixes

* byScore cleanup score sorting

* remove pointless scaling in `/` Duration operator

* revert using libp2p org for daemon

* interop fixes

* fixes and cleanup

* remove heartbeat assertion, minor debug fixes

* logging improvements and cleaning up

* (to revert) add some traces

* add explicit topic to gossip rpcs

* pubsub merge fixes and type fix in switch

* Revert "(to revert) add some traces"

This reverts commit 4663eaab6cc336c81cee50bc54025cf0b7bcbd99.

* cleanup some now irrelevant todo

* shuffle peers anyway as score might be disabled

* add missing shuffle

* old merge fix

* more merge fixes

* debug improvements

* re-enable gossip internal tests

* add gossip10 fallback (dormant but tested)

* split gossipsub internal tests into 1.0 and 1.1

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2020-09-21 11:16:29 +02:00

126 lines
3.7 KiB
YAML

strategy:
maxParallel: 10
matrix:
# Nim requires enforcing ARCH="x86" and UCPU
# for 32-bit targets as it seems like Azure machines are 64-bit
# TEST_LANG env variable support TODO
Windows_32bit:
VM: 'windows-latest'
ARCH: x86
PLATFORM: x86
TEST_LANG: c
Windows_64bit:
VM: 'windows-latest'
PLATFORM: x64
TEST_LANG: c
pool:
vmImage: $(VM)
variables:
V: 0 # Scripts verbosity, 1 for debugging build scripts
steps:
- task: CacheBeta@1
displayName: 'cache Nim binaries'
inputs:
key: NimBinaries | $(Agent.OS) | $(PLATFORM) | "$(Build.SourceBranchName)" | "v5"
path: NimBinaries
- task: CacheBeta@1
displayName: 'cache Go libp2p daemon'
inputs:
key: p2pdCache | $(Agent.OS) | $(PLATFORM) | "v3"
path: p2pdCache
- task: CacheBeta@1
displayName: 'cache MinGW-w64'
inputs:
key: mingwCache | 8_1_0 | $(PLATFORM) | "v2"
path: mingwCache
- powershell: |
Set-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem' -Name 'LongPathsEnabled' -Value 1
displayName: 'long path support'
- bash: |
set -e
# custom MinGW-w64 versions for both 32-bit and 64-bit, since we need a 64-bit build of p2pd
echo "Installing MinGW-w64"
install_mingw() {
mkdir -p mingwCache
cd mingwCache
if [[ ! -e "$MINGW_FILE" ]]; then
curl -OLsS "$MINGW_URL"
fi
7z x -y -bd "$MINGW_FILE" >/dev/null
mkdir -p /c/custom
mv "$MINGW_DIR" /c/custom/
cd ..
}
# 32-bit
MINGW_FILE="i686-8.1.0-release-posix-dwarf-rt_v6-rev0.7z"
MINGW_URL="https://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win32/Personal%20Builds/mingw-builds/8.1.0/threads-posix/dwarf/${MINGW_FILE}"
MINGW_DIR="mingw32"
install_mingw
# 64-bit
MINGW_FILE="x86_64-8.1.0-release-posix-seh-rt_v6-rev0.7z"
MINGW_URL="https://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Personal%20Builds/mingw-builds/8.1.0/threads-posix/seh/${MINGW_FILE}"
MINGW_DIR="mingw64"
install_mingw
if [[ $PLATFORM == "x86" ]]; then
MINGW_DIR="mingw32"
else
MINGW_DIR="mingw64"
fi
export PATH="/c/custom/${MINGW_DIR}/bin:${PATH}"
echo "PATH=${PATH}"
which gcc
gcc -v
# detect number of cores
export ncpu="$NUMBER_OF_PROCESSORS"
[[ -z "$ncpu" || $ncpu -le 0 ]] && ncpu=2
echo "Found ${ncpu} cores"
if [[ $PLATFORM == "x86" ]]; then
choco --version
choco install --x86 openssl
export PATH="/c/Program Files (x86)/OpenSSL-Win32/bin:${PATH}"
echo "PATH=${PATH}"
fi
# build nim from our own branch - this to avoid the day-to-day churn and
# regressions of the fast-paced Nim development while maintaining the
# flexibility to apply patches
curl -OLsS https://raw.githubusercontent.com/status-im/nimbus-build-system/master/scripts/build_nim.sh
env MAKE="mingw32-make -j${ncpu}" ARCH_OVERRIDE=$(PLATFORM) bash build_nim.sh Nim csources dist/nimble NimBinaries
export PATH="${PWD}/Nim/bin:${PATH}"
echo "PATH=${PATH}"
# install and build go-libp2p-daemon
go version
export GOPATH="${PWD}/go"
export PATH="${GOPATH}/bin:${PATH}"
echo "PATH=${PATH}"
# we can't seem to be able to build a 32-bit p2pd
env PATH="/c/custom/mingw64/bin:${PATH}" bash scripts/build_p2pd.sh p2pdCache v0.3.0
# install dependencies
nimble refresh
nimble install -y --depsOnly
# run tests
nimble test
nimble examples_build
displayName: 'build and test'